Checking to see if a record exists and if so update else insert

Question

I've decided to post this as a sticky given the frequency this question is asked.

For those of you wishing to build a package that determines if a source row exists in the destination and if so update it else insert it, this link is for you.

http://blogs.conchango.com/jamiethomson/archive/2006/09/12/SSIS_3A00_-Checking-if-a-row-exists-and-if-it-does_2C00_-has-it-changed.aspx

Thanks Jamie!

If you want to do a similar concept to Jamie's blog post above, but with the Konesan's Checksum Transformation to quickly compare MANY fields, you can visit here:
http://www.ssistalk.com/2007/03/09/ssis-using-a-checksum-to-determine-if-a-row-has-changed/

Phil

Answer 1

This is exactly what I was looking for.

I appreciate that.
Fahad

Answer 2

It's an awesome article - and I used this method in my 1st SSIS package.

However, if number of records is high (over 1000) the OLE Update Command performs slowly. So in that case you might better off loading source data to staging table and creating Stored Procedure which Updates then Inserts (and Deletes) data in the destination table.

Using Lookup transform in the way described in article is quite efficient (compared to other methods like merge join) - and can be useful in a variety of situations

Answer 3

TheViewMaster,
Yes, very valid points. Each developer will have to decide based on the pros and cons of each method.

Personally, I wouldn't go to the extreme that you did, but would rather dump just the updates to a staging table and then outside of the data flow use an Execute SQL task to perform the batch update. I don't want to spend the cost to insert the data twice (once into staging table, and then again into the destination table using your idea).

Answer 4

Can somebody point me to a simple example of this.

I have a flat file and I want to check if the record exist in the file exist in the table, if it does I want to update and if it does not I want to insert.

Answer 5

This is very helpful. Thanks!

I am completely new to the SQL Server Integration Services. I went through your link and understand how you filter out diffrent records. Could you pointting out how you actually update you existing records ?

 

Thanks!

Answer 6

I have cloned the example above and it does not seem to be picking up the new or changed records.  Is there any way to troubleshoot this.

Answer 7

I use OLE DB Command update the records. Thanks!

Answer 8

Hi phill,

Great article,  I loved it.

This is what i had been looking for since long time.

One doubt is there in my mind,  my current assignment i have to transform about 40K rows from a database.

Will this method work for me or there is any other better way.. ?

regards

Sudhir Kesharwani

 

Answer 9

Sudhir Kesharwani wrote:

Hi phill, Great article,  I loved it. This is what i had been looking for since long time. One doubt is there in my mind,  my current assignment i have to transform about 40K rows from a database. Will this method work for me or there is any other better way.. ? regards Sudhir Kesharwani

You need to likely send your changed records to a temporary table and then perform a batch update using an Execute SQL Task on the control flow.  Using the OLE DB Command transformation isn't very efficient for high numbers of rows.

Answer 10

Phil,
As we discussed about updating/inserting techniques a while ago(see http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1211321&SiteID=1) you told me SSIS can perform this operation much faster than using plain SQL...  I put the emphasize on the "much faster".

I have created a performance test package comparing the 2 methods:
1) stored proc with UPDATE and INSERT statements
2) Data flow using look up/conditional split

I have a data flow task to create a data test set with a variable number of rows (I used Jamie's script component source example) with IDs (used as PK) always ranging from 1 to nbrRows.  Each row is constituted of 5 int, 4 string (10, 20, 40 and 80 characters) and 1 datetime with values chosen randomly.

So what I do is calling the package with nbrRows=1, 10, 100...  The first time 1 new row is added.  The second time, id=1 already exists so 1 row is updated, the other 9 are added.  And so on...  I then perform a second round of tests with the same numbers, the only difference being that the destination table is now full (therefore performing only updates).

What I've found out so far is that SSIS is faster only when inserting a certain number of rows (approx . 1000 to 100000).  SSIS performed on avg. 30% better with 50000 rows while at 1 million rows, SQL performed better by 5%.

When the data was updated (second round), SQL was always faster.

Of course, these figures are based on my tests which may not reflect actual usage.  I run SQL/SSIS in parallel, the tests are done in batch, ....  I thought about using DBCC DROPCLEANBUFFER but it doesn't seem to change much.

Maybe my way of comparing is totally flawed...  Have you ever tried comparing the two methods in terms of performance?  How did you conclude the SSIS way is much faster??

Leonce

Answer 11

Leonce,

First, when using fast load option on the destination, there are two parameters which need to be adjusted based on the rows you are working with.  Max Insert Commit Size and Rows Per Batch.  What were the values for these set at?

Also note, that when possible, using a SQL Server Destination over the OLE DB Destination would be preferred.

SSIS should be no different than using BULK INSERT (it seems to be just a glorified wrapper to that).  Performing updates via an OLE DB Command will always be slower than a batch update using SQL.  (Only because the OLE DB Command is executed for every row in the data flow)

Phil

Answer 12

I left the default values for Commit Size and Rows Per Batch ('' and 0).  You suggest I should adjust these values depending on the number of rows transferred?  What would typical values be for a table with 100000 rows?  Does it dependent on the row size?

I use OLEDB Destination, I'll try with SQL Server Destination.

I forgot to mention I use your method for updates (dump into table then update).  Do you think SSIS can be faster than plain SQL for updates?  I would tend to think it's impossible.

Leonce

Answer 13

fleo wrote:

I left the default values for Commit Size and Rows Per Batch ('' and 0). You suggest I should adjust these values depending on the number of rows transferred? What would typical values be for a table with 100000 rows? Does it dependent on the row size?

MICS should be set to whatever you wish the batches to be committed.  So if you want to commit in 10,000 row batches, set MICS to 10,000.  Rows Per Batch should be set to 100,000 in this case.

fleo wrote:

I forgot to mention I use your method for updates (dump into table then update). Do you think SSIS can be faster than plain SQL for updates? I would tend to think it's impossible.

True.  Using an Execute SQL task will likely have a slight slowness to it compared to straight SQL, but in the end they do the same thing, so they should be REAL close.

Answer 14

Let us say that a package uses merge join option to insert new rows and it uses OLE DB Destination with fast load, table lock, and batch size (1000) options. The table lock option on the destination forces us to put a NOLOCK hint in the OLE DB Source query (on the destination table). Otherwise, they run into locking contentions (one is requesting exclusive table lock when the other is requesting shared table lock). This works most of the times but occationally I see that SQL Server does not honour the hint (one scenario: when the target table does not have a clustered index on the join key column) and runs into locking contentions. Have you seen this? Do you rely on NOLOCK hint too? Thanks for your help in advance.

Answer 15

bkallich wrote:

Let us say that a package uses merge join option to insert new rows and it uses OLE DB Destination with fast load, table lock, and batch size (1000) options. The table lock option on the destination forces us to put a NOLOCK hint in the OLE DB Source query (on the destination table). Otherwise, they run into locking contentions (one is requesting exclusive table lock when the other is requesting shared table lock). This works most of the times but occationally I see that SQL Server does not honour the hint (one scenario: when the target table does not have a clustered index on the join key column) and runs into locking contentions. Have you seen this? Do you rely on NOLOCK hint too? Thanks for your help in advance.

I don't use costly merge joins if I can avoid it because the data has to go through a sort upfront.  Using a lookup component will cache the records first, before data has a chance to hit the destination table.  With that said, I haven't used a merge join in my situations for checking to see if a record exists because the amount of rows hasn't been high enough for me to warrant avoiding the use of a lookup.

Answer 16

Thanks Phil. I will try the lookup transformation instead. I asked the same question on your link (http://blogs.conchango.com/jamiethomson/archive/2006/09/12/SSIS_3A00_-Checking-if-a-row-exists-and-if-it-does_2C00_-has-it-changed.aspx) too and Jamie had the comment to the same effect. Sorry about asking the same question twice.

Answer 17

Hi everybody,

Any one tell me how to use "the batch update" as the solution above.

I have to update over 20.000 records by using cursor in a proc. It runs vey slowly.

Thanks

Answer 18

hoo....that a great article. really helps alot.

 

but however, i want to know can it be done between two different database? like between MySQL and SQL Server Mobile Edition?? well, you see...i have a system that uses MySQL in desktop and ive install the sql server mobile edition into my pocket pc. the question is ..can only selected data be transfer between those two? if yes, how? by the way, the system in desktop that uses the MySQL is alrdy a complete system and i hv been asked to build its mobile version, which is a TOTALLY new exprience to me. any idea???

 

thank you in advance. it would be very much appreciated. save my world.

 

Answer 19

Hi all,

I too tried using the same way. But i have strucked when i used oracle oledb provider for oledb command. i have got the following error.
Error at Data flow task[oledb command[8717]]: An oledb error has occured. Error Code:0x80040E51. An Ole db record is available. Sourece:"OraOledb" Descrition:"Provider cannot derive parameter information and SetParameterInfo has not been called"

Unable to retreive destination column descriptions from the parameters of the sql command

Please help me in this regarding.

Regards,
Roopa

Answer 20

hi Phil,

 

I am new this technology,thanks for sharing a good knowldge.

Its a great pleasure to get good knowledge from this forum.

 

maruthi

Answer 21

Well, i usually use 2 SQL Statements for this situation...

Till now, all other methods rather than this seem to affect performance. I know that probably my maintenance is not that user friendly but i know its efficient and old enough to be considered safe enough

Maybe in the future, MS will release a special task for this... Who knows? Or maybe we can start an opensource one... why not?

 

Best Regards,

 

 

Answer 22

Luis Simões wrote:

Well, i usually use 2 SQL Statements for this situation... Till now, all other methods rather than this seem to affect performance. I know that probably my maintenance is not that user friendly but i know its efficient and old enough to be considered safe enough Maybe in the future, MS will release a special task for this... Who knows? Or maybe we can start an opensource one... why not?   Best Regards,

Microsoft has implemented a MERGE T-SQL function in SQL Server 2008...  That would also work.

See John Welch's post here about it: http://agilebi.com/cs/blogs/jwelch/archive/2007/06/16/using-the-merge-statement-in-sql-server-2008.aspx

Answer 23

Hi Phil,

I just wondering, if we have millions records of data, is it possible to use this method since it used to read each record ? I mean, I usually use Inner Join, Right Join and Left Join to find the deleted record, inserted record or updated record. How about the performance of this method comparing the Join method ?

Best regards,


Hery

Answer 24

JOINs are usually faster, assuming appropriate indexes exist. However, using lookups is extremely useful in situations where the source data is not in a relational table, or the source table and the reference table are on two different servers. Also, lookups cache data, so millions of rows aren't necessarily a problem.

 

Answer 25

Phil Brammer wrote:

Microsoft has implemented a MERGE T-SQL function in SQL Server 2008...  That would also work. See John Welch's post here about it: http://agilebi.com/cs/blogs/jwelch/archive/2007/06/16/using-the-merge-statement-in-sql-server-2008.aspx

 

It appears that the domain name for this URL has been, how do you verb this, "squatted upon".

 

Answer 26

John Saunders wrote:

Phil Brammer wrote: Microsoft has implemented a MERGE T-SQL function in SQL Server 2008...  That would also work. See John Welch's post here about it: http://agilebi.com/cs/blogs/jwelch/archive/2007/06/16/using-the-merge-statement-in-sql-server-2008.aspx   It appears that the domain name for this URL has been, how do you verb this, "squatted upon".

 

Why do you say this?  The link works fine for me.  The DNS servers for the domain both respond, and return the correct IP address for the domain (along with the WWW record)

 

 

 

 

Answer 27

The site was down for a few hours (John Saunders alerted me to it - thanks again for that, John). I have the site hosted with one provider and the name registered with another one. For some reason, they choose not to make me aware that domain registration was expiring. It was fixed within about 3 hours of it going down, and should be good from this point forward.

Answer 28

Hi Phil, first of all, great post and many thanks for sharing this article.

 

I like a lot the first solution, which I addopted.

http://blogs.conchango.com/jamiethomson/archive/2006/09/12/SSIS_3A00_-Checking-if-a-row-exists-and-if-it-does_2C00_-has-it-changed.aspx

I just have a question to make. Since we do this lookup to see if it is an update or insert, in the data-flow, I would like to proceed to update and insert tasks. For insert, I use OLE DB DESTINATION inside of the data-flow (when we get the "error" from lookup). Till here, no problem.

 

But for update task, what you suggest? I think that I should take my key field, and update by this reference, right? but where? Can this be done at the end of this data-flow task, where there are the UNION ALL tasks? Or can this only be done outside data-flow, in the Control Flow, with SQL Task? If so, I should take in consideration, if data-flow ended with error or correctly, to know if it was an insert, or can I do the update respectively. Am I right, or I missed something here?

 

Thanks.

Answer 29

You can use an OLE DB Command to issue Updates inside the data flow. That can slow performance, though, since the updates are issued row by row. A better performing technique is to send the rows to be updated to a second OLE DB Destination so that they are saved in a working table, and then use an Execute SQL after the data flow to perform a batch update.

 

Answer 30

Hi all.

Thanks jwelch for the reply. Well, I'm trying to make the update inside the data-flow, using OLE DB DESTINATION. Here is the link: http://i85.photobucket.com/albums/k61/marcoadf/dts.jpg

 

As we can see, I have the lookup before. The insert is working fine with the "error" from lookup. I just have a OLE DB DESTINATION to the table where I wanna insert and mapp the columns. 

 

My problem is how can I make the update here. How can I make my sql to update (column x=1) where the line to be updated is equal to the column returned from lookup (y=?). That's the only way I can make the update right? Or the solution is something like:

 

UPDATE T

SET X=?

FROM TABLE T

WHERE Y=?

 

and the ? are ordered as the resultset coming from lookup, am I right? This means that first column must be the updatable column and the second ? must be the column_cod that is the key to update right?

 

Thanks in advance.

 

Answer 31

You can't do this. You will have to use an OLE DB Command Transform component instead.

 

Answer 32

marcoadf wrote:

Hi all. Thanks jwelch for the reply. Well, I'm trying to make the update inside the data-flow, using OLE DB DESTINATION. Here is the link: http://i85.photobucket.com/albums/k61/marcoadf/dts.jpg   As we can see, I have the lookup before. The insert is working fine with the "error" from lookup. I just have a OLE DB DESTINATION to the table where I wanna insert and mapp the columns.    My problem is how can I make the update here. How can I make my sql to update (column x=1) where the line to be updated is equal to the column returned from lookup (y=?). That's the only way I can make the update right? Or the solution is something like:   UPDATE T SET X=? FROM TABLE T WHERE Y=?   and the ? are ordered as the resultset coming from lookup, am I right? This means that first column must be the updatable column and the second ? must be the column_cod that is the key to update right?   Thanks in advance.

 

In the second OLE DB Destination, you need to write the rows to a second, working table. Then you'd issue a batch UPDATE in an Execute SQL.

 

UPDATE table

SET colA = tableB.colA

FROM table, tableB

WHERE table.ID = tableB.ID