We Keep Coding

How to we skip error with SQLBulkCopy [duplicate]

I am trying to insert huge amount of data into SQL server. My destination table has an unique index called "Hash".
I would like to replace my SqlDataAdapter implementation with SqlBulkCopy. In SqlDataAapter there is a property called "ContinueUpdateOnError", when set to true adapter.Update(table) will insert all the rows possible and tag the error rows with RowError property.
The question is how can I use SqlBulkCopy to insert data as quickly as possible while keeping track of which rows got inserted and which rows did not (due to the unique index)?
Here is the additional information:
The process is iterative, often set on a schedule to repeat.
The source and destination tables can be huge, sometimes millions of rows.
Even though it is possible to check for the hash values first, it requires two transactions per row (first for selecting the hash from destination table, then perform the insertion). I think in the adapter.update(table)'s case, it is faster to check for the RowError than checking for hash hits per row.

SqlBulkCopy, has very limited error handling facilities, by default it doesn't even check constraints.
However, its fast, really really fast.
If you want to work around the duplicate key issue, and identify which rows are duplicates in a batch. One option is:
start tran
Grab a tablockx on the table select all current "Hash" values and chuck them in a HashSet.
Filter out the duplicates and report.
Insert the data
commit tran
This process will work effectively if you are inserting huge sets and the size of the initial data in the table is not too huge.
Can you please expand your question to include the rest of the context of the problem.
EDIT
Now that I have some more context here is another way you can go about it:
Do the bulk insert into a temp table.
start serializable tran
Select all temp rows that are already in the destination table ... report on them
Insert the data in the temp table into the real table, performing a left join on hash and including all the new rows.
commit the tran
That process is very light on round trips, and considering your specs should end up being really fast;

Slightly different approach than already suggested; Perform the SqlBulkCopy and catch the SqlException thrown:
Violation of PRIMARY KEY constraint 'PK_MyPK'. Cannot insert duplicate
key in object 'dbo.MyTable'. **The duplicate key value is (17)**.
You can then remove all items from your source from ID 17, the first record that was duplicated. I'm making assumptions here that apply to my circumstances and possibly not yours; i.e. that the duplication is caused by the exact same data from a previously failed SqlBulkCopy due to SQL/Network errors during the upload.

Note: This is a recap of Sam's answer with slightly more details
Thanks to Sam for the answer. I have put it in an answer due to comment's space constraints.
Deriving from your answer I see two possible approaches:
Solution 1:
start tran
grab all possible hit "hash" values by doing "select hash in destinationtable where hash in (val1, val2, ...)
filter out duplicates and report
insert data
commit tran
solution 2:
Create temp table to mirror the
schema of destination table
bulk insert into the temp table
start serializable transaction
Get duplicate rows: "select hash from
tempTable where
tempTable.hash=destinationTable.hash"
report on duplicate rows
Insert the data in the temp table
into the destination table: "select * into destinationTable from temptable left join temptable.hash=destinationTable.hash where destinationTable.hash is null"
commit the tran
Since we have two approaches, it comes down to which approach is the most optimized? Both approaches have to retrieve the duplicate rows and report while the second approach requires extra:
temp table creation and delete
one more sql command to move data from temp to destination table
depends on the percentage of hash collision, it also transfers a lot of unnecessary data across the wire
If these are the only solutions, it seems to me that the first approach wins. What do you guys think? Thanks!

MySQL insert/update records very slow

I found out mysql update records very slow.
I have a logic whereby, if the records exist then, it will update, else it will insert.
but the total record for source database having 14k rows, and it only successfully inserted each row in few sec, therefore, it is very slow, how can i make it 1 minutes to insert 14k rows?
by the way, i'm using c#
foreach (row in rows)
checkrecords(row); // if exist update, else create.
Regards,
MH

If exists may be the root of the problem.
Make sure you are using a unique index on your check whether the record exists or not.

Try setting values:
innodb_flush_log_at_trx_commit=2
innodb_flush_method=O_DIRECT (for non-windows machine)
innodb_buffer_pool_size=25GB (currently it is close to 21GB)
innodb_doublewrite=0
innodb_support_xa=0
innodb_thread_concurrency=0...1000 (try different values, beginning with 200)
References:
MySQL docs for description of different variables.
MySQL Server Setting Tuning
MySQL Performance Optimization basics
Hope it helps...
Please refer Why is MySQL InnoDB insert so slow?
Source

Restructure the query and be sure the indexes are set up correctly.
Load the rows to be updated into a temporary table. Then do the insert/update as:
insert into t(. . .)
select . . .
from temptable tt
on duplicate key update col1 = values(col1), . . .;
Make sure that the conditions that you are checking for the existence of the record are combined into a unique index. So, if you are looking for a combination of three columns, then make sure you have a unique index on t(col1, col2, col3).

Can't insert to sqlite using c#, "id's are not unique" but in the sqlite app it works

My SQL query is
string stringSQL = "Insert into table(clientid, contractorid, driverid)
values (1,1,last_insert_rowid() + 1)"
ExecuteNonQuery(stringSQL)
And the error I get is:
Error: abort due to constraint
violation(clientid, contractorid,
driverid are not unique.
Btw those columns are my primary keys!
Is there an issue using SQLite's functions in c# vs 2010?
Thx I advance

From the last_insert_rowid() documentation:
The last_insert_rowid() function returns the ROWID of the last row insert from the database connection which invoked the function.
Note the "from the database connection" part... which means if there's already data in your table before you open this connection, presumably it's going to start from 0 or 1 again, and end up with a conflict. In other words, I don't think you can use this as a general way of incrementing row IDs.
I'd expect this to work if the table was empty before opening your current database connection, and if that connection is the only thing to have inserted data into that table though... you might want to test that part, just to make sure we understand what's going on.
That's assuming the docs are correct, of course - I've never actually used sqlite myself...

Violation of UNIQUE KEY Contstraints - Cannot insert duplicate key

I am pulling out my hair over here. I have a table that has a UNIQUE Key (IX_tblTable) and the unique key is on a column Number. I am parsing some data from the web and storing it in the table. My latest collection of data from the web contains number THAT ARE NOT CONTAINED IN THE DATABASE. so I am getting data from the site and all of the data I am getting in unique, there are no duplicates and the numbers in the list that is returned are not in the database.
I keep getting this error everytime I try to update the database, what is the best way to trap the error to see which number is throwing the error. I am storing everything that comes back in an object list and when it is done running I have 131 records that need to be inserted and I can not see which one is throwing this error. What is the bast way to trap it?
EDIT: I am using sql server 2005, wirtting in C# and using Linq2SQL. I can not post any c# code at this time for proprietary reasons...

Can you just disable your constraint for a while and see what duplicates save? Later you can remove duplicates and re-enable it.

Create a copy of the table without a primary key or uniqueness constraint (having the column, just not as a primary). Modify your code to insert into that table. Run it. Select values having more than one duplicate.

You can use an algorithm called binary search to find the 'wrong' data.
Suppose you have 131 lines of INSERT INTO ... VALUES(...) and you want to catch the one that's causing the error, you can divide your 131 lines into two pices(first 66 and last 65). Now run that 66 and 65 INSERTs separatelly, and see which throws the error. Continue to 'divide an try' until you get to one single line. (That's 10 tries in the worst case).

Are you controlling the lifecycle of your datacontext?
Insert 5
SubmitChanges (record inserted, no exception)
Insert 5
SubmitChanges (duplicateKeyException on 5)
Insert 6
SubmitChanges (duplicateKeyException on 5)

Why not use Begin Try... End Try.. Begin Catch... End Catch... in the SQL store procedure (I assume you use the SP to insert data) to capture the Row that causes the unique constraint violation?

SqlBulkCopy Error handling / continue on error

I am trying to insert huge amount of data into SQL server. My destination table has an unique index called "Hash".
I would like to replace my SqlDataAdapter implementation with SqlBulkCopy. In SqlDataAapter there is a property called "ContinueUpdateOnError", when set to true adapter.Update(table) will insert all the rows possible and tag the error rows with RowError property.
The question is how can I use SqlBulkCopy to insert data as quickly as possible while keeping track of which rows got inserted and which rows did not (due to the unique index)?
Here is the additional information:
The process is iterative, often set on a schedule to repeat.
The source and destination tables can be huge, sometimes millions of rows.
Even though it is possible to check for the hash values first, it requires two transactions per row (first for selecting the hash from destination table, then perform the insertion). I think in the adapter.update(table)'s case, it is faster to check for the RowError than checking for hash hits per row.

SqlBulkCopy, has very limited error handling facilities, by default it doesn't even check constraints.
However, its fast, really really fast.
If you want to work around the duplicate key issue, and identify which rows are duplicates in a batch. One option is:
start tran
Grab a tablockx on the table select all current "Hash" values and chuck them in a HashSet.
Filter out the duplicates and report.
Insert the data
commit tran
This process will work effectively if you are inserting huge sets and the size of the initial data in the table is not too huge.
Can you please expand your question to include the rest of the context of the problem.
EDIT
Now that I have some more context here is another way you can go about it:
Do the bulk insert into a temp table.
start serializable tran
Select all temp rows that are already in the destination table ... report on them
Insert the data in the temp table into the real table, performing a left join on hash and including all the new rows.
commit the tran
That process is very light on round trips, and considering your specs should end up being really fast;

Slightly different approach than already suggested; Perform the SqlBulkCopy and catch the SqlException thrown:
Violation of PRIMARY KEY constraint 'PK_MyPK'. Cannot insert duplicate
key in object 'dbo.MyTable'. **The duplicate key value is (17)**.
You can then remove all items from your source from ID 17, the first record that was duplicated. I'm making assumptions here that apply to my circumstances and possibly not yours; i.e. that the duplication is caused by the exact same data from a previously failed SqlBulkCopy due to SQL/Network errors during the upload.

Note: This is a recap of Sam's answer with slightly more details
Thanks to Sam for the answer. I have put it in an answer due to comment's space constraints.
Deriving from your answer I see two possible approaches:
Solution 1:
start tran
grab all possible hit "hash" values by doing "select hash in destinationtable where hash in (val1, val2, ...)
filter out duplicates and report
insert data
commit tran
solution 2:
Create temp table to mirror the
schema of destination table
bulk insert into the temp table
start serializable transaction
Get duplicate rows: "select hash from
tempTable where
tempTable.hash=destinationTable.hash"
report on duplicate rows
Insert the data in the temp table
into the destination table: "select * into destinationTable from temptable left join temptable.hash=destinationTable.hash where destinationTable.hash is null"
commit the tran
Since we have two approaches, it comes down to which approach is the most optimized? Both approaches have to retrieve the duplicate rows and report while the second approach requires extra:
temp table creation and delete
one more sql command to move data from temp to destination table
depends on the percentage of hash collision, it also transfers a lot of unnecessary data across the wire
If these are the only solutions, it seems to me that the first approach wins. What do you guys think? Thanks!