I have a c#.net application in which I need to insert the default value from application to sql server by using sql bulkcopy.
Example:
SqlColumnMapping("src_col1","dest_col1");
SqlColumnMapping("src_col2","dest_col2");
in "dest_col3", I would like to insert default value.
How could I map it in app and how the default value can be inserted in database?
Thanks
Hint: do not use SqlBulkCopy - that thing has tons of problems. Most around locking, default values also are in the game.
Use it against a temporary table ;)
THis is what I do.
Create a temp table with the proper field structure. You can make fields nullable here if they have a default value (information_schema can help you find it). THis step can be automated - 100% and it is not that hard.
SqlBulkCopy into the temp table. No locking issues.
After that you can run updates for default values ;)
INSERT INTO the final table.
Problems with SqlBulkCopy locking:
Locks the table. Exclusively.
It does not wait. It tries to get a lock, immediately. If that fails it retries. If the table is busy, it never gets the lock as it never waits until it gets one - and every new request is end of the queue.
We got hit badly by that in a ETL scenario some years back.
On top, as you found out, you can not work with default values.
I actually have that stuff totally isolated now in a separate bulk loader class and am just in the process of allowing this to UPDATE rows (by merging from the temp table).
Here's how you do it. Create a DataTable object that has the same structure as your desitination table, except remove the columns that have a default value. If you are using DataSet Designer in Visual Studio, you can remove the columns that have default values from the TableAdapter.
Using an SqlConnection called "connection" and a DataTable object called "table", your code would look something like:
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(connection))
{
foreach (System.Data.DataColumn c in table.Columns)
{
bulkCopy.ColumnMappings.Add(c.ColumnName, c.ColumnName);
}
bulkCopy.DestinationTableName = table.TableName;
bulkCopy.WriteToServer(table);
}
Again, in order to use this method, you have to ensure that your DataTable object does not contain the columns that you would like to insert with default values.
Related
I have a DataSet with two TableAdapters (1 to many relationship) that was created using visual studio 2010's Configuration Wizard.
I make a call to an external source and populate a Dictionary with the results. These results should be all of the entries in the database. To synchronize the DB I don't want to just clear all of the tables and then repopulate them like dropping the tables and creating them with new data in sql.
Is there a clean way possibly using the TableAdapter.Fill() method or do I have to loop through the two tables row by row and decide if it stay or gets deleted and then add the new entries? What is the best approach to make the data that is in the dictionary be the only data in my two tables with the DataSet?
First Question: if it's the same DB why do you have 2 tables with the same information?
To the question at hand: that largley depend on the sizes. If the tables are not big then use a transaction, clear the table (DELETE * FROM TABLE or whatever) and write your data in there again.
If the tables are big on the other hand the question is: can you load all this into your dictionary?
Of course you have to ask yourself what happens to inconsistent data (another user/app changed the data while you had it in your dictionary).
If this takes to long you could remember what you did to the data - that means: flag the changed data and remember the deleted keys and new inserted rows and make your updates based on that.
Both can be achieved by remembering the Filled DataTable and use this as backing field or by implementing your own mechanisms.
In any way I would recommend think on the problem: do you really need the dictionary? Why not make queries against the database to get the data? Or only cache a part of the data for quick access?
PS: the update method on you DataAdapter will do all the work (changing the changed, removing the deleted and inserting the new datarows but it will update the DataTable/Set so this will only work once)
It could be that it is quicker to repopulate the entire table than to itterate through and decide what record go / stay. Could you not do the process of deciding if a records is deleteed via an sql statement ? (Delete from table where active = false) if you want them to stay in the database but not in the dataset (select * from table where active = true)
You could have a date field and select all records that have been added since the date you late 'pooled' the database (select * from table where active = true and date-added > #12:30#)
I'm having an issue with writing back to my Access Database (.accdb) through using a DataAdapter.
Currently, I have a Fill method which fills a DataSet Table up with data. This piece of code is currently sitting within my Form_Load().
// TODO: This line of code loads data into the 'hCAliasDataSet.Topics' table. You can move, or remove it, as needed.
this.topicsTableAdapter.Fill(this.hCAliasDataSet.Topics);
Then I have an cmdAdd_Click() event, this is obviously to add a new row into the Topcis table that sits within the hCAliasDataSet.
// Create a new row, append it to Topics table.
DataRow DR;
DR = this.hCAliasDataSet.Tables["Topics"].NewRow();
this.hCAliasDataSet.Tables["Topics"].Rows.Add(DR);
Next, I've created some code to caputre the input of one of the column values.
// Capture user input
sTopicName = Interaction.InputBox("Please Enter Topic Name", "Topic Name", null, 100, 100);
// Set the Topic value for the new Row
DR["Topic"] = sTopicName;
My problem, I'm assuming is here, where I call the dataAdapter to update the Topics table. I manually check the database, and there aren't any new rows created?
// Commit the changes back to the hCAlias DataBase (hcalias.accdb).
this.topicsTableAdapter.Update(this.hCAliasDataSet.Topics);
Edit: I believe I'm needing to create an INSERT query, for my TableAdapter, would this be correct?
The adapter should generate the insert statement automatically for you behind the scenes.
Your code looks right, but it's possible that you have a constraint on one of your columns that makes it unable to save. (like a non-nullable column that you didn't specify any data for). But, you'd usually get an exception if there was a constraint that cancelled the insert.
You can try one of these alternatives, but they're basically the same thing:
this.topicsTableAdapter.Update(this.hCAliasDataSet);
this.topicsTableAdapter.Update(DR);
this.topicsTableAdapter.Insert(sTopicName); // add any other columns in here
You should call AcceptChanges in your dataset:
hCAliasDataSet.AcceptChanges();
And then commit to the database with your TableAdapter.
The Update() method just updates existing records, you'll need to use the TableAdapter Insert() method to add a new row. VC# will have created a default Insert() method for you, (which may be overloaded)... but there will be a method that will let you explicitly insert values....
For example...
this.topicsTableAdapter.Insert(int Column1, string Column2 etc etc)
This will create a new row in your database and populate it with the values you specify.
I want to insert many rows (constructed from Entity Framework objects) to SQL Server. The problem is, some of string properties have length exceeded length of column in database, which causes an exception, and then all of rows will unable to insert into database.
So I wonder that if there is a way to tell SqlBulkCopy to automatically truncate any over-length rows? Of course, I can check and substring each property if it exceeds the limited length, before insert it in to a DataTable, but it would slow down the whole program.
Always use a staging/load table for bulk actions.
Then you can process, clean, scrub etc the data before flushing to the real table. This includes, LEFTs, lookups, de-duplications etc
So:
Load a staging table with wide columns
Flush from staging to "real" table using INSERT realtable (..) SELECT LEFT(..), .. FROM Staging
Unfortunately there is no direct way of doing that with SqlBulkCopy. SQL Bulk Inserts are by nature almost "dumb" but that's why they are so fast. They aren't even logged (except capturing SqlRowsCopied event) so if something fails, there's not much information. What you're looking for would in a way, be counter to the purpose of this class
But there can be 2 possible ways:
You can try using SqlBulkCopyOptionsEnumeration (passed to SqlBulkCopy() Constructor) and use SqlBulkCopyOptions.CheckConstraints (Check constraints while data is being inserted. By default, constraints are not checked.).
Or you can use SqlBulkCopyOptions.FireTriggers Enumeration (When specified, cause the server to fire the insert triggers for the rows being inserted into the database.) and handle the exception in SQL Server Insert Trigger.
Try Using SQLTransaction Class while using SQLBulkCopy Class
i have a many-to-many relationship table in a typed DataSet.
For convenience on an update i'm deleting old relations before i'm adding the new(maybe the same as before).
Now i wonder if this way is failsafe or if i should ensure only to delete which are really deleted(for example with LINQ) and only add that one which are really new.
In SQL-Server is a unique constraint defined for the relation table, the two foreign keys are a composite primary key.
Is the order the DataAdapter updates the DataRows which RowState are <> Unchanged predictable or not?
In other words: is it possible that DataAdapter.Update(DataTable) will result in an exception when the key already exists?
This is the datamodel:
This is part of the code(LbSymptomCodes is an ASP.Net ListBox):
Dim daTrelRmaSymptomCode As New ERPModel.dsRMATableAdapters.trelRMA_SymptomCodeTableAdapter
For Each oldTrelRmaSymptomCodeRow As ERPModel.dsRMA.trelRMA_SymptomCodeRow In thisRMA.GettrelRMA_SymptomCodeRows
oldTrelRmaSymptomCodeRow.Delete()
Next
For Each item As ListItem In LbSymptomCodes.Items
If item.Selected Then
Dim newTrelRmaSymptomCodeRow As ERPModel.dsRMA.trelRMA_SymptomCodeRow = Services.dsRMA.trelRMA_SymptomCode.NewtrelRMA_SymptomCodeRow
newTrelRmaSymptomCodeRow.fiRMA = Services.IdRma
newTrelRmaSymptomCodeRow.fiSymptomCode = CInt(item.Value)
Services.dsRMA.trelRMA_SymptomCode.AddtrelRMA_SymptomCodeRow(newTrelRmaSymptomCodeRow)
End If
Next
daTrelRmaSymptomCode.Update(Services.dsRMA.trelRMA_SymptomCode)
Thank you in advance.
I think that the DataAdapter in ADO.NET is clever enough to perform the delete/inserts in the correct order.
However, if you really want to ensure that updates are done in the correct order you should do it manually by using the Select method to return an array of data rows for each particular row state. You could then call the Update method on the array of data rows
DataTable tbl = ds.Tables["YourTable"];
// Process any Deleted rows first
adapter.Update(tbl.Select(null, null, DataViewRowState.Deleted));
// Process any Updated/Modified rows
adapter.Update(tbl.Select(null, null, DataViewRowState.ModifiedCurrent));
// Process the Inserts last
adapter.Update(tbl.Select(null, null, DataViewRowState.Added));
Not sure about the DA but in theory DB transactions should be performed in the following order Deletes, Inserts, Updates.
looking at msdn the exact wording for the update method is
Blockquote
Attempts to save all changes in the DataTable to the database. (This includes removing any rows deleted from the table, adding rows inserted to the table, and updating any rows in the table that have changed.)
Blockquote
In regards to your solution of deleting items and possibly re-inserting the same items, typically speaking this should be avoided because it creates a load on the DB. In high volume applications you want to do everything you can to minimize calls to the DB as they are very expensive; computation time, from determining which row updates are spurious, is cheap.
I have a piece of code which copies data from an excel spreadsheet to a MSSQL table using DataReader and SqlBulkCopy. It worked fine until I created a primary key on the table and now it fails. I am first deleting the contents of the SQL table before filling it again with the data from excel.
As it is only a small amount of data I am moving, I wondered if there was a better way to do this than using BulkCopy?
Update: below is the relative code and the error I receive is:
"The given value of type String from the data source cannot be converted to type float of the specified target column."
using (OleDbConnection connection = new OleDbConnection(excelConnectionString))
{
connection.Open();
OleDbCommand cmd = new OleDbCommand
("SELECT Name, Date, Amount FROM ExcelNamedRange", connection);
using (OleDbDataReader dr = cmd.ExecuteReader())
{
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(sqlConnectionString))
{
bulkCopy.DestinationTableName = "SqlTable";
bulkCopy.WriteToServer(dr);
}
}
}
SqlBulkCopy automatically maps the fields. But since you added a primary key that default mapping is no longer valid.
You will have to set ColumnMapping to tell your SqlBulkCopy object explicityly how to map the fields.
Do this for all your fields, except the primary key (assuming you use an identity on the PK).
For example:
_bulkCopyEngine.ColumnMappings.Add("fieldname_from", "fieldname_to");
Creating a primary key, suggests you are enforcing a domain constraint (a good thing).
Therefore, your actual problem is not that you need another way to perform the bulk insert, but that you need to find out why you have duplicate keys (the precise reason for enforcing the PK).
BulkCopy should work just fine, so your problem seems to be a duplicate key (what's the error message?). You either have data that that is wrong there, or the primary key you created is too narrow.
What you could also do is push the data into a staging table first (no keys/ indexes etc, just a plain table) and then use an update (merge when on 2008) statement to put it into the actual table.
GJ
ok that seems to be a different problem altogether, there seems to be a value in year ExcelNamedRange that cannot be cast as one of the columns in SqlTable. Can you see any? Maybe division by 0 error etc?
Also make sure the columns line up. Not sure exactly how SqlBulkCopy maps the columns, I think it just puts the first column from NamedRange into the first column of SqlTale etc. So make sure they;re in the right order. (or see what happens if you change the names)
The BulkCopy is fastest way, how to insert data into MSSQL from C#.