Related
I will be taking in around 1000 records at a given time and I have to determine if they are existing records or new records.
If they are existing I have to update the records, if new then just insert them. I will not know if any of them will be existing or if they all will be existing.
I thought that it might be best to do one query to the database and try to find if any of them exist in the db and store them in memory and check that collection in memory and check that.
Originally I was told I 1 field would be enough to determine uniqueness. So I thought I could just do 1 big in clause against 1 field in the database but now I found out that is not the case and I need to use 3 fields to determine if the record is existing or now.
This is basically an and clause
select * from where columnA = "1" and ColumnB = "2" and ColumnC = "3"
How can I properly right this in C# ado.net?
I am guessing I going to need to have like some super where clause?
select * from where (columnA = "1" and ColumnB = "2" and ColumnC = "3") or (columnA = "4" and ColumnB = "5" and ColumnC = "6") or [....998 more conditional clauses)
I am open to better ideas if possible. I still think doing it in 1 shot is better than doing 1000 separate queries.
I can only help you to write query for your request
var recordCount = 1000;
var query = "SELECT * FROM TableName WHERE";
for (var i = 1; i < recordCount - 2; i += 3)
{
query += " (columnA = " + i + " and ColumnB = " + (i + 1) + " and ColumnC = " + (i + 2) + ") or ";
}
I feel kinda silly writing this answer, because I think that you should be able to put the pieces together a complete answer from other posts - but this is not an exact duplicate of either of the questions I have in mind.
There already are questions and answers in Stackoverflow dealing with this issue - however in my search I only found answers that are not thread safe, and most of them are using merge.
There are different questions and answers I can refer you to such as
my answer to Adding multiple parameterized variables to a database in c#
where you can see how to work with table valued parameters on c#, and to Aaron Bertrand's answer to Using a if condition in an insert SQL Server where you can see how to create a safe upsert - however I didn't find any answer that covers this completely - so here you go:
First you need to create a user defined table type in your database:
CERATE TYPE MyTableType AS TABLE
(
Column1 int NOT NULL,
Column2 int NOT NULL,
Column3 int NOT NULL,
-- rest of the columns in your table goes here
PRIMARY KEY (Column1, Column2, Column3)
)
Then, you create the stored procedure:
CREATE stp_UpsertMyTable
(
#MyTableType dbo.MyTableType readonly -- table valued parameters must be readonly
)
AS
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
BEGIN TRANSACTION;
UPDATE t
SET t.column4 = tvp.column4,
t.column5 = tvp.column5 -- and so on for all columns that are not part of the key
FROM dbo.MyTable AS t
INNER JOIN #MyTableType AS tvp
ON t.Column1 = tvp.Column1
AND t.Column2 = tvp.Column2
AND t.Column3 = tvp.Column3;
-- Note: <ColumnsList> should be replaced with the actual columns in the table
INSERT dbo.MyTable(<ColumnsList>)
SELECT <ColumnsList>
FROM #MyTableType AS tvp
WHERE NOT EXISTS
(
SELECT 1 FROM dbo.MyTable t
WHERE t.Column1 = tvp.Column1
AND t.Column2 = tvp.Column2
AND t.Column3 = tvp.Column3
);
COMMIT TRANSACTION;
GO
Then, the c# part is simple:
DataTable dt = new DataTable();
dt.Columns.Add("Column1", typeof(int));
dt.Columns.Add("Column2", typeof(int));
dt.Columns.Add("Column3", typeof(int));
dt.Columns.Add("Column4", typeof(string));
dt.Columns.Add("Column5", typeof(string));
// Fill your data table here
using (var con = new SqlConnection("ConnectionString"))
{
using(var cmd = new SqlCommand("stp_UpsertMyTable", con))
{
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.Add("#MyTable", SqlDbType.Structured).Value = dt;
con.Open();
cmd.ExecuteNonQuery();
}
}
Now you have a complete and safe upsert using a table valued parameter with only one round trip between c# and sql server.
I have this code:
string insertSql =
"INSERT INTO aspnet_GameProfiles(UserId,GameId) VALUES(#UserId, #GameId)";
using (SqlConnection myConnection = new SqlConnection(myConnectionString))
{
myConnection.Open();
SqlCommand myCommand = new SqlCommand(insertSql, myConnection);
myCommand.Parameters.AddWithValue("#UserId", newUserId);
myCommand.Parameters.AddWithValue("#GameId", newGameId);
myCommand.ExecuteNonQuery();
myConnection.Close();
}
When I insert into this table, I have an auto_increment int primary key column called GamesProfileId, how can i get the last inserted one after this so I can use that id to insert into another table?
For SQL Server 2005+, if there is no insert trigger, then change the insert statement (all one line, split for clarity here) to this
INSERT INTO aspnet_GameProfiles(UserId,GameId)
OUTPUT INSERTED.ID
VALUES(#UserId, #GameId)
For SQL Server 2000, or if there is an insert trigger:
INSERT INTO aspnet_GameProfiles(UserId,GameId)
VALUES(#UserId, #GameId);
SELECT SCOPE_IDENTITY()
And then
Int32 newId = (Int32) myCommand.ExecuteScalar();
You can create a SqlCommand with CommandText equal to
INSERT INTO aspnet_GameProfiles(UserId, GameId) OUTPUT INSERTED.ID VALUES(#UserId, #GameId)
and execute int id = (int)command.ExecuteScalar.
This MSDN article will give you some additional techniques.
string insertSql =
"INSERT INTO aspnet_GameProfiles(UserId,GameId) VALUES(#UserId, #GameId)SELECT SCOPE_IDENTITY()";
int primaryKey;
using (SqlConnection myConnection = new SqlConnection(myConnectionString))
{
myConnection.Open();
SqlCommand myCommand = new SqlCommand(insertSql, myConnection);
myCommand.Parameters.AddWithValue("#UserId", newUserId);
myCommand.Parameters.AddWithValue("#GameId", newGameId);
primaryKey = Convert.ToInt32(myCommand.ExecuteScalar());
myConnection.Close();
}
This will work.
I had the same need and found this answer ..
This creates a record in the company table (comp), it the grabs the auto ID created on the company table and drops that into a Staff table (staff) so the 2 tables can be linked, MANY staff to ONE company. It works on my SQL 2008 DB, should work on SQL 2005 and above.
===========================
CREATE PROCEDURE [dbo].[InsertNewCompanyAndStaffDetails]
#comp_name varchar(55) = 'Big Company',
#comp_regno nchar(8) = '12345678',
#comp_email nvarchar(50) = 'no1#home.com',
#recID INT OUTPUT
-- The '#recID' is used to hold the Company auto generated ID number that we are about to grab
AS
Begin
SET NOCOUNT ON
DECLARE #tableVar TABLE (tempID INT)
-- The line above is used to create a tempory table to hold the auto generated ID number for later use. It has only one field 'tempID' and its type INT is the same as the '#recID'.
INSERT INTO comp(comp_name, comp_regno, comp_email)
OUTPUT inserted.comp_id INTO #tableVar
-- The 'OUTPUT inserted.' line above is used to grab data out of any field in the record it is creating right now. This data we want is the ID autonumber. So make sure it says the correct field name for your table, mine is 'comp_id'. This is then dropped into the tempory table we created earlier.
VALUES (#comp_name, #comp_regno, #comp_email)
SET #recID = (SELECT tempID FROM #tableVar)
-- The line above is used to search the tempory table we created earlier where the ID we need is saved. Since there is only one record in this tempory table, and only one field, it will only select the ID number you need and drop it into '#recID'. '#recID' now has the ID number you want and you can use it how you want like i have used it below.
INSERT INTO staff(Staff_comp_id)
VALUES (#recID)
End
-- So there you go. You can actually grab what ever you want in the 'OUTPUT inserted.WhatEverFieldNameYouWant' line and create what fields you want in your tempory table and access it to use how ever you want.
I was looking for something like this for ages, with this detailed break down, I hope this helps.
In pure SQL the main statement kools like:
INSERT INTO [simbs] ([En]) OUTPUT INSERTED.[ID] VALUES ('en')
Square brackets defines the table simbs and then the columns En and ID, round brackets defines the enumeration of columns to be initiated and then the values for the columns, in my case one column and one value. The apostrophes enclose a string
I will explain you my approach:
It might be not easy to understand but i hope useful to get the big picture around using the last inserted id. Of course there are alternative easier approaches. But I have reasons to keep mine. Associated functions are not included, just their names and parameter names.
I use this method for medical artificial intelligence
The method check if the wanted string exist in the central table (1). If the wanted string is not in the central table "simbs", or if duplicates are allowed, the wanted string is added to the central table "simbs" (2). The last inseerted id is used to create associated table (3).
public List<int[]> CreateSymbolByName(string SymbolName, bool AcceptDuplicates)
{
if (! AcceptDuplicates) // check if "AcceptDuplicates" flag is set
{
List<int[]> ExistentSymbols = GetSymbolsByName(SymbolName, 0, 10); // create a list of int arrays with existent records
if (ExistentSymbols.Count > 0) return ExistentSymbols; //(1) return existent records because creation of duplicates is not allowed
}
List<int[]> ResultedSymbols = new List<int[]>(); // prepare a empty list
int[] symbolPosition = { 0, 0, 0, 0 }; // prepare a neutral position for the new symbol
try // If SQL will fail, the code will continue with catch statement
{
//DEFAULT und NULL sind nicht als explizite Identitätswerte zulässig
string commandString = "INSERT INTO [simbs] ([En]) OUTPUT INSERTED.ID VALUES ('" + SymbolName + "') "; // Insert in table "simbs" on column "En" the value stored by variable "SymbolName"
SqlCommand mySqlCommand = new SqlCommand(commandString, SqlServerConnection); // initialize the query environment
SqlDataReader myReader = mySqlCommand.ExecuteReader(); // last inserted ID is recieved as any resultset on the first column of the first row
int LastInsertedId = 0; // this value will be changed if insertion suceede
while (myReader.Read()) // read from resultset
{
if (myReader.GetInt32(0) > -1)
{
int[] symbolID = new int[] { 0, 0, 0, 0 };
LastInsertedId = myReader.GetInt32(0); // (2) GET LAST INSERTED ID
symbolID[0] = LastInsertedId ; // Use of last inserted id
if (symbolID[0] != 0 || symbolID[1] != 0) // if last inserted id succeded
{
ResultedSymbols.Add(symbolID);
}
}
}
myReader.Close();
if (SqlTrace) SQLView.Log(mySqlCommand.CommandText); // Log the text of the command
if (LastInsertedId > 0) // if insertion of the new row in the table was successful
{
string commandString2 = "UPDATE [simbs] SET [IR] = [ID] WHERE [ID] = " + LastInsertedId + " ;"; // update the table by giving to another row the value of the last inserted id
SqlCommand mySqlCommand2 = new SqlCommand(commandString2, SqlServerConnection);
mySqlCommand2.ExecuteNonQuery();
symbolPosition[0] = LastInsertedId; // mark the position of the new inserted symbol
ResultedSymbols.Add(symbolPosition); // add the new record to the results collection
}
}
catch (SqlException retrieveSymbolIndexException) // this is executed only if there were errors in the try block
{
Console.WriteLine("Error: {0}", retrieveSymbolIndexException.ToString()); // user is informed about the error
}
CreateSymbolTable(LastInsertedId); //(3) // Create new table based on the last inserted id
if (MyResultsTrace) SQLView.LogResult(LastInsertedId); // log the action
return ResultedSymbols; // return the list containing this new record
}
I tried the above but they didn't work, i found this thought, that works a just fine for me.
var ContactID = db.GetLastInsertId();
Its less code and i easy to put in.
Hope this helps someone.
You can also use a call to SCOPE_IDENTITY in SQL Server.
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using System.Data.SqlClient;
namespace DBDemo2
{
public partial class Form1 : Form
{
string connectionString = "Database=company;Uid=sa;Pwd=mypassword";
System.Data.SqlClient.SqlConnection connection;
System.Data.SqlClient.SqlCommand command;
SqlParameter idparam = new SqlParameter("#eid", SqlDbType.Int, 0);
SqlParameter nameparam = new SqlParameter("#name", SqlDbType.NChar, 20);
SqlParameter addrparam = new SqlParameter("#addr", SqlDbType.NChar, 10);
public Form1()
{
InitializeComponent();
connection = new System.Data.SqlClient.SqlConnection(connectionString);
connection.Open();
command = new System.Data.SqlClient.SqlCommand(null, connection);
command.CommandText = "insert into employee(ename, city) values(#name, #addr);select SCOPE_IDENTITY();";
command.Parameters.Add(nameparam);
command.Parameters.Add(addrparam);
command.Prepare();
}
private void Form1_Load(object sender, EventArgs e)
{
}
private void buttonSave_Click(object sender, EventArgs e)
{
try
{
int id = Int32.Parse(textBoxID.Text);
String name = textBoxName.Text;
String address = textBoxAddress.Text;
command.Parameters[0].Value = name;
command.Parameters[1].Value = address;
SqlDataReader reader = command.ExecuteReader();
if (reader.HasRows)
{
reader.Read();
int nid = Convert.ToInt32(reader[0]);
MessageBox.Show("ID : " + nid);
}
/*int af = command.ExecuteNonQuery();
MessageBox.Show(command.Parameters["ID"].Value.ToString());
*/
}
catch (NullReferenceException ne)
{
MessageBox.Show("Error is : " + ne.StackTrace);
}
catch (Exception ee)
{
MessageBox.Show("Error is : " + ee.StackTrace);
}
}
private void buttonSave_Leave(object sender, EventArgs e)
{
}
private void Form1_Leave(object sender, EventArgs e)
{
connection.Close();
}
}
}
There are all sorts of ways to get the Last Inserted ID but the easiest way I have found is by simply retrieving it from the TableAdapter in the DataSet like so:
<Your DataTable Class> tblData = new <Your DataTable Class>();
<Your Table Adapter Class> tblAdpt = new <Your Table Adapter Class>();
/*** Initialize and update Table Data Here ***/
/*** Make sure to call the EndEdit() method ***/
/*** of any Binding Sources before update ***/
<YourBindingSource>.EndEdit();
//Update the Dataset
tblAdpt.Update(tblData);
//Get the New ID from the Table Adapter
long newID = tblAdpt.Adapter.InsertCommand.LastInsertedId;
Hope this Helps ...
After inserting any row you can get last inserted id by below line of query.
INSERT INTO aspnet_GameProfiles(UserId,GameId)
VALUES(#UserId, #GameId);
SELECT ##IDENTITY
If you're using executeScalar:
cmd.ExecuteScalar();
result_id=cmd.LastInsertedId.ToString();
Maybe this answer helps as well as my database seems to have no column specified as "IDENTITY" (which is needed for "SELECT SCOPE_IDENTITY()" or "##IDENTITY" calls). Also my "ID" column was of type "binary(16)" so I needed to convert the output like stated below:
string returnId = BitConverter.ToString((byte[])cmd.ExecuteScalar()).Replace("-", "");
// skip the replace if you handle the hyphen otherwise
Use SELECT SCOPE_IDENTITY() in query
After this:
INSERT INTO aspnet_GameProfiles(UserId, GameId) OUTPUT INSERTED.ID VALUES(#UserId, #GameId)
Execute this
int id = (int)command.ExecuteScalar;
It will work
INSERT INTO aspnet_GameProfiles(UserId,GameId) VALUES(#UserId, #GameId)";
then you can just access to the last id by ordering the table in desc way.
SELECT TOP 1 UserId FROM aspnet_GameProfiles ORDER BY UserId DESC.
set ANSI_NULLS ON
set QUOTED_IDENTIFIER ON
GO
CREATE PROC [dbo].[spCountNewLastIDAnyTableRows]
(
#PassedTableName as NVarchar(255),
#PassedColumnName as NVarchar(225)
)
AS
BEGIN
DECLARE #ActualTableName AS NVarchar(255)
DECLARE #ActualColumnName as NVarchar(225)
SELECT #ActualTableName = QUOTENAME( TABLE_NAME )
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_NAME = #PassedTableName
SELECT #ActualColumnName = QUOTENAME( COLUMN_NAME )
FROM INFORMATION_SCHEMA.COLUMNS
WHERE COLUMN_NAME = #PassedColumnName
DECLARE #sql AS NVARCHAR(MAX)
SELECT #sql = 'select MAX('+ #ActualColumnName + ') + 1 as LASTID' + ' FROM ' + #ActualTableName
EXEC(#SQL)
END
I'm trying to delete records from my SQL Server database table by passing multiple values as comma separated string using the below query and ado.net ExecuteNonQuery() method. Here I am passing the table name, column name and comma separated values.
string delQuery = "DELETE FROM " + delTblName +
" WHERE " + delColumnName + " IN (" + toDel+ ")";
The deletion process is successful if the full list of values in toDel has no issues. Assume that if there is a foreign key conflict with any of the value, the entire statement will be terminated and no records will be deleted.
Since the number of elements in toDel variable is big I cannot use sequential processing and also not able to use a stored procedure as per the requirement.
I need to delete based on the successful elements and return the erroneous ones from this method. Any help would be appreciated.
Used the below code using Parameters
try
{
using (var sc = new SqlConnection(dbConnString))
using (var cmd = sc.CreateCommand())
{
sc.Open();
cmd.CommandText = "DELETE FROM #delTblName WHERE #delColumnName IN ( #valConcat) ";
cmd.Parameters.AddWithValue("#delTblName", tblName);
cmd.Parameters.AddWithValue("#delColumnName", delColumnName);
cmd.Parameters.AddWithValue("#valConcat", nosConcat);
cmd.CommandType = CommandType.Text;
rowsAffected = rowsAffected + cmd.ExecuteNonQuery();
}
}
catch
{
}
Got this error - Must declare the table variable "#delTblName".
Regardless of using stored procedures or not, you have two contradicting requirements:
You want to delete rows in batches, not one-by-one. For the sake of speed.
You want the delete operation to complete partially ignoring possible constraint violations.
I don't know how to make DELETE statement work partially. One statement is the minimal atomic amount of work, so if one row out of 1000 violates the constraint, the whole statement will be rolled back.
This leads to the following generic idea. Before attempting to DELETE in one statement that affects many rows, make sure that the whole list of affected rows can be deleted. Explicitly check that there is nothing in the data that can prevent the rows that you are going to delete from being deleted. The actual check depends on your constraints. Identify those rows that can't be deleted and remove them from the main DELETE list.
Example
Two tables - TestMaster and TestDetails with a one-to-many relationship.
CREATE TABLE [dbo].[TestMaster](
[ID] [int] NOT NULL,
[MasterData] [nvarchar](50) NOT NULL,
CONSTRAINT [PK_TestMaster] PRIMARY KEY CLUSTERED
(
[ID] ASC
))
CREATE TABLE [dbo].[TestDetails](
[ID] [int] NOT NULL,
[MasterID] [int] NOT NULL,
[DetailData] [nvarchar](50) NOT NULL,
CONSTRAINT [PK_TestDetails] PRIMARY KEY CLUSTERED
(
[ID] ASC
))
GO
ALTER TABLE [dbo].[TestDetails] WITH CHECK
ADD CONSTRAINT [FK_TestDetails_TestMaster] FOREIGN KEY([MasterID])
REFERENCES [dbo].[TestMaster] ([ID])
GO
ALTER TABLE [dbo].[TestDetails] CHECK CONSTRAINT [FK_TestDetails_TestMaster]
GO
Some sample data:
INSERT INTO [dbo].[TestMaster] ([ID],[MasterData]) VALUES
(1, 'Master1'),
(2, 'Master2'),
(3, 'Master3'),
(4, 'Master4');
INSERT INTO [dbo].[TestDetails] ([ID],[MasterID],[DetailData]) VALUES
(10, 1, 'Detail10'),
(11, 1, 'Detail11'),
(20, 2, 'Detail20');
Simple attempt to DELETE items (1, 2, 3, 4):
DELETE FROM [dbo].[TestMaster]
WHERE ID IN (1, 2, 3, 4);
This fails:
Msg 547, Level 16, State 0, Line 13
The DELETE statement conflicted with the REFERENCE constraint "FK_TestDetails_TestMaster". The conflict occurred in database "AdventureWorks2014", table "dbo.TestDetails", column 'MasterID'.
The statement has been terminated.
We can delete only those Master rows that don't have corresponding Details. In this example it is only 3, 4.
Find MasterID which can't be deleted first:
SELECT DISTINCT [dbo].[TestDetails].MasterID
FROM [dbo].[TestDetails]
WHERE [dbo].[TestDetails].MasterID IN (1, 2, 3, 4);
Which returns:
MasterID
1
2
Edit your original list of IDs and remove conflicting IDs from it, then you can run the final DELETE:
DELETE FROM [dbo].[TestMaster]
WHERE ID IN (3, 4);
Single query
Instead of running a separate SELECT, retrieving a list of conflicting IDs to the client over the network, adjusting the original list of IDs, running the final DELETE you can make a single query that does it all. For this simple example it can look like this:
DELETE FROM [dbo].[TestMaster]
WHERE
[dbo].[TestMaster].ID IN (1, 2, 3, 4)
AND [dbo].[TestMaster].ID NOT IN
(
SELECT [dbo].[TestDetails].MasterID
FROM [dbo].[TestDetails]
WHERE [dbo].[TestDetails].MasterID IN (1, 2, 3, 4)
);
This query will delete only two rows with IDs 3 and 4.
By Default if your sql statment fails then the transaction will be rolled back. And you cannot meet your need if you are doing through single statement where you need to get the erroneous one
May be you can follow the approach given below
try
{
using (var sc = new SqlConnection(dbConnString))
using (var cmd = sc.CreateCommand())
{
sc.Open();
var nosConcat = "1,2,3,4,5";
string failedIds = string.Empty;
var ids = nosConcat.Split(',');
string #delTblName = "sometable";
string #delColumnName = "somecolumn";
for(int i = 0 ; i < ids.Length ; i++)
{
try
{
cmd.CommandText = "DELETE FROM " + #delTblName + " WHERE " + #delColumnName + " = #valConcat ";
cmd.Parameters.AddWithValue("#valConcat", ids[i]);
cmd.CommandType = CommandType.Text;
cmd.ExecuteNonQuery();
}
catch (SqlException ex)
{
if (ex.Errors[0].Number == 444) //Use the actual error number that you get on foreign key confilict
failedIds += "," + ids[i];
}
}
}
}
catch
{
}
If you needed to just delete, it would have been an easy answer - batch the delete statements and wrap them in a transaction. But since you want the keys that caused an issue, and you cannot use stored procedures, that makes the problem tricky.
Here is a thought experiment that might get you closer to what you want. The idea is this:
Process the delete in batches. I see no way around that without using stored procedures.
In each batch:
Try to delete all the records (maximum "window" size)
If the delete fails, try to delete the records in a smaller "window".
Keep trying and decreasing the "window" until a problem record is found.
On a single error add that value to the list of errors.
Continue checking each record.
if the next delete succeeds, confidently increase the "window" size.
The "window" size adjusts the number of records in the batch to process.
At the end you would have deleted what you can, and will have a list of values that failed to delete.
Unfortunately you cannot run a delete and get the list of values that gave problems (database transactions don't work like that) but the "binary search-like" method above will at least be faster than checking each value in the 15k list... only if there are a a small number of conflicts. Otherwise it won't be any faster.
Here is the code that details the steps:
int batchSize = 1024; // define this elsewhere
// allValuesToDel is your list of values. I've just defined it as an empty list
var allValuesToDel = new List<int>();
var valueErrors = new List<int>();
for (int i = 0; i < allValuesToDel.Count; i += batchSize)
{
using (var sc = new SqlConnection(dbConnString))
{
sc.Open();
var valueBatch = allValuesToDel.Skip(i)
.Take(batchSize)
// annotating each value with a 'processed' flag allows us to track them
.Select(k => new { Value = k, Processed = false }).ToList();
// the starting window size
int windowSize = valueBatch.Count;
while (valueBatch.Count > 0 && valueBatch.Any(o => !o.Processed))
{
// get the unprocessed values within the window
var valuesToDel = valueBatch.Where(k => !k.Processed).Take(windowSize).ToList();
string nosConcat = string.Join(",", valuesToDel.Select(k => k.Value));
try
{
using (var cmd = sc.CreateCommand())
{
cmd.CommandText = string.Format("DELETE FROM {0} WHERE {1} IN ({2})", tblName, delColumnName, nosConcat);
cmd.ExecuteNonQuery();
}
// on success we can mark them as processed
valuesToDel.ForEach(k => k.Processed = true);
// Since delete worked, let's try on more records
windowSize = windowSize * 2;
}
catch (SqlException ex)
{
if (windowSize == 1)
{
// we found a value that failed - add that to the list of failures
valueErrors.Add(valuesToDel[0].Value);
valuesToDel.ForEach(k => k.Processed = true); // mark it as processed
}
// decrease the window size (until 1) to try and catch the problematic record
windowSize = (windowSize / 2) + 1;
}
}
}
}
I have two identical SQL tables (one is work table and the other is history table). I want to be able to write a record from the work table to the history table whenever a user selects a record to be deleted. The two tables are a clone of each other just to save deleted records to a history file.
Both tables contain an Identity Column (first column) and an image datatype column.
What i want to know is what is the best way to do this. Here is what I have, but I keep getting an error saying:
"Unable to cast object of type 'System.String' to type 'System.Byte[]'."
string sql = "SELECT * FROM EDI10000 WHERE File_Id = " + fid;
DataTable dt1 = Vit.GetDataRecords(sql);
if (dt1.Rows.Count > 0)
{
DataRow dr = dt1.Rows[0];
int fileId = int.Parse(fid);
string tpID = Vit.GetFieldValue(dr, "Tp_Id");
string fName = Vit.GetFieldValue(dr, "File_Name");
int fileSz = int.Parse(Vit.GetFieldValue(dr, "File_Size"));
string conType = Vit.GetFieldValue(dr, "Content_Type");
object obj = Vit.GetFieldValue(dr, "File_Data");
Byte[] byteData = (Byte[])obj;
MemoryStream ms = new MemoryStream(byteData);
object[] Insert = new object[] { fileId, tpID, fName, fileSz, ms, conType };
string sql2 = string.Format(#"INSERT INTO EDI10500 VALUES ({0}, '{1}', '{2}', {3}, '{4}', '{5}')", Insert);
Vit.UpdateDB(sql2);
}
Any help would be appreciated.
Thank you
Here is a ROUGH example of a trigger:
CREATE TRIGGER [dbo].[tr_SourceTableName_Audit] ON [dbo].[SourceTableName] FOR DELETE AS
BEGIN
SET NOCOUNT ON;
BEGIN TRY
INSERT INTO AuditTable (cols)
Select * from deleted
END TRY
BEGIN CATCH
-- In this case, I just want to eat the error.
END CATCH
END
I need to insert a huge CSV-File into 2 Tables with a 1:n relationship within a mySQL Database.
The CSV-file comes weekly and has about 1GB, which needs to be append to the existing data.
Each of them 2 tables have a Auto increment Primary Key.
I've tried:
Entity Framework (takes most time of all approaches)
Datasets (same)
Bulk Upload (doesn't support multiple tables)
MySqlCommand with Parameters (needs to be nested, my current approach)
MySqlCommand with StoredProcedure including a Transaction
Any further suggestions?
Let's say simplified this is my datastructure:
public class User
{
public string FirstName { get; set; }
public string LastName { get; set; }
public List<string> Codes { get; set; }
}
I need to insert from the csv into this database:
User (1-n) Code
+---+-----+-----+ +---+---+-----+
|PID|FName|LName| |CID|PID|Code |
+---+-----+-----+ +---+---+-----+
| 1 |Jon | Foo | | 1 | 1 | ed3 |
| 2 |Max | Foo | | 2 | 1 | wst |
| 3 |Paul | Foo | | 3 | 2 | xsd |
+---+-----+-----+ +---+---+-----+
Here a sample line of the CSV-file
Jon;Foo;ed3,wst
A Bulk load like LOAD DATA LOCAL INFILE is not possible because i have restricted writing rights
Referring to your answer i would replace
using (MySqlCommand myCmdNested = new MySqlCommand(cCommand, mConnection))
{
foreach (string Code in item.Codes)
{
myCmdNested.Parameters.Add(new MySqlParameter("#UserID", UID));
myCmdNested.Parameters.Add(new MySqlParameter("#Code", Code));
myCmdNested.ExecuteNonQuery();
}
}
with
List<string> lCodes = new List<string>();
foreach (string code in item.Codes)
{
lCodes.Add(String.Format("('{0}','{1}')", UID, MySqlHelper.EscapeString(code)));
}
string cCommand = "INSERT INTO Code (UserID, Code) VALUES " + string.Join(",", lCodes);
using (MySqlCommand myCmdNested = new MySqlCommand(cCommand, mConnection))
{
myCmdNested.ExecuteNonQuery();
}
that generates one insert statement instead of item.Count
Given the great size of data, the best approach (performance wise) is to leave as much data processing to the database and not the application.
Create a temporary table that the data from the .csv file will be temporarily saved.
CREATE TABLE `imported` (
`id` int(11) NOT NULL,
`firstname` varchar(45) DEFAULT NULL,
`lastname` varchar(45) DEFAULT NULL,
`codes` varchar(450) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Loading the data from the .csv to this table is pretty straightforward. I would suggest the use of MySqlCommand (which is also your current approach). Also, using the same MySqlConnection object for all INSERT statements will reduce the total execution time.
Then to furthermore process the data, you can create a stored procedure that will handle it.
Assuming these two tables (taken from your simplified example):
CREATE TABLE `users` (
`PID` int(11) NOT NULL AUTO_INCREMENT,
`FName` varchar(45) DEFAULT NULL,
`LName` varchar(45) DEFAULT NULL,
PRIMARY KEY (`PID`)
) ENGINE=InnoDB AUTO_INCREMENT=3737 DEFAULT CHARSET=utf8;
and
CREATE TABLE `codes` (
`CID` int(11) NOT NULL AUTO_INCREMENT,
`PID` int(11) DEFAULT NULL,
`code` varchar(45) DEFAULT NULL,
PRIMARY KEY (`CID`)
) ENGINE=InnoDB AUTO_INCREMENT=15 DEFAULT CHARSET=utf8;
you can have the following stored procedure.
CREATE DEFINER=`root`#`localhost` PROCEDURE `import_data`()
BEGIN
DECLARE fname VARCHAR(255);
DECLARE lname VARCHAR(255);
DECLARE codesstr VARCHAR(255);
DECLARE splitted_value VARCHAR(255);
DECLARE done INT DEFAULT 0;
DECLARE newid INT DEFAULT 0;
DECLARE occurance INT DEFAULT 0;
DECLARE i INT DEFAULT 0;
DECLARE cur CURSOR FOR SELECT firstname,lastname,codes FROM imported;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1;
OPEN cur;
import_loop:
LOOP FETCH cur INTO fname, lname, codesstr;
IF done = 1 THEN
LEAVE import_loop;
END IF;
INSERT INTO users (FName,LName) VALUES (fname, lname);
SET newid = LAST_INSERT_ID();
SET i=1;
SET occurance = (SELECT LENGTH(codesstr) - LENGTH(REPLACE(codesstr, ',', '')) + 1);
WHILE i <= occurance DO
SET splitted_value =
(SELECT REPLACE(SUBSTRING(SUBSTRING_INDEX(codesstr, ',', i),
LENGTH(SUBSTRING_INDEX(codesstr, ',', i - 1)) + 1), ',', ''));
INSERT INTO codes (PID, code) VALUES (newid, splitted_value);
SET i = i + 1;
END WHILE;
END LOOP;
CLOSE cur;
END
For every row in the source data, it makes an INSERT statement for the user table. Then there is a WHILE loop to split the comma separated codes and make for each one an INSERT statement for the codes table.
Regarding the use of LAST_INSERT_ID(), it is reliable on a PER CONNECTION basis (see doc here). If the MySQL connection used to run this stored procedure is not used by other transactions, the use of LAST_INSERT_ID() is safe.
The ID that was generated is maintained in the server on a per-connection basis. This means that the value returned by the function to a given client is the first AUTO_INCREMENT value generated for most recent statement affecting an AUTO_INCREMENT column by that client. This value cannot be affected by other clients, even if they generate AUTO_INCREMENT values of their own. This behavior ensures that each client can retrieve its own ID without concern for the activity of other clients, and without the need for locks or transactions.
Edit: Here is the OP's variant that omits the temp-table imported. Instead of inserting the data from the .csv to the imported table, you call the SP to directly store them to your database.
CREATE DEFINER=`root`#`localhost` PROCEDURE `import_data`(IN fname VARCHAR(255), IN lname VARCHAR(255),IN codesstr VARCHAR(255))
BEGIN
DECLARE splitted_value VARCHAR(255);
DECLARE done INT DEFAULT 0;
DECLARE newid INT DEFAULT 0;
DECLARE occurance INT DEFAULT 0;
DECLARE i INT DEFAULT 0;
INSERT INTO users (FName,LName) VALUES (fname, lname);
SET newid = LAST_INSERT_ID();
SET i=1;
SET occurance = (SELECT LENGTH(codesstr) - LENGTH(REPLACE(codesstr, ',', '')) + 1);
WHILE i <= occurance DO
SET splitted_value =
(SELECT REPLACE(SUBSTRING(SUBSTRING_INDEX(codesstr, ',', i),
LENGTH(SUBSTRING_INDEX(codesstr, ',', i - 1)) + 1), ',', ''));
INSERT INTO codes (PID, code) VALUES (newid, splitted_value);
SET i = i + 1;
END WHILE;
END
Note: The code to split the codes is taken from here (MySQL does not provide a split function for strings).
I developed my WPF application application using the Entity Framework and used SQL server database and needed to read data from an excel file and had to insert that data into 2 tables that has relationship between them. For roughly about 15000 rows in excel it used to take around 4 hours of time. Then what I did was I used a block of 500 rows per insert and this speeded up my insertion to unbelievalbe fast and now it takes mere 3-5 seconds to import that same data.
So I would suggest you add your rows to a Context like 100/200/500 at a time and then call the SaveChanges method (if you really want to be using EF). There are other helpful tips as well to speed up the performance for EF. Please read this for your reference.
var totalRecords = TestPacksData.Rows.Count;
var totalPages = (totalRecords / ImportRecordsPerPage) + 1;
while (count <= totalPages)
{
var pageWiseRecords = TestPacksData.Rows.Cast<DataRow>().Skip(count * ImportRecordsPerPage).Take(ImportRecordsPerPage);
count++;
Project.CreateNewSheet(pageWiseRecords.ToList());
Project.CreateNewSpool(pageWiseRecords.ToList());
}
And here is the CreateNewSheet method
/// <summary>
/// Creates a new Sheet record in the database
/// </summary>
/// <param name="row">DataRow containing the Sheet record</param>
public void CreateNewSheet(List<DataRow> rows)
{
var tempSheetsList = new List<Sheet>();
foreach (var row in rows)
{
var sheetNo = row[SheetFields.Sheet_No.ToString()].ToString();
if (string.IsNullOrWhiteSpace(sheetNo))
continue;
var testPackNo = row[SheetFields.Test_Pack_No.ToString()].ToString();
TestPack testPack = null;
if (!string.IsNullOrWhiteSpace(testPackNo))
testPack = GetTestPackByTestPackNo(testPackNo);
var existingSheet = GetSheetBySheetNo(sheetNo);
if (existingSheet != null)
{
UpdateSheet(existingSheet, row);
continue;
}
var isometricNo = GetIsometricNoFromSheetNo(sheetNo);
var newSheet = new Sheet
{
sheet_no = sheetNo,
isometric_no = isometricNo,
ped_rev = row[SheetFields.PED_Rev.ToString()].ToString(),
gpc_rev = row[SheetFields.GPC_Rev.ToString()].ToString()
};
if (testPack != null)
{
newSheet.test_pack_id = testPack.id;
newSheet.test_pack_no = testPack.test_pack_no;
}
if (!tempSheetsList.Any(l => l.sheet_no == newSheet.sheet_no))
{
DataStore.Context.Sheets.Add(newSheet);
tempSheetsList.Add(newSheet);
}
}
try
{
DataStore.Context.SaveChanges();
**DataStore.Dispose();** This is very important. Dispose the context
}
catch (DbEntityValidationException ex)
{
// Create log for the exception here
}
}
CreateNewSpool is ditto same method except for the fields name and table name, because it updates a child table. But the idea is the same
1 - Add a column VirtualId to User table & class.
EDITED
2 - Assign numbers in a loop for the VirtualId (use negative numbers starting -1 to avoid collisions in the last step) field in each User Object. For each Code c object belonging to User u object set the c.UserId = u.VirtualId.
3 - Bulk load Users into User table, Bulk load Codes into Code table.
4- UPDATE CODE C,USER U SET C.UserId = U.Id WHERE C.UserId = U.VirtualId.
NOTE : If you have a FK Constraint on Code.UserId you can drop it and re-add it after the Insert.
public class User
{
public int Id { get; set; }
public string FirstName { get; set; }
public string LastName { get; set; }
public int VirtualId { get; set; }
}
public class Code
{
public int Id { get; set; }
public string Code { get; set; }
public string UserId { get; set; }
}
Can you break the CSV into two files?
E.g. Suppose your file has the following columns:
... A ... | ... B ...
a0 | b0
a0 | b1
a0 | b2 <-- data
a1 | b3
a1 | b4
So one set of A might have multiple B entries. After you break it apart, you get:
... A ...
a0
a1
... B ...
b0
b1
b2
b3
b4
Then you bulk insert them separately.
Edit: Pseudo code
Based on the conversation, something like:
DataTable tableA = ...; // query schema for TableA
DataTable tableB = ...; // query schmea for TableB
List<String> usernames = select distinct username from TableA;
Hashtable htUsername = new Hashtable(StringComparer.InvariantCultureIgnoreCase);
foreach (String username in usernames)
htUsername[username] = "";
int colUsername = ...;
foreach (String[] row in CSVFile) {
String un = row[colUsername] as String;
if (htUsername[un] == null) {
// add new row to tableA
DataRow row = tableA.NewRow();
row["Username"] = un;
// etc.
tableA.Rows.Add(row);
htUsername[un] = "";
}
}
// bulk insert TableA
select userid, username from TableA
Hashtable htUserId = new Hashtable(StringComparer.InvariantCultureIgnoreCase);
// htUserId[username] = userid;
int colUserId = ...;
foreach (String[] row in CSVFile) {
String un = row[colUsername] as String;
int userid = (int) htUserId[un];
DataRow row = tableB.NewRow();
row[colUserId] = userId;
// fill in other values
tableB.Rows.Add(row);
if (table.Rows.Count == 65000) {
// bulk insert TableB
var t = tableB.Clone();
tableB.Dispose();
tableB = t;
}
}
if (tableB.Rows.Count > 0)
// bulk insert TableB
AFAIK the insertions done in a table are sequential while the insertions in different table can be done in parallel. Open two separate new connections to the same database and then insert in parallel maybe by using Task Parallel Library.
However, if there are integrity constraints about 1:n relationship between the tables, then:
Insertions might fail and thus any parallel insert approach would be wrong. Clearly then your best bet would be to do sequential inserts only, one table after the other.
You can try and sort the data of both tables write the InsertInto method written below such that insert in second table will happen only after you are done inserting the data in the first one.
Edit: Since you have requested, if there is a possibility for you to perform the inserts in parallel, following is the code template you can use.
private void ParallelInserts()
{
..
//Other code in the method
..
//Read first csv into memory. It's just a GB so should be fine
ReadFirstCSV();
//Read second csv into memory...
ReadSecondCSV();
//Because the inserts will last more than a few CPU cycles...
var taskFactory = new TaskFactory(TaskCreationOptions.LongRunning, TaskContinuationOptions.None)
//An array to hold the two parallel inserts
var insertTasks = new Task[2];
//Begin insert into first table...
insertTasks[0] = taskFactory.StartNew(() => InsertInto(commandStringFirst, connectionStringFirst));
//Begin insert into second table...
insertTasks[1] = taskFactory.StartNew(() => InsertInto(commandStringSecond, connectionStringSecond));
//Let them be done...
Task.WaitAll(insertTasks);
Console.WriteLine("Parallel insert finished.");
}
//Defining the InsertInto method which we are passing to the tasks in the method above
private static void InsertInto(string commandString, string connectionString)
{
using (/*open a new connection using the connectionString passed*/)
{
//In a while loop, iterate until you have 100/200/500 rows
while (fileIsNotExhausted)
{
using (/*commandString*/)
{
//Execute command to insert in bulk
}
}
}
}
When you say "efficiently" are you talking memory, or time?
In terms of improving the speed of the inserts, if you can do multiple value blocks per insert statement, you can get 500% improvement in speed. I did some benchmarks on this over in this question: Which is faster: multiple single INSERTs or one multiple-row INSERT?
My approach is described in the answer, but simply put, reading in up to say 50 "rows" (to be inserted) at once and bundling them into a single INSERT INTO(...), VALUES(...),(...),(...)...(...),(...) type statement seems to really speed things up. At least, if you're restricted from not being able to bulk load.
Another approach btw if you have live data you can't drop indexes on during the upload, is to create a memory table on the mysql server without indexes, dump the data there, and then do an INSERT INTO live SELECT * FROM mem. Though that uses more memory on the server, hence the question at the start of this answer about "what do you mean by 'efficiently'?" :)
Oh, and there's probably nothing wrong with iterating through the file and doing all the first table inserts first, and then doing the second table ones. Unless the data is being used live, I guess. In that case you could definitely still use the bundled approach, but the application logic to do that is a lot more complex.
UPDATE: OP requested example C# code for multivalue insert blocks.
Note: this code assumes you have a number of structures already configured:
tables List<string> - table names to insert into
fieldslist Dictionary<string, List<String>> - list of field names for each table
typeslist Dictionary<string, List<MySqlDbType>> - list of MySqlDbTypes for each table, same order as the field names.
nullslist Dictionary<string, List<Boolean>> - list of flags to tell if a field is nullable or not, for each table (same order as field names).
prikey Dictionary<string, string> - list of primary key field name, per table (note: this doesn't support multiple field primary keys, though if you needed it you could probably hack it in - I think somewhere I have a version that does support this, but... meh).
theData Dictionary<string, List<Dictionary<int, object>>> - the actual data, as a list of fieldnum-value dictionaries, per table.
Oh yeah, and the localcommand is MySqlCommand created by using CreateCommand() on the local MySqlConnection object.
Further note: I wrote this quite a while back when I was kind of starting. If this causes your eyes or brain to bleed, I apologise in advance :)
const int perinsert = 50;
foreach (string table in tables)
{
string[] fields = fieldslist[table].ToArray();
MySqlDbType[] types = typeslist[table].ToArray();
bool[] nulls = nullslist[table].ToArray();
int thisblock = perinsert;
int rowstotal = theData[table].Count;
int rowsremainder = rowstotal % perinsert;
int rowscopied = 0;
// Do the bulk (multi-VALUES block) INSERTs, but only if we have more rows than there are in a single bulk insert to perform:
while (rowscopied < rowstotal)
{
if (rowstotal - rowscopied < perinsert)
thisblock = rowstotal - rowscopied;
// Generate a 'perquery' multi-VALUES prepared INSERT statement:
List<string> extravals = new List<string>();
for (int j = 0; j < thisblock; j++)
extravals.Add(String.Format("(#{0}_{1})", j, String.Join(String.Format(", #{0}_", j), fields)));
localcmd.CommandText = String.Format("INSERT INTO {0} VALUES{1}", tmptable, String.Join(",", extravals.ToArray()));
// Now create the parameters to match these:
for (int j = 0; j < thisblock; j++)
for (int i = 0; i < fields.Length; i++)
localcmd.Parameters.Add(String.Format("{0}_{1}", j, fields[i]), types[i]).IsNullable = nulls[i];
// Keep doing bulk INSERTs until there's less rows left than we need for another one:
while (rowstotal - rowscopied >= thisblock)
{
// Queue up all the VALUES for this block INSERT:
for (int j = 0; j < thisblock; j++)
{
Dictionary<int, object> row = theData[table][rowscopied++];
for (int i = 0; i < fields.Length; i++)
localcmd.Parameters[String.Format("{0}_{1}", j, fields[i])].Value = row[i];
}
// Run the query:
localcmd.ExecuteNonQuery();
}
// Clear all the paramters - we're done here:
localcmd.Parameters.Clear();
}
}