I have a stored procedure that looks like that:
InsertItem: INSERT INTO (IN itemId INT, name TEXT);
Is there a way I could execute a bulk of it?
like instead of executing something like that:
using (MySqlConnection connection = new MySqlConnection(_connectionString))
{
connection.Open();
foreach (Item item in GetItems())
{
using (MySqlCommand command = new MySqlCommand("InsertItem", connection))
{
command.CommandType = CommandType.StoredProcedure;
command.Parameters.AddWithValue("#itemId", item.ItemId);
command.Parameters.AddWithValue("#name", item.Name);
command.ExecuteNonQuery();
}
}
}
I'm trying to achieve code looking like that without successing:
using (MySqlConnection connection = new MySqlConnection(_connectionString))
{
connection.Open();
using (MySqlCommandBulk command = new MySqlCommand("InsertItem", connection))
{
command.CommandType = CommandType.StoredProcedure;
for (Item item in GetItems())
{
MySqlCommandBulkItem bulkItem = new MySqlCommandBulkItem();
bulkItem["itemId"] = item.ItemId;
bulkItem["name"] = item.Name;
command.BulkItems.Add(bulkItem);
}
command.Execute();
}
}
My point is that the command will send all of the data at once, and will not send each query alone.
Any ideas?
The Oracle connector for the Dotnet framework allows the use of arrays in place of scalars on parameters. But the MySQL connector doesn't.
There are two ways to accelerate bulk loads in MySQL.
One of them applies to InnoDB tables but doesn't help with MyISAM tables. Start a transaction. Then, after every few hundred rows, COMMIT it and start another one. That will commit your table inserts in bunches, which is faster than autocommiting them individually.
The other is to use MySQL's LOAD DATA INFILE command to slurp up a data file and bulk-insert it into the database. This is very fast, but you have to be diligent about formatting your file correctly.
Related
I am quit busy turning a old classic asp website to a .NET site. also i am now using SQL Server.
Now I have some old code
strsql = "select * FROM tabel WHERE ID = " & strID & " AND userid = " & struserid
rs1.open strsql, strCon, 2, 3
if rs1.eof THEN
rs1.addnew
end if
if straantal <> 0 THEN
rs1("userid") = struserid
rs1("verlangid") = strID
rs1("aantal") = straantal
end if
rs1.update
rs1.close
I want to use this in SQL Server. The update way. How can I do this?
How can I check if the datareader is EOF/EOL
How can I insert a row id it is EOF/EOL
How can I update a row or delete a row with one function?
If you want to use raw SQL commands you can try something like this
using (SqlConnection cnn = new SqlConnection(_connectionString))
using (SqlCommand cmd = new SqlCommand())
{
cnn.Open();
cmd.Connection = cnn;
// Example of reading with SqlDataReader
cmd.CommandText = "select sql query here";
using (SqlDataReader reader = cmd.ExecuteReader())
{
while (reader.Read())
{
myList.Add((int)reader[0]);
}
}
// Example of updating row
cmd.CommandText = "update sql query here";
cmd.ExecuteNonQuery();
}
It depends on the method you use... Are you going to use Entity Framework and LINQ? Are you going to use a straight SQL Connection? I would highly recommend going down the EF route but a simple straight SQL snippet would look something like:
using (var connection = new SqlConnection("Your connection string here"))
{
connection.Open();
using (var command = new SqlCommand("SELECT * FROM xyz ETC", connection))
{
// Process results
using (SqlDataReader reader = command.ExecuteReader())
{
while (reader.Read())
{
int userId = (int)reader["UserID"];
string somethingElse = (string)reader["AnotherField"];
// Etc, etc...
}
}
}
// To execute a query (INSERT, UPDATE, DELETE etc)
using (var commandExec = new SqlCommand("DELETE * FROM xyz ETC", connection))
{
commandExec.ExecuteNonQuery();
}
}
You will note the various elements wrapped in using, that is because you need to release the memory / connection when you have finished. This should answer your question quickly but as others have suggested (including me) I would investigate Entity Framework as it is much more powerful but has a learning curve attached to it!
You can use SQL store procedure for Update. And call this store procedure through C#.
Create procedure [dbo].[xyz_Update]
(
#para1
#para2
)
AS
BEGIN
Update tablename
Set Fieldname1=#para1,
Set Feildname2=#para2
end
Problem statement.
Basically I get 3 - 50 parameters that come back from a web service as a NVP array I then need to loop over them create the SQL command parameters for each and call the stored procedure. Is there a more efficient way to handle it than the approach below?
using (SqlConnection connection = new SqlConnection(connectionString))
{
connection.Open();
using (SqlCommand cm = connection.CreateCommand())
{
cm.CommandText = "MySproc";
cm.CommandType = CommandType.StoredProcedure;
foreach (var field in row)
{
cm.Parameters.AddWithValue("#" + field.Key.ToString(), field.Value.ToString());
}
cm.ExecuteNonQuery();
}
}
I personally use the ISNULL or COALESCE in the WHERE clause of the stored procedure. Unless your looking to do it inside your c#...
http://blogs.x2line.com/al/archive/2004/03/01/189.aspx
I have an array containing a list of file paths that I want to insert into an SQL database.
string[] filePaths = Directory.GetFiles(#"C:\Test Folder");
I am confident in setting up and connecting to the database, I just can't figure out how to take each element and place it in a new row in the database.
Thanks,
Matt
It depends on which technology you are using (Please note that when inserting lots of rows the use of SqlBulkCopy is recommended).
ADO.NET
foreach (var path in filePaths)
{
var command = new SqlCommand("INSERT INTO mytable(col1) VALUES(#param1)", connection);
command.Parameters.AddWithValue("#param1", path);
command.ExecuteNonQuery();
}
LINQ-to-SQL
var projection = filePaths.Select(a => new MyTable() { Col1 = a });
myContext.InsertAllOnSubmit(projection);
myContext.SubmitChanges();
LINQ-to-Entities
foreach (var path in filePaths)
{
myModel.MyTable.AddObject(new MyTable() { Col1 = path });
}
myModel.SaveChanges();
foreach(string fp in filePaths)
{
InsertIntoDb(fp);
}
//Method
public void InsertIntoDb(string insert)
{
SqlConnection con = //Setup DB connection
SqlCommand command = new SqlCommand();
command.Connection = con;
command.CommandText = "Insert #insert into Table";
command.Parameters.AddWithValue("#insert", insert);
command.ExecuteNonQuery();
}
This leaves out a lot, but it should point you in the right direction. Essentially, you want to set up your connection, instantiate a SqlCommand object where the command text is the SQL Text to insert your value (Better would be a stored procedure or some other sanitized way to insert the data to avoid Sql Injection), and then call ExecuteNonQuery() to insert the actual data.
The most efficient way is to use SqlBulkCopy
you will have to project your data into a DataTable, DataRow[], etc. OR IDataReader (which is more efficient - refer this discussion and example) in order to use SqlBulkCopy
Ok, so here's the problem I have to solve. I need to write a method in C# that will modify a table in SQL Server 2008. The table could potentially contain millions of records. The modifications include altering the table by adding a new column and then calculating and setting the value of the new field for every row in the table.
Adding the column is not a problem. It's setting the values efficiently that is the issue. I don't want to read in the whole table into a DataTable and then update and commit for obvious reasons. I'm thinking that I would like to use a cursor to iterate over the rows in the table and update them one by one. I haven't done a whole lot of ADO.NET development, but it is my understanding that only read-only server side (firehose) cursors are supported.
So what is the correct way to go about doing something like this (preferably with some sample code in C#)? Stored procedures or other such modifications to the DB are not allowed.
jpgoody,
Here is an example to chew on using the NerdDinner database and some SQLConnection, SQLCommand, and SQLDataReader objects. It adds one day to each of the Event Dates in the Dinners table.
using System;
using System.Data.SqlClient;
namespace NerdDinner
{
public class Class1
{
public void Execute()
{
SqlConnection readerConnection = new SqlConnection(Properties.Settings.Default.ConnectionString);
readerConnection.Open();
SqlCommand cmd = new SqlCommand("SELECT DinnerID, EventDate FROM Dinners", readerConnection);
SqlDataReader reader = cmd.ExecuteReader();
SqlConnection writerConnection = new SqlConnection(Properties.Settings.Default.ConnectionString);
writerConnection.Open();
SqlCommand writerCommand = new SqlCommand("", writerConnection);
while (reader.Read())
{
int DinnerID = reader.GetInt32(0);
DateTime EventDate = reader.GetDateTime(1);
writerCommand.CommandText = "UPDATE Dinners SET EventDate = '" + EventDate.AddDays(1).ToString() + "' WHERE DinnerID = " + DinnerID.ToString();
writerCommand.ExecuteNonQuery();
}
}
}
}
Your problem looks like something that you should be solving using T-SQL and not C#, unless there is some business rule that you are picking up dynamically and calculating the column values T-SQL should be the way to go. Just write a stored procedure or just open up Management studio and write the code to make your changes.
If this does not help then please elaborate on what exactly you want to do to the table, then we can help you figure out if this can be done via T-SQL or not.
[EDIT] you can do something like this
string sql = " USE " + paramDbName;
sql+= " ALTER TABLE XYZ ADD COLUMN " + param1 + " datatype etc, then put semicolon to separate the commands as well"
sql+= " UPDATE XYZ SET Columnx = " + some logic here
cmd.CommandText = sql;
cmd.ExecuteNonQuery();
get this executed on the required instance of Sql Server 2008.
If you have too many lines of text then use StringBuilder.
Here's a suggestion:
You can read data using a DataReader , create a update command for current row and add it to a list of commands.Then run update commands in a transaction.
something like this:
var commands=new List<SqlCommand>();
while(dr.Read())
{
var cmd=new SqlCommand();
cmd.CommandText="Add your command text here";
commands.Add(cmd);
}
using(var cnn=new SqlConnection("Connection String"))
{
IDbTransaction transaction;
try
{
cnn.Open();
transaction=cnn.BeginTransaction();
foreach(var cmd in commands)
{
cmd.Transaction=transaction;
cmd.ExecuteNonQuery();
cmd.Dispose();
}
transaction.Commit();
}
catch(SqlException)
{
if(transaction!=null)
transaction.Rollback();
throw;
}
}
Replaces Question: Update multiple rows into SQL table
Here's a Code Snippet to update an exam results set.
DB structure is as given, but I can submit Stored Procedures for inclusion (Which are a pain to modify, so I save that until the end.)
The question: Is there a better way using SQL server v 2005.,net 2.0 ?
string update = #"UPDATE dbo.STUDENTAnswers
SET ANSWER=#answer
WHERE StudentID =#ID and QuestionNum =#qnum";
SqlCommand updateCommand = new SqlCommand( update, conn );
conn.Open();
string uid = Session["uid"].ToString();
for (int i= tempStart; i <= tempEnd; i++)
{
updateCommand.Parameters.Clear();
updateCommand.Parameters.AddWithValue("#ID",uid);
updateCommand.Parameters.AddWithValue("#qnum",i);
updateCommand.Parameters.AddWithValue("#answer", Request.Form[i.ToString()]);
try
{
updateCommand.ExecuteNonQuery();
}
catch { }
}
A few things stand out:
You don't show where the SqlConnection is instantiated, so it's not clear that you're disposing it properly.
You shouldn't be swallowing exceptions in the loop - better to handle them in a top level exception handler.
You're instantiating new parameters on each iteration through the loop - you could just reuse the parameters.
Putting this together it could look something like the following (if you don't want to use a transaction, i.e. don't care if some but not all updates succeed):
using (SqlConnection conn = new SqlConnection(connectionString))
{
conn.Open();
using (SqlCommand updateCommand = new SqlCommand(update, conn))
{
string uid = Session["uid"].ToString();
updateCommand.Parameters.AddWithValue("#ID", uid);
updateCommand.Parameters.AddWithValue("#qnum", i);
updateCommand.Parameters.Add("#answer", System.Data.SqlDbType.VarChar);
for (int i = tempStart; i <= tempEnd; i++)
{
updateCommand.Parameters["#answer"] = Request.Form[i.ToString()];
updateCommand.ExecuteNonQuery();
}
}
}
Or to use a transaction to ensure all or nothing:
using (SqlConnection conn = new SqlConnection(connectionString))
{
conn.Open();
using (SqlTransaction transaction = conn.BeginTransaction())
{
using (SqlCommand updateCommand = new SqlCommand(update, conn, transaction))
{
string uid = Session["uid"].ToString();
updateCommand.Parameters.AddWithValue("#ID", uid);
updateCommand.Parameters.AddWithValue("#qnum", i);
updateCommand.Parameters.Add("#answer", System.Data.SqlDbType.VarChar);
for (int i = tempStart; i <= tempEnd; i++)
{
updateCommand.Parameters["#answer"] = Request.Form[i.ToString()];
updateCommand.ExecuteNonQuery();
}
transaction.Commit();
}
} // Transaction will be disposed and rolled back here if an exception is thrown
}
Finally, another problem is that you are mixing UI code (e.g. Request.Form) with data access code. It would be more modular and testable to separate these - e.g. by splitting your application into UI, Business Logic and Data Access layers.
For 30 updates I think you're on the right track, although the comment about the need for a using around updateCommand is correct.
We've found the best performing way to do bulk updates (>100 rows) is via the SqlBulkCopy class to a temporary table followed by a stored procedure call to populate the live table.
An issue I see is when you are opening your connection.
I would at least before every update call the open and then close the connection after the update.
If your loop takes time to execute you will have your connection open for a long time.
It is a good rule to never open your command until you need it.
You can bulk insert using OpenXML. Create an xml document containing all your questions and answers and use that to insert the values.
Edit: If you stick with your current solution, I would at least wrap your SqlConnection and SqlCommand in a using block to make sure they get disposed.
emit a single update that goes against a values table:
UPDATE s SET ANSWER=a FROM dbo.STUDENTAnswers s JOIN (
SELECT 1 as q, 'answer1' as a
UNION ALL SELECT 2, 'answer2' -- etc...
) x ON s.QuestionNum=x.q AND StudentID=#ID
so you just put this together like this:
using(SqlCommand updateCommand = new SqlCommand()) {
updateCommand.CommandType = CommandType.Text;
updateCommand.Connection = conn;
if (cn.State != ConnectionState.Open) conn.Open();
StringBuilder sb = new StringBuilder("UPDATE s SET ANSWER=a FROM dbo.STUDENTAnswers s JOIN (");
string fmt = "SELECT {0} as q, #A{0} as a";
for(int i=tempStart; i<tempEnd; i++) {
sb.AppendFormat(fmt, i);
fmt=" UNION ALL SELECT {0},#A{0}";
updateCommand.Parameters.AddWithValue("#A"+i.ToString(), Request.Form[i.ToString()]);
}
sb.Append(") x ON s.QuestionNum=x.q AND StudentID=#ID");
updateCommand.CommandText = sb.ToString();
updateCommand.Parameters.AddWithValue("#ID", uid);
updateCommand.ExecuteNonQuery();
}
This has the advantages of being an all other nothing operation (like if you'd wrapped several updates in a transaction) and will run faster since:
The table and associated indexes are looked at/updated once
You only pay for the latency between your application and the database server once, rather than on each update