Replaces Question: Update multiple rows into SQL table
Here's a Code Snippet to update an exam results set.
DB structure is as given, but I can submit Stored Procedures for inclusion (Which are a pain to modify, so I save that until the end.)
The question: Is there a better way using SQL server v 2005.,net 2.0 ?
string update = #"UPDATE dbo.STUDENTAnswers
SET ANSWER=#answer
WHERE StudentID =#ID and QuestionNum =#qnum";
SqlCommand updateCommand = new SqlCommand( update, conn );
conn.Open();
string uid = Session["uid"].ToString();
for (int i= tempStart; i <= tempEnd; i++)
{
updateCommand.Parameters.Clear();
updateCommand.Parameters.AddWithValue("#ID",uid);
updateCommand.Parameters.AddWithValue("#qnum",i);
updateCommand.Parameters.AddWithValue("#answer", Request.Form[i.ToString()]);
try
{
updateCommand.ExecuteNonQuery();
}
catch { }
}
A few things stand out:
You don't show where the SqlConnection is instantiated, so it's not clear that you're disposing it properly.
You shouldn't be swallowing exceptions in the loop - better to handle them in a top level exception handler.
You're instantiating new parameters on each iteration through the loop - you could just reuse the parameters.
Putting this together it could look something like the following (if you don't want to use a transaction, i.e. don't care if some but not all updates succeed):
using (SqlConnection conn = new SqlConnection(connectionString))
{
conn.Open();
using (SqlCommand updateCommand = new SqlCommand(update, conn))
{
string uid = Session["uid"].ToString();
updateCommand.Parameters.AddWithValue("#ID", uid);
updateCommand.Parameters.AddWithValue("#qnum", i);
updateCommand.Parameters.Add("#answer", System.Data.SqlDbType.VarChar);
for (int i = tempStart; i <= tempEnd; i++)
{
updateCommand.Parameters["#answer"] = Request.Form[i.ToString()];
updateCommand.ExecuteNonQuery();
}
}
}
Or to use a transaction to ensure all or nothing:
using (SqlConnection conn = new SqlConnection(connectionString))
{
conn.Open();
using (SqlTransaction transaction = conn.BeginTransaction())
{
using (SqlCommand updateCommand = new SqlCommand(update, conn, transaction))
{
string uid = Session["uid"].ToString();
updateCommand.Parameters.AddWithValue("#ID", uid);
updateCommand.Parameters.AddWithValue("#qnum", i);
updateCommand.Parameters.Add("#answer", System.Data.SqlDbType.VarChar);
for (int i = tempStart; i <= tempEnd; i++)
{
updateCommand.Parameters["#answer"] = Request.Form[i.ToString()];
updateCommand.ExecuteNonQuery();
}
transaction.Commit();
}
} // Transaction will be disposed and rolled back here if an exception is thrown
}
Finally, another problem is that you are mixing UI code (e.g. Request.Form) with data access code. It would be more modular and testable to separate these - e.g. by splitting your application into UI, Business Logic and Data Access layers.
For 30 updates I think you're on the right track, although the comment about the need for a using around updateCommand is correct.
We've found the best performing way to do bulk updates (>100 rows) is via the SqlBulkCopy class to a temporary table followed by a stored procedure call to populate the live table.
An issue I see is when you are opening your connection.
I would at least before every update call the open and then close the connection after the update.
If your loop takes time to execute you will have your connection open for a long time.
It is a good rule to never open your command until you need it.
You can bulk insert using OpenXML. Create an xml document containing all your questions and answers and use that to insert the values.
Edit: If you stick with your current solution, I would at least wrap your SqlConnection and SqlCommand in a using block to make sure they get disposed.
emit a single update that goes against a values table:
UPDATE s SET ANSWER=a FROM dbo.STUDENTAnswers s JOIN (
SELECT 1 as q, 'answer1' as a
UNION ALL SELECT 2, 'answer2' -- etc...
) x ON s.QuestionNum=x.q AND StudentID=#ID
so you just put this together like this:
using(SqlCommand updateCommand = new SqlCommand()) {
updateCommand.CommandType = CommandType.Text;
updateCommand.Connection = conn;
if (cn.State != ConnectionState.Open) conn.Open();
StringBuilder sb = new StringBuilder("UPDATE s SET ANSWER=a FROM dbo.STUDENTAnswers s JOIN (");
string fmt = "SELECT {0} as q, #A{0} as a";
for(int i=tempStart; i<tempEnd; i++) {
sb.AppendFormat(fmt, i);
fmt=" UNION ALL SELECT {0},#A{0}";
updateCommand.Parameters.AddWithValue("#A"+i.ToString(), Request.Form[i.ToString()]);
}
sb.Append(") x ON s.QuestionNum=x.q AND StudentID=#ID");
updateCommand.CommandText = sb.ToString();
updateCommand.Parameters.AddWithValue("#ID", uid);
updateCommand.ExecuteNonQuery();
}
This has the advantages of being an all other nothing operation (like if you'd wrapped several updates in a transaction) and will run faster since:
The table and associated indexes are looked at/updated once
You only pay for the latency between your application and the database server once, rather than on each update
Related
I am trying to insert into table from array of Json as well as select records from SQL Server DB table.
When executing the below method it is almost taking more than 10 minutes.
public async Task CreateTableAsync(string formsJson, string connectionString)
{
SqlConnection con = new SqlConnection(connectionString);
List<FormsJson> listOfformsJson = JsonConvert.DeserializeObject<List<FormsJson>>(formsJson);
foreach (var form in listOfformsJson)
{
string formId = Guid.NewGuid().ToString();
//insert into forms Table
string formQuery = "insert into Forms([FormId]) values(#FormId)";
using (var cmd = new SqlCommand(formQuery, con))
{
cmd.CommandTimeout = 120;
//Pass values to Parameters
cmd.Parameters.AddWithValue("#FormId", formId);
if (con.State == System.Data.ConnectionState.Closed)
{
con.Open();
}
cmd.ExecuteNonQuery();
}
//relationship between forms and ETypes,get all the eTypes and fill
foreach (var typeOf in form.TypeOf)
{
//get all the eTypeIds for this typeof field
string query = "select Id from ETypes Where TypeOf = #typeOf";
List<string> eTypeIdList = new List<string>();
using (var sqlcmd = new SqlCommand(query, con))
{
sqlcmd.CommandTimeout = 120;
//Pass values to Parameters
sqlcmd.Parameters.AddWithValue("#typeOf", typeOf);
if (con.State == System.Data.ConnectionState.Closed)
{
con.Open();
}
SqlDataReader sqlDataReader = sqlcmd.ExecuteReader();
while (sqlDataReader.Read())
{
string eTypeId = sqlDataReader[0].ToString();
eTypeIdList.Add(eTypeId);
}
sqlDataReader.Close();
}
//insert into Forms ETypes Relationship
string fe_query = "";
foreach (var eTypeId in eTypeIdList)
{
fe_query = "insert into Forms_ETypes([Form_Id],[EType_Id]) values (#Form_Id,#EType_Id)";
if (con.State == System.Data.ConnectionState.Closed)
{
con.Open();
}
using (var fesqlcmd = new SqlCommand(fe_query, con))
{
fesqlcmd.CommandTimeout = 120;
//Pass values to Parameters
fesqlcmd.Parameters.AddWithValue("#Form_Id", formId);
fesqlcmd.Parameters.AddWithValue("#EType_Id", eTypeId);
fesqlcmd.ExecuteNonQuery();
}
}
}
}
}
Outer foreach(...listofformsJson) loop more than hundreds of records.
And same for the inner loop around hundreds of rows.
In between commandTimeout, keeping the open connection with server statements.
Any help to optimize the time and remove/add ADO statements.
The primary issue here is that you are pulling all of the data out of the database and then, row by row, inserting it back in. This is not optimal from the database's point of view. It is great at dealing with sets - but you are treating the set as lots of individual rows. Thus it becomes slow.
From a set-based standpoint, you have only two statements that you need to run:
Insert the Forms row
Insert the Forms_ETypes rows (as a set, not one at a time)
1) should be what you have now:
insert into Forms([FormId]) values(#FormId)
2) should be something like:
insert Forms_ETypes([Form_Id],[EType_Id]) SELECT #FormId, Id from ETypes Where TypeOf IN ({0});
using this technique to pass in your form.TypeOf values. Note this assumes you have fewer than 500 entries in form.TypeOf. If you have many (e.g. greater than 500) then using a UDT is a better approach (note some info on UDTs suggest that you need to use a stored proc, but that isn't the case).
This will enable you to run just two SQL statements - the first, then the second (vs possibly thousands with your current solution).
This will save time for two reasons:
The database engine didn't need to pass the data over the wire twice (from your DB server to your application, and back again).
You enabled the database engine to do a large set based operation, rather than lots of smaller operations (with latency due to the request-response nature of the loop).
I have the following code:
SqlConnection connection1, connection2;
SqlCommand command1, command2;
SqlDataReader reader1, reader2;
using (connection1 = new SqlConnection("connection string here"))
{
using (command1 = new SqlCommand(#"SELECT * FROM [SERVER1].[DATABASE1].[TABLE1] WHERE COL1 = #COL1 AND COL2 = #COL2", connection1))
{
command1.Parameters.Add("#COL1", SqlDbType.VarChar, 255).Value = TextBox1.Text;
command1.Parameters.Add("#COL2", SqlDbType.VarChar, 255).Value = TextBox2.Text;
connection1.Open();
using (reader1 = command1.ExecuteReader())
{
while (reader1.Read())
{
int COL3Index = reader1.GetOrdinal("COL3");
Console.Write("### LOOP 1 ###");
Console.Write(reader1.GetDouble(COL3Index));
using (connection2 = new SqlConnection("same connection string here"))
{
using (command2 = new SqlCommand(#"SELECT * FROM [SERVER1].[DATABASE1].[TABLE2] WHERE COL1 = #COL1", connection1))
{
command2.Parameters.Add("#COL1", SqlDbType.Float).Value = reader1.GetDouble(COL3Index);
connection2.Open();
using (reader2 = command2.ExecuteReader())
{
while (reader2.Read())
{
int COL2Index = reader2.GetOrdinal("COL2");
Console.Write("### LOOP 2 ###");
Console.Write(reader2.GetDouble(COL2Index));
}
}
}
}
}
}
}
}
Basically 2 of everything, I will be needing to do this 5 times, i.e. loop within loop within loop within loop within loop...
The first loop on its own works, but the second one does not work and gives the following error:
There is already an open DataReader associated with this Command which
must be closed first.
on the line:
using (reader2 = command2.ExecuteReader())
How can I get this to work as I need to embed loops
This is the definition of Select N+1 and should be avoided if possible. I would recommend using something like Entity Framework and eagerly loading the child values.
If not possible to avoid, loop though your entire reader1 results, assign to a local collection, close reader1, and then iterate through the local collection and load based on the local values.
You have no reason to open the connection twice if you have MARS enabled to the same database/connection string. This can be done by adding "MultipleActiveResultSets=True" to your connection string.
Additionally, You can use a DataAdapter to load the data into a DataSet/DataTable and then query the DataSet. However, this assumes that your tables aren't too big that you can load them into memory, otherwise an ORM would be a better option.
An ORM solution such as LINQ-to-SQL or LINQ-to-Entities via Entity Framework (as mentioned in Mike Cole's answer) could really help you out here so you don't need to worry about writing these queries and handling the connections. Instead you just rely on a DataContext to handle the connections.
I am quit busy turning a old classic asp website to a .NET site. also i am now using SQL Server.
Now I have some old code
strsql = "select * FROM tabel WHERE ID = " & strID & " AND userid = " & struserid
rs1.open strsql, strCon, 2, 3
if rs1.eof THEN
rs1.addnew
end if
if straantal <> 0 THEN
rs1("userid") = struserid
rs1("verlangid") = strID
rs1("aantal") = straantal
end if
rs1.update
rs1.close
I want to use this in SQL Server. The update way. How can I do this?
How can I check if the datareader is EOF/EOL
How can I insert a row id it is EOF/EOL
How can I update a row or delete a row with one function?
If you want to use raw SQL commands you can try something like this
using (SqlConnection cnn = new SqlConnection(_connectionString))
using (SqlCommand cmd = new SqlCommand())
{
cnn.Open();
cmd.Connection = cnn;
// Example of reading with SqlDataReader
cmd.CommandText = "select sql query here";
using (SqlDataReader reader = cmd.ExecuteReader())
{
while (reader.Read())
{
myList.Add((int)reader[0]);
}
}
// Example of updating row
cmd.CommandText = "update sql query here";
cmd.ExecuteNonQuery();
}
It depends on the method you use... Are you going to use Entity Framework and LINQ? Are you going to use a straight SQL Connection? I would highly recommend going down the EF route but a simple straight SQL snippet would look something like:
using (var connection = new SqlConnection("Your connection string here"))
{
connection.Open();
using (var command = new SqlCommand("SELECT * FROM xyz ETC", connection))
{
// Process results
using (SqlDataReader reader = command.ExecuteReader())
{
while (reader.Read())
{
int userId = (int)reader["UserID"];
string somethingElse = (string)reader["AnotherField"];
// Etc, etc...
}
}
}
// To execute a query (INSERT, UPDATE, DELETE etc)
using (var commandExec = new SqlCommand("DELETE * FROM xyz ETC", connection))
{
commandExec.ExecuteNonQuery();
}
}
You will note the various elements wrapped in using, that is because you need to release the memory / connection when you have finished. This should answer your question quickly but as others have suggested (including me) I would investigate Entity Framework as it is much more powerful but has a learning curve attached to it!
You can use SQL store procedure for Update. And call this store procedure through C#.
Create procedure [dbo].[xyz_Update]
(
#para1
#para2
)
AS
BEGIN
Update tablename
Set Fieldname1=#para1,
Set Feildname2=#para2
end
I can't seem to find why this function doesn't insert records into the database. :(
I get no error messages or whatsoever, just nothing in the database.
EDIT: this is how my query looks now .. still nothing ..
connection.Open();
XmlNodeList nodeItem = rssDoc.SelectNodes("/edno23/posts/post");
foreach (XmlNode xn in nodeItem)
{
cmd.Parameters.Clear();
msgText = xn["message"].InnerText;
C = xn["user_from"].InnerText;
avatar = xn["user_from_avatar"].InnerText;
string endhash = GetMd5Sum(msgText.ToString());
cmd.Parameters.Add("#endhash",endhash);
cmd.CommandText = "Select * FROM posts Where hash=#endhash";
SqlCeDataReader reader = cmd.ExecuteReader();
while (reader.Read())
{
string msgs = reader["hash"].ToString();
if (msgs != endhash || msgs == null)
{
sql = "INSERT INTO posts([user],msg,avatar,[date],hash) VALUES(#username,#messige,#userpic,#thedate,#hash)";
cmd.CommandText = sql;
cmd.Parameters.Add("#username", C);
cmd.Parameters.Add("#messige", msgText.ToString());
cmd.Parameters.Add("#userpic", avatar.ToString());
cmd.Parameters.Add("#thedate", dt);
cmd.Parameters.Add("#hash", endhash);
cmd.ExecuteNonQuery();// executes query
adapter.Update(data);// saves the changes
}
}
reader.Close();
}
connection.Close();
Does nodeItem actually have any items in it? If not, the contents of the foreach loop aren't being executed.
What's the adapter and data being used for? The queries and updates seem be done via other commands and readers.
What does 'hash' actually contain? If it's a hash, why are you hashing the content of the hash inside the while loop? If not, why is it being compared against a hash in the query SELECT * FROM posts WHERE hash = #endhash?
Won't closing the connection before the end of the while loop invalidate the reader used to control the loop?
Lots of things going on here...
You are using the command 'cmd' to loop over records with a datareader, and then using the same 'cmd' command inside the while statement to execute an insert statement. You declared another command 'cmdAdd' before but don't seem to use it anywhere; is that what you intended to use for the insert statement?
You also close your data connection inside the while loop that iterates over your datareader. You are only going to read one record and then close the connection to your database that way; if your conditional for inserting is not met, you're not going to write anything to the database.
EDIT:
You really should open and close the connection to the database outside the foreach on the xmlnodes. If you have 10 nodes to loop over, the db connection is going to be opened and closed 10 times (well, connection pooling will probably prevent that, but still...)
You are also loading the entire 'posts' table into a dataset for seemingly no reason. You're not changing any of the values in the dataset yet you are calling an update on it repeatedly (at "save teh shanges"). If the 'posts' table is even remotely large, this is going to suck a lot of memory for no reason (on a handheld device, no less).
Is anything returned from "Select * FROM posts Where hash=#endhash"?
If not, nothing inside the while loop matters....
Why are you closing the Database Connection inside the while loop?
The code you posted should throw an exception when you try to call cmd.ExecuteNonQuery() with an unopen DB connection object.
SqlCeCommand.ExecuteNonQuery() method returns the number of rows affected.
Why don't you check whether it is returning 1 or not in the debugger as shown below?
int rowsAffectedCount = cmd.ExecuteNonQuery();
Hope it helps :-)
You've got some issues with not implementing "using" blocks. I've added some to your inner code below. The blocks for the connection and select command are more wishful thinking on my part. I hope you're doing the same with the data adapter.
using (var connection = new SqlCeConnection(connectionString))
{
connection.Open();
var nodeItem = rssDoc.SelectNodes("/edno23/posts/post");
foreach (XmlNode xn in nodeItem)
{
using (
var selectCommand =
new SqlCeCommand(
"Select * FROM posts Where hash=#endhash",
connection))
{
var msgText = xn["message"].InnerText;
var c = xn["user_from"].InnerText;
var avatar = xn["user_from_avatar"].InnerText;
var endhash = GetMd5Sum(msgText);
selectCommand.Parameters.Add("#endhash", endhash);
selectCommand.CommandText =
"Select * FROM posts Where hash=#endhash";
using (var reader = selectCommand.ExecuteReader())
{
while (reader.Read())
{
var msgs = reader["hash"].ToString();
if (msgs == endhash && msgs != null)
{
continue;
}
const string COMMAND_TEXT =
"INSERT INTO posts([user],msg,avatar,[date],hash) VALUES(#username,#messige,#userpic,#thedate,#hash)";
using (
var insertCommand =
new SqlCeCommand(
COMMAND_TEXT, connection))
{
insertCommand.Parameters.Add("#username", c);
insertCommand.Parameters.Add(
"#messige", msgText);
insertCommand.Parameters.Add(
"#userpic", avatar);
insertCommand.Parameters.Add("#thedate", dt);
insertCommand.Parameters.Add(
"#hash", endhash);
insertCommand.ExecuteNonQuery();
// executes query
}
adapter.Update(data); // saves teh changes
}
reader.Close();
}
}
}
connection.Close();
}
Of course with the additional nesting, parts should be broken out as separate methods.
I suspect your problem is that you're trying to reuse the same SqlCeCommand instances.
Try making a new SqlCeCommand within the while loop. Also, you can use the using statement to close your data objects.
Why are you calling adapter.Update(data) since you're not changing the DataSet at all? I suspect you want to call adapter.Fill(data). The Update method will save any changes in the DataSet to the database.
How to debug programs: http://www.drpaulcarter.com/cs/debug.php
Seriously, can you post some more information about where it's working? Does it work if you use SQL Server Express instead of SQL CE? If so, can you break out SQL Profiler and take a look at the SQL commands being executed?
Ok, so here's the problem I have to solve. I need to write a method in C# that will modify a table in SQL Server 2008. The table could potentially contain millions of records. The modifications include altering the table by adding a new column and then calculating and setting the value of the new field for every row in the table.
Adding the column is not a problem. It's setting the values efficiently that is the issue. I don't want to read in the whole table into a DataTable and then update and commit for obvious reasons. I'm thinking that I would like to use a cursor to iterate over the rows in the table and update them one by one. I haven't done a whole lot of ADO.NET development, but it is my understanding that only read-only server side (firehose) cursors are supported.
So what is the correct way to go about doing something like this (preferably with some sample code in C#)? Stored procedures or other such modifications to the DB are not allowed.
jpgoody,
Here is an example to chew on using the NerdDinner database and some SQLConnection, SQLCommand, and SQLDataReader objects. It adds one day to each of the Event Dates in the Dinners table.
using System;
using System.Data.SqlClient;
namespace NerdDinner
{
public class Class1
{
public void Execute()
{
SqlConnection readerConnection = new SqlConnection(Properties.Settings.Default.ConnectionString);
readerConnection.Open();
SqlCommand cmd = new SqlCommand("SELECT DinnerID, EventDate FROM Dinners", readerConnection);
SqlDataReader reader = cmd.ExecuteReader();
SqlConnection writerConnection = new SqlConnection(Properties.Settings.Default.ConnectionString);
writerConnection.Open();
SqlCommand writerCommand = new SqlCommand("", writerConnection);
while (reader.Read())
{
int DinnerID = reader.GetInt32(0);
DateTime EventDate = reader.GetDateTime(1);
writerCommand.CommandText = "UPDATE Dinners SET EventDate = '" + EventDate.AddDays(1).ToString() + "' WHERE DinnerID = " + DinnerID.ToString();
writerCommand.ExecuteNonQuery();
}
}
}
}
Your problem looks like something that you should be solving using T-SQL and not C#, unless there is some business rule that you are picking up dynamically and calculating the column values T-SQL should be the way to go. Just write a stored procedure or just open up Management studio and write the code to make your changes.
If this does not help then please elaborate on what exactly you want to do to the table, then we can help you figure out if this can be done via T-SQL or not.
[EDIT] you can do something like this
string sql = " USE " + paramDbName;
sql+= " ALTER TABLE XYZ ADD COLUMN " + param1 + " datatype etc, then put semicolon to separate the commands as well"
sql+= " UPDATE XYZ SET Columnx = " + some logic here
cmd.CommandText = sql;
cmd.ExecuteNonQuery();
get this executed on the required instance of Sql Server 2008.
If you have too many lines of text then use StringBuilder.
Here's a suggestion:
You can read data using a DataReader , create a update command for current row and add it to a list of commands.Then run update commands in a transaction.
something like this:
var commands=new List<SqlCommand>();
while(dr.Read())
{
var cmd=new SqlCommand();
cmd.CommandText="Add your command text here";
commands.Add(cmd);
}
using(var cnn=new SqlConnection("Connection String"))
{
IDbTransaction transaction;
try
{
cnn.Open();
transaction=cnn.BeginTransaction();
foreach(var cmd in commands)
{
cmd.Transaction=transaction;
cmd.ExecuteNonQuery();
cmd.Dispose();
}
transaction.Commit();
}
catch(SqlException)
{
if(transaction!=null)
transaction.Rollback();
throw;
}
}