We Keep Coding

How to get better performance from LINQ-to-SQL large dataset update

I have a c# console application that is updating a database with about 320,000 records. Basically it is encrypting a password in each record in a loop, then calling DatabaseContext.SubmitChanges(). The "UPDATE" part of the code takes about 20 seconds. I had to CTRL-C the app because it's taking over 15 minutes to do the "SubmitChanges" part: this is part of a time-sensitive system that should not be down for more than a couple minutes.
I ran SQL Profiler and I'm seeing queries like this for each update:
exec sp_executesql N'UPDATE [dbo].[PointRecord]
SET [PtPassword] = #p19
WHERE ([ID] = #p0) AND ([PtLocation] = #p1) AND ([PtIPAddress] = #p2) AND ([PtPort] = #p3) AND ([PtUsername] = #p4) AND ([PtPassword] = #p5) AND ([PtGWParam1] = #p6) AND ([PtGWParam2] = #p7) AND ([PtGWParam3] = #p8) AND ([PtGWParam4] = #p9) AND ([PtTag] = #p10) AND ([PtCapture] = #p11) AND ([PtGroup] = #p12) AND ([PtNumSuccess] = #p13) AND ([PtNumFailure] = #p14) AND ([PtControllerType] = #p15) AND ([PtControllerVersion] = #p16) AND ([PtAssocXMLGroupID] = #p17) AND ([PtErrorType] IS NULL) AND ([PtPollInterval] = #p18)',N'#p0 int,#p1 nvarchar(4000),#p2 nvarchar(4000),#p3 nvarchar(4000),#p4 nvarchar(4000),#p5 nvarchar(4000),#p6 nvarchar(4000),#p7 nvarchar(4000),#p8 nvarchar(4000),#p9 nvarchar(4000),#p10 nvarchar(4000),#p11 int,#p12 nvarchar(4000),#p13 int,#p14 int,#p15 nvarchar(4000),#p16 nvarchar(4000),#p17 int,#p18 int,#p19 nvarchar(4000)',#p0=296987,#p1=N'1234 Anytown USA',#p2=N'10.16.31.20',#p3=N'80',#p4=N'username1',#p5=N'password1',#p6=N'loadmon.htm?PARM2=21',#p7=N'>Operating Mode',#p8=N'',#p9=N'',#p10=N'1234 Anytown USA\HLTH SERVICE LTS\Operating Modeloadmon',#p11=0,#p12=N'1234 Anytown USA',#p13=0,#p14=0,#p15=N'DeviceA',#p16=N'3.5.0.2019.0219',#p17=309,#p18=15,#p19=N'hk+MUoeVMG69pOB3DHYB8g=='
As you can see, the "WHERE" part is asking for EVERY SINGLE FIELD to match, when this is an indexed table, using unique primary key "ID". This is really time-consuming. Is there any way to get this to only use "WHERE ID=[value]"?

I understand now that checking every field is a requirement of concurrency checking in EF. To bypass, methods outside of LINQ are required. I ended up using a variation of what Mr. Petrov and Mr. Harvey suggested, using ExecuteCommand since I am updating the database, not querying for data. Here is sample code, in case it can help others with a similar issue.
It uses LINQ to get the records to update and the record count for user feedback.
It uses ExecuteCommand to update the records. I am actually updating three tables (only one is shown in the sample below), hence the use of a transaction object.
The EncryptPassword method is not shown. It is what I use to update the records. You should replace that with whatever update logic suits your needs.
static void Main(string[] args)
{
DatabaseHelpers.Initialize();
if (DatabaseHelpers.PasswordsEncrypted)
{
Console.WriteLine("DatabaseHelpers indicates that passwords are already encrypted. Exiting.");
return;
}
// Note that the DatabaseHelpers.DbContext is in a helper library,
// it is a copy of the auto-generated EF 'DataClasses1DataContext'.
// It has already been opened using a generated connection string
// (part of DatabaseHelpers.Initialize()).
// I have altered some of the variable names to hide confidential information.
try
{
// show user what's happening
Console.WriteLine("Encrypting passwords...");
// flip switch on encryption methods
DatabaseHelpers.PasswordsEncrypted = true;
int recordCount = 0;
// Note: Using LINQ to update the records causes an unacceptable delay because of the concurrency checking
// where the UPDATE statement (at SubmitChanges) checks EVERY field instead of just the ID
// and we don't care about that!
// We have to set up an explicit transaction in order to use with context.ExecuteCommand statements
// start transaction - all or nothing
DatabaseHelpers.DbContext.Transaction = DatabaseHelpers.DbContext.Connection.BeginTransaction();
// update non-null and non-empty passwords in groups
Console.Write("Updating RecordGroups");
List<RecordGroup> recordGroups = (from p in DatabaseHelpers.DbContext.RecordGroups
where p.RecordPassword != null && p.RecordPassword != string.Empty
select p).ToList();
recordCount = recordGroups.Count;
foreach (RecordGroup rGroup in recordGroups)
{
// bypass LINQ-to-SQL
DatabaseHelpers.DbContext.ExecuteCommand("UPDATE RecordGroup SET RecordPassword={0} WHERE ID={1}", DatabaseHelpers.EncryptPassword(rGroup.RecordPassword), rGroup.ID);
Console.Write('.');
}
// show user what's happening
Console.WriteLine("\nCommitting transaction...");
DatabaseHelpers.DbContext.Transaction.Commit();
// display results
Console.WriteLine($"Updated {recordCount} RecordGroup passwords. Exiting.");
}
catch (Exception ex)
{
Console.WriteLine($"\nThere was an error executing the password encryption process: {ex}");
DatabaseHelpers.DbContext.Transaction.Rollback();
}
}

Reading results of SP with OdbcDataReader gives Error 22018 - Error in assignment

Ok I don't get this. I haven't used ODBC classes before but figuered it's nothing special for basic use. And it does work except in this case.
I need to execute stored procedure without parameters via ODBC connection and get the results, parse the rows into objects and insert them in my local DB. And it worked with test data but now fails with live data, while customer is able to execute the same PS via some other tool... The real trouble is that I have to run it on live server, so I can't debug, instead I created small project which writes output into TextBox. Anyway, here's the code:
var ODBCConnection = new OdbcConnection();
ODBCConnection.ConnectionString = "something...";
//using command "exec schema.spName" or "exec schema.spName()" or "{ call schema.spName()}" runs the procedure
//putting only name "schema.spName" gives ERROR [42000]
var cmd = new OdbcCommand("exec schema.spName())", ODBCConnection);
cmd.CommandType = CommandType.StoredProcedure;
DbReader = cmd.ExecuteReader();
int fCount = DbReader.FieldCount;
infoBox1.Text += System.Environment.NewLine + "Results:";
for (int i = 0; i < fCount; i++)
{
String fName = DbReader.GetName(i);
infoBox1.Text += fName + "|";
}
This list all the column names in result and there are 20 columns.
while (DbReader.Read())
{
var row = new RowClass();
for (int i = 0; i < fCount; i++)
{
object val = DbReader.GetValue(i);
//check which column this is and parse it to set properties of RowClass
//Expected values are string, int and decimal
}
}
This works for the first 10 rows but breaks when it tries to read for following columns with error:
ERROR [22018] [Cache ODBC][State : 22005][Native Code 22005]
[path to .exe]
Error in assignment
No StackTrace no InnerException.
I tried skipping 11th column because it started there, but breaks for every column after the first 10.
I am clueless... if it read there are 20 fields then wtf... null values are not problem because it returns DBNull, it works on other places (not executing SP but doing select queries).
Client executed sp connectin from same net environment and send me picshot, and csv of data. Nothing strane in data itself.
Anyone had this before? Should I use something else for instead of OdbcDataReader?
Thank you.

OK, I managed to figure it out thanks to customer admin who was executing the same procedure by some browser SQL tool. There I saw that 11th column was of type Date. ODBC returned type INT for that column. So I had to call another admin who set up ODBC and he changed column type to varchar so now it finally works.
I guess I can now assume how DataReader works if one wrong column type made all subsequent columns impossible to read as well.
I don't feel like accepting my own answer, so if someone would be kind enough to give me few links about how ODBC works and how DataReader works, not just how to use it, and maybe can quote some text that explains this behavior for this kind of mistake...

SQL DBGeography second insert fails

This one is a strange one. I am trying to save a polygon from Google maps into MS SQL, via an MVC controller. The problem is that the first time I do it, it works, the second time it gives me the error:
The incoming tabular data stream (TDS) remote procedure call (RPC) protocol stream is incorrect. Parameter 3 ("#2"): The supplied value is not a valid instance of data type geography. Check the source data for invalid values. An example of an invalid value is data of numeric type with scale greater than precision.
I am using EntityFramework 6.1.3, code first. The error appears on the commit line below:
var newPoly = new GenericPolygon()
{
Name = webShape.Name,
PolyShape = shapePolygon,
IsEnabled = true,
IsDeleted = false
};
_unitOfWork.PolygonRepository.Add(newPoly);
_unitOfWork.Commit();
The SQL table structure is the same as the class except that it has an int ID identity column as well, and the name is a varchar(255). The PolyShape column is of type geography.
The shapePolygon variable is defined like this, with the class adding a read-only property called "LongLat", which is used to switch from the Google LatLong to the MS LongLat format:
var shapePolygon = DbGeography.PolygonFromText("POLYGON((" + webShape.LongLat + "))", 4326);
The commit line itself calls the db context save method (I'm using UoW pattern to cut down on code):
this.context.SaveChanges();
I can't for the life of me figure out why it works once, and then not again, unless I restart my VS (running VS 2013 with IIS Express - SQL 2008 R2 Enterprise on a server).
Any help or pointers would be appreciated :-)

I seem to have narrowed down on the issue, and whilst it is more of a workaround than an answer this may help someone else.
The issue is the version of SQL Server, namely SQL 2008 R2 10.50.4000. I migrated my database to SQL Server 2012 build 11.0.5058, after which the code worked, every time.
Hope this helps someone!

I just had this and solved it by reversing the points in the polygon. Apparently SQL Server is left handed with these things or something.
So instead of having a string concatenation like strGeog += string.Format("{0} {1}, ", latlong[0], latlong[1]); I changed it to:
foreach (XmlNode xnPoly in xmlPolyList)
{
strGeog = "";
firstlatlong = null;
if (xnPoly["coordinates"] != null)
{
latlongpairs = xnPoly["coordinates"].InnerText.Replace("\n", "").Split(' ');
foreach (string ll in latlongpairs)
{
latlong = ll.Split(',');
if (firstlatlong == null) firstlatlong = latlong;
strGeog = string.Format("{0} {1}, ", latlong[0], latlong[1]) + strGeog;
}
}
if (strGMPoly.Length > 0)
{
strGeog = strGeog.Substring(0, strGeog.Length - 2); //trim off the last comma and space
strGeog = "POLYGON((" + string.Format("{0} {1} ", firstlatlong[0], firstlatlong[1]) + strGeog + "))"; // conversion from WKT needs it to come back to the first point.
}
i++;
dbPCPoly = new PostCodePolygon();
dbPCPoly.geog = DbGeography.PolygonFromText(strGeog, 4326);
LocDB.PostCodePolygons.Add(dbPCPoly);
LocDB.SaveChanges();
Console.WriteLine(string.Format("Added Polygon {0} for Postcode ({1})", dbPCPoly.PCPolyID, dbPC.PostCodeName));
}

SqLite C# extremely slow on update

I'm really struggling to iron out this issue. When I use the following code to update my database for large numbers of records it runs extremely slow. I've got 500,000 records to update which takes nearly an hour. During this operation, the journal file grows slowly with little change on the main SQLite db3 file - is this normal?
The operation only seems to be a problem when I have large numbers or records to update - it runs virtually instantly on smaller numbers of records.
Some other operations are performed on the database prior to this code running so could they be some how tying up the database? I've tried to ensure that all other connections are closed properly.
Thanks for any suggestions
using (SQLiteConnection sqLiteConnection = new SQLiteConnection("Data Source=" + _case.DatabasePath))
{
sqLiteConnection.Open();
using (SQLiteCommand sqLiteCommand = new SQLiteCommand("begin", sqLiteConnection))
{
sqLiteCommand.ExecuteNonQuery();
sqLiteCommand.CommandText = "UPDATE CaseFiles SET areaPk = #areaPk, KnownareaPk = #knownareaPk WHERE mhash = #mhash";
var pcatpk = sqLiteCommand.CreateParameter();
var pknowncatpk = sqLiteCommand.CreateParameter();
var pmhash = sqLiteCommand.CreateParameter();
pcatpk.ParameterName = "#areaPk";
pknowncatpk.ParameterName = "#knownareaPk";
pmhash.ParameterName = "#mhash";
sqLiteCommand.Parameters.Add(pcatpk);
sqLiteCommand.Parameters.Add(pknowncatpk);
sqLiteCommand.Parameters.Add(pmhash);
foreach (CatItem CatItem in _knownFiless)
{
if (CatItem.FromMasterHashes == true)
{
pcatpk.Value = CatItem.areaPk;
pknowncatpk.Value = CatItem.areaPk;
pmhash.Value = CatItem.mhash;
}
else
{
pcatpk.Value = CatItem.areaPk;
pknowncatpk.Value = null;
pmhash.Value = CatItem.mhash;
}
sqLiteCommand.ExecuteNonQuery();
}
sqLiteCommand.CommandText = "end";
sqLiteCommand.ExecuteNonQuery();
sqLiteCommand.Dispose();
sqLiteConnection.Close();
}
sqLiteConnection.Close();
}

The first thing to ensure that you have an index on mhash.
Group commands into batches.
Use more than one thread.
Or [inserted]
Bulk import the records to a temporary table. Create an index on the mhash column. Perform a single update statement to update the records.

You need to wrap everything inside a transaction otherwise I believe SQLite will create and commit one for you for every update ... hence the slowness. You clearly know that looking at your code but I am not sure using "Begin" and "End" commands achieve the same result here, you might end up with empty transaction at start and finish instead of one wrapping everything. Try something like this instead just in case:
using (SQLiteTransaction mytransaction = myconnection.BeginTransaction())
{
using (SQLiteCommand mycommand = new SQLiteCommand(myconnection))
{
SQLiteParameter myparam = new SQLiteParameter();
mycommand.CommandText = "YOUR QUERY HERE";
mycommand.Parameters.Add(myparam);
foreach (CatItem CatItem in _knownFiless)
{
...
mycommand.ExecuteNonQuery();
}
}
mytransaction.Commit();
}

This part is most certainly your problem.
foreach (CatItem CatItem in _knownFiless)
{
....
sqLiteCommand.ExecuteNonQuery();
}
You are looping a List(?) and executing a query against the database. That is not a good way to do it. Because database calls are quite expensive. So you might consider using another way of updating these items.

The SQL code appears to be okay. The C# code is not wrong, but it has some redundancy (explicit close/dispose is not needed since you're using a using already).
There is a for loop on _knownFiless (intended with double s?), could that run slowly possibly? It is unusual to run a query in a for loop against the DB, rather you should create a query with the respective set of parameters. Consider that (especially without an index on the hash) you will perform n * m operations (n being the run count of the for loop, m being the table size).
Considering that m is around 500k, and assuming that m = n you will get 250,000,000,000 operations. That may well last an hour.
Former connections or operations should have no effect as far as I know.
You should also ensure that the internal structure of the database is not causing problems. Is there a compound index that is affected from this operation? Any foreign keys / complex contraints?

trying to read sql statement c#

string queryStr = "select max(patient_history_date_bio) " +
"as med_date, medication_name from biological where " +
"(patient_id = " + patientID.patient_id + ") " +
"group by medication_name;";
using (var conn = new SqlConnection(connStr))
using (var cmd = new SqlCommand(queryStr, conn))
{
conn.Open();
using (SqlDataReader rdr = cmd.ExecuteReader())
{
int count = 0;
while (rdr.Read())
{
MessageBox.Show("test");
med.medication_date[count] = new DateTime();
med.medication_date[count] = DateTime.Parse(rdr["med_date"].
ToString());
MessageBox.Show("test2");
med.medication_name[count] = rdr["medication_name"].ToString();
count++;
}
}
conn.Close();
}
so i'm trying to read this sql statement. "test" message box displays, but not "test2". I tried running the sql statement in VS by itself (in the server explorer), and the sql statement works. it gives me what i want. but somehow, the code doesn't work... does anyone see why?

Assuming patient_id is some sort of integer (or is it a Guid), my assumption is that the current culture of your program is causing the ToString method call on int to be formatted in a way that is returning something that the parser can't parse (e.g. "1,234,567").
Generally, the way you are executing this statement is not a best-practice. While you might not be susceptible to injection attacks if the id is indeed an int (you are most definitely open to them if it's a string), you generally want to parameterize the queries.
The reason for this is not only to protect against injection attacks, but because it will properly format the parameter in the query string according to the type.
Another thing to point out about your code is how you are retrieving the values from the reader. You are effectively calling ToString on the DateTime instance, then calling Parse on the string to get a DateTime back.
This effectively burns cycles. All you need to do is cast (unbox in the case of value types) the value back.
So where you have:
med.medication_date[count] = DateTime.Parse(rdr["med_date"].
ToString());
You should have:
med.medication_date[count] = (DateTime) rdr["med_date"];
All that being said, as to why the second message box is not showing, my first guess is that you are executing this in an event handler in a Windows Forms application, and that an exception is being thrown.
I think that what you will find is that if medication_date is an array, then it hasn't been initialized and you are getting a NullReferenceException or something about the array index being out of bounds.

What is med.medication_date?
If it is an array, maybe it hasn't been initialized yet.
If it is a list, you should assign to it using med.medication_date.Add(value);
Alternatively, as everyone else is saying, the date time conversion may be at fault. Try replacing
MessageBox.Show("test");
With
MessageBox.Show(rdr["med_date"].ToString());

you should direct your debug stuff to the output window...it's muche easier to follow the flow.
system.diagnostics.debug.writeline(rdr["med_date"].ToString());

Without more info, it looks like the line
med.medication_date[count] = DateTime.Parse(rdr["med_date"].ToString());
throws an exception due to an unrecognised date, and the exception is being swallowed by a handler higher up.