We Keep Coding

How to get better performance from LINQ-to-SQL large dataset update

I have a c# console application that is updating a database with about 320,000 records. Basically it is encrypting a password in each record in a loop, then calling DatabaseContext.SubmitChanges(). The "UPDATE" part of the code takes about 20 seconds. I had to CTRL-C the app because it's taking over 15 minutes to do the "SubmitChanges" part: this is part of a time-sensitive system that should not be down for more than a couple minutes.
I ran SQL Profiler and I'm seeing queries like this for each update:
exec sp_executesql N'UPDATE [dbo].[PointRecord]
SET [PtPassword] = #p19
WHERE ([ID] = #p0) AND ([PtLocation] = #p1) AND ([PtIPAddress] = #p2) AND ([PtPort] = #p3) AND ([PtUsername] = #p4) AND ([PtPassword] = #p5) AND ([PtGWParam1] = #p6) AND ([PtGWParam2] = #p7) AND ([PtGWParam3] = #p8) AND ([PtGWParam4] = #p9) AND ([PtTag] = #p10) AND ([PtCapture] = #p11) AND ([PtGroup] = #p12) AND ([PtNumSuccess] = #p13) AND ([PtNumFailure] = #p14) AND ([PtControllerType] = #p15) AND ([PtControllerVersion] = #p16) AND ([PtAssocXMLGroupID] = #p17) AND ([PtErrorType] IS NULL) AND ([PtPollInterval] = #p18)',N'#p0 int,#p1 nvarchar(4000),#p2 nvarchar(4000),#p3 nvarchar(4000),#p4 nvarchar(4000),#p5 nvarchar(4000),#p6 nvarchar(4000),#p7 nvarchar(4000),#p8 nvarchar(4000),#p9 nvarchar(4000),#p10 nvarchar(4000),#p11 int,#p12 nvarchar(4000),#p13 int,#p14 int,#p15 nvarchar(4000),#p16 nvarchar(4000),#p17 int,#p18 int,#p19 nvarchar(4000)',#p0=296987,#p1=N'1234 Anytown USA',#p2=N'10.16.31.20',#p3=N'80',#p4=N'username1',#p5=N'password1',#p6=N'loadmon.htm?PARM2=21',#p7=N'>Operating Mode',#p8=N'',#p9=N'',#p10=N'1234 Anytown USA\HLTH SERVICE LTS\Operating Modeloadmon',#p11=0,#p12=N'1234 Anytown USA',#p13=0,#p14=0,#p15=N'DeviceA',#p16=N'3.5.0.2019.0219',#p17=309,#p18=15,#p19=N'hk+MUoeVMG69pOB3DHYB8g=='
As you can see, the "WHERE" part is asking for EVERY SINGLE FIELD to match, when this is an indexed table, using unique primary key "ID". This is really time-consuming. Is there any way to get this to only use "WHERE ID=[value]"?

I understand now that checking every field is a requirement of concurrency checking in EF. To bypass, methods outside of LINQ are required. I ended up using a variation of what Mr. Petrov and Mr. Harvey suggested, using ExecuteCommand since I am updating the database, not querying for data. Here is sample code, in case it can help others with a similar issue.
It uses LINQ to get the records to update and the record count for user feedback.
It uses ExecuteCommand to update the records. I am actually updating three tables (only one is shown in the sample below), hence the use of a transaction object.
The EncryptPassword method is not shown. It is what I use to update the records. You should replace that with whatever update logic suits your needs.
static void Main(string[] args)
{
DatabaseHelpers.Initialize();
if (DatabaseHelpers.PasswordsEncrypted)
{
Console.WriteLine("DatabaseHelpers indicates that passwords are already encrypted. Exiting.");
return;
}
// Note that the DatabaseHelpers.DbContext is in a helper library,
// it is a copy of the auto-generated EF 'DataClasses1DataContext'.
// It has already been opened using a generated connection string
// (part of DatabaseHelpers.Initialize()).
// I have altered some of the variable names to hide confidential information.
try
{
// show user what's happening
Console.WriteLine("Encrypting passwords...");
// flip switch on encryption methods
DatabaseHelpers.PasswordsEncrypted = true;
int recordCount = 0;
// Note: Using LINQ to update the records causes an unacceptable delay because of the concurrency checking
// where the UPDATE statement (at SubmitChanges) checks EVERY field instead of just the ID
// and we don't care about that!
// We have to set up an explicit transaction in order to use with context.ExecuteCommand statements
// start transaction - all or nothing
DatabaseHelpers.DbContext.Transaction = DatabaseHelpers.DbContext.Connection.BeginTransaction();
// update non-null and non-empty passwords in groups
Console.Write("Updating RecordGroups");
List<RecordGroup> recordGroups = (from p in DatabaseHelpers.DbContext.RecordGroups
where p.RecordPassword != null && p.RecordPassword != string.Empty
select p).ToList();
recordCount = recordGroups.Count;
foreach (RecordGroup rGroup in recordGroups)
{
// bypass LINQ-to-SQL
DatabaseHelpers.DbContext.ExecuteCommand("UPDATE RecordGroup SET RecordPassword={0} WHERE ID={1}", DatabaseHelpers.EncryptPassword(rGroup.RecordPassword), rGroup.ID);
Console.Write('.');
}
// show user what's happening
Console.WriteLine("\nCommitting transaction...");
DatabaseHelpers.DbContext.Transaction.Commit();
// display results
Console.WriteLine($"Updated {recordCount} RecordGroup passwords. Exiting.");
}
catch (Exception ex)
{
Console.WriteLine($"\nThere was an error executing the password encryption process: {ex}");
DatabaseHelpers.DbContext.Transaction.Rollback();
}
}

Why MySQL return is always 1 in C# code but not when I test in stored procedure?

I am developing an ASP.net application using MySQL and have a question related to a stored procedure return value.
This is my stored procedure:
CREATE DEFINER=`pcg`#`%` PROCEDURE `UpdatePreSellerProfile`(
IN UserID INT(11),
IN SellerImageID INT(11),
IN BusinessImageID INT(11),
OUT ProfileUpdated INT(1)
)
BEGIN
SET #Approved = 'APPROVED';
UPDATE user SET
SELLER_IMAGE_ID = COALESCE((SELECT IMAGE_ID FROM image_url WHERE IMAGE_USER_ID = UserID AND IMAGE_ID=SellerImageID),SELLER_IMAGE_ID),
SELLER_BUSINESS_LOGO_ID = COALESCE((SELECT IMAGE_ID FROM image_url WHERE IMAGE_USER_ID = UserID AND IMAGE_ID=BusinessImageID),SELLER_BUSINESS_LOGO_ID)
WHERE (USER_LOGIN_ID = UserID AND USER_PROFILE_STATUS = #Approved);
SET ProfileUpdated = ROW_COUNT();
END
When I test this code with following MySQL script I get 0 (#ProfileUpdated) always when there is no update.
call UpdatePreSellerProfile(#UserID, #SellerImageID, #BusinessImageID ,#ProfileUpdated);
But when I check this in my C# code, it is always showing 1 (ProfileUpdated).
if (oMySQLConnecion.State == System.Data.ConnectionState.Open)
{
MySqlCommand oCommand = new MySqlCommand("UpdatePreSellerProfile", oMySQLConnecion);
oCommand.CommandType = System.Data.CommandType.StoredProcedure;
MySqlParameter sqlProfileUpdated = new MySqlParameter("#ProfileUpdated", MySqlDbType.VarString);
sqlProfileUpdated.Direction = System.Data.ParameterDirection.Output;
oCommand.Parameters.Add(sqlProfileUpdated);
oCommand.Parameters.AddWithValue("#UserID", UserID);
oCommand.Parameters.AddWithValue("#SellerImageID", oSeller.SellerImageID);
oCommand.Parameters.AddWithValue("#BusinessImageID", oSeller.BusinessLogoID);
oCommand.ExecuteNonQuery();
Int16 ProfileUpdated = Convert.ToInt16(oCommand.Parameters["#ProfileUpdated"].Value);
if (ProfileUpdated > 0) // <<-- Should be greater only if it is updated is sucessfull
{
oDBStatus.Type = DBOperation.SUCCESS;
oDBStatus.Message.Add(DBMessageType.SUCCESSFULLY_DATA_UPDATED);
}
else
{
oDBStatus.Type = DBOperation.ERROR;
oDBStatus.Message.Add(DBMessageType.ERROR_NO_RECORDS_UPDATED);
}
oMySQLConnecion.Close();
}
Why is the difference between MySQL script vs C# code?

Unless you have set the UseAffectedRows connection string option, it defaults to false. This means:
When false (default), the connection reports found rows instead of changed (affected) rows. Set to true to report only the number of rows actually changed by UPDATE or INSERT … ON DUPLICATE KEY UPDATE statements.
Additionally, from the documentation of the ROW_COUNT function:
For UPDATE statements, the affected-rows value by default is the number of rows actually changed. If you specify the CLIENT_FOUND_ROWS flag to mysql_real_connect() when connecting to mysqld [ed. note: this is the same as UseAffectedRows], the affected-rows value is the number of rows “found”; that is, matched by the WHERE clause.
Thus, the UPDATE user statement in your stored procedure will return the number of rows that were found by the query, not the number that were actually updated.
To fix this, either:
Set UseAffectedRows=true; in your connection string; this may cause changes to other UPDATE queries.
Add more conditions to the WHERE clause, e.g., WHERE ... AND SELLER_IMAGE_ID != SellerImageID AND SELLER_BUSINESS_LOGO_ID != BusinessImageID, to make sure the row is only found and updated if it actually needs to change.

Error from SQL query

Currently I'm working on cleaning up some code on the backend of an application I'm contracted for maintenance to. I ran across a method where a call is being made to the DB via Oracle Data Reader. After examining the SQL, I realized it was not necessary to make the call to open up Oracle Data Reader seeing how the object being loaded up was already within the Context of our Entity Framework. I changed the code to follow use of the Entity Model instead. Below are the changes I made.
Original code
var POCs = new List<TBLPOC>();
Context.Database.Connection.Open();
var cmd = (OracleCommand)Context.Database.Connection.CreateCommand();
OracleDataReader reader;
var SQL = string.Empty;
if (IsAssociate == 0)
SQL = #"SELECT tblPOC.cntPOC,INITCAP(strLastName),INITCAP(strFirstName)
FROM tblPOC,tblParcelToPOC
WHERE tblParcelToPOC.cntPOC = tblPOC.cntPOC AND
tblParcelToPOC.cntAsOf = 0 AND
tblParcelToPOC.cntParcel = " + cntParcel + " ORDER BY INITCAP(strLastName)";
else
SQL = #"SELECT cntPOC,INITCAP(strLastName),INITCAP(strFirstName)
FROM tblPOC
WHERE tblPOC.cntPOC NOT IN ( SELECT cntPOC
FROM tblParcelToPOC
WHERE cntParcel = " + cntParcel + #"
AND cntAsOf = 0 )
AND tblPOC.ysnActive = 1 ORDER BY INITCAP(strLastName)";
cmd.CommandText = SQL;
cmd.CommandType = CommandType.Text;
using (reader = cmd.ExecuteReader())
{
while (reader.Read())
{
POCs.Add(new TBLPOC { CNTPOC = (decimal)reader[0],
STRLASTNAME = reader[1].ToString(),
STRFIRSTNAME = reader[2].ToString() });
}
}
Context.Database.Connection.Close();
return POCs;
Replacement code
var sql = string.Empty;
if (IsAssociate == 0)
sql = string.Format(#"SELECT tblPOC.cntPOC,INITCAP(strLastName),INITCAP(strFirstName)
FROM tblPOC,tblParcelToPOC
WHERE tblParcelToPOC.cntPOC = tblPOC.cntPOC
AND tblParcelToPOC.cntAsOf = 0
AND tblParcelToPOC.cntParcel = {0}
ORDER BY INITCAP(strLastName)",
cntParcel);
else
sql = string.Format(#"SELECT cntPOC,INITCAP(strLastName), INITCAP(strFirstName)
FROM tblPOC
WHERE tblPOC.cntPOC NOT IN (SELECT cntPOC
FROM tblParcelToPOC
WHERE cntParcel = {0}
AND cntAsOf = 0)
AND tblPOC.ysnActive = 1
ORDER BY INITCAP(strLastName)",
cntParcel);
return Context.Database.SqlQuery<TBLPOC>(sql, "0").ToList<TBLPOC>();
The issue I'm having right now is when the replacement code is executed, I get the following error:
The data reader is incompatible with the specified 'TBLPOC'. A member of the type 'CNTPOCORGANIZATION', does not have a corresponding column in the data reader with the same name.
The field cntPOCOrganization does exist within tblPOC, as well as within the TBLPOC Entity. cntPOCOrganization is a nullable decimal (don't ask why decimal, I myself don't get why the previous contractors used decimals versus ints for identifiers...). However, in the past code and the newer code, there is no need to fill that field. I'm confused on why it is errors out on that particular field.
If anyone has any insight, I would truly appreciate it. Thanks.
EDIT: So after thinking on it a bit more and doing some research, I think I know what the issue is. In the Entity Model for TBLPOC, the cntPOCOrganization field is null, however, there is an object tied to this Entity Model called TBLPOCORGANIZATION. I'm pondering if it's trying to fill it. It too has cntPOCOrganization within itself and I'm guessing that maybe it is trying to fill itself and is what is causing the issue.
That maybe possibly why the previous contractor wrote the Oracle Command versus run it through the Entity Framework. I'm going to revert back for time being (on a deadline and really don't want to play too long with it). Thanks!

This error is issued when your EF entity model does not match the query result. If you post your entity model you are trying to fetch this in, the SQL can be fixed. In general you need to use:
sql = string.Format(#"SELECT tblPOC.cntPOC AS <your_EF_model_property_name_here>,INITCAP(strLastName) AS <your_EF_model_property_name_here>,INITCAP(strFirstName) AS <your_EF_model_property_name_here>
FROM tblPOC,tblParcelToPOC
WHERE tblParcelToPOC.cntPOC = tblPOC.cntPOC
AND tblParcelToPOC.cntAsOf = 0
AND tblParcelToPOC.cntParcel = {0}
ORDER BY INITCAP(strLastName)",
cntParcel);

Getting empty answer when I should get nothing

I'm working on a C# program that uses a SQL Server Compact database. I have a query where I want to select the highest number in a specific field that looks like this:
SELECT MAX(nr2) FROM TABLE WHERE nr1 = '10'
This works as inteneded when there is a row where nr1 is 10. But I would expect to not get an answer when that row doesn't exist, but instead I get an empty field. So in my C# code I have:
text = result[0].ToString();
When I get a value from my SQL query the string contains a number and when the specified row doesn't exist I get an empty string.
This isn't really a big problem but I would be able to do the following check:
if (result.Count > 0)
Instead of:
if (result[0].ToString() == "")
which I have to do at the moment since count is always larger than 0.

Talk about using a sledgehammer to crack a nut, but...
I don't test it with C# code, but in SQL Server Management Studio, if you run...
SELECT MAX(nr2) FROM TABLE WHERE nr1 = '10' HAVING MAX(nr2) IS NOT NULL
, the result is an empty collection, not a collection with one null (or empty) element.
NOTE: My answer is based on this SO Answer. It seems that MAX and COUNT SQL functions returns always a single row collection.

That SQL statement will always return a result... if the base query returns no result then the value of max() is null !
if you are using ADO.NEt, you could use ExecuteScalar, here an example :
private int GetIDNum()
{
SqlConnection connection = new SqlConnection("connectionstring");
using(SqlCommand command = new SqlCommand("SELECT MAX(nr2) FROM TABLE WHERE nr1 = '10'", connection))
{
try
{
connection.Open();
object result = command.ExecuteScalar();
if( result != null && result != DBNull.Value )
{
return Convert.ToInt32( result );
}
else
{
return 0;
}
}
finally
{
connection.Close();
}
}
}

Problem in Manual ID Code

My Auto ID is already working. But, I tried a Manual ID code and insert the value in my database.. My Questions is how i able to get the value of the manual id. Yes, it's simple but not in a read only.
Please check my code:
Deleting - Its working!
CMSInsurance oInsuranceDelete = new CMSInsurance();
Insurance oInsurance = new Insurance();
List<InsuranceLabel> lstName = oInsuranceDelete.RetrieveInsuranceList();
foreach (InsuranceLabel item in lstName)
{
var code = e.CommandArgument;
if (item.InsuranceCode.ID == code.ToString())
{
oInsurance.InsuranceCode = item.InsuranceCode; //proper getting the value of Insurance Code instead of txtCode.text or e.CommandArgument
oInsuranceDelete.DeleteInsurance(oInsurance);
bind();
}
}
Inserting Auto ID code - Its Working!
oInsurance.Name = txtName.text // for string
CMSInsurance oCMSInsurance = new CMSnsurance();
oCMSInsurance.CreateInsurance(oInsurance);
Inserting Manual ID Code - Not working..
//Get the Manual value of Insurance Code
//For Insert a New Insurance
CMSInsurance oCMSInsuranceManual= new CMSInsurance ();
List<InsuranceLabel> lstList = oCMSInsuranceManual.RetrieveInsuranceList();
foreach (InsuranceLabel item in lstList)
{
if (item.InsuranceCode.ID != Session["AutoCode"])
{
oInsurance.InsuranceCode = item.InsuranceCode; // I can't insert a string like txtCode.. is there anyway to solve this issue?
}
}
Thanks!

Like tSQL, SQL Anywhere does not 'normally' allow for inserts into an identity field:
SQL Server does not allow INSERTs into
identity columns unless the
identity_insert option is set to on
you can find this and more on SQL at Sybase SQL Anywhere User's Guide