We Keep Coding

How to get better performance from LINQ-to-SQL large dataset update

I have a c# console application that is updating a database with about 320,000 records. Basically it is encrypting a password in each record in a loop, then calling DatabaseContext.SubmitChanges(). The "UPDATE" part of the code takes about 20 seconds. I had to CTRL-C the app because it's taking over 15 minutes to do the "SubmitChanges" part: this is part of a time-sensitive system that should not be down for more than a couple minutes.
I ran SQL Profiler and I'm seeing queries like this for each update:
exec sp_executesql N'UPDATE [dbo].[PointRecord]
SET [PtPassword] = #p19
WHERE ([ID] = #p0) AND ([PtLocation] = #p1) AND ([PtIPAddress] = #p2) AND ([PtPort] = #p3) AND ([PtUsername] = #p4) AND ([PtPassword] = #p5) AND ([PtGWParam1] = #p6) AND ([PtGWParam2] = #p7) AND ([PtGWParam3] = #p8) AND ([PtGWParam4] = #p9) AND ([PtTag] = #p10) AND ([PtCapture] = #p11) AND ([PtGroup] = #p12) AND ([PtNumSuccess] = #p13) AND ([PtNumFailure] = #p14) AND ([PtControllerType] = #p15) AND ([PtControllerVersion] = #p16) AND ([PtAssocXMLGroupID] = #p17) AND ([PtErrorType] IS NULL) AND ([PtPollInterval] = #p18)',N'#p0 int,#p1 nvarchar(4000),#p2 nvarchar(4000),#p3 nvarchar(4000),#p4 nvarchar(4000),#p5 nvarchar(4000),#p6 nvarchar(4000),#p7 nvarchar(4000),#p8 nvarchar(4000),#p9 nvarchar(4000),#p10 nvarchar(4000),#p11 int,#p12 nvarchar(4000),#p13 int,#p14 int,#p15 nvarchar(4000),#p16 nvarchar(4000),#p17 int,#p18 int,#p19 nvarchar(4000)',#p0=296987,#p1=N'1234 Anytown USA',#p2=N'10.16.31.20',#p3=N'80',#p4=N'username1',#p5=N'password1',#p6=N'loadmon.htm?PARM2=21',#p7=N'>Operating Mode',#p8=N'',#p9=N'',#p10=N'1234 Anytown USA\HLTH SERVICE LTS\Operating Modeloadmon',#p11=0,#p12=N'1234 Anytown USA',#p13=0,#p14=0,#p15=N'DeviceA',#p16=N'3.5.0.2019.0219',#p17=309,#p18=15,#p19=N'hk+MUoeVMG69pOB3DHYB8g=='
As you can see, the "WHERE" part is asking for EVERY SINGLE FIELD to match, when this is an indexed table, using unique primary key "ID". This is really time-consuming. Is there any way to get this to only use "WHERE ID=[value]"?

I understand now that checking every field is a requirement of concurrency checking in EF. To bypass, methods outside of LINQ are required. I ended up using a variation of what Mr. Petrov and Mr. Harvey suggested, using ExecuteCommand since I am updating the database, not querying for data. Here is sample code, in case it can help others with a similar issue.
It uses LINQ to get the records to update and the record count for user feedback.
It uses ExecuteCommand to update the records. I am actually updating three tables (only one is shown in the sample below), hence the use of a transaction object.
The EncryptPassword method is not shown. It is what I use to update the records. You should replace that with whatever update logic suits your needs.
static void Main(string[] args)
{
DatabaseHelpers.Initialize();
if (DatabaseHelpers.PasswordsEncrypted)
{
Console.WriteLine("DatabaseHelpers indicates that passwords are already encrypted. Exiting.");
return;
}
// Note that the DatabaseHelpers.DbContext is in a helper library,
// it is a copy of the auto-generated EF 'DataClasses1DataContext'.
// It has already been opened using a generated connection string
// (part of DatabaseHelpers.Initialize()).
// I have altered some of the variable names to hide confidential information.
try
{
// show user what's happening
Console.WriteLine("Encrypting passwords...");
// flip switch on encryption methods
DatabaseHelpers.PasswordsEncrypted = true;
int recordCount = 0;
// Note: Using LINQ to update the records causes an unacceptable delay because of the concurrency checking
// where the UPDATE statement (at SubmitChanges) checks EVERY field instead of just the ID
// and we don't care about that!
// We have to set up an explicit transaction in order to use with context.ExecuteCommand statements
// start transaction - all or nothing
DatabaseHelpers.DbContext.Transaction = DatabaseHelpers.DbContext.Connection.BeginTransaction();
// update non-null and non-empty passwords in groups
Console.Write("Updating RecordGroups");
List<RecordGroup> recordGroups = (from p in DatabaseHelpers.DbContext.RecordGroups
where p.RecordPassword != null && p.RecordPassword != string.Empty
select p).ToList();
recordCount = recordGroups.Count;
foreach (RecordGroup rGroup in recordGroups)
{
// bypass LINQ-to-SQL
DatabaseHelpers.DbContext.ExecuteCommand("UPDATE RecordGroup SET RecordPassword={0} WHERE ID={1}", DatabaseHelpers.EncryptPassword(rGroup.RecordPassword), rGroup.ID);
Console.Write('.');
}
// show user what's happening
Console.WriteLine("\nCommitting transaction...");
DatabaseHelpers.DbContext.Transaction.Commit();
// display results
Console.WriteLine($"Updated {recordCount} RecordGroup passwords. Exiting.");
}
catch (Exception ex)
{
Console.WriteLine($"\nThere was an error executing the password encryption process: {ex}");
DatabaseHelpers.DbContext.Transaction.Rollback();
}
}

EF 5 Conditional Mapping

I'm using EF 5 Database first approach in my MVC application. all of my tables uses a Field called Deleted which is a boolean field to mark a record is deleted.
I'm trying to get rid of the requirement of having to check Deleted == false every time I query my database. The very straightforward way of doing this is to use a conditional mapping in the edmx file where EF always return data that are not deleted. That's all good.
But the problem of doing this condition mapping is that, when I want to allow the user to delete some record for e.g Address from their address book I don't have access to Delete field from EF as I used it in the conditional mapping and therefore I have to look for another option to allow user to delete a record.
The way I thought is to create a stored proc that handle the delete query and call it when I want to delete the record.
Is there a better way of doing this? Is it possible to make the Delete field accessible even it is used in the conditional mapping?

I have a working solution for Soft Delete in Entity Framework Code First that may help.
The key is that you add a discriminator to every model that you want to be able to soft delete. In code first that is done like this:
modelBuilder.Entity<Foo>().Map(m => m.Requires("IsDeleted").HasValue(false));
This makes it invisible to the context and therefore you have to do the deletes using sql.
If this is the equivalent of your "conditional mapping" in Database First then one way to modify the sql is to override SaveChanges and run sql from there:
public override int SaveChanges()
{
foreach (var entry in ChangeTracker.Entries()
.Where(p => p.State == EntityState.Deleted
&& p.Entity is ModelBase))//I do have a base class for entities with a single
//"ID" property - all my entities derive from this,
//but you could use ISoftDelete here
SoftDelete(entry);
return base.SaveChanges();
}
private void SoftDelete(DbEntityEntry entry)
{
var e = entry.Entity as ModelBase;
string tableName = GetTableName(e.GetType());
Database.ExecuteSqlCommand(
String.Format("UPDATE {0} SET IsDeleted = 1 WHERE ID = #id", tableName)
, new SqlParameter("id", e.ID));
//Marking it Unchanged prevents the hard delete
//entry.State = EntityState.Unchanged;
//So does setting it to Detached:
//And that is what EF does when it deletes an item
//http://msdn.microsoft.com/en-us/data/jj592676.aspx
entry.State = EntityState.Detached;
}
Method used to Get Table Name explained here
That is the way I used to do it. Probably irrelevant to your Database First approach in EF5, but I have now moved to doing it in stored procedures. EF6 Code First generates CreateStoredProcedure calls in Migration files. I replace these with this.CreateDeleteProcedure("dbo.Foo_Delete", "[dbo].[Foos]"); - which is a call to my own extension method:
public static class MigrationExtensions
{
internal static string DeleteSqlFormat
{
//I also hard delete anything deleted more than a day ago in the same table
get { return "DELETE FROM {0} WHERE IsDeleted = 1 AND DATEADD(DAY, 1, DeletedAt) < GETUTCDATE(); UPDATE {0} SET IsDeleted = 1, DeletedAt = GETUTCDATE() WHERE ID = #ID;"; }
}
internal static void CreateDeleteProcedure(this DbMigration migration, string procName, string tableName)
{
migration.CreateStoredProcedure(
procName,
p => new
{
ID = p.Int(),
},
body:
string.Format(MigrationExtensions.DeleteSqlFormat, tableName)
);
}
}

EntityFramework 4.1, context.Entities.Add sometimes set FK to NULL

I have this weird problem that has burned way more hours than it should.
My main question is this:
What may cause EF 4.1 Code First to set a foreign key to NULL when an entity is added?
The problem is this: I have a list of users on file, and these users must be inserted to my database if they're not already there.
I have something like this:
foreach (var u in usersFromFile) {
var userProfile = context.Users
.FirstOrDefault(user=>
user.EmployeeId == u.EmployeeId && user.CompanyId == 1);
if (userProfile == null) {
User newUser = new User();
newUser.EmployeeId = u.EmployeeId;
newUser.CompanyId = 1;
context.Users.Add(newUser); //This will sometimes set CompanyId = NULL
}
}
context.SaveChanges();
Some users won't be added to the Users table correctly.
They get CompanyId == NULL, and as such they do not belong to the company.
I also tried injecting SQL directly like so:
var query = #"INSERT INTO [dbo].[Users]
([CompanyId],[EmployeeId]) VALUES (3,#emplid)";
context.Database.ExecuteSqlCommand(query, new SqlParameter[] {
new SqlParameter("emplid", u.EmployeeId)});
I have tried to access the Users list on the Company object. That does not work.
I have tried to use context.Users.Create() instead of new User(). Does not change anything.
I have tried to inject SQL, still the same problem. That exact SQL works, if ran from the Studio Manager.
I have tried to context.SaveChanges() after each add, nothing changed.
I know for a fact that the state of the entity about to be added is correct, also in the cases where CompanyId is set to NULL.
Could there be something with my underlying database?
Thank you so much for your time and help!

Try the following:
foreach (var u in usersFromFile) {
if (context.Users.Any(
user=>
user.EmployeeId == u.EmployeeId && user.CompanyId == 1)
)
{
User newUser = new User();
newUser.EmployeeId = u.EmployeeId;
newUser.CompanyId = 1;
context.Users.Add(newUser); //This will sometimes set CompanyId = NULL
}
}
context.SaveChanges();
The Any() function checks wether a user based on the given query exists.
Also, don't forget to add the context.savechanges to make sure every added record gets put in the database.
Lastly, you checked if user.CompanyId = 1, should be == 1

I ended up transforming the list of users to a series of SQL statemens, and running them through context.Database.ExecuteSqlCommand(sql).
It's dirty but it works.
If anyone has any good ideas as to why the FK CompanyId on the user entity is sometimes set to NULL I'd we very happy if you share your ideas.

If condition with Entity Framework

In my asp.net application I have two pages like new students and edit students. On the new student page I am passing general details like first name, last name, mobile number, email and register number.
Here RegNo should be unique. I am using Entity Framework for database connection. I'm checking with the condition to avoid the same RegNo being entered, like:
DataObject.Entities dataEntities = new DataObject.Entities();
if (!dataEntities.Students.Any(s => s.RegNo == RegNo))
{
// my code here.
}
The same way for edit option, when try to change the RegNo. If it is allotted to some other student, it should not go into the update code.
I know if I use the same condition here, it will fail, because the RegNo is there in the database for this student (the one am trying to update), so if the RegNo is allotted for this particular student and not for other students it should be accepted, otherwise should go to else part.
I don't know how to check this using Entity Framework. Can anyone help me,please?
I've a column StudentId, it's an autoincrement column
I tried like
if (!dataEntities.Students.Any(s => s.RegNo == RegNo && s.StudentId != StudentId))
{
}
still it's not working.....

if(!dataEntities.Students.Any(s=>s.RegNo == RegNo && s != studentBeingUpdated))
Replace studentBeingUpdated with a variable containing a reference to the student that you are currently updating.

you can set in database level.. ie if you set auto increment then you dont need to edit/ reassign id to someone else and there is no need for separate management for this.
Please tell me if i understood mistakenly. :)

I just declared an object for the student table and tried like,
DataObject.Student student = dataEntities.Students.First(s => s.StudentId ==
StudentId);
if (!dataEntities.Students.Any(s => s.RegNo == RegNo &&
s.StudentId != student.StudentId))
{
}
else
{
throw new exception("RegNo already exists!");
}
and its working for me

Quick way to detect if a DataContext table or view exists

Currently, I'm developing an application that depends on (and thus connects to) various databases via LINQ-to-SQL. For one of the databases, the connection string may vary and is thus configurable - however, the schema of this database is identical for all connection strings.
Because of the configurable connection string, I want to validate the DataContext during the startup of my application, to make sure that all tables and views my application uses, are available.
The Table<T> objects in the DataContext object are always initialized - even if the corresponding SQL table or view doesn't have any records.
So then. Currently, the validation check is performed as follows:
bool valid = _dataContext.Articles.Count() > 0
&& _dataContext.Customers.Count() > 0
&& _dataContext.Orders.Count() > 0;
While this does work, the determination of the value of valid takes quite some time (every record of each Table is touched), which ultimately results in a time out. So, is there a faster, more reliable way to determine whether or not a Table<T> of a certain DataContext really exists as a table in the corresponding database?

Here is an (untested) idea:
Grab the name of your table. You can hard code it in, or you can grab it programmatically via
TableAttribute attribute = (TableAttribute)typeof(MyTableObject)
.GetCustomAttributes(typeof(TableAttribute), true)
.Single();
string name = attribute.Name;
MyTableObject is the LINQ-to-SQL generated object contained in your Table, i.e., the generic parameter T in Table<T>.
(TableAttribute is in System.Data.Linq.Mapping.)
Use the DataContext.ExecuteQuery method as in
var db = new MyDataContext();
var results = db.ExecuteQuery<string>("SELECT name FROM dbo.sysobjects WHERE xtype = 'U'");
bool hasTable = results.Any(s => "dbo." + s == name);

A slight change on Jason's answer (I gave him an upvote :))
public bool TableExistsInDatabase<T>()
{
TableAttribute attribute = (TableAttribute)typeof(T)
.GetCustomAttributes(typeof(TableAttribute), true)
.Single();
var result = ExecuteQuery<bool>(
String.Format(
"IF OBJECT_ID('{0}', 'U') IS NOT NULL
SELECT CAST(1 AS BIT) ELSE
SELECT CAST(0 AS BIT)", attribute.Name));
return result.First();
}