We Keep Coding

Compare local list to DataBase

I have a local List with entities, some hundreds, and I have a SQL Server table where I store the ID of the successful processed entities, some millions. I would like to know, which entities form my local set are not yet processed i.e. are not in the SQL Table.
The first approach is to iterate through the local list with the following Linq statement:
Entity entity = db.Entities.FirstOrDefault(m => m.ID == ID);
if (entity == null) { NewList.Add(ID) }
the NewList would then contain all the new entities. However this is very slow.
In LINQ, how would you send the entire local list to the SQL Server with one call and then return the ones not in the SQL table?
Do you really have to create a temporary table with my local list, then left-join on the already processed table and return the ones with a null?

Use .Contains method to retrieve already processed ids
and Except to create list of not yet processed ids.
var localList = new List<int> { 1, 2, 3 };
var processed = db.Entities
.Where(entity => localList.Contains(entity.Id))
.Select(entity => entity.Id)
.ToList();
var notProcessed = localList.Except(processed).ToList();
It will depend on provider, but .Contains should generate sql like:
SELECT Id FROM Entity WHERE Id IN (1, 2, 3)

suggestion:
create a temp table and insert your IDs
select your result on the SQL side
EDIT:
"Can you do that in LINQ?"
TL;DR:
yes* but that's an ugly piece of work, write the SQL yourself
*)depends on what you mean with "in" LINQ, because that is not in the scope of LINQ. In other words: a LINQ expression is one layer too abstract, but if you happen to have an LINQ accessible implementation for this, you can use this in your LINQ statements
on the LINQ expression side you have something like:
List<int> lst = new List<int>() { 1,2,3 };
List<int> result = someQueryable.Where(x=>lst.Contains(x.ID)).Select(x=>x.ID).ToList();
the question now is: what happens on the SQL side (assuming the queryable leads us to a SQL database)?
the queryable provider (e.g. Entity Framework) somehow has to translate that into SQL, execute it and come back with the result
here would be the place to modify the translation...
for example examine the expression tree with regard to the object that is the target for the Contains(...) call and if it is more than just a few elements, go for the temp table approach...
the very same LINQ expression can be translated into different SQL commands. The provider decides how the translation has to be done.
if your provider lacks support for large Contains(...) cases, you will probably experience poor performance... good thing is usually nobody forces you to use it this way ... you can skip linq for performance optimized queries, or you could write a provider extension yourself but then you are not on the "doing something with LINQ"-side but extending the functionality of your LINQ provider
if you are not developing a large scalable product that will be deployed to work with different DB-Backends, it is usually not worth the effort... the easier way to go is to write the sql yourself and just use the raw sql option of your db connection

Speeding up LINQ queries (selecting data from table)

I have written a code which looks like this:
using(var ctx = new myentitiesContext())
{
var currentLoggedUser = ctx.Users.FirstOrDefault(x=>x.Email==User.Identity.Name);
var items = ctx.Items.Where(x=>x.Sales>0 && x.UserId==currentLoggedUser.UserId).ToList();
}
As you can see it's a simple select from the DB. But the tricky part is that sometimes I can select a large quantity of data (50-100k records at a time). So what I've been wondering, are there any ways to tweak the LINQ to perform faster when the data is being pulled out of the table?
I've already created indexes in my table on FK UserId, so that part is done.
My question here is, is there any way to speed up LINQ queries via some tweaks in context configuration section, or perhaps by creating compiled queries, or via some other method ?
P.S. guys, would something like this work good:
ctx.Configuration.AutoDetectChangesEnabled = false;
// my queries...
ctx.Configuration.AutoDetectChangesEnabled = true;

In addition with the things that the rest of the users have written. You could disable lazy loading. That way if the Items Db Table has references to other tables they will not get loaded along with the Items unless you absolutely need it. Check these links
thecodegarden
mehdi
One more think that i would recommend is that you must log the sql queries that your linq expressions create and try to optimise them with your DBA. You could do this by adding an Action<string> delegate on the DbContext.Database.Log that will emit everything between a connection.Open() and a connection.Close(). You could also take the sql query out of your IQueryableor IQueryable<T> calling the .ToString() method on your IQueryable variable.

You should make projection first. For example, this:
var items = ctx.Items.Where(x=>x.Sales>0 && x.UserId==currentLoggedUser.UserId).ToList();
will be better if you write it like this:
var items = ctx.Items.Where(x.UserId==currentLoggedUser.UserId).Where(x2=>x2.Sales>0 ).ToList();
And if you don't need all the object you have to use the "Select" clause before the "Where" and project just the properties that you need to minimize the cost, like this:
ctx.Items.Select(e=>new {e.UserID,e.Sales}).Where(x.UserId==currentLoggedUser.UserId).Where(x2=>x2.Sales>0 ).ToList();

How do I update multiple Entity models in one SQL statement?

I had the following:
List<Message> unreadMessages = this.context.Messages
.Where( x =>
x.AncestorMessage.MessageID == ancestorMessageID &&
x.Read == false &&
x.SentTo.Id == userID ).ToList();
foreach(var unreadMessage in unreadMessages)
{
unreadMessage.Read = true;
}
this.context.SaveChanges();
But there must be a way of doing this without having to do 2 SQL queries, one for selecting the items, and one for updating the list.
How do i do this?

Current idiomatic support in EF
As far as I know, there is no direct support for "bulk updates" yet in Entity Framework (there has been an ongoing discussion for bulk operation support for a while though, and it is likely it will be included at some point).
(Why) Do you want to do this?
It is clear that this is an operation that, in native SQL, can be achieved in a single statement, and provides some significant advantages over the approach followed in your question. Using the single SQL statement, only a very small amount of I/O is required between client and DB server, and the statement itself can be completely executed and optimized by the DB server. No need to transfer to and iterate through a potentially large result set client side, just to update one or two fields and send this back the other way.
How
So although not directly supported by EF, it is still possible to do this, using one of two approaches.
Option A. Handcode your SQL update statement
This is a very simple approach, that does not require any other tools/packages and can be performed Async as well:
var sql = "UPDATE TABLE x SET FIELDA = #fieldA WHERE FIELDB = #fieldb";
var parameters = new SqlParameter[] { ..., ... };
int result = db.Database.ExecuteSqlCommand(sql, parameters);
or
int result = await db.Database.ExecuteSqlCommandAsync(sql, parameters);
The obvious downside is, well breaking the nice linqy paradigm and having to handcode your SQL (possibly for more than one target SQL dialect).
Option B. Use one of the EF extension/utility packages
Since a while, a number of open source nuget packages are available that offer specific extensions to EF. A number of them do provide a nice "linqy" way to issue a single update SQL statement to the server. Two examples are:
Entity Framework Extended Library that allows performing a bulk update using a statement like:
context.Messages.Update(
x => x.Read == false && x.SentTo.Id == userID,
x => new Message { Read = true });
It is also available on github
EntityFramework.Utilities that allows performing a bulk update using a statement like:
EFBatchOperation
.For(context, context.Messages)
.Where(x => x.Read == false && x.SentTo.Id == userID)
.Update(x => x.Read, x => x.Read = true);
It is also available on github
And there are definitely other packages and libraries out there that provide similar support.

Even SQL has to do this in two steps in a sense, in that an UPDATE query with a WHERE clause first runs the equivalent of a SELECT behind the scenes, filtering via the WHERE clause, then applying the update. So really, I don't think you need to be worried about improving this.
Further, the reason why it's broken into two steps like this in LINQ is precisely for performance reasons. You want that "select" to be as minimal as possible, i.e. you don't want to load any more objects from the database into in memory objects than you have to. Only then do you alter objects (in the foreach).
If you really want to run a native UPDATE on the SQL side, you could use a System.Data.SqlClient.SqlCommand to issue the update, instead of having LINQ give you back objects that you then update. That will be faster, but then you conceptually move some of your logic out of your C# code object model space into the database model space (you are doing things in the database, not in your object space), even if the SqlCommand is being issued from your code.

EF5 generating "imaginary" columns in select statements

We are using C#. VS2012, EF 5.0, and Oracle 11g. Approach is code first. I have a table that is defined, and it is plainly visible in looking at the code that it is defined with all the correct columns (and none that are not there.)
Still, when I run certain LINQ queries (joins) and attempt to select the results into a new object, things break. Here is the LINQ:
IQueryable<CheckWage> query =
from clientWage in context.ClientWages
join paycheckWage in context.PaycheckWages
on
new {clientWage.PermanentClientId, clientWage.WageId} equals
new {paycheckWage.PermanentClientId, paycheckWage.WageId}
where
(paycheckWage.PermanentClientId == Session.PermanentClientId) &&
(clientWage.PermanentClientId == Session.PermanentClientId)
select new CheckWage
{
CWage = clientWage,
PWage = paycheckWage
};
Now, here is the SQL it emits (as captured by Devart's DbMonitor tool):
SELECT
"Extent1".ASSOCIATE_NO,
"Extent1".PCLIENT_ID,
"Extent1".CLIENT_NO,
"Extent1".CLIENT_NAME,
"Extent1".ADDRESS1,
"Extent1".ADDRESS2,
"Extent1".CITY,
"Extent1".STATE,
"Extent1".ZIP,
"Extent1".COUNTRY,
"Extent1".CLIENT_TYPE,
"Extent1".DOING_BUSINESS_AS,
"Extent1".CONTACT,
"Extent1".PHONE,
"Extent1".EXTENSION,
"Extent1".FAX,
"Extent1".FAX_EXTENSION,
"Extent1".EMAIL,
"Extent1".NEXTEMP,
"Extent1".PAY_FREQ,
"Extent1".EMPSORT,
"Extent1".DIVUSE,
"Extent1".CLIENT_ACCESS_TYPE,
"Extent1".AUTOPAY_WAGE_ID,
"Extent1".FEIN,
"Extent1".HR_MODULE,
"Extent1".BANK_CODE,
"Extent1".ACH_DAYS,
"Extent1".ACH_COLLECT,
"Extent1".UPDATED,
"Extent1".IAT_FLAG,
"Extent1".ORIG_EMAIL,
"Extent1"."R1",
"Extent1"."R2"
FROM INSTANTPAY.CLIENT "Extent1"
WHERE "Extent1".PCLIENT_ID = :EntityKeyValue1'
There are no such columns as "R1" and "R2." I am guessing is has something to do with the join into a new object type with two properties, but I am pulling my hair out trying to figure out what I've done or haven't done that is resulting in this errant SQL. Naturally, the error from the Oracle server is "ORA-00904: "Extent1"."R2": invalid identifier." Strange that is doesn't choke on R1, but perhaps it only lists the last error or something...
Thanks in advance,
Peter
5/23/2014: I left out an important detail. The SQL is emitted when I attempt to drill into one of the CheckWage objects (using Lazy loading), as both of the contained objects have a navigation property to the "Client" entity. I can access the client table just fine in other LINQ queries that do not use a join, it is only this one that creates the "R1" and "R2" in the SELECT statement.
Peter

LinqPad, using multiple datacontexts - DevForce

I have bought the premium version of LINQPad. I thought it would also be possible to perform cross data base queries with DevForce models.
There are two ways to do this. The simplest is the drag-and-drop
approach: hold down the Ctrl key while dragging additional databases
from the Schema Explorer to the query editor. To access those
additional databases in your queries, use database.table notation,
e.g., Northwind.Regions.Take(100). The databases that you query must
reside on the same server.
The second approach is to list the extra database(s) that you want to
query in the connection properties dialog. This dialog also lets you
choose databases from linked servers. Here's how to proceed:
Add a new LINQ to SQL connection.
Choose Specify New or Existing Database and choose the primary database that you want to query.
Click the Include Additional Databases checkbox and pick the extra database(s) you want to include. You can also choose databases from
linked servers in this dialog.
Source
But obviously there isn't any way, is it? Anyone a solution for this?

Cross-database querying works only with standard SQL Server connections, with databases on the same server or on linked servers. The main rationale is to ensure server-side joining (otherwise you'd end up pulling entire tables back to the client whenever you joined).
I have considered adding a feature to LINQPad to allow arbitrary cross-database queries, because sometimes it would be useful even with client-side joining. However, getting this to work with custom data contexts (such as DevForce or Entity Framework) turned out to be really tricky, and so the feature ended up in the "too-hard basket". A major problem was dealing with namespace/assembly/app.config conflicts.
Bear in mind that there's nothing to stop you from pressing F4 and adding a reference to an assembly containing an additional datacontext. Of course, you'd have to manually instantiate the second data context, but that shouldn't be a huge deal. You'll still get autocompletion, and you'll still be able to see its schema in the tree view if you create a separate connection for it. And functionally, that's what you'd end up with anyway, if LINQPad supported multi-connection queries.
What's special about LINQPad's cross-database querying support for SQL Server is that it does something you couldn't otherwise do simply by adding a reference to another assembly, which is to allow efficient cross-database querying by leveraging server-side joins.

You can instantiate as many contexts as you like to disparate SQL instances and execute pseudo cross database joins, copy data, etc. Note, joins across contexts are performed locally so you must call ToList(), ToArray(), etc to execute the queries using their respective data sources individually before joining. In other words if you "inner" join 10 rows from DB1.TABLE1 with 20 rows from DB2.TABLE2, both sets (all 30 rows) must be pulled into memory on your local machine before Linq performs the join and returns the related/intersecting set (20 rows max per example).
//EF6 context not selected in Linqpad Connection dropdown
var remoteContext = new YourContext();
remoteContext.Database.Connection.ConnectionString = "Server=[SERVER];Database="
+ "[DATABASE];Trusted_Connection=false;User ID=[SQLAUTHUSERID];Password="
+ "[SQLAUTHPASSWORD];Encrypt=True;";
remoteContext.Database.Connection.Open();
var DB1 = new Repository(remoteContext);
//EF6 connection to remote database
var remote = DB1.GetAll<Table1>()
.Where(x=>x.Id==123)
//note...depending on the default Linqpad connection you may get
//"EntityWrapperWithoutRelationships" results for
//results that include a complex type. you can use a Select() projection
//to specify only simple type columns
.Select(x=>new { x.Col1, x.Col1, etc... })
.Take(1)
.ToList().Dump(); // you must execute query by calling ToList(), ToArray(),
// etc before joining
//Linq-to-SQL default connection selected in Linqpad Connection dropdown
Table2.Where(x=>x.Id = 123)
.ToList() // you must execute query by calling ToList(), ToArray(),
// etc before joining
.Join(remote, a=> a.d, b=> (short?)b.Id, (a,b)=>new{b.Col1, b.Col2, a.Col1})
.Dump();
localContext.Database.Connection.Close();
localContext = null;