We Keep Coding

Compare local list to DataBase

I have a local List with entities, some hundreds, and I have a SQL Server table where I store the ID of the successful processed entities, some millions. I would like to know, which entities form my local set are not yet processed i.e. are not in the SQL Table.
The first approach is to iterate through the local list with the following Linq statement:
Entity entity = db.Entities.FirstOrDefault(m => m.ID == ID);
if (entity == null) { NewList.Add(ID) }
the NewList would then contain all the new entities. However this is very slow.
In LINQ, how would you send the entire local list to the SQL Server with one call and then return the ones not in the SQL table?
Do you really have to create a temporary table with my local list, then left-join on the already processed table and return the ones with a null?

Use .Contains method to retrieve already processed ids
and Except to create list of not yet processed ids.
var localList = new List<int> { 1, 2, 3 };
var processed = db.Entities
.Where(entity => localList.Contains(entity.Id))
.Select(entity => entity.Id)
.ToList();
var notProcessed = localList.Except(processed).ToList();
It will depend on provider, but .Contains should generate sql like:
SELECT Id FROM Entity WHERE Id IN (1, 2, 3)

suggestion:
create a temp table and insert your IDs
select your result on the SQL side
EDIT:
"Can you do that in LINQ?"
TL;DR:
yes* but that's an ugly piece of work, write the SQL yourself
*)depends on what you mean with "in" LINQ, because that is not in the scope of LINQ. In other words: a LINQ expression is one layer too abstract, but if you happen to have an LINQ accessible implementation for this, you can use this in your LINQ statements
on the LINQ expression side you have something like:
List<int> lst = new List<int>() { 1,2,3 };
List<int> result = someQueryable.Where(x=>lst.Contains(x.ID)).Select(x=>x.ID).ToList();
the question now is: what happens on the SQL side (assuming the queryable leads us to a SQL database)?
the queryable provider (e.g. Entity Framework) somehow has to translate that into SQL, execute it and come back with the result
here would be the place to modify the translation...
for example examine the expression tree with regard to the object that is the target for the Contains(...) call and if it is more than just a few elements, go for the temp table approach...
the very same LINQ expression can be translated into different SQL commands. The provider decides how the translation has to be done.
if your provider lacks support for large Contains(...) cases, you will probably experience poor performance... good thing is usually nobody forces you to use it this way ... you can skip linq for performance optimized queries, or you could write a provider extension yourself but then you are not on the "doing something with LINQ"-side but extending the functionality of your LINQ provider
if you are not developing a large scalable product that will be deployed to work with different DB-Backends, it is usually not worth the effort... the easier way to go is to write the sql yourself and just use the raw sql option of your db connection

Is Queryable.OrderBy unstable for SQL Server database?

OrderBy is stable for LINQ to Objects, but MSDN on Queryable.OrderBy doesn't mention if it is stable or not.
I guess it depends on the provider implementation. Is it unstable for SQL Server? Because it looks so. I did a quick look at Queryable source code, but it is not obvious from there.
I need to order a collection before other operations and I want to use IQueryable, rather than IEnumerable for the sake of performance.
// All the timestamps are the same and I am getting inconsistent
// results by running it multiple times, first few pages return the same results
var result = data.OrderBy(i => i.TimeStamp).Skip(start).Take(length);
but if I use
var result = data.ToList().OrderBy(i => i.TimeStamp).Skip(start).Take(length);
It works just fine, but I lose performance boost from LINQ to SQL. It seems combination of Queryable OrderBy/Skip/Take produce inconsistent results.
SQL Code generated seems fine to me:
SELECT
...
FROM [dbo].[Table] AS [Extent1]
ORDER BY [Extent1].[TimeStamp] ASC
OFFSET 0 ROWS FETCH NEXT 10 ROWS ONLY

In Linq-to-Entities LINQ queries are translated into SQL queries so Linq-to-Objects implementation of OrderBy doesn't matter. You should look at your database implementation of ORDER BY. If you are using MS SQL you can find in docs that:
To achieve stable results between query requests using OFFSET and FETCH, the following conditions must be met:
(...)
The ORDER BY clause contains a column or combination of columns that are guaranteed to be unique.
So ORDER BY for the same values does not guarantee the same order so limiting it could provide different results set. To solve this you can simply sort by some additional column that has unique values e.g. id. So basically you will have:
var result = data
.OrderBy(i => i.TimeStamp)
.ThenBy(i => i.Id)
.Skip(start)
.Take(length);

I take it that by "stable", you mean consistent. If you didn't have the ORDER BY in a SQL query, the order of the data is not guaranteed for each time you run the query. It will simply return all of the data in whatever order is most efficient for the server. When you add the ORDER BY, it will sort that data. Since you are sorting data where all of the sort values are the same, no rows are being reordered, so the ordered data is in an order you don't expect. If you need a specific order, you will need to add a secondary sort column such as an ID.
It is a best to never assume the order of data coming back from the server unless you explicitly define what that order is.

Linq query with Entity Framework yields no result

I am doing a query with linq and entity framework. The database I'm using is Oracle. Here is the code :
Entities bdd = contextWrapper.GetContext();
data = (from table in bdd.COP_PRDTICSOURES
where (table.IDTTIC==ticketId && table.IDTPRD==productId)
select table).AsEnumerable();
When I look at the variable bdd.COP_PRDTIC_SOURES using a debugger, it contains an entry matching my two criteria. However, after the execution of the query, the data variable contains no result.
Is there something wrong with my syntax ?
Some additional information:
The entity i'm looking for isn't commited in the database when I perform this query. It's created before in the same transaction.
I use a foreach on the data after, so it's not a problem of lazy loading.

AsEnumerable does not load data from DB. Calling it you are not executing the actual query, just changing how it is going to be executed in its entirety.
Use ToArray or ToList instead to load data explicitly. Or call foreach on this collection.

At what point do I need to off-load the work of a query to the DB?

I have a web app, and I'm connecting to a DB with entity framework.
To select all Employee records of a particular department, for example, I can quite easily write:
....Employees.Where(o => o.Department == "HR").ToList();
Works fine. But is it most optimal?
Should this Where clause be incorporated into a stored procedure or view? Or does my entity framework code do the job of converting it to SQL anyway?
We've had performance problems in our team in the past from when people pull records into memory and then do the filtering in .net instead of at a database level. I'm trying to avoid this happening again so want to be crystal clear on what I must avoid.

If Employees is provided by Entity Framework then the Where() will be translated into SQL and sent to the database. It is only the point that you materialise the objects does it take the filters you have applied before and turn them into SQL. Anything after that point is just plain LINQ to objects.
Methods that cause materialisation to happen include things like .ToList() and .ToArray() (there are more, but these two are probably the most common).
If you want to see what is happening on SQL Server, you should open up the SQL Profiler and have a look at the queries that are being sent.

We've had performance problems in our team in the past from when people pull records into memory and then do the filtering in .net instead of at a database level.
Just as an addendum to Colin's answer, and to target the quote above, the way to avoid this is to make sure your database queries are fully constructed with IQueryable<T> first, before enumerating the results with a call such as .ToList(), or .ToArray().
As an example, consider the following:
IEnumerable<Employee> employees = context.Employees;
// other code, before executing the following
var hrEmployees = employees.Where(o => o.Department == "HR").ToList();
The .ToList() will enumerate the results grabbing all of the employees from the context first, and then performing the filtering on the client. This isn't going to perform very well if you've got a lot of employees to contend with, and it's certainly not going to scale very well.
Compare that with this:
IQueryable<Employee> employees = context.Employees;
// other code, before executing the following
var hrEmployees = employees.Where(o => o.Department == "HR").ToList();
IQueryable<T> derives from IEnumerable<T>. The difference between them is that IQueryable<T> has a query provider built in, and the query you construct is represented as an expression tree. That means it's not evaluated until a call that enumerates the results of the query, such as .ToList().
In the second example above, the query provider will execute SQL to fetch only those employees that belong to the HR department, performing the filtering on the database itself.

Query Execution of SP while using it in LINQ to SQL

I have a query that returns a resultset. And I want to apply filter and sorting on the resultset.
Can someone help me understand if I use the query in LINQ (I'm using EF 4.0), will I be able to get deferred executuin so that when i apply filter/sort in entity model, the execution happens only one time (deffered)
Thanks in advance!
Regards,
Bhavik

If the query takes no parameters, then yes, as you could make a view that calls that sproc, expose the view in your model, then query it.
If it takes parameters, then if you need the sort/filter done server-side, then I think you'd have to add a wrapper sproc (or modify the existing one) to pass in the sort and filter to perform (basically, do it manually, but at least server-side).
Alternatively, you could write the sql to do it server side (sproc results into temp table, then select from that temp table and apply filtering, still manually) and then ExecuteStoreQuery

No, you can't defer the execution of a linq filtering to the stored proc in sql. The stored proc will be executed first, a resultset will be returned, you can then cast it to a list of your object types, once done that you can filter using Linq.
You can easily cast the resultset to a list of your objects using context.Translate<>
Have a look to these links :
enter link description here
List item
Of course the query (in your code) will not be evaluated until you cast it to a list, so you can concatenate all the filtering you want to your resultset and then call the ToList() to get the results.