We Keep Coding

Why are multiple where in LINQ so slow?

Using C# and Linq to SQL, I found that my query with multiple where is orders of magnitude slower than with a single where / and.
Here is the query
using (TeradiodeDataContext dc = new TeradiodeDataContext())
{
var filterPartNumberID = 71;
var diodeIDsInBlades = (from bd in dc.BladeDiodes
select bd.DiodeID.Value).Distinct();
var diodesWithTestData = (from t in dc.Tests
join tt in dc.TestTypes on t.TestTypeID equals tt.ID
where tt.DevicePartNumberID == filterPartNumberID
select t.DeviceID.Value).Distinct();
var result = (from d in dc.Diodes
where d.DevicePartNumberID == filterPartNumberID
where diodesWithTestData.Contains(d.ID)
where !diodeIDsInBlades.Contains(d.ID)
orderby d.Name
select d);
var list = result.ToList();
// ~15 seconds
}
However, when the condition in the final query is this
where d.DevicePartNumberID == filterPartNumberID
& diodesWithTestData.Contains(d.ID)
& !diodeIDsInBlades.Contains(d.ID)
// milliseconds
it is very fast.
Comparing the SQL in result before calling ToList(), here are the queries (value 71 manually added in place of #params)
-- MULTIPLE WHERE
SELECT [t0].[ID], [t0].[Name], [t0].[M2MID], [t0].[DevicePartNumberID], [t0].[Comments], [t0].[Hold]
FROM [dbo].[Diode] AS [t0]
WHERE (NOT (EXISTS(
SELECT NULL AS [EMPTY]
FROM (
SELECT DISTINCT [t2].[value]
FROM (
SELECT [t1].[DiodeID] AS [value]
FROM [dbo].[BladeDiode] AS [t1]
) AS [t2]
) AS [t3]
WHERE [t3].[value] = [t0].[ID]
))) AND (EXISTS(
SELECT NULL AS [EMPTY]
FROM (
SELECT DISTINCT [t6].[value]
FROM (
SELECT [t4].[DeviceID] AS [value], [t5].[DevicePartNumberID]
FROM [dbo].[Test] AS [t4]
INNER JOIN [dbo].[TestType] AS [t5] ON [t4].[TestTypeID] = ([t5].[ID])
) AS [t6]
WHERE [t6].[DevicePartNumberID] = (71)
) AS [t7]
WHERE [t7].[value] = [t0].[ID]
)) AND ([t0].[DevicePartNumberID] = 71)
ORDER BY [t0].[Name]
and
-- SINGLE WHERE
SELECT [t0].[ID], [t0].[Name], [t0].[M2MID], [t0].[DevicePartNumberID], [t0].[Comments], [t0].[Hold]
FROM [dbo].[Diode] AS [t0]
WHERE ([t0].[DevicePartNumberID] = 71) AND (EXISTS(
SELECT NULL AS [EMPTY]
FROM (
SELECT DISTINCT [t3].[value]
FROM (
SELECT [t1].[DeviceID] AS [value], [t2].[DevicePartNumberID]
FROM [dbo].[Test] AS [t1]
INNER JOIN [dbo].[TestType] AS [t2] ON [t1].[TestTypeID] = ([t2].[ID])
) AS [t3]
WHERE [t3].[DevicePartNumberID] = (71)
) AS [t4]
WHERE [t4].[value] = [t0].[ID]
)) AND (NOT (EXISTS(
SELECT NULL AS [EMPTY]
FROM (
SELECT DISTINCT [t6].[value]
FROM (
SELECT [t5].[DiodeID] AS [value]
FROM [dbo].[BladeDiode] AS [t5]
) AS [t6]
) AS [t7]
WHERE [t7].[value] = [t0].[ID]
)))
ORDER BY [t0].[Name]
The two SQL queries execute in < 1 second in SSMS and produce the same results.
So I'm wondering why the first is slower on the LINQ side. It's worrying to me because I know I've used multiple where elsewhere, without being aware of a such a severe performance impact.
This question even has answered with both multiple & and where. And this answer even suggests using multiple where clauses.
Can anyone explain why this happens in my case?

Because writing like this
if (someParam1 != 0)
{
myQuery = myQuery.Where(q => q.SomeField1 == someParam1)
}
if (someParam2 != 0)
{
myQuery = myQuery.Where(q => q.SomeField2 == someParam2)
}
is NOT(upd) the same as (in case when someParam1 and someParam2 != 0)
myQuery = from t in Table
where t.SomeField1 == someParam1
&& t.SomeField2 == someParam2
select t;
is (NOT deleted) the same as
myQuery = from t in Table
where t.SomeField1 == someParam1
where t.SomeField2 == someParam2
select t;
UPD
Yes, I do mistake. Second query is same, first is not same.
First and Second queries not EXACTLY the same. Let me show you what I mean.
1st query with lamda-expression writen as
t.Where(r => t.SomeField1 == someParam1 && t.SomeField2 == someParam2)
2nd query as
t.Where(r => r.SomeField1 == someParam1).Where(r => r.SomeField2 == someParam2)
In this case in generated SQL Predicate with SomeField2 goes first (it is important, see below)
In 1st case we getting this SQL:
SELECT <all field from Table>
FROM table t
WHERE t.SomeField1 = :someParam1
AND t.SomeField2 = :someParam2
In 2 case the SQL is:
SELECT <all field from Table>
FROM table t
WHERE t.SomeField2 = :someParam2
AND t.SomeField1 = :someParam1
As we see there are 2 'same' SQLs. As we see, the OP's SQLs are also 'same', they are different in order of predicates in WHERE clause (as in my example). And I guess that SQL optimizer generate 2 different execution plans and may be(!!!) doing NOT EXISTS, then EXISTS and then filtering take more time than do first filtering and after that do EXISTS and NOT EXISTS
UPD2
It is a 'problem' of Linq Provider (ORM). I'm using another ORM (linq2db), and it generates for me EXACTLY the same SQLs in both cases.

EntityFramework generating poor tsql (linqpad generates much better)

I have a database structure as below
Family(1) ----- (*) FamilyPersons -----(1)Person(1)------() Expenses (1) -----(0..1)GroceriesDetails
Let me explain that relation, Family can have one or more than one person , we have a mapping table FamilyPersons between Family and Persons. Now each person can enter his expenses which go into Expenses Table. Expense Table has a column ExpenseType (groceries, entertainemnet etc)
and details of each of these expenses goes into their own Tables, so we have a GroceriesDetails table (similarly we have other tables), so we have 1 to 0..1 relation between Expense and Groceries.
Now I am writing a query to get Complete GroceriesDetails for a family
GroceriesDetails.Where (g => g.Expenses.Person.FamilyPersons.Any(fp =>
fp.FamilyId == 1) && g.Expenses.ExpenseType == "GC" )
For this the sql generated by EF is
SELECT
[Extent1].[Id] AS [Id],
[Extent1].[Amount] AS [Amount]
FROM [dbo].[GroceriesDetails] AS [Extent1]
INNER JOIN (SELECT [Extent3].[Id] AS [Id1]
FROM [dbo].[Expenses] AS [Extent2]
INNER JOIN [dbo].[GroceriesDetails] AS [Extent3] ON [Extent2].[Id] = [Extent3].[Id]
WHERE N'GC' = [Extent2].[ExpenseType] ) AS [Filter1] ON [Extent1].[Id] = [Filter1].[Id1]
WHERE EXISTS (SELECT
1 AS [C1]
FROM [dbo].[Expenses] AS [Extent4]
INNER JOIN [dbo].[GroceriesDetails] AS [Extent5] ON [Extent4].[Id] = [Extent5].[Id]
INNER JOIN [dbo].[FamilyPerson] AS [Extent6] ON [Extent4].[PersonId] = [Extent6].[PersonId]
WHERE ([Extent1].[Id] = [Extent5].[Id]) AND (1 = [Extent6].[FamilyId])
)
In this query there is a full table join between Expenses and GroceriesDetails tables which is causing performance issues.
Whereas Linqpad generates a much better SQL
SELECT [t0].[Id], [t0].[Amount]
FROM [GroceriesDetails] AS [t0]
INNER JOIN [Expenses] AS [t1] ON [t1].[Id] = [t0].[Id]
WHERE (EXISTS(
SELECT NULL AS [EMPTY]
FROM [Expenses] AS [t2]
INNER JOIN [Person] AS [t3] ON [t3].[Id] = [t2].[PersonId]
CROSS JOIN [FamilyPerson] AS [t4]
WHERE ([t4].[FamilyId] = #p0) AND ([t2].[Id] = [t0].[Id]) AND ([t4].[PersonId] =
[t3].[Id])
)) AND ([t1].[ExpenseType] = #p1)
Please note that we are using WCF data services so this query is written against a WCF data service reference, so I can't traverse from top (family) to bottom (Groceries) as OData allows only one level of select.
Any help on optimizing this code is appreciated.

From the comments I learned that LinqPad uses Linq2SQL while the app uses EF, and that explains the difference.
The thing is that you have zero control on how EF generates SQL.
The only thing you can do is to rewrite your LINQ query to make it "closer" to desired SQL.
For example, instead of
GroceriesDetails.Where (g => g.Expenses.Person.FamilyPersons.Any(fp => fp.FamilyId == 1)
&& g.Expenses.ExpenseType == "GC" )
you can try to write something like (pseudocode):
from g in GrosseriesDetails
join e in Expenses on g.Id = e.GrosseryId
join p in Persons on p.Id = e.PersonId
join f in FamilyPersons on f.PersonId = p.Id
where f.FamilyId == 1 && e.ExpenseType == "GC"
It almost always helps as it tells an ORM a straightforward way to transform it into SQL. The idea is that the expression tree in the "original" case is more complex compare to the proposed scenario, and by simplifying the expression tree we make translator's job easier and more straightforward.
But besides manipulating the LINQ there is no control over how it generates SQL from the expression tree.

LinqToEntities produces incorrect SQL (Doubled Subquery)

I have a LinqToEntities query that double produces a subquery when creating the SQL. This causes the result set to come back with 0-3 results, every time the query is run. The subquery on its own produces a single random result (as it should). What is going on here?
The LINQ query:
from jpj in JobProviderJobs
where jpj.JobID == 4725
&& jpj.JobProviderID == (from jp2 in JobProviderJobs
where jp2.JobID == 4725
orderby Guid.NewGuid()
select jp2.JobProviderID).FirstOrDefault()
select new
{
JobProviderID = jpj.JobProviderID
}
Produce this SQL:
SELECT
[Filter2].[JobID] AS [JobID],
[Filter2].[JobProviderID1] AS [JobProviderID]
FROM (SELECT [Extent1].[JobID] AS [JobID], [Extent1].[JobProviderID] AS [JobProviderID1], [Limit1].[JobProviderID] AS [JobProviderID2]
FROM [dbo].[JobProviderJob] AS [Extent1]
LEFT OUTER JOIN (SELECT TOP (1) [Project1].[JobProviderID] AS [JobProviderID]
FROM ( SELECT
NEWID() AS [C1],
[Extent2].[JobProviderID] AS [JobProviderID]
FROM [dbo].[JobProviderJob] AS [Extent2]
WHERE 4725 = [Extent2].[JobID]
) AS [Project1]
ORDER BY [Project1].[C1] ASC ) AS [Limit1] ON 1 = 1
WHERE 4725 = [Extent1].[JobID] ) AS [Filter2]
LEFT OUTER JOIN (SELECT TOP (1) [Project2].[JobProviderID] AS [JobProviderID]
FROM ( SELECT
NEWID() AS [C1],
[Extent3].[JobProviderID] AS [JobProviderID]
FROM [dbo].[JobProviderJob] AS [Extent3]
WHERE 4725 = [Extent3].[JobID]
) AS [Project2]
ORDER BY [Project2].[C1] ASC ) AS [Limit2] ON 1 = 1
WHERE [Filter2].[JobProviderID1] = (CASE WHEN ([Filter2].[JobProviderID2] IS NULL) THEN 0 ELSE [Limit2].[JobProviderID] END)
EDIT:
So changing the subquery to this works, but I have no idea why
(from jp2 in JobProviderJobs
where jp2.JobID == 4725
orderby Guid.NewGuid()
select jp2).FirstOrDefault().JobProviderID

It's doing this due to the expected behavior of FirstOrDefault(). Calling FirstOrDefault() on an empty set of JobProviderJobs would produce a null value, but calling it on an empty set of ints would produce a 0. Recognizing this, LINQ to Entities tries to invoke a case statement at the end of the query to ensure that if there are no matching JobProviderJobs, the result of the select will be 0 instead of null.
In most cases, recreating the inner projection would cause no problems, but your use of NewGuid() obviously throws this logic off.
You found one solution. Another one would be to cast the result of the inner expression thusly:
from jpj in JobProviderJobs
where jpj.JobID == 4725
&& jpj.JobProviderID == (from jp2 in JobProviderJobs
where jp2.JobID == 4725
orderby Guid.NewGuid()
select (int?) jp2.JobProviderID).FirstOrDefault()
select new
{
JobProviderID = jpj.JobProviderID
}
A more correct implementation would reuse the initial SQL projection rather than creating two equivalent projections. It's likely that parser simply isn't complex enough to handle this properly, and the developers never saw any need to fix it because SQL Server should be able to optimize the identical projections away as long as they are deterministic.
You should probably log a bug report about this if one doesn't already exist.

Linq's static function 'Contain' is messing up with its content on a nested query?

I'm trying to reduce the number of database queries and in doing so I'm attempting to retrieve a series of hierarchical entities (that were previously fetched recursively) in one go.
I've got a Labels property holding a set of strings:
public string[] Labels { get; set; } // new string[] {"{{a}}", "{{b}}", "{{c}}", "{{d}}", "{{e}}"};
that I'm using to construct a first query:
var IDeferredTopLabels=
db.labels
.Where(l =>
l.site_id == this.site_id &&
this.Labels.Contains(l.name_for_code)
)
.Select(l => new LabelWithParentId { Label = l, ParentId = null });
The above, if inspected while debugging, would generate the following TSQL
SELECT [t0].[id], [t0].[name_for_code], [t0].[site_id], [t0].[priority_level]
FROM [dbo].[labels] AS [t0]
WHERE ([t0].[site_id] = 15) AND ([t0].[name_for_code] IN ('{{a}}', '{{b}}', '{{c}}', '{{d}}', '{{e}}'))
So far so good, following the IDeferredToplabels declaration I'm extending the first query with a Union specifying how child Label entities should be fetched:
var IDeferredWithChildLabels = IDeferredTopLabels
.Union(
db.label__labels
.Where(ll =>
db.labels
.Where(l =>
l.site_id == this.site_id &&
this.Labels.Contains(l.name_for_code)
)
.Select(l => l.id)
.Contains(ll.parent_label_id)
)
.Select(ll => new LabelWithParentId { Label = ll.label1, ParentId = ll.parent_label_id})
);
Now if I inspect the TSQL generate is:
SELECT [t4].[id], [t4].[name_for_code], [t4].[site_id], [t4].[priority_level], [t4].[value] AS [ParentId]
FROM (
SELECT [t0].[id], [t0].[name_for_code], [t0].[site_id], [t0].[priority_level], NULL AS [value]
FROM [dbo].[labels] AS [t0]
WHERE ([t0].[site_id] = 15) AND ([t0].[name_for_code] IN ('{{a}}', '{{b}}', '{{c}}', '{{d}}', '{{e}}'))
UNION
SELECT [t2].[id], [t2].[name_for_code], [t2].[site_id], [t2].[priority_level], [t1].[parent_label_id] AS [value]
FROM [dbo].[label__label] AS [t1]
INNER JOIN [dbo].[labels] AS [t2] ON [t2].[id] = [t1].[label_id]
WHERE EXISTS(
SELECT NULL AS [EMPTY]
FROM [dbo].[labels] AS [t3]
WHERE ([t3].[id] = [t1].[parent_label_id]) AND ([t3].[site_id] = 15) AND ([t3].[name_for_code] IN ('{{a}}', '{{b}}', '{{c}}', '{{a}}'0, '{{a}}'1))
)
) AS [t4]
If you see at the end of this second TSQL Query, the part IN ('{{a}}', '{{b}}', '{{c}}', '{{a}}'0, '{{a}}'1)), the set of strings have some unespected 0 and 1 next to the second last and last element plus it is no longer the initial set of strings (ie '{{a}}', '{{b}}', '{{c}}', '{{d}}', '{{e}}') but instead '{{a}}', '{{b}}', '{{c}}', '{{a}}', '{{a}}'.
What I'm I doing wrong!?
I'm clueless, I appreciate your help, thank you.
EDIT:
I just tried to use an int[] and compare the id rather then the name_for_code and I'm still getting 1 and 0 appended to the last 2 elements of the array as well as the array elements are wrong:
SELECT [t4].[id], [t4].[name_for_code], [t4].[site_id], [t4].[priority_level], [t4].[value] AS [ParentId]
FROM (
SELECT [t0].[id], [t0].[name_for_code], [t0].[site_id], [t0].[priority_level], NULL AS [value]
FROM [dbo].[labels] AS [t0]
WHERE ([t0].[site_id] = 15) AND ([t0].[id] IN (1, 2, 3, 4, 5))
UNION
SELECT [t2].[id], [t2].[name_for_code], [t2].[site_id], [t2].[priority_level], [t1].[parent_label_id] AS [value]
FROM [dbo].[label__label] AS [t1]
INNER JOIN [dbo].[labels] AS [t2] ON [t2].[id] = [t1].[label_id]
WHERE EXISTS(
SELECT NULL AS [EMPTY]
FROM [dbo].[labels] AS [t3]
WHERE ([t3].[id] = [t1].[parent_label_id]) AND ([t3].[site_id] = 15) AND ([t3].[id] IN (1, 2, 3, 10, 11))
)
) AS [t4]

OK I found the problem and actually it was not happening when running the code but only in debug mode and specifically it seems it is a bug in Scott Gu's LINQ to SQL Debug Visualizer
That is why the TSQL that I was grabbing was not parameterized, #David B you made a very good point!
I will leave a comment on Scott Gu's page pointing to this question.
Thanks all!

Try making a copy of Labels and using that in the second part of the query (the Union). EG:
string[] LabelsCopy = new string[5];
Labels.CopyTo(LabelsCopy, 0);
Then set a breakpoint after the query and inspect the contents of each array (or print out the contents of both arrays with Console.WriteLine(String.Join(',',Labels))).
It's extremely unlikely that .Contains() is changing the contents of Labels, but if it is then you'll have proof.

whats wrong in this LINQ synatx?

I am trying to convert a SQL query to LINQ. Somehow my count(distinct(x)) logic does not seem to be working correctly. The original SQL is quite efficient(or so i think), but the generated SQL is not even returning the correct result.
I am trying to fix this LINQ to do what the original SQL is doing, AND in an efficient way as the original query is doing. Help here would be really apreciated as I am stuck here :(
SQL which is working and I need to make a comparable LINQ of:
SELECT [t1].[PersonID] AS [personid]
FROM [dbo].[Code] AS [t0]
INNER JOIN [dbo].[phonenumbers] AS [t1] ON [t1].[PhoneCode] = [t0].[Code]
INNER JOIN [dbo].[person] ON [t1].[PersonID]= [dbo].[Person].PersonID
WHERE ([t0].[codetype] = 'phone') AND (
([t0].[CodeDescription] = 'Home') AND ([t1].[PhoneNum] = '111')
OR
([t0].[CodeDescription] = 'Work') AND ([t1].[PhoneNum] = '222') )
GROUP BY [t1].[PersonID] HAVING COUNT(DISTINCT([t1].[PhoneNum]))=2
The LINQ which I made is approximately as below:
var ids = context.Code.Where(predicate);
var rs = from r in ids
group r by new { r.phonenumbers.person.PersonID} into g
let matchcount=g.Select(p => p.phonenumbers.PhoneNum).Distinct().Count()
where matchcount ==2
select new
{
personid = g.Key
};
Unfortunately, the above LINQ is NOT generating the correct result, and is actually internally getting generated to the SQL shown below. By the way, this generated query is also reading ALL the rows(about 19592040) around 2 times due to the COUNTS :( Wich is a big performance issue too. Please help/point me to the right direction.
Declare #p0 VarChar(10)='phone'
Declare #p1 VarChar(10)='Home'
Declare #p2 VarChar(10)='111'
Declare #p3 VarChar(10)='Work'
Declare #p4 VarChar(10)='222'
Declare #p5 VarChar(10)='2'
SELECT [t9].[PersonID], (
SELECT COUNT(*)
FROM (
SELECT DISTINCT [t13].[PhoneNum]
FROM [dbo].[Code] AS [t10]
INNER JOIN [dbo].[phonenumbers] AS [t11] ON [t11].[PhoneType] = [t10].[Code]
INNER JOIN [dbo].[Person] AS [t12] ON [t12].[PersonID] = [t11].[PersonID]
INNER JOIN [dbo].[phonenumbers] AS [t13] ON [t13].[PhoneType] = [t10].[Code]
WHERE ([t9].[PersonID] = [t12].[PersonID]) AND ([t10].[codetype] = #p0) AND ((([t10].[codetype] = #p1) AND ([t11].[PhoneNum] = #p2)) OR (([t10].[codetype] = #p3) AND ([t11].[PhoneNum] = #p4)))
) AS [t14]
) AS [cnt]
FROM (
SELECT [t3].[PersonID], (
SELECT COUNT(*)
FROM (
SELECT DISTINCT [t7].[PhoneNum]
FROM [dbo].[Code] AS [t4]
INNER JOIN [dbo].[phonenumbers] AS [t5] ON [t5].[PhoneType] = [t4].[Code]
INNER JOIN [dbo].[Person] AS [t6] ON [t6].[PersonID] = [t5].[PersonID]
INNER JOIN [dbo].[phonenumbers] AS [t7] ON [t7].[PhoneType] = [t4].[Code]
WHERE ([t3].[PersonID] = [t6].[PersonID]) AND ([t4].[codetype] = #p0) AND ((([t4].[codetype] = #p1) AND ([t5].[PhoneNum] = #p2)) OR (([t4].[codetype] = #p3) AND ([t5].[PhoneNum] = #p4)))
) AS [t8]
) AS [value]
FROM (
SELECT [t2].[PersonID]
FROM [dbo].[Code] AS [t0]
INNER JOIN [dbo].[phonenumbers] AS [t1] ON [t1].[PhoneType] = [t0].[Code]
INNER JOIN [dbo].[Person] AS [t2] ON [t2].[PersonID] = [t1].[PersonID]
WHERE ([t0].[codetype] = #p0) AND ((([t0].[codetype] = #p1) AND ([t1].[PhoneNum] = #p2)) OR (([t0].[codetype] = #p3) AND ([t1].[PhoneNum] = #p4)))
GROUP BY [t2].[PersonID]
) AS [t3]
) AS [t9]
WHERE [t9].[value] = #p5
Thanks!

I think the issue might be new { r.phonenumbers.person.PersonID}.
Why are you newing up a new object here rather than just grouping by r.phonenumbers.person directly? new {} is going to be a different object every time which will never group.
After grouping by person I would Select a group => new {person = group.person, phoneNumbers = group.person.phonenumbers} and then perform the check for how many phone numbers they have and then any final projection.

Drats! It looks like the fault was on my side(GIGO principle!)
In my ORM, I had created the associations from right to left, instead of the other way round. I think that was the issue.
Only issue left now is that somehow the LINQ is generating one INNER JOIN twice, and that is keeping the final result to be retrieved correctly. If i comment it out i nthe generated sql, i am getting correct result. That's the only issue now and I guess i'll open a new question for that. Thanks for your time!.