We Keep Coding

What is the difference between these LINQ queries

I've been fooling around with some LINQ over Entities and I'm getting strange results and I would like to get an explanation...
Given the following LINQ query,
// Sample # 1
IEnumerable<GroupInformation> groupingInfo;
groupingInfo = from a in context.AccountingTransaction
group a by a.Type into grp
select new GroupInformation()
{
GroupName = grp.Key,
GroupCount = grp.Count()
};
I get the following SQL query (taken from SQL Profiler):
SELECT
1 AS [C1],
[GroupBy1].[K1] AS [Type],
[GroupBy1].[A1] AS [C2]
FROM ( SELECT
[Extent1].[Type] AS [K1],
COUNT(1) AS [A1]
FROM [dbo].[AccountingTransaction] AS [Extent1]
GROUP BY [Extent1].[Type]
) AS [GroupBy1]
So far so good.
If I change my LINQ query to:
// Sample # 2
groupingInfo = context.AccountingTransaction.
GroupBy(a => a.Type).
Select(grp => new GroupInformation()
{
GroupName = grp.Key,
GroupCount = grp.Count()
});
it yields to the exact same SQL query. Makes sense to me.
Here comes the interesting part... If I change my LINQ query to:
// Sample # 3
IEnumerable<AccountingTransaction> accounts;
IEnumerable<IGrouping<object, AccountingTransaction>> groups;
IEnumerable<GroupInformation> groupingInfo;
accounts = context.AccountingTransaction;
groups = accounts.GroupBy(a => a.Type);
groupingInfo = groups.Select(grp => new GroupInformation()
{
GroupName = grp.Key,
GroupCount = grp.Count()
});
the following SQL is executed (I stripped a few of the fields from the actual query, but all the fields from the table (~ 15 fields) were included in the query, twice):
SELECT
[Project2].[C1] AS [C1],
[Project2].[Type] AS [Type],
[Project2].[C2] AS [C2],
[Project2].[Id] AS [Id],
[Project2].[TimeStamp] AS [TimeStamp],
-- <snip>
FROM ( SELECT
[Distinct1].[Type] AS [Type],
1 AS [C1],
[Extent2].[Id] AS [Id],
[Extent2].[TimeStamp] AS [TimeStamp],
-- <snip>
CASE WHEN ([Extent2].[Id] IS NULL) THEN CAST(NULL AS int) ELSE 1 END AS [C2]
FROM (SELECT DISTINCT
[Extent1].[Type] AS [Type]
FROM [dbo].[AccountingTransaction] AS [Extent1] ) AS [Distinct1]
LEFT OUTER JOIN [dbo].[AccountingTransaction] AS [Extent2] ON [Distinct1].[Type] = [Extent2].[Type]
) AS [Project2]
ORDER BY [Project2].[Type] ASC, [Project2].[C2] ASC
Why are the SQLs generated are so different? After all, the exact same code is executed, it's just that sample # 3 is using intermediate variables to get the same job done!
Also, if I do:
Console.WriteLine(groupingInfo.ToString());
for sample # 1 and sample # 2, I get the exact same query that was captured by SQL Profiler, but for sample # 3, I get:
System.Linq.Enumerable+WhereSelectEnumerableIterator`2[System.Linq.IGrouping`2[System.Object,TestLinq.AccountingTransaction],TestLinq.GroupInformation]
What is the difference? Why can't I get the SQL Query generated by LINQ if I split the LINQ query in multiple instructions?
The ulitmate goal is to be able to add operators to the query (Where, OrderBy, etc.) at run-time.
BTW, I've seen this behavior in EF 4.0 and EF 6.0.
Thank you for your help.

The reason is because in your third attempt you're referring to accounts as IEnumerable<AccountingTransaction> which will cause the query to be invoked using Linq-To-Objects (Enumerable.GroupBy and Enumerable.Select)
On the other hand, in your first and second attempts the reference to AccountingTransaction is preserved as IQueryable<AccountingTransaction> and the query will be executed using Linq-To-Entities which will then transform it to the appropriate SQL statement.

Optimising LINQ To Entities Without Returning Large Anonymous Type Set

I'm currently trying to optimise some of my LINQ to Entity queries (EF 6.1.1) so that the generated SQL is more efficient. I've managed to refactor my statements so that the SQL is pretty much how I would write it, however I am running into issue with the return objects from the query.
One relatively simple example would be where I originally I had lambda syntax with something like this:
var originalQuery = context.WorkstationConfigurations
.Include(config => config.WorkstationOptions)
.Include(config => config.WorkstationOptions.Select(option => option.ImageWorkstationOption))
.Include(config => config.WorkstationOptions.Select(option => option.ImageWorkstationOption).Select(iwo => iwo.Image))
.Include(config => config.WorkstationPowerBarTileAssignments);
List<WorkstationConfiguration> originalResult = originalQuery.ToList();
This correctly returned a list of 3 WorkstationConfiguration entites, with the object graph all correctly hooked up, but it produced pretty horrific SQL with multiple selects, unions, order by's, etc:
SELECT
[UnionAll1].[RowVersion] AS [C1],
[UnionAll1].[WorkstationConfigurationID] AS [C2],
[UnionAll1].[Name] AS [C3],
...
...
FROM (SELECT
CASE WHEN ([Join2].[Name1] IS NULL) THEN CAST(NULL AS int) ELSE 1 END AS [C1],
[Extent1].[RowVersion] AS [RowVersion],
[Extent1].[WorkstationConfigurationID] AS [WorkstationConfigurationID],
[Extent1].[Name] AS [Name],
[Extent1].[ConfigurationType] AS [ConfigurationType],
[Extent1].[RowVersion] AS [RowVersion1],
[Join2].[Name1] AS [Name1],
...
...
FROM [dbo].[WorkstationConfiguration] AS [Extent1]
LEFT OUTER JOIN (SELECT [Extent2].[Name] AS [Name1], [Extent2].[LineNumber] AS [LineNumber1], [Extent2].[WorkstationConfigurationID] AS [WorkstationConfigurationID1], [Extent2].[TypeCode] AS [TypeCode], [Extent2].[RowVersion] AS [RowVersion1], [Extent3].[Name] AS [Name2], [Extent3].[LineNumber] AS [LineNumber2], [Extent3].[WorkstationConfigurationID] AS [WorkstationConfigurationID2], [Extent3].[ImageID] AS [ImageID1], [Extent3].[RowVersion] AS [RowVersion2], [Extent4].[ImageID] AS [ImageID2], [Extent4].[ImageData] AS [ImageData], [Extent4].[SizeInBytes] AS [SizeInBytes], [Extent4].[WidthInPixels] AS [WidthInPixels], [Extent4].[HeightInPixels] AS [HeightInPixels], [Extent4].[ImageHash] AS [ImageHash], [Extent4].[Description] AS [Description], [Extent4].[ImageTypeID] AS [ImageTypeID], [Extent4].[ContentType] AS [ContentType], [Extent4].[ThumbnailImageData] AS [ThumbnailImageData], [Extent4].[RowVersion] AS [RowVersion3]
FROM [dbo].[WorkstationOption] AS [Extent2]
LEFT OUTER JOIN [dbo].[ImageWorkstationOption] AS [Extent3] ON ([Extent2].[WorkstationConfigurationID] = [Extent3].[WorkstationConfigurationID]) AND ([Extent2].[LineNumber] = [Extent3].[LineNumber]) AND ([Extent2].[Name] = [Extent3].[Name])
LEFT OUTER JOIN [dbo].[Image] AS [Extent4] ON [Extent3].[ImageID] = [Extent4].[ImageID] ) AS [Join2] ON [Extent1].[WorkstationConfigurationID] = [Join2].[WorkstationConfigurationID1]
UNION ALL
SELECT
2 AS [C1],
[Extent5].[RowVersion] AS [RowVersion],
[Extent5].[WorkstationConfigurationID] AS [WorkstationConfigurationID],
[Extent5].[Name] AS [Name],
[Extent5].[ConfigurationType] AS [ConfigurationType],
[Extent5].[RowVersion] AS [RowVersion1],
CAST(NULL AS int) AS [C2],
CAST(NULL AS int) AS [C3],
CAST(NULL AS int) AS [C4],
...
...
FROM [dbo].[WorkstationConfiguration] AS [Extent5]
INNER JOIN [dbo].[WorkstationPowerBarTileAssignment] AS [Extent6] ON [Extent5].[WorkstationConfigurationID] = [Extent6].[WorkstationConfigurationID]) AS [UnionAll1]
ORDER BY [UnionAll1].[WorkstationConfigurationID] ASC, [UnionAll1].[C1] ASC
So after a bit of googling I found that query syntax is better for multiple joins/complex queries, so I refactored to this:
var optimisedQuery = from workstationConfiguration in context.WorkstationConfigurations
from workstationOption in workstationConfiguration.WorkstationOptions
from tileAssignment in workstationConfiguration.WorkstationPowerBarTileAssignments.DefaultIfEmpty()
select new
{
a = workstationConfiguration,
b = workstationOption,
c = workstationOption.ImageWorkstationOption,
d = workstationOption.ImageWorkstationOption.Image,
e = tileAssignment
};
var optimisedResult = optimisedQuery.ToList();
This produced pretty much perfect SQL (single select, correct joins) like I would have written if I had hand rolled it:
SELECT
[Extent1].[RowVersion] AS [RowVersion],
[Extent1].[WorkstationConfigurationID] AS [WorkstationConfigurationID],
[Extent1].[Name] AS [Name],
[Extent1].[ConfigurationType] AS [ConfigurationType],
[Extent2].[Name] AS [Name1],
...
...
FROM [dbo].[WorkstationConfiguration] AS [Extent1]
INNER JOIN [dbo].[WorkstationOption] AS [Extent2] ON [Extent1].[WorkstationConfigurationID] = [Extent2].[WorkstationConfigurationID]
LEFT OUTER JOIN [dbo].[WorkstationPowerBarTileAssignment] AS [Extent3] ON [Extent1].[WorkstationConfigurationID] = [Extent3].[WorkstationConfigurationID]
LEFT OUTER JOIN [dbo].[ImageWorkstationOption] AS [Extent4] ON ([Extent2].[Name] = [Extent4].[Name]) AND ([Extent2].[LineNumber] = [Extent4].[LineNumber]) AND ([Extent2].[WorkstationConfigurationID] = [Extent4].[WorkstationConfigurationID])
LEFT OUTER JOIN [dbo].[Image] AS [Extent5] ON [Extent4].[ImageID] = [Extent5].[ImageID]
Entity framework takes care of populating the object graph correctly (all objects and nav properties hooked up correctly), but I end up with one anonymous type entry for every row returned from the database (316), rather than a set of 3 distinct WorkstationConfiguration objects. So if I manually pull out the unique WorkstationConfiguration objects ('a') from the result set then I have the same end result, but with optimised SQL.
Is there a way I can get the optimised SQL, with the child entities all populated/object graph all automatically hooked up, without getting the superfluous set of anonymous types (i.e. get a List<WorkstationConfiguration> directly returned from my optimised LINQ statement)?

EntityFramework generating poor tsql (linqpad generates much better)

I have a database structure as below
Family(1) ----- (*) FamilyPersons -----(1)Person(1)------() Expenses (1) -----(0..1)GroceriesDetails
Let me explain that relation, Family can have one or more than one person , we have a mapping table FamilyPersons between Family and Persons. Now each person can enter his expenses which go into Expenses Table. Expense Table has a column ExpenseType (groceries, entertainemnet etc)
and details of each of these expenses goes into their own Tables, so we have a GroceriesDetails table (similarly we have other tables), so we have 1 to 0..1 relation between Expense and Groceries.
Now I am writing a query to get Complete GroceriesDetails for a family
GroceriesDetails.Where (g => g.Expenses.Person.FamilyPersons.Any(fp =>
fp.FamilyId == 1) && g.Expenses.ExpenseType == "GC" )
For this the sql generated by EF is
SELECT
[Extent1].[Id] AS [Id],
[Extent1].[Amount] AS [Amount]
FROM [dbo].[GroceriesDetails] AS [Extent1]
INNER JOIN (SELECT [Extent3].[Id] AS [Id1]
FROM [dbo].[Expenses] AS [Extent2]
INNER JOIN [dbo].[GroceriesDetails] AS [Extent3] ON [Extent2].[Id] = [Extent3].[Id]
WHERE N'GC' = [Extent2].[ExpenseType] ) AS [Filter1] ON [Extent1].[Id] = [Filter1].[Id1]
WHERE EXISTS (SELECT
1 AS [C1]
FROM [dbo].[Expenses] AS [Extent4]
INNER JOIN [dbo].[GroceriesDetails] AS [Extent5] ON [Extent4].[Id] = [Extent5].[Id]
INNER JOIN [dbo].[FamilyPerson] AS [Extent6] ON [Extent4].[PersonId] = [Extent6].[PersonId]
WHERE ([Extent1].[Id] = [Extent5].[Id]) AND (1 = [Extent6].[FamilyId])
)
In this query there is a full table join between Expenses and GroceriesDetails tables which is causing performance issues.
Whereas Linqpad generates a much better SQL
SELECT [t0].[Id], [t0].[Amount]
FROM [GroceriesDetails] AS [t0]
INNER JOIN [Expenses] AS [t1] ON [t1].[Id] = [t0].[Id]
WHERE (EXISTS(
SELECT NULL AS [EMPTY]
FROM [Expenses] AS [t2]
INNER JOIN [Person] AS [t3] ON [t3].[Id] = [t2].[PersonId]
CROSS JOIN [FamilyPerson] AS [t4]
WHERE ([t4].[FamilyId] = #p0) AND ([t2].[Id] = [t0].[Id]) AND ([t4].[PersonId] =
[t3].[Id])
)) AND ([t1].[ExpenseType] = #p1)
Please note that we are using WCF data services so this query is written against a WCF data service reference, so I can't traverse from top (family) to bottom (Groceries) as OData allows only one level of select.
Any help on optimizing this code is appreciated.

From the comments I learned that LinqPad uses Linq2SQL while the app uses EF, and that explains the difference.
The thing is that you have zero control on how EF generates SQL.
The only thing you can do is to rewrite your LINQ query to make it "closer" to desired SQL.
For example, instead of
GroceriesDetails.Where (g => g.Expenses.Person.FamilyPersons.Any(fp => fp.FamilyId == 1)
&& g.Expenses.ExpenseType == "GC" )
you can try to write something like (pseudocode):
from g in GrosseriesDetails
join e in Expenses on g.Id = e.GrosseryId
join p in Persons on p.Id = e.PersonId
join f in FamilyPersons on f.PersonId = p.Id
where f.FamilyId == 1 && e.ExpenseType == "GC"
It almost always helps as it tells an ORM a straightforward way to transform it into SQL. The idea is that the expression tree in the "original" case is more complex compare to the proposed scenario, and by simplifying the expression tree we make translator's job easier and more straightforward.
But besides manipulating the LINQ there is no control over how it generates SQL from the expression tree.

Entity Framework performing different on multiple computers

I am at a loss, So I came here to see what everyone thinks about this.
Background: Textbox using jquery autocomplete w/ EF 5
Problem: Entity is performing a inner join, but the syntax shows a left join is being performing. (Both PC's are running Win 7 and latest framework)
Anomaly: It works the right way (left outer join) on my PC. But on my buddys PC, it shows an inner join. Both files are binary equals. (Actually being pulled from git)
Here is the code:
public JsonResult AutoCompleteName(string term)
{
using (var db = new PersonnelContext())
{
return this.Json((from r in db.Personnel
join per in db.PersonnelEmployee on r.Id equals per.Personnel_Id
join dep in db.RefDepartment on per.Department equals dep.Department into rfdp
from g in rfdp.DefaultIfEmpty()
where r.First_Name.ToLower().Contains(term.ToLower()) | r.Last_Name.ToLower().Contains(term.ToLower()) | (r.First_Name.ToLower() + " " + r.Last_Name.ToLower()).Contains(term.ToLower())
select new { firstname = r.First_Name, lastname = r.Last_Name, department = g.DeptDesc ?? "None", per.Personnel_Id, per.Pernr }).OrderBy(a => a.firstname).ToArray(), JsonRequestBehavior.AllowGet);
}
}
Here is the intellitrace from the PC where its working fine: (Note I took out some due to confidentiality)
FROM ( SELECT
[Extent1].[Id] AS [Id],
[Extent1].[First_Name] AS [First_Name],
[Extent1].[Last_Name] AS [Last_Name],
[Extent2].[Personnel_Id] AS [Personnel_Id],
[Extent2].[Pernr] AS [Pernr],
CASE WHEN ([Extent3].[Dept_Desc] IS NULL) THEN N'None' ELSE [Extent3].[Dept_Desc] END AS [C1]
FROM [Core].[Personnel] AS [Extent1]
INNER JOIN [Core].[Personnel_Employee] AS [Extent2] ON [Extent1].[Id] = [Extent2].[Personnel_Id]
LEFT OUTER JOIN [Core].[Ref_Department] AS [Extent3] ON [Extent2].[Department] = [Extent3].[Department]
WHERE (( CAST(CHARINDEX(LOWER(#p__linq__0), LOWER([Extent1].[First_Name])) AS int)) > 0) OR (( CAST(CHARINDEX(LOWER(#p__linq__1), LOWER([Extent1].[Last_Name])) AS int)) > 0) OR (( CAST(CHARINDEX(LOWER(#p__linq__2), LOWER([Extent1].[First_Name]) + N' ' + LOWER([Extent1].[Last_Name])) AS int)) > 0)
) AS [Project1]
ORDER BY [Project1].[First_Name] ASC"
As you can see, its performing a Left Outer Join, just as the syntax suggests.
Here is the Intellitrace from PC 2: (Exact same code!)
FROM ( SELECT
[Extent1].[Id] AS [Id],
[Extent1].[First_Name] AS [First_Name],
[Extent1].[Last_Name] AS [Last_Name],
[Extent2].[Personnel_Id] AS [Personnel_Id],
[Extent2].[Pernr] AS [Pernr],
CASE WHEN ([Extent3].[Dept_Desc] IS NULL) THEN N'None' ELSE [Extent3].[Dept_Desc] END AS [C1]
FROM [Core].[Personnel] AS [Extent1]
INNER JOIN [Core].[Personnel_Employee] AS [Extent2] ON [Extent1].[Id] = [Extent2].[Personnel_Id]
INNER JOIN [Core].[Ref_Department] AS [Extent3] ON [Extent2].[Department] = [Extent3].[Department]
WHERE (( CAST(CHARINDEX(LOWER(#p__linq__0), LOWER([Extent1].[First_Name])) AS int)) > 0) OR (( CAST(CHARINDEX(LOWER(#p__linq__1), LOWER([Extent1].[Last_Name])) AS int)) > 0) OR (( CAST(CHARINDEX(LOWER(#p__linq__2), LOWER([Extent1].[First_Name]) + N' ' + LOWER([Extent1].[Last_Name])) AS int)) > 0)
) AS [Project1]
ORDER BY [Project1].[First_Name] ASC"
Here is another strange anomaly:
When deployed to QA, it works fine on both PC's!! There is no consistency here for us to know whats going on.
Any ideas here? I have no idea what going on. We have completely wiped away our working versions, and pulled down again from git. Same thing happens. If no one has any ideas, we may need to use a proc. But I would at least like to know whats going on.
** Edit ** Answer:
I really have no idea why this works, but i got it working. Here is the query now:
return this.Json((from r in db.Personnel
join per in db.PersonnelEmployee on r.Id equals per.Personnel_Id
where r.First_Name.ToLower().Contains(term.ToLower()) | r.Last_Name.ToLower().Contains(term.ToLower()) | (r.First_Name.ToLower() + " " + r.Last_Name.ToLower()).Contains(term.ToLower())
join dep in db.RefDepartment
.Where(x => x.DeptDesc != null || x.DeptDesc == null) on per.Department equals dep.Department into rfdp
from g in rfdp.DefaultIfEmpty()
select new {
firstname = r.First_Name,
lastname = r.Last_Name,
department = (g == null) ? "None" : g.DeptDesc,
per.Personnel_Id,
per.Pernr }).ToArray(), JsonRequestBehavior.AllowGet);
I think this one works because it explicitly looks for DeptDesc being null and not null.... I am just glad its working. Thanks for everyone who looked at this.

I was able to get this fixed with the following modified query:
return this.Json((from r in db.Personnel
join per in db.PersonnelEmployee on r.Id equals per.Personnel_Id
where r.First_Name.ToLower().Contains(term.ToLower()) | r.Last_Name.ToLower().Contains(term.ToLower()) | (r.First_Name.ToLower() + " " + r.Last_Name.ToLower()).Contains(term.ToLower())
join dep in db.RefDepartment
.Where(x => x.DeptDesc != null || x.DeptDesc == null) on per.Department equals dep.Department into rfdp
from g in rfdp.DefaultIfEmpty()
select new {
firstname = r.First_Name,
lastname = r.Last_Name,
department = (g == null) ? "None" : g.DeptDesc,
per.Personnel_Id,
per.Pernr }).ToArray(), JsonRequestBehavior.AllowGet);
As I said in the edits: I'm not 100% sure why this works instead of the first, but im glad it does!

This sounds suspiciously like an already 'fixed' issue with EF... have a look at this duplicated join issue
They note that removing the orderby removes the duplicate... worth a test eh

How to convert following SQL Query into a LINQ to SQL query

I have following SQL Query and would like to convert to LINQ to SQL which I will use in entity framework 5.0
var internationalDesksList =
from internationalDesks in _context.InternationalDesks
from subsection in
_context.Subsections.Where(
s =>
internationalDesks.EBALocationId == s.LocationId ||
internationalDesks.FELocationId == s.LocationId).DefaultIfEmpty()
where subsection.PublicationId == 1
select new {internationalDesks.Id, subsection.LocationId};
I have referred the following posts and answers. Though no luck.
LINQ to SQL - Left Outer Join with multiple join conditions Linq
left join on multiple (OR) conditions
When I tried this query in LINQPad I got the following answer which is correct.
-- Region Parameters
DECLARE #p0 Int = 1
-- EndRegion
SELECT [t0].[Id], [t1].[Id] AS [Id1]
FROM [InternationalDesks] AS [t0]
LEFT OUTER JOIN [Subsection] AS [t1] ON (([t0].[FELocationId]) = [t1].[LocationId]) OR (([t0].[EBALocationId]) = [t1].[LocationId])
WHERE [t1].[PublicationId] = #p0
However in entity framework 5 ( DBContext ) it is not providing me with the correct query. When I checked in SQL profiler all columns in subsection table is selected. That's it.
Following is the result:
SELECT
[Extent1].[Id] AS [Id],
[Extent1].[Description] AS [Description],
[Extent1].[PracticeAreaId] AS [PracticeAreaId],
[Extent1].[LocationId] AS [LocationId],
...
FROM [dbo].[Subsection] AS [Extent1]
Don't know what could be problem. Please help me.

With LINQ you can't do LEFT OUTER JOIN on some boolean expression, only equijoins are supported. So, you can generate CROSS JOIN this way:
var internationalDesksList =
from internationalDesks in _context.InternationalDesks
from subsection in _context.Subsections
where subsection.PublicationId == 1 &&
(internationalDesks.EBALocationId == subsection.LocationId ||
internationalDesks.FELocationId == subsection.LocationId)
select new {
internationalDesks.Id,
subsection.LocationId
};
EF 5 will generate following SQL:
SELECT
[Extent1].[Id] AS [Id],
[Extent2].[LocationId] AS [LocationId]
FROM [dbo].[InternationalDesks] AS [Extent1]
CROSS JOIN [dbo].[Subsections] AS [Extent2]
WHERE (1 = [Extent2].[PublicationId]) AND
([Extent1].[EBALocationId] = [Extent2].[LocationId] OR
[Extent1].[FELocationId] = [Extent2].[LocationId])
As you can see, only required columns are selected. I also checked this query in LINQ to SQL - following query is generated:
DECLARE #p0 Int = 1
SELECT [t0].[Id], [t1].[LocationId]
FROM [InternationalDesks] AS [t0], [Subsections] AS [t1]
WHERE ([t1].[PublicationId] = #p0) AND
(([t0].[EBALocationId] = [t1].[LocationId]) OR
([t0].[FELocationId] = [t1].[LocationId]))