We Keep Coding

Linq Left Outer Join with Count

I want to create this SQL query:
SELECT
a.[Seat],
b.[PlayerId],
b.[UserName],
b.[NickName],
COUNT(c.PlayerId) AS Trophy
FROM [dbo].[tbl_PlayerTableSeat] AS a
INNER JOIN [dbo].[tbl_Player] AS b ON a.[PlayerId] = b.[PlayerId]
INNER JOIN [dbo].[tbl_GameVirtualTable] AS d ON d.GameVirtualTableId = a.GameVirtualTableId
LEFT OUTER JOIN [dbo].[tbl_PlayerTableWinning] AS c ON a.[PlayerId] = c.[PlayerId] AND c.GameTableId = d.GameTableId
WHERE a.GameVirtualTableId = 36
GROUP BY a.[Seat], b.[PlayerId], b.[UserName], b.[NickName]
I have this Linq
var virtualTableSeatList = (from s in db.PlayerTableSeat
join p in db.Player on s.PlayerId equals p.PlayerId
join v in db.GameVirtualTable on s.GameVirtualTableId equals v.GameVirtualTableId
join w in db.PlayerTableWinning on new { X1 = s.PlayerId, X2 = v.GameTableId } equals new { X1 = w.PlayerId, X2 = w.GameTableId } into gj
from g in gj.DefaultIfEmpty()
where s.GameVirtualTableId == virtualGameTableId
group new { p, s } by new { p.PlayerId, s.Seat, p.NickName, p.UserName } into grp
select new VirtualTableSeatDto
{
PlayerId = grp.Key.PlayerId,
Seat = grp.Key.Seat,
NickName = grp.Key.NickName,
UserName = grp.Key.UserName,
Trophy = grp.Count()
}
).ToList();
From SQL Profiler, the Linq generates this SQL query:
exec sp_executesql N'SELECT
[GroupBy1].[K2] AS [PlayerId],
CAST( [GroupBy1].[K1] AS int) AS [C1],
[GroupBy1].[K4] AS [NickName],
[GroupBy1].[K3] AS [UserName],
[GroupBy1].[A1] AS [C2]
FROM ( SELECT
[Extent1].[Seat] AS [K1],
[Extent2].[PlayerId] AS [K2],
[Extent2].[UserName] AS [K3],
[Extent2].[NickName] AS [K4],
COUNT(1) AS [A1]
FROM [dbo].[tbl_PlayerTableSeat] AS [Extent1]
INNER JOIN [dbo].[tbl_Player] AS [Extent2] ON [Extent1].[PlayerId] = [Extent2].[PlayerId]
INNER JOIN [dbo].[tbl_GameVirtualTable] AS [Extent3] ON [Extent1].[GameVirtualTableId] = [Extent3].[GameVirtualTableId]
LEFT OUTER JOIN [dbo].[tbl_PlayerTableWinning] AS [Extent4] ON ([Extent1].[PlayerId] = [Extent4].[PlayerId]) AND ([Extent3].[GameTableId] = [Extent4].[GameTableId])
WHERE [Extent1].[GameVirtualTableId] = #p__linq__0
GROUP BY [Extent1].[Seat], [Extent2].[PlayerId], [Extent2].[UserName], [Extent2].[NickName]
) AS [GroupBy1]',N'#p__linq__0 int',#p__linq__0=36
I want to change COUNT(1) AS [A1] to COUNT([Extent4].[PlayerId]) AS [A1]
so it can return correct data.
I have no idea how to change the LinQ
Trophy = grp.Count()
so that it can count PlayerId of PlayerTableWinning instead of COUNT(1)
Updated: #Ivan Stoev
By adding the g into the group.
group new { p, s, g }
And sum the group
Trophy = grp.Sum(item => item.w != null ? 1 : 0)
It return the correct answer. However, it is using SUM instead of count. The SQL query generated is as below:
exec sp_executesql N'SELECT
[GroupBy1].[K2] AS [PlayerId],
CAST( [GroupBy1].[K1] AS int) AS [C1],
[GroupBy1].[K4] AS [NickName],
[GroupBy1].[K3] AS [UserName],
[GroupBy1].[A1] AS [C2]
FROM ( SELECT
[Filter1].[K1] AS [K1],
[Filter1].[K2] AS [K2],
[Filter1].[K3] AS [K3],
[Filter1].[K4] AS [K4],
SUM([Filter1].[A1]) AS [A1]
FROM ( SELECT
[Extent1].[Seat] AS [K1],
[Extent2].[PlayerId] AS [K2],
[Extent2].[UserName] AS [K3],
[Extent2].[NickName] AS [K4],
CASE WHEN ( NOT (([Extent4].[GameTableId] IS NULL) AND ([Extent4].[PlayerId] IS NULL) AND ([Extent4].[GameRoundId] IS NULL))) THEN 1 ELSE 0 END AS [A1]
FROM [dbo].[tbl_PlayerTableSeat] AS [Extent1]
INNER JOIN [dbo].[tbl_Player] AS [Extent2] ON [Extent1].[PlayerId] = [Extent2].[PlayerId]
INNER JOIN [dbo].[tbl_GameVirtualTable] AS [Extent3] ON [Extent1].[GameVirtualTableId] = [Extent3].[GameVirtualTableId]
LEFT OUTER JOIN [dbo].[tbl_PlayerTableWinning] AS [Extent4] ON ([Extent1].[PlayerId] = [Extent4].[PlayerId]) AND ([Extent3].[GameTableId] = [Extent4].[GameTableId])
WHERE [Extent1].[GameVirtualTableId] = #p__linq__0
) AS [Filter1]
GROUP BY [K1], [K2], [K3], [K4]
) AS [GroupBy1]',N'#p__linq__0 int',#p__linq__0=36

The only (but significant) difference between SQL COUNT(field) and COUNT(1) is that the former is excluding the NULL values, which when applied to the normally required field from the right side of a left outer join like in your case produces a different result when there are no matching records - the former returns 0 while the latter returns 1.
The "natural" LINQ equivalent would be Count(field != null), but that unfortunately is translated to a quite different SQL by the current EF query provider. So in such cases I personally use the closer equivalent expression Sum(field != null ? 1 : 0) which produces a much better SQL.
In order to apply the above to your query, you'll need an access to w inside the grouping, so change
group new { p, s }
to
group new { p, s, w }
and then use
Trophy = grp.Sum(item => item.w != null ? 1 : 0)

Optimize linq query with count()

I have the following query that works fine
var myList = (from p in db.full
group p by p.object into g
orderby g.Count() descending
select new StringIntType
{
str = g.Key,
nbr = g.Count()
}).Take(50).ToList();
The problem is that it's a bit slow due to the fact that i'm using count(), which is translated to count(*).
I need to know if is there a way to use count(object),
Here is what i got in sql server profiler
exec sp_executesql N'SELECT TOP (50)
[Project1].[C2] AS [C1],
[Project1].[object] AS [object],
[Project1].[C1] AS [C2]
FROM ( SELECT
[GroupBy1].[A1] AS [C1],
[GroupBy1].[K1] AS [object],
1 AS [C2]
FROM ( SELECT
[Extent1].[object], AS [K1],
COUNT(1) AS [A1]
FROM (SELECT
[full].[mc_host_class] AS [mc_host_class],
[full].[event_handle] AS [event_handle],
[full].[mc_host_address] AS [mc_host_address],
[full].[mc_object_class] AS [mc_object_class],
[full].[mc_object] AS [mc_object],
[full].[mc_incident_time] AS [mc_incident_time],
[full].[date_reception] AS [date_reception],
[full].[status] AS [status],
[full].[mc_owner] AS [mc_owner],
[full].[msg] AS [msg],
[full].[duration] AS [duration],
[full].[repeat_count] AS [repeat_count],
[full].[mc_date_modification] AS [mc_date_modification],
[full].[event_class] AS [event_class],
[full].[bycn_ticket_remedy] AS [bycn_ticket_remedy],
[full].[mc_host] AS [mc_host],
[full].[acknowledge_by] AS [acknowledge_by],
[full].[acknowledge_by_time] AS [acknowledge_by_time],
[full].[assigned_by] AS [assigned_by],
[full].[assigned_to] AS [assigned_to],
[full].[assigned_by_time] AS [assigned_by_time],
[full].[closed_b y] AS [closed_by],
[full].[closed_by_time] AS [closed_by_time],
[full].[blacked_out] AS [blacked_out],
[full].[bycn_liaison_type] AS [bycn_liaison_type],
[full].[bycn_liaison_debit] AS [bycn_liaison_debit],
[full].[cause] AS [cause],
[full].[mc_location] AS [mc_location],
[full].[mc_parameter] AS [mc_parameter]
FROM [dbo].[full] AS [full]) AS [Extent1]
GROUP BY [Extent1].[object],
) AS [GroupBy1]
) AS [Project1]
ORDER BY [Project1].[C1] DESC',N'#p__linq__0 datetime2(7),#p__linq__1 datetime2(7)',#p__linq__0='2015-03-14 00:00:00',#p__linq__1='2015-04-15 00:00:00'

Perhaps couple optimisations can do the trick:
Do the take first, before selecting
Count groups only once using a let keyword
So metacode (this code written in notepad and won't compile!)
var topFifty = (
from p in db.full
group p by p.object into g
let groupedCount = g.Count()
orderby groupedCount descending
select p.key, groupedCount
)
.Take(50).ToList();
var topFifty.Select(x => new StringIntType
{
str = x.Key,
nbr = x.Count
}).ToList();

What is the difference between these LINQ queries

I've been fooling around with some LINQ over Entities and I'm getting strange results and I would like to get an explanation...
Given the following LINQ query,
// Sample # 1
IEnumerable<GroupInformation> groupingInfo;
groupingInfo = from a in context.AccountingTransaction
group a by a.Type into grp
select new GroupInformation()
{
GroupName = grp.Key,
GroupCount = grp.Count()
};
I get the following SQL query (taken from SQL Profiler):
SELECT
1 AS [C1],
[GroupBy1].[K1] AS [Type],
[GroupBy1].[A1] AS [C2]
FROM ( SELECT
[Extent1].[Type] AS [K1],
COUNT(1) AS [A1]
FROM [dbo].[AccountingTransaction] AS [Extent1]
GROUP BY [Extent1].[Type]
) AS [GroupBy1]
So far so good.
If I change my LINQ query to:
// Sample # 2
groupingInfo = context.AccountingTransaction.
GroupBy(a => a.Type).
Select(grp => new GroupInformation()
{
GroupName = grp.Key,
GroupCount = grp.Count()
});
it yields to the exact same SQL query. Makes sense to me.
Here comes the interesting part... If I change my LINQ query to:
// Sample # 3
IEnumerable<AccountingTransaction> accounts;
IEnumerable<IGrouping<object, AccountingTransaction>> groups;
IEnumerable<GroupInformation> groupingInfo;
accounts = context.AccountingTransaction;
groups = accounts.GroupBy(a => a.Type);
groupingInfo = groups.Select(grp => new GroupInformation()
{
GroupName = grp.Key,
GroupCount = grp.Count()
});
the following SQL is executed (I stripped a few of the fields from the actual query, but all the fields from the table (~ 15 fields) were included in the query, twice):
SELECT
[Project2].[C1] AS [C1],
[Project2].[Type] AS [Type],
[Project2].[C2] AS [C2],
[Project2].[Id] AS [Id],
[Project2].[TimeStamp] AS [TimeStamp],
-- <snip>
FROM ( SELECT
[Distinct1].[Type] AS [Type],
1 AS [C1],
[Extent2].[Id] AS [Id],
[Extent2].[TimeStamp] AS [TimeStamp],
-- <snip>
CASE WHEN ([Extent2].[Id] IS NULL) THEN CAST(NULL AS int) ELSE 1 END AS [C2]
FROM (SELECT DISTINCT
[Extent1].[Type] AS [Type]
FROM [dbo].[AccountingTransaction] AS [Extent1] ) AS [Distinct1]
LEFT OUTER JOIN [dbo].[AccountingTransaction] AS [Extent2] ON [Distinct1].[Type] = [Extent2].[Type]
) AS [Project2]
ORDER BY [Project2].[Type] ASC, [Project2].[C2] ASC
Why are the SQLs generated are so different? After all, the exact same code is executed, it's just that sample # 3 is using intermediate variables to get the same job done!
Also, if I do:
Console.WriteLine(groupingInfo.ToString());
for sample # 1 and sample # 2, I get the exact same query that was captured by SQL Profiler, but for sample # 3, I get:
System.Linq.Enumerable+WhereSelectEnumerableIterator`2[System.Linq.IGrouping`2[System.Object,TestLinq.AccountingTransaction],TestLinq.GroupInformation]
What is the difference? Why can't I get the SQL Query generated by LINQ if I split the LINQ query in multiple instructions?
The ulitmate goal is to be able to add operators to the query (Where, OrderBy, etc.) at run-time.
BTW, I've seen this behavior in EF 4.0 and EF 6.0.
Thank you for your help.

The reason is because in your third attempt you're referring to accounts as IEnumerable<AccountingTransaction> which will cause the query to be invoked using Linq-To-Objects (Enumerable.GroupBy and Enumerable.Select)
On the other hand, in your first and second attempts the reference to AccountingTransaction is preserved as IQueryable<AccountingTransaction> and the query will be executed using Linq-To-Entities which will then transform it to the appropriate SQL statement.

Is Linq Include broken when used with joins and where clauses?

I have been experimenting trying to get the following Linq working without joy. I'm convinced that it's right, but that might just be my bad Linq. I originally added this as a answer to a similar question here:
Linq-to-entities - Include() method not loading
But as it's a very old question, and mine is more specific, I figured it would do better as an explicit question.
In the linked question, Alex James gives two interesting solutions, however if you try them and check the SQL, it's horrible.
The example I was working on is:
var theRelease = from release in context.Releases
where release.Name == "Hello World"
select release;
var allProductionVersions = from prodVer in context.ProductionVersions
where prodVer.Status == 1
select prodVer;
var combined = (from release in theRelease
join p in allProductionVersions on release.Id equals p.ReleaseID
select release).Include(release => release.ProductionVersions);
var allProductionsForChosenRelease = combined.ToList();
This follows the simpler of the two examples. Without the include it produces the perfectly respectable sql:
SELECT
[Extent1].[Id] AS [Id],
[Extent1].[Name] AS [Name]
FROM [dbo].[Releases] AS [Extent1]
INNER JOIN [dbo].[ProductionVersions] AS [Extent2] ON [Extent1].[Id] = [Extent2].[ReleaseID]
WHERE ('Hello World' = [Extent1].[Name]) AND (1 = [Extent2].[Status])
But with, OMG:
SELECT
[Project1].[Id1] AS [Id],
[Project1].[Id] AS [Id1],
[Project1].[Name] AS [Name],
[Project1].[C1] AS [C1],
[Project1].[Id2] AS [Id2],
[Project1].[Status] AS [Status],
[Project1].[ReleaseID] AS [ReleaseID]
FROM ( SELECT
[Extent1].[Id] AS [Id],
[Extent1].[Name] AS [Name],
[Extent2].[Id] AS [Id1],
[Extent3].[Id] AS [Id2],
[Extent3].[Status] AS [Status],
[Extent3].[ReleaseID] AS [ReleaseID],
CASE WHEN ([Extent3].[Id] IS NULL) THEN CAST(NULL AS int) ELSE 1 END AS [C1]
FROM [dbo].[Releases] AS [Extent1]
INNER JOIN [dbo].[ProductionVersions] AS [Extent2] ON [Extent1].[Id] = [Extent2].[ReleaseID]
LEFT OUTER JOIN [dbo].[ProductionVersions] AS [Extent3] ON [Extent1].[Id] = [Extent3].[ReleaseID]
WHERE ('Hello World' = [Extent1].[Name]) AND (1 = [Extent2].[Status])
) AS [Project1]
ORDER BY [Project1].[Id1] ASC, [Project1].[Id] ASC, [Project1].[C1] ASC
Total garbage. The key point to note here is the fact that it returns the outer joined version of the table which has not been limited by status=1.
This results in the WRONG data being returned:
Id Id1 Name C1 Id2 Status ReleaseID
2 1 Hello World 1 1 2 1
2 1 Hello World 1 2 1 1
Note that the status of 2 is being returned there, despite our restriction. It simply does not work.
If I have gone wrong somewhere, I would be delighted to find out, as this is making a mockery of Linq. I love the idea, but the execution doesn't seem to be usable at the moment.
Out of curiosity, I tried the LinqToSQL dbml rather than the LinqToEntities edmx that produced the mess above:
SELECT [t0].[Id], [t0].[Name], [t2].[Id] AS [Id2], [t2].[Status], [t2].[ReleaseID], (
SELECT COUNT(*)
FROM [dbo].[ProductionVersions] AS [t3]
WHERE [t3].[ReleaseID] = [t0].[Id]
) AS [value]
FROM [dbo].[Releases] AS [t0]
INNER JOIN [dbo].[ProductionVersions] AS [t1] ON [t0].[Id] = [t1].[ReleaseID]
LEFT OUTER JOIN [dbo].[ProductionVersions] AS [t2] ON [t2].[ReleaseID] = [t0].[Id]
WHERE ([t0].[Name] = #p0) AND ([t1].[Status] = #p1)
ORDER BY [t0].[Id], [t1].[Id], [t2].[Id]
Slightly more compact - weird count clause, but overall same total FAIL.
Please tell me I've missed something obvious, as I really want to like Linq!

Okay, after another evening of head scratching I cracked it.
In LinqToSQL:
using (var context = new TestSQLModelDataContext())
{
context.DeferredLoadingEnabled = false;
DataLoadOptions ds = new DataLoadOptions();
ds.LoadWith<ProductionVersion>(prod => prod.Release);
context.LoadOptions = ds;
var combined = from release in context.Releases
where release.Name == "Hello World"
select from prodVer in release.ProductionVersions
where prodVer.Status == 1
select prodVer;
var allProductionsForChosenRelease = combined.ToList();
}
This produces the much more reasonable SQL:
SELECT [t2].[Id], [t2].[Status], [t2].[ReleaseID], [t0].[Id] AS [Id2], [t0].[Name], (
SELECT COUNT(*)
FROM [dbo].[ProductionVersions] AS [t3]
WHERE ([t3].[Status] = 1) AND ([t3].[ReleaseID] = [t0].[Id])
) AS [value]
FROM [dbo].[Releases] AS [t0]
OUTER APPLY (
SELECT [t1].[Id], [t1].[Status], [t1].[ReleaseID]
FROM [dbo].[ProductionVersions] AS [t1]
WHERE ([t1].[Status] =1) AND ([t1].[ReleaseID] = [t0].[Id])
) AS [t2]
WHERE [t0].[Name] = 'Hello World'
ORDER BY [t0].[Id], [t2].[Id]
Which produces the correct results:
Id Status ReleaseID Id2 Name value
2 1 1 1 Hello World 1
And in LinqToEntities (I couldn't get the Include syntax to work, so I use the quirk where including the desired table in the results links it up correctly):
using (var context = new TestEntities1())
{
var combined = (from release in context.Releases
where release.Name == "Hello World"
select from prodVer in release.ProductionVersions
where prodVer.Status == 1
select new { prodVer, Release =prodVer.Release });
var allProductionsForChosenRelease = combined.ToList();
}
And this produces the SQL:
SELECT
[Project1].[Id] AS [Id],
[Project1].[C1] AS [C1],
[Project1].[Id1] AS [Id1],
[Project1].[Status] AS [Status],
[Project1].[ReleaseID] AS [ReleaseID],
[Project1].[Id2] AS [Id2],
[Project1].[Name] AS [Name]
FROM ( SELECT
[Extent1].[Id] AS [Id],
[Join1].[Id1] AS [Id1],
[Join1].[Status] AS [Status],
[Join1].[ReleaseID] AS [ReleaseID],
[Join1].[Id2] AS [Id2],
[Join1].[Name] AS [Name],
CASE WHEN ([Join1].[Id1] IS NULL) THEN CAST(NULL AS int) ELSE 1 END AS [C1]
FROM [dbo].[Releases] AS [Extent1]
LEFT OUTER JOIN (SELECT [Extent2].[Id] AS [Id1], [Extent2].[Status] AS [Status], [Extent2].[ReleaseID] AS [ReleaseID], [Extent3].[Id] AS [Id2], [Extent3].[Name] AS [Name]
FROM [dbo].[ProductionVersions] AS [Extent2]
INNER JOIN [dbo].[Releases] AS [Extent3] ON [Extent2].[ReleaseID] = [Extent3].[Id] ) AS [Join1] ON ([Extent1].[Id] = [Join1].[ReleaseID]) AND (1 = [Join1].[Status])
WHERE 'Hello World' = [Extent1].[Name]
) AS [Project1]
ORDER BY [Project1].[Id] ASC, [Project1].[C1] ASC
Which is fairly mental, but it does work.
Id C1 Id1 Status ReleaseID Id2 Name
1 1 2 1 1 1 Hello World
All of which leads me to the conclusion that Linq is far from finished. It can be used, but with extreme caution. Use it as a strongly typed and compile time checked, but laborious/error prone, way of writing bad SQL. It's a trade-off. You get more security at the C# end, but man it's a lot harder than writing SQL!

Taking a second look, I now understand the elusive effect of the Include.
Just as in plain SQL, a join in LINQ will repeat results when the right side of the join is the "n" end of a 1-n association.
Let's assume you have one Release with two ProductionVersions. Without the Include, the join will give you two identical Releases, because after all the statement selects releases. Now when you add the Include, EF will not only return two releases, but will also fully populate their ProductionVersions collections.
Looking a bit deeper, in the context's cache, it appears that EF really only materialized just 1 Release and 2ProductionVersions. It's just that the releases are returned twice in the final result set.
In a way, you got what you asked for: give me releases, multiplied by their number of versions. But that's not what you intended to ask.
What you (probably) intended reveals a weak spot in EF's toolbox: we can't Include partial collections. I think you tried to get releases populated with ProductionVersions of Status = 1 only. If possible, you'd rather have done this:
context.Releases.Include(r => r.ProductionVersions.Where(v => v.Status == 1))
.Where(r => r.Name == "Hello World")
But that throws an exception:
The Include path expression must refer to a navigation property defined on the type. Use dotted paths for reference navigation properties and the Select operator for collection navigation properties.
Parameter name: path
This "filtered include" problem has been noted before and until the EF team (or a contributor) decides to grab this issue we have to do with elaborate work-arounds. I described a common one here.

Linq to SQL sort by fields in 2 entities (parent-child)

I have 2 entities called Request and Days. Request has many days and what I am having problems with is properly sorting my entities the way I want to.
Days has a certain field called Hours and I need to sort it first by the Hours field in days (however, first are ALL of the fields that have only one day) and then, by then number of Days in a Request.
I've tried many orderby/thenby combinations and can't get this quite right.
Here is a recent one I've tried:
sortingFunction = x => x.Days.OrderBy(h => h.Hours).Count();
Any help with this?

from r in db.Requests
let daysCount = r.Days.Count()
orderby daysCount == 1 ? r.Days.FirstOrDefault().Hours : Int32.MaxValue,
daysCount
select r
Generated SQL query will look like:
SELECT
[Project4].[Id] AS [Id],
[Project4].[Foo] AS [Foo]
FROM ( SELECT
CASE WHEN ((1 = [Project3].[C1]) AND ([Project3].[C1] IS NOT NULL))
THEN [Project3].[C2] ELSE 2147483647 END AS [C1],
[Project3].[Id] AS [Id],
[Project3].[Foo] AS [Foo]
[Project3].[C1] AS [C2]
FROM ( SELECT
[Project1].[Id] AS [Id],
[Project1].[Foo] AS [Foo]
[Project1].[C1] AS [C1], // count of days
(SELECT TOP (1) // C2 is hours of first day
[Extent3].[Hours] AS [Hours]
FROM [dbo].[Days] AS [Extent3]
WHERE [Project1].[Id] = [Extent3].[RequestId]) AS [C2]
FROM (SELECT
[Extent1].[Id] AS [Id],
[Extent1].[Foo] AS [Foo]
(SELECT
COUNT(1) AS [A1]
FROM [dbo].[Days] AS [Extent2]
WHERE [Extent1].[Id] = [Extent2].[RequestId]) AS [C1]
FROM [dbo].[Requests] AS [Extent1]
) AS [Project1]
) AS [Project3]
) AS [Project4]
ORDER BY [Project4].[C1] ASC, [Project4].[C2] ASC