We Keep Coding

Linq to SQL - Order By related Table using a Where Clause

I have two related tables lets say:
Table Products (Main Table)
ID
Name
Type
Table Parts (Child of Products, contains detailled Information)
ID
ProductID
PartName
PartValue
I would like to get Products ordered by the value of a specific part (e.G. Engine)
I came up with the following:
// Code to construct a query to get all desired products
products = products.OrderBy(c => c.Parts.Where(d => d.PartName == "Engine").Select(d => d.Value).FirstOrDefault());
This works but is too slow. Can I improve the query or will I have to redsign my database so I won't be sorting like this in the first place?
Generated SQL-Query:
SELECT
[Project3].[ID] AS [ID],
[Project3].[Name] AS [Name]
FROM ( SELECT
[Project2].[ID] AS [ID],
[Project2].[Name] AS [Name],
[Project2].[C1] AS [C1]
FROM ( SELECT
[Extent1].[ID] AS [ID],
[Extent1].[Name] AS [Name],
(SELECT TOP (1)
[Extent2].[PartValue] AS [PartValue]
FROM [dbo].[Parts] AS [Extent2]
WHERE ([Extent1].[ID] = [Extent2].[ProductID]) AND (N'Engine' = [Extent2].[PartName])) AS [C1]
FROM [dbo].[Products] AS [Extent1]
WHERE (1 = [Extent1].[Type])
) AS [Project2]
) AS [Project3]
ORDER BY [Project3].[C1] ASC

If you want to try a join, it would be a bit like this:
Products
.Join(Parts, pd => pd.ID, pt => pt.ProductID, (pd, pt) => new { pd.Name, pt.PartName, pt.PartValue })
.Where(x => x.PartName == "Engine")
.OrderBy(x => x.PartValue);
This would result in a single select statement without the inner selects from before

Linq Left Outer Join with Count

I want to create this SQL query:
SELECT
a.[Seat],
b.[PlayerId],
b.[UserName],
b.[NickName],
COUNT(c.PlayerId) AS Trophy
FROM [dbo].[tbl_PlayerTableSeat] AS a
INNER JOIN [dbo].[tbl_Player] AS b ON a.[PlayerId] = b.[PlayerId]
INNER JOIN [dbo].[tbl_GameVirtualTable] AS d ON d.GameVirtualTableId = a.GameVirtualTableId
LEFT OUTER JOIN [dbo].[tbl_PlayerTableWinning] AS c ON a.[PlayerId] = c.[PlayerId] AND c.GameTableId = d.GameTableId
WHERE a.GameVirtualTableId = 36
GROUP BY a.[Seat], b.[PlayerId], b.[UserName], b.[NickName]
I have this Linq
var virtualTableSeatList = (from s in db.PlayerTableSeat
join p in db.Player on s.PlayerId equals p.PlayerId
join v in db.GameVirtualTable on s.GameVirtualTableId equals v.GameVirtualTableId
join w in db.PlayerTableWinning on new { X1 = s.PlayerId, X2 = v.GameTableId } equals new { X1 = w.PlayerId, X2 = w.GameTableId } into gj
from g in gj.DefaultIfEmpty()
where s.GameVirtualTableId == virtualGameTableId
group new { p, s } by new { p.PlayerId, s.Seat, p.NickName, p.UserName } into grp
select new VirtualTableSeatDto
{
PlayerId = grp.Key.PlayerId,
Seat = grp.Key.Seat,
NickName = grp.Key.NickName,
UserName = grp.Key.UserName,
Trophy = grp.Count()
}
).ToList();
From SQL Profiler, the Linq generates this SQL query:
exec sp_executesql N'SELECT
[GroupBy1].[K2] AS [PlayerId],
CAST( [GroupBy1].[K1] AS int) AS [C1],
[GroupBy1].[K4] AS [NickName],
[GroupBy1].[K3] AS [UserName],
[GroupBy1].[A1] AS [C2]
FROM ( SELECT
[Extent1].[Seat] AS [K1],
[Extent2].[PlayerId] AS [K2],
[Extent2].[UserName] AS [K3],
[Extent2].[NickName] AS [K4],
COUNT(1) AS [A1]
FROM [dbo].[tbl_PlayerTableSeat] AS [Extent1]
INNER JOIN [dbo].[tbl_Player] AS [Extent2] ON [Extent1].[PlayerId] = [Extent2].[PlayerId]
INNER JOIN [dbo].[tbl_GameVirtualTable] AS [Extent3] ON [Extent1].[GameVirtualTableId] = [Extent3].[GameVirtualTableId]
LEFT OUTER JOIN [dbo].[tbl_PlayerTableWinning] AS [Extent4] ON ([Extent1].[PlayerId] = [Extent4].[PlayerId]) AND ([Extent3].[GameTableId] = [Extent4].[GameTableId])
WHERE [Extent1].[GameVirtualTableId] = #p__linq__0
GROUP BY [Extent1].[Seat], [Extent2].[PlayerId], [Extent2].[UserName], [Extent2].[NickName]
) AS [GroupBy1]',N'#p__linq__0 int',#p__linq__0=36
I want to change COUNT(1) AS [A1] to COUNT([Extent4].[PlayerId]) AS [A1]
so it can return correct data.
I have no idea how to change the LinQ
Trophy = grp.Count()
so that it can count PlayerId of PlayerTableWinning instead of COUNT(1)
Updated: #Ivan Stoev
By adding the g into the group.
group new { p, s, g }
And sum the group
Trophy = grp.Sum(item => item.w != null ? 1 : 0)
It return the correct answer. However, it is using SUM instead of count. The SQL query generated is as below:
exec sp_executesql N'SELECT
[GroupBy1].[K2] AS [PlayerId],
CAST( [GroupBy1].[K1] AS int) AS [C1],
[GroupBy1].[K4] AS [NickName],
[GroupBy1].[K3] AS [UserName],
[GroupBy1].[A1] AS [C2]
FROM ( SELECT
[Filter1].[K1] AS [K1],
[Filter1].[K2] AS [K2],
[Filter1].[K3] AS [K3],
[Filter1].[K4] AS [K4],
SUM([Filter1].[A1]) AS [A1]
FROM ( SELECT
[Extent1].[Seat] AS [K1],
[Extent2].[PlayerId] AS [K2],
[Extent2].[UserName] AS [K3],
[Extent2].[NickName] AS [K4],
CASE WHEN ( NOT (([Extent4].[GameTableId] IS NULL) AND ([Extent4].[PlayerId] IS NULL) AND ([Extent4].[GameRoundId] IS NULL))) THEN 1 ELSE 0 END AS [A1]
FROM [dbo].[tbl_PlayerTableSeat] AS [Extent1]
INNER JOIN [dbo].[tbl_Player] AS [Extent2] ON [Extent1].[PlayerId] = [Extent2].[PlayerId]
INNER JOIN [dbo].[tbl_GameVirtualTable] AS [Extent3] ON [Extent1].[GameVirtualTableId] = [Extent3].[GameVirtualTableId]
LEFT OUTER JOIN [dbo].[tbl_PlayerTableWinning] AS [Extent4] ON ([Extent1].[PlayerId] = [Extent4].[PlayerId]) AND ([Extent3].[GameTableId] = [Extent4].[GameTableId])
WHERE [Extent1].[GameVirtualTableId] = #p__linq__0
) AS [Filter1]
GROUP BY [K1], [K2], [K3], [K4]
) AS [GroupBy1]',N'#p__linq__0 int',#p__linq__0=36

The only (but significant) difference between SQL COUNT(field) and COUNT(1) is that the former is excluding the NULL values, which when applied to the normally required field from the right side of a left outer join like in your case produces a different result when there are no matching records - the former returns 0 while the latter returns 1.
The "natural" LINQ equivalent would be Count(field != null), but that unfortunately is translated to a quite different SQL by the current EF query provider. So in such cases I personally use the closer equivalent expression Sum(field != null ? 1 : 0) which produces a much better SQL.
In order to apply the above to your query, you'll need an access to w inside the grouping, so change
group new { p, s }
to
group new { p, s, w }
and then use
Trophy = grp.Sum(item => item.w != null ? 1 : 0)

Linq query executed from code slowly than from studio

I have a Linq-to-Entities query:
var result = (from sls in context.tbl_sales
where sls.saleDate == d
join clnt in context.tbl_clients on sls.clientId equals clnt.clientId
group sls.quantity by clnt.distributorId into grp
select new
{ distributorId = grp.Key,
Quantity = grp.Sum(e => e) })
.ToDictionary(e => e.distributorId, e => e.Quantity);
EF generates this SQL query:
set #p__linq__0='01.11.2015 0:00:00'
SELECT
1 AS [C1],
[GroupBy1].[K1] AS [distributorId],
[GroupBy1].[A1] AS [C2]
FROM ( SELECT
[Extent2].[distributorId] AS [K1],
SUM([Extent1].[quantity]) AS [A1]
FROM [sales].[tbl_sales] AS [Extent1]
INNER JOIN [sales].[tbl_clients] AS [Extent2] ON [Extent1].[clientId] = [Extent2].[clientId]
WHERE [Extent1].[saleDate] = #p__linq__0
GROUP BY [Extent2].[distributorId]
) AS [GroupBy1]
This query executed from code and it took about 3-10 minutes to complete.
If I copy the query and run it from SQL Server Management Studio, it takes ~1 second.
Why is it so slow from code?
update:
sql profiler trace result:
exec sp_executesql N'SELECT
1 AS [C1],
[GroupBy1].[K1] AS [distributorId],
[GroupBy1].[A1] AS [C2]
FROM ( SELECT
[Extent2].[distributorId] AS [K1],
SUM([Extent1].[quantity]) AS [A1]
FROM [sales].[tbl_sales] AS [Extent1]
INNER JOIN [sales].[tbl_clients] AS [Extent2] ON [Extent1].[clientId] = [Extent2].[clientId]
WHERE [Extent1].[saleDate] = #p__linq__0
GROUP BY [Extent2].[distributorId]
) AS [GroupBy1]',N'#p__linq__0 datetime2(7)',#p__linq__0='2015-10-01 00:00:00'
update2:
if entity framework generate exec sp_executesql index by saleDate column not workin?

What is the difference between these LINQ queries

I've been fooling around with some LINQ over Entities and I'm getting strange results and I would like to get an explanation...
Given the following LINQ query,
// Sample # 1
IEnumerable<GroupInformation> groupingInfo;
groupingInfo = from a in context.AccountingTransaction
group a by a.Type into grp
select new GroupInformation()
{
GroupName = grp.Key,
GroupCount = grp.Count()
};
I get the following SQL query (taken from SQL Profiler):
SELECT
1 AS [C1],
[GroupBy1].[K1] AS [Type],
[GroupBy1].[A1] AS [C2]
FROM ( SELECT
[Extent1].[Type] AS [K1],
COUNT(1) AS [A1]
FROM [dbo].[AccountingTransaction] AS [Extent1]
GROUP BY [Extent1].[Type]
) AS [GroupBy1]
So far so good.
If I change my LINQ query to:
// Sample # 2
groupingInfo = context.AccountingTransaction.
GroupBy(a => a.Type).
Select(grp => new GroupInformation()
{
GroupName = grp.Key,
GroupCount = grp.Count()
});
it yields to the exact same SQL query. Makes sense to me.
Here comes the interesting part... If I change my LINQ query to:
// Sample # 3
IEnumerable<AccountingTransaction> accounts;
IEnumerable<IGrouping<object, AccountingTransaction>> groups;
IEnumerable<GroupInformation> groupingInfo;
accounts = context.AccountingTransaction;
groups = accounts.GroupBy(a => a.Type);
groupingInfo = groups.Select(grp => new GroupInformation()
{
GroupName = grp.Key,
GroupCount = grp.Count()
});
the following SQL is executed (I stripped a few of the fields from the actual query, but all the fields from the table (~ 15 fields) were included in the query, twice):
SELECT
[Project2].[C1] AS [C1],
[Project2].[Type] AS [Type],
[Project2].[C2] AS [C2],
[Project2].[Id] AS [Id],
[Project2].[TimeStamp] AS [TimeStamp],
-- <snip>
FROM ( SELECT
[Distinct1].[Type] AS [Type],
1 AS [C1],
[Extent2].[Id] AS [Id],
[Extent2].[TimeStamp] AS [TimeStamp],
-- <snip>
CASE WHEN ([Extent2].[Id] IS NULL) THEN CAST(NULL AS int) ELSE 1 END AS [C2]
FROM (SELECT DISTINCT
[Extent1].[Type] AS [Type]
FROM [dbo].[AccountingTransaction] AS [Extent1] ) AS [Distinct1]
LEFT OUTER JOIN [dbo].[AccountingTransaction] AS [Extent2] ON [Distinct1].[Type] = [Extent2].[Type]
) AS [Project2]
ORDER BY [Project2].[Type] ASC, [Project2].[C2] ASC
Why are the SQLs generated are so different? After all, the exact same code is executed, it's just that sample # 3 is using intermediate variables to get the same job done!
Also, if I do:
Console.WriteLine(groupingInfo.ToString());
for sample # 1 and sample # 2, I get the exact same query that was captured by SQL Profiler, but for sample # 3, I get:
System.Linq.Enumerable+WhereSelectEnumerableIterator`2[System.Linq.IGrouping`2[System.Object,TestLinq.AccountingTransaction],TestLinq.GroupInformation]
What is the difference? Why can't I get the SQL Query generated by LINQ if I split the LINQ query in multiple instructions?
The ulitmate goal is to be able to add operators to the query (Where, OrderBy, etc.) at run-time.
BTW, I've seen this behavior in EF 4.0 and EF 6.0.
Thank you for your help.

The reason is because in your third attempt you're referring to accounts as IEnumerable<AccountingTransaction> which will cause the query to be invoked using Linq-To-Objects (Enumerable.GroupBy and Enumerable.Select)
On the other hand, in your first and second attempts the reference to AccountingTransaction is preserved as IQueryable<AccountingTransaction> and the query will be executed using Linq-To-Entities which will then transform it to the appropriate SQL statement.

Linq to SQL - Joins and Skip+Take

I have Linq-to-SQL code that works with a many-to-many relationship, but note that the relationship itself has its own set of attributes (in this case, Products are in Many Categories, and each product-in-category relation has its own SortOrder attribute).
I have a Linq-to-SQL block that returns matching Products with Category membership information. When I execute the code it generates optimised T-SQL code like so:
exec sp_executesql N'SELECT [t0].[ProductId], [t0].[Name], [t1].[ProductId] AS [ProductId2], [t1].[CategoryId], [t1].[SortOrder] AS [SortOrder2], [t2].[CategoryId] AS [CategoryId2], [t2].[Name] AS [Name2] (
SELECT COUNT(*)
FROM [dbo].[ProductsInCategories] AS [t3]
INNER JOIN [dbo].[Categories] AS [t4] ON [t4].[CategoryId] = [t3].[CategoryId]
WHERE [t3].[ProductId] = [t0].[ProductId]
) AS [value]
FROM [dbo].[Products] AS [t0]
LEFT OUTER JOIN ([dbo].[ProductsInCategories] AS [t1]
INNER JOIN [dbo].[Categories] AS [t2] ON [t2].[CategoryId] = [t1].[CategoryId]) ON [t1].[ProductId] = [t0].[ProductId]
WHERE (([t0].[OwnerId]) = #p0) AND ([t0].[Visible] = 1)
ORDER BY [t0].[SortOrder], [t0].[Name], [t0].[ProductId], [t1].[CategoryId]',N'#p0 bigint',#p0=3
However, when I add paging instructions (i.e.".Skip(0).Take(50)") to the Linq expression the generated SQL becomes this:
exec sp_executesql N'SELECT TOP (50) [t0].[ProductId], [t0].[Name]
FROM [dbo].[Products] AS [t0]
WHERE (([t0].[OwnerId]) = #p0) AND ([t0].[Visible] = 1)
ORDER BY [t0].[SortOrder], [t0].[Name]',N'#p0 bigint',#p0=3
Which means the Category membership information isn't loaded anymore, so Linq-to-SQL then executes the manual loading code 50 times over (one for each member in the returned set):
exec sp_executesql N'SELECT [t0].[ProductId], [t0].[CategoryId], [t0].[SortOrder], [t1].[CategoryId] AS [CategoryId2], [t1].[Name]
FROM [dbo].[ProductsInCategories] AS [t0]
INNER JOIN [dbo].[Categories] AS [t1] ON [t1].[CategoryId] = [t0].[CategoryId]
WHERE [t0].[ProductId] = #x1',N'#x1 bigint',#x1=1141
(obviously the "#x1" ID parameter varies for each result from the original query).
So clearly Linq paging breaks the query and causes it to load data separately. Is there a way around this or should I do paging in my own software?
...fortunately the number of products in the database is small enough (<500) to do this, but it just feels dirty because there could be tens of thousands of products, and this just wouldn't be a good query.
EDIT:
Here is my Linq:
DataLoadOptions dlo = new DataLoadOptions();
dlo.LoadWith<Product>( p => p.ProductsInCategories );
dlo.LoadWith<ProductsInCategory>( pic => pic.Category );
this.LoadOptions = dlo;
query = from p in this.Products
select p;
// The lines below are added conditionally:
query = query.OrderBy( p => p.SortOrder ).ThenBy( p => p.Name );
query = query.Where( p => p.Visible );
query = query.Where( p => p.Name.Contains( filter ) || p.Description.Contains( filter ) );
query = query.Where( p => p.OwnerId == siteId );
The skip/take lines are added optionally, and are the only differences that cause the different T-SQL generation (as far as I know):
IQueryable<Product> query = GetProducts( siteId, category, filter, showHidden, sortBySortOrder );
///////////////////////////////////
total = query.Count();
var pagedProducts = query.Skip( pageIndex * pageSize ).Take( pageSize );
return pagedProducts;

An alternative answer which first pages the products and then selects products and categories in a parent-child structure would be like this:
var filter = "a";
var pageSize = 2;
var pageIndex = 1;
// get the correct products
var query = Products.AsQueryable();
query = query.Where (q => q.Name.Contains(filter));
query = query.OrderBy (q => q.SortOrder).ThenBy(q => q.Name);
// do paging
query = query.Skip(pageSize*pageIndex).Take(pageSize);
// now get products + categories as tree structure
var query2 = query.Select(
q=>new
{
q.Name,
Categories=q.ProductsInCategories.Select (pic => pic.Category)
});
Which produces a single SQL statement
-- Region Parameters
DECLARE #p0 NVarChar(1000) = '%a%'
DECLARE #p1 Int = 2
DECLARE #p2 Int = 2
-- EndRegion
SELECT [t2].[Name], [t4].[CategoryId], [t4].[Name] AS [Name2], [t4].[Visible], (
SELECT COUNT(*)
FROM (
SELECT [t5].[CategoryId]
FROM [ProductsInCategories] AS [t5]
WHERE [t5].[ProductId] = [t2].[ProductId]
) AS [t6]
INNER JOIN [Categories] AS [t7] ON [t7].[CategoryId] = [t6].[CategoryId]
) AS [value]
FROM (
SELECT [t1].[ProductId], [t1].[Name], [t1].[ROW_NUMBER]
FROM (
SELECT ROW_NUMBER() OVER (ORDER BY [t0].[SortOrder], [t0].[Name], [t0].[ProductId]) AS [ROW_NUMBER], [t0].[ProductId], [t0].[Name]
FROM [Products] AS [t0]
WHERE [t0].[Name] LIKE #p0
) AS [t1]
WHERE [t1].[ROW_NUMBER] BETWEEN #p1 + 1 AND #p1 + #p2
) AS [t2]
LEFT OUTER JOIN ([ProductsInCategories] AS [t3]
INNER JOIN [Categories] AS [t4] ON [t4].[CategoryId] = [t3].[CategoryId]) ON [t3].[ProductId] = [t2].[ProductId]
ORDER BY [t2].[ROW_NUMBER], [t3].[CategoryId], [t3].[ProductId]

Here is a workaround: you should construct your query based on all your conditions, perform ordering there but select only the primary key on your Product table (let's assume this is ProductId column).
The next step is to take the total count (to calculate rows should be skipped and taken),
and the last step is to select all the records from your Product table whose ProductIds are in the query (note: Skip and Take extension methods should be applied to query, not to the new select itself).
This will get you a SELECT statement similar to yours (from the first example) with related entities.
EDIT:
Just created a similar DB structure (according to the original SQL from the question):
Then used:
using (var db = new TestDataContext())
{
DataLoadOptions options = new DataLoadOptions();
options.LoadWith<Product>(p => p.ProductsInCategories);
options.LoadWith<ProductsInCategory>(pic => pic.Category);
db.LoadOptions = options;
var filter = "product";
var pageIndex = 1;
var pageSize = 10;
var query = db.Products
.OrderBy(p => p.SortOrder)
.ThenBy(p => p.Name)
.Where(p => p.Name.Contains(filter) || p.Description.Contains(filter))
.Select(p => p.ProductId);
var total = query.Count();
var products = db.Products
.Where(p => query.Skip(pageIndex * pageSize).Take(pageSize).Contains(p.ProductId))
.ToList();
}
After the .ToList() call, products variable hold products with product categories with categories. This also produced 2 SQL statement, one - for .Count() statement:
exec sp_executesql N'SELECT COUNT(*) AS [value]
FROM [dbo].[Products] AS [t0]
WHERE ([t0].[Name] LIKE #p0) OR ([t0].[Description] LIKE #p1)',N'#p0 nvarchar(4000),#p1 nvarchar(4000)',#p0=N'%product%',#p1=N'%product%'
and another one for .ToList():
exec sp_executesql N'SELECT [t0].[ProductId], [t0].[Name], [t0].[Description], [t0].[SortOrder], [t1].[ProductId] AS [ProductId2], [t1].[CategoryId], [t1].[SortOrder] AS [SortOrder2], [t2].[CategoryId] AS [CategoryId2], [t2].[Name] AS [Name2], (
SELECT COUNT(*)
FROM (
SELECT NULL AS [EMPTY]
FROM [dbo].[ProductsInCategories] AS [t6]
INNER JOIN [dbo].[Category] AS [t7] ON [t7].[CategoryId] = [t6].[CategoryId]
WHERE [t6].[ProductId] = [t0].[ProductId]
) AS [t8]
) AS [value]
FROM [dbo].[Products] AS [t0]
LEFT OUTER JOIN ([dbo].[ProductsInCategories] AS [t1]
INNER JOIN [dbo].[Category] AS [t2] ON [t2].[CategoryId] = [t1].[CategoryId]) ON [t1].[ProductId] = [t0].[ProductId]
WHERE EXISTS(
SELECT NULL AS [EMPTY]
FROM (
SELECT [t4].[ProductId]
FROM (
SELECT ROW_NUMBER() OVER (ORDER BY [t3].[SortOrder], [t3].[Name], [t3].[ProductId]) AS [ROW_NUMBER], [t3].[ProductId]
FROM [dbo].[Products] AS [t3]
WHERE ([t3].[Name] LIKE #p0) OR ([t3].[Description] LIKE #p1)
) AS [t4]
WHERE [t4].[ROW_NUMBER] BETWEEN #p2 + 1 AND #p2 + #p3
) AS [t5]
WHERE [t5].[ProductId] = [t0].[ProductId]
)
ORDER BY [t0].[ProductId], [t1].[CategoryId]',N'#p0 nvarchar(4000),#p1 nvarchar(4000),#p2 int,#p3 int',#p0=N'%product%',#p1=N'%product%',#p2=10,#p3=10
No more extra queries (as SQL Server Profiler said).