We Keep Coding

How to implement Row_Number() over (Partition By Order By) in EntityFramework Core 5?

I have a table and fields as following
Table
T {
RunDate datetime,
A varchar,
B varchar,
C varchar,
D varchar,
E varchar,
}
I want to run the Sql with LINQ
SQL
Select
t.*
From
(
SELECT *,ROW_NUMBER() OVER (PARTITION BY A,B,C,D ORDER BY RunDate DESC) AS RowNumber
From T
Where A == 'xx' && B == 'XX' && RunDate <= 'dddd'
) AS t
Where t.RowNumber=1
I tried
LINQ
T.AsNoTracking()
.Where(x => x.Product == "XXX" && x.ProductSubCode == "AAA" && x.RunDate <= DateTime.Now.AddDays(-1))
.GroupBy(x => new {x.A,x.B,x.C,x.D})
.Select(g => g.OrderByDescending(x => x.RunDate).FirstOrDefault())
.ToList()
This can only run in EF Core 6 but can't run in EF Core 5.
Now I want to know how to implement above SQL in EF Core 5?
And it can run in EF Core 6, but will generate a very complex sql, I also want to know there is a better way to solve this case?
LINQ generated SQL
SELECT [t0].[RunDate], [t0].[A], [t0].[B], [t0].[C], [t0].[D], [t0].[E]
FROM (
SELECT [m].[A], [m].[B], [m].[C], [m].[D]
FROM [T] AS [m]
WHERE (([m].[A] = N'XXX') AND ([m].[B] = N'AAA')) AND ([m].[RunDate] <= DATEADD(day, CAST(0.0E0 AS int), GETDATE()))
GROUP BY [m].[A], [m].[B], [m].[C], [m].[D]
) AS [t]
LEFT JOIN (
SELECT [t1].[RunDate], [t1].[C], [t1].[D], [t1].[E], [t1].[A], [t1].[B]
FROM (
SELECT [m0].[RunDate], [m0].[A], [m0].[B], [m0].[C], [m0].[D], [m0].[E], ROW_NUMBER() OVER(PARTITION BY [m0].[A], [m0].[B], [m0].[C], [m0].[D] ORDER BY [m0].[RunDate] DESC) AS [row]
FROM [T] AS [m0]
WHERE (([m0].[A] = N'XXX') AND ([m0].[B] = N'AAA')) AND ([m0].[RunDate] <= DATEADD(day, CAST(0.0E0 AS int), GETDATE()))
) AS [t1]
WHERE [t1].[row] <= 1
) AS [t0] ON ((([t].[A] = [t0].[A]) AND ([t].[B] = [t0].[B])) AND ([t].[C] = [t0].[C])) AND (([t].[D] = [t0].[D]) OR ([t].[D] IS NULL AND [t0].[D] IS NULL))
GO

Is it possible to convert this SQL query to linq?

I need to count three values on a single table. In plain SQL, it is written like this way:
select
count (*) as num_products,
sum(case when CreatedAt > '{sql.ToSqlDate(_CreatedAfter)}' then 1 else 0 end) num_new,
sum(case when UpdatedAt > '{sql.ToSqlDate(_UpdatedAfter)}' then 1 else 0 end) num_updated
from
Products
While switching to EF Core, I tried to convert it to Linq, like this
var res = (from p in _db.Products
let total = _db.Products.Count()
let NewProducts = _db.Products.Count(s => s.CreatedAt > crDate.Date)
let UpdatedProducts = _db.Products.Count(s => s.UpdatedAt > updDate.Date)
select new { total, NewProducts, UpdatedProducts } );
var response = res.ToList();
but the resulting SQL query seems not optimized
SELECT
(SELECT COUNT(*) FROM [Products] AS [p0]) AS [total],
(SELECT COUNT(*) FROM [Products] AS [s]
WHERE [s].[CreatedAt] > '2019-07-31') AS [NewProducts],
(SELECT COUNT(*) FROM [Products] AS [s0]
WHERE [s0].[UpdatedAt] > '2019-07-01') AS [UpdatedProducts]
FROM
[Products] AS [p]
Maybe somebody can help to translate the original SQL query to linq?
tia
ish

A more literal translation of that query, that generates a query more likely to execute in a single scan of the target table would be:
var q =
from p in db.Products
select new
{
p.Id,
NewProduct = p.CreatedAt > DateTime.Parse("2019-07-31") ? 1 : 0,
UpdatedProduct = p.UpdatedAt > DateTime.Parse("2019-07-01") ? 1 : 0
} into counts
group counts by 1 into grouped
select new
{
ProductCount = grouped.Count(),
NewProductCount = grouped.Sum(r => r.NewProduct),
UpdatedProductCount = grouped.Sum(r => r.UpdatedProduct)
};
Which translates to something like:
SELECT COUNT(*) AS [ProductCount],
SUM([t].[NewProduct]) AS [NewProductCount],
SUM([t].[UpdatedProduct]) AS [UpdatedProductCount]
FROM (
SELECT [p].[Id], CASE
WHEN [p].[CreatedAt] > #__Parse_0
THEN 1 ELSE 0
END AS [NewProduct], CASE
WHEN [p].[UpdatedAt] > #__Parse_1
THEN 1 ELSE 0
END AS [UpdatedProduct], 1 AS [Key]
FROM [Products] AS [p]
) AS [t]
GROUP BY [t].[Key]

You do not need a from clause in your linq because you aren't not going over the rows. just use three statements:
var total = _db.Products.Count();
var NewProducts = _db.Products.Count(s => s.CreatedAt > crDate.Date);
var UpdatedProducts = _db.Products.Count(s => s.UpdatedAt > updDate.Date) ;

Why are multiple where in LINQ so slow?

Using C# and Linq to SQL, I found that my query with multiple where is orders of magnitude slower than with a single where / and.
Here is the query
using (TeradiodeDataContext dc = new TeradiodeDataContext())
{
var filterPartNumberID = 71;
var diodeIDsInBlades = (from bd in dc.BladeDiodes
select bd.DiodeID.Value).Distinct();
var diodesWithTestData = (from t in dc.Tests
join tt in dc.TestTypes on t.TestTypeID equals tt.ID
where tt.DevicePartNumberID == filterPartNumberID
select t.DeviceID.Value).Distinct();
var result = (from d in dc.Diodes
where d.DevicePartNumberID == filterPartNumberID
where diodesWithTestData.Contains(d.ID)
where !diodeIDsInBlades.Contains(d.ID)
orderby d.Name
select d);
var list = result.ToList();
// ~15 seconds
}
However, when the condition in the final query is this
where d.DevicePartNumberID == filterPartNumberID
& diodesWithTestData.Contains(d.ID)
& !diodeIDsInBlades.Contains(d.ID)
// milliseconds
it is very fast.
Comparing the SQL in result before calling ToList(), here are the queries (value 71 manually added in place of #params)
-- MULTIPLE WHERE
SELECT [t0].[ID], [t0].[Name], [t0].[M2MID], [t0].[DevicePartNumberID], [t0].[Comments], [t0].[Hold]
FROM [dbo].[Diode] AS [t0]
WHERE (NOT (EXISTS(
SELECT NULL AS [EMPTY]
FROM (
SELECT DISTINCT [t2].[value]
FROM (
SELECT [t1].[DiodeID] AS [value]
FROM [dbo].[BladeDiode] AS [t1]
) AS [t2]
) AS [t3]
WHERE [t3].[value] = [t0].[ID]
))) AND (EXISTS(
SELECT NULL AS [EMPTY]
FROM (
SELECT DISTINCT [t6].[value]
FROM (
SELECT [t4].[DeviceID] AS [value], [t5].[DevicePartNumberID]
FROM [dbo].[Test] AS [t4]
INNER JOIN [dbo].[TestType] AS [t5] ON [t4].[TestTypeID] = ([t5].[ID])
) AS [t6]
WHERE [t6].[DevicePartNumberID] = (71)
) AS [t7]
WHERE [t7].[value] = [t0].[ID]
)) AND ([t0].[DevicePartNumberID] = 71)
ORDER BY [t0].[Name]
and
-- SINGLE WHERE
SELECT [t0].[ID], [t0].[Name], [t0].[M2MID], [t0].[DevicePartNumberID], [t0].[Comments], [t0].[Hold]
FROM [dbo].[Diode] AS [t0]
WHERE ([t0].[DevicePartNumberID] = 71) AND (EXISTS(
SELECT NULL AS [EMPTY]
FROM (
SELECT DISTINCT [t3].[value]
FROM (
SELECT [t1].[DeviceID] AS [value], [t2].[DevicePartNumberID]
FROM [dbo].[Test] AS [t1]
INNER JOIN [dbo].[TestType] AS [t2] ON [t1].[TestTypeID] = ([t2].[ID])
) AS [t3]
WHERE [t3].[DevicePartNumberID] = (71)
) AS [t4]
WHERE [t4].[value] = [t0].[ID]
)) AND (NOT (EXISTS(
SELECT NULL AS [EMPTY]
FROM (
SELECT DISTINCT [t6].[value]
FROM (
SELECT [t5].[DiodeID] AS [value]
FROM [dbo].[BladeDiode] AS [t5]
) AS [t6]
) AS [t7]
WHERE [t7].[value] = [t0].[ID]
)))
ORDER BY [t0].[Name]
The two SQL queries execute in < 1 second in SSMS and produce the same results.
So I'm wondering why the first is slower on the LINQ side. It's worrying to me because I know I've used multiple where elsewhere, without being aware of a such a severe performance impact.
This question even has answered with both multiple & and where. And this answer even suggests using multiple where clauses.
Can anyone explain why this happens in my case?

Because writing like this
if (someParam1 != 0)
{
myQuery = myQuery.Where(q => q.SomeField1 == someParam1)
}
if (someParam2 != 0)
{
myQuery = myQuery.Where(q => q.SomeField2 == someParam2)
}
is NOT(upd) the same as (in case when someParam1 and someParam2 != 0)
myQuery = from t in Table
where t.SomeField1 == someParam1
&& t.SomeField2 == someParam2
select t;
is (NOT deleted) the same as
myQuery = from t in Table
where t.SomeField1 == someParam1
where t.SomeField2 == someParam2
select t;
UPD
Yes, I do mistake. Second query is same, first is not same.
First and Second queries not EXACTLY the same. Let me show you what I mean.
1st query with lamda-expression writen as
t.Where(r => t.SomeField1 == someParam1 && t.SomeField2 == someParam2)
2nd query as
t.Where(r => r.SomeField1 == someParam1).Where(r => r.SomeField2 == someParam2)
In this case in generated SQL Predicate with SomeField2 goes first (it is important, see below)
In 1st case we getting this SQL:
SELECT <all field from Table>
FROM table t
WHERE t.SomeField1 = :someParam1
AND t.SomeField2 = :someParam2
In 2 case the SQL is:
SELECT <all field from Table>
FROM table t
WHERE t.SomeField2 = :someParam2
AND t.SomeField1 = :someParam1
As we see there are 2 'same' SQLs. As we see, the OP's SQLs are also 'same', they are different in order of predicates in WHERE clause (as in my example). And I guess that SQL optimizer generate 2 different execution plans and may be(!!!) doing NOT EXISTS, then EXISTS and then filtering take more time than do first filtering and after that do EXISTS and NOT EXISTS
UPD2
It is a 'problem' of Linq Provider (ORM). I'm using another ORM (linq2db), and it generates for me EXACTLY the same SQLs in both cases.

Convert T-SQL to Fluent Linq C# with the same SQL generated output

I've the following schema in my database:
The point is to have for each User, his last read datetime per Thread.
I can also check this dates by Forums & Universes with and aggregation if needed.
A Forum is considered Read, if all Thread are Read.
A Thread is considered Read, if the User's Lastview is more or equal to the last created post date (from Post.CreatedAt).
I've made the following T-SQL request to achieve this goals per Forums:
SELECT
F.Id,
CASE WHEN SUM(V.IsRead) = COUNT(V.IsRead) THEN 1 ELSE 0 END AS IsRead
FROM Forum F
LEFT JOIN Thread T ON T.Id_Forum = F.Id
LEFT JOIN
(
SELECT
P.Id_Thread,
CASE WHEN MAX(P.CreatedAt) < MAX(V.LastView) THEN 1 ELSE 0 END AS IsRead
FROM Post P
INNER JOIN Thread T ON P.Id_Thread = T.Id
INNER JOIN Forum F ON T.Id_Forum = F.Id
LEFT JOIN Thread_View V ON P.Id_Thread = V.Id_Thread AND V.Id_User = #Id_User
WHERE F.Id_Universe = #Id_Universe
GROUP BY P.Id_Thread
) V ON T.Id = V.Id_Thread
WHERE F.Id_Universe = #Id_Universe
GROUP BY F.Id
ORDER BY F.Id
It works perferctly, BUT, i now want to create this request from Linq using EntityFramework... And I'm stuck...
Here is what I've made, but it generates a very much complicated query, and i'm afraid about performance...
var viewsQuery = context.Posts
.Where(p => p.Thread.Forum.Id_Universe == idUniverse)
.GroupJoin
(
context.Thread_Views.Where(v => v.Id_User == idUser),
p => p.Id_Thread,
v => v.Id_Thread,
(p, v) => new { Id_Thread = p.Id_Thread, Id_Forum = p.Thread.Id_Forum, CreatedAt = p.CreatedAt, LastView = v.Max(_v => _v.LastView) }
)
.Select(r => new { Id_Thread = r.Id_Thread, Id_Forum = r.Id_Forum, IsRead = r.CreatedAt < r.LastView });
var forumQuery = context.Forums.Where(f => f.Id_Universe == idUniverse).GroupJoin
(
viewsQuery.DefaultIfEmpty(),
forum => forum.Id,
view => view.Id_Forum,
(forum, views) => new
{
Forum = forum.Id,
IsRead = views.Any() && views.All(v => v.IsRead),
}
);
The SQL output generated by linq is this (slightly edited to be more readable), and it's so ugly...
SELECT
[Project1].[Id] AS [Id],
CASE WHEN
(
(
EXISTS
(
SELECT
1 AS [C1]
FROM ( SELECT 1 AS X ) AS [SingleRowTable1]
INNER JOIN
(
SELECT [Extent3].[Id_Forum] AS [Id_Forum], [Extent4].[Id_Universe] AS [Id_Universe]
FROM [dbo].[Post] AS [Extent2]
INNER JOIN [dbo].[Thread] AS [Extent3] ON [Extent2].[Id_Thread] = [Extent3].[Id]
INNER JOIN [dbo].[Forum] AS [Extent4] ON [Extent3].[Id_Forum] = [Extent4].[Id] ) AS [Join2] ON 1 = 1
WHERE ([Join2].[Id_Universe] = 3) AND ([Project1].[Id] = [Join2].[Id_Forum])
)
)
AND
(
NOT EXISTS
(
SELECT
1 AS [C1]
FROM ( SELECT 1 AS X ) AS [SingleRowTable2]
INNER JOIN
(
SELECT
[Project3].[CreatedAt] AS [CreatedAt],
[Project3].[Id_Forum] AS [Id_Forum],
(
SELECT MAX([Extent8].[LastView]) AS [A1]
FROM [dbo].[Thread_View] AS [Extent8]
WHERE ([Extent8].[Id_User] = 79775) AND ([Project3].[Id_Thread] = [Extent8].[Id_Thread])
) AS [C1]
FROM
(
SELECT
[Extent5].[Id_Thread] AS [Id_Thread],
[Extent5].[CreatedAt] AS [CreatedAt],
[Extent6].[Id] AS [Id],
[Extent6].[Id_Forum] AS [Id_Forum]
FROM [dbo].[Post] AS [Extent5]
INNER JOIN [dbo].[Thread] AS [Extent6] ON [Extent5].[Id_Thread] = [Extent6].[Id]
INNER JOIN [dbo].[Forum] AS [Extent7] ON [Extent6].[Id_Forum] = [Extent7].[Id]
WHERE [Extent7].[Id_Universe] = 3
) AS [Project3]
) AS [Project4] ON 1 = 1
WHERE
(
(
(
CASE
WHEN ([Project4].[CreatedAt] < [Project4].[C1]) THEN cast(1 as bit)
WHEN ( NOT ([Project4].[CreatedAt] < [Project4].[C1])) THEN cast(0 as bit)
END
) <> 1
)
OR
(
CASE
WHEN ([Project4].[CreatedAt] < [Project4].[C1]) THEN cast(1 as bit)
WHEN ( NOT ([Project4].[CreatedAt] < [Project4].[C1])) THEN cast(0 as bit)
END IS NULL
)
) AND ([Project1].[Id] = [Project4].[Id_Forum])
)
)
) THEN cast(1 as bit)
ELSE cast(0 as bit) END AS [C1]
FROM
(
SELECT
[Extent1].[Id] AS [Id]
FROM [dbo].[Forum] AS [Extent1]
WHERE [Extent1].[Id_Universe] = 3
) AS [Project1]
EDIT : Found a working linq query with the same result, but it is so ugly i'm very afraid about performance...

var forumsWhichAreUnread = from forum on context.Forums
let isUnRead = forum.Threads.All(thread =>
thread.ThreadViews
.Where(view => view.UserId = userId)
.Max(view => view.Lastview)
<
thread.Posts.Max(post => post.CreatedAt))
select new {forum, isUnRead};
Translation
Get each forum in context.Forums
Only give me forums where every single thread...
the largest threadview.LastView time is smaller than the Latest Post Date.
Select out those forum.

LINQ Where in MAX

I currently have this linq statement:
from s in SubContentRevisions
where s.SubContentID.Equals("e3f319f1-65cc-4799-b84d-309941dbc1da")
&& s.RevisionNumber == (SubContentRevisions.Max(s1 => s1.RevisionNumber))
select s
which generates this SQL (according to LINQPad):
-- Region Parameters
DECLARE #p0 UniqueIdentifier = 'e3f319f1-65cc-4799-b84d-309941dbc1da'
-- EndRegion
SELECT [t0].[SubContentRevisionID], [t0].[SubContentID], [t0].[RevisionNumber], [t0].[RevisionText], [t0].[CreatedDate], [t0].[ModifiedDate]
FROM [SubContentRevision] AS [t0]
WHERE ([t0].[SubContentID] = #p0) AND ([t0].[RevisionNumber] = ((
SELECT MAX([t1].[RevisionNumber])
FROM [SubContentRevision] AS [t1]
)))
How can I make it generate this SQL statement? I can't seem to find anything related anywhere. (I need it to add the where clause to the subquery)
-- Region Parameters
DECLARE #p0 UniqueIdentifier = 'e3f319f1-65cc-4799-b84d-309941dbc1da'
-- EndRegion
SELECT [t0].[SubContentRevisionID], [t0].[SubContentID], [t0].[RevisionNumber], [t0].[RevisionText], [t0].[CreatedDate], [t0].[ModifiedDate]
FROM [SubContentRevision] AS [t0]
WHERE ([t0].[SubContentID] = #p0) AND ([t0].[RevisionNumber] = ((
SELECT MAX([t1].[RevisionNumber])
FROM [SubContentRevision] AS [t1]
WHERE [SubContentID] = #p0 -- **********Adds the where clause**********
)))

I think you want:
from s in SubContentRevisions
where s.SubContentID.Equals("e3f319f1-65cc-4799-b84d-309941dbc1da")
&& s.RevisionNumber == (SubContentRevisions.Where(s.SubContentID.Equals("..."))
.Max(s1 => s1.RevisionNumber))
select s
Or, more clearly:
var specificSubContents = SubContentRevisions.Where(s =>
s.SubContentID.Equals("e3f319f1-65cc-4799-b84d-309941dbc1da")
var query = from s in specificSubContents
where s.RevisionNumber = s.Max(s1 => s1.RevisionNumber)
select s;
Alternatively, it sounds like you could actually do:
var latest = (from s in SubContentRevisions
where s.SubContentID.Equals("e3f319f1-65cc-4799-b84d-309941dbc1da")
orderby s.RevisionNumber descending
select s).FirstOrDefault();

How about adding the where clause to the subquery (max):
from s in SubContentRevisions
where s.SubContentID.Equals("e3f319f1-65cc-4799-b84d-309941dbc1da")
&& s.RevisionNumber == (SubContentRevisions
.Where(s1 => s1.SubContentID.Equals(s.SubContentID))
.Max(s1 => s1.RevisionNumber))
select s