We Keep Coding

Improving LINQ query for many-to-many relation

I have a database with the following schema:
Now, I'm trying to pull all landingpages for a domain and sort those by the first UrlFilter's FilterType that matches a certain group. This is the LINQ I've come up with so far:
var baseQuery = DbSet.AsNoTracking()
.Where(e => EF.Functions.Contains(EF.Property<string>(e, "Url"), $"\"{searchTerm}*\""))
.Where(e => e.DomainLandingPages.Select(lp => lp.DomainId).Contains(domainId));
var count = baseQuery.Count();
var page = baseQuery
.Select(e => new
{
LandingPage = e,
UrlFilter = e.LandingPageUrlFilters.FirstOrDefault(f => f.UrlFilter.GroupId == groupId)
})
.Select(e => new
{
e.LandingPage,
FilterType = e.UrlFilter == null ? UrlFilterType.NotCovered : e.UrlFilter.UrlFilter.UrlFilterType
})
.OrderBy(e => e.FilterType)
.Skip(10).Take(75).ToList();
Now, while this technically works, it's quite slow with execution times ranging from 10-30 seconds, which is not good enough for the use case. The LINQ is translated to the following SQL:
SELECT [l1].[Id], [l1].[LastUpdated], [l1].[Url], CASE
WHEN (
SELECT TOP(1) [l].[LandingPageId]
FROM [LandingPageUrlFilters] AS [l]
INNER JOIN [UrlFilters] AS [u] ON [l].[UrlFilterId] = [u].[Id]
WHERE ([l1].[Id] = [l].[LandingPageId]) AND ([u].[GroupId] = #__groupId_3)) IS NULL THEN 4
ELSE (
SELECT TOP(1) [u0].[UrlFilterType]
FROM [LandingPageUrlFilters] AS [l0]
INNER JOIN [UrlFilters] AS [u0] ON [l0].[UrlFilterId] = [u0].[Id]
WHERE ([l1].[Id] = [l0].[LandingPageId]) AND ([u0].[GroupId] = #__groupId_3))
END AS [FilterType]
FROM [LandingPages] AS [l1]
WHERE CONTAINS([l1].[Url], #__Format_1) AND #__domainId_2 IN (
SELECT [d].[DomainId]
FROM [DomainLandingPages] AS [d]
WHERE [l1].[Id] = [d].[LandingPageId]
)
ORDER BY CASE
WHEN (
SELECT TOP(1) [l2].[LandingPageId]
FROM [LandingPageUrlFilters] AS [l2]
INNER JOIN [UrlFilters] AS [u1] ON [l2].[UrlFilterId] = [u1].[Id]
WHERE ([l1].[Id] = [l2].[LandingPageId]) AND ([u1].[GroupId] = #__groupId_3)) IS NULL THEN 4
ELSE (
SELECT TOP(1) [u2].[UrlFilterType]
FROM [LandingPageUrlFilters] AS [l3]
INNER JOIN [UrlFilters] AS [u2] ON [l3].[UrlFilterId] = [u2].[Id]
WHERE ([l1].[Id] = [l3].[LandingPageId]) AND ([u2].[GroupId] = #__groupId_3))
END
OFFSET #__p_4 ROWS FETCH NEXT #__p_5 ROWS ONLY
Now my question is, how can I improve the execution time of this? Either by SQL or LINQ
EDIT: So I've been tinkering with some raw SQL and this is what I've come up with:
with matched_urls as (
select l.id, min(f.urlfiltertype) as Filter
from landingpages l
join landingpageurlfilters lpf on lpf.landingpageid = l.id
join urlfilters f on lpf.urlfilterid = f.id
where f.groupid = #groupId
and contains(Url, '"barz*"')
group by l.id
) select l.id, 5 as Filter
from landingpages l
where #domainId in (
select domainid
from domainlandingpages dlp
where l.id = dlp.landingpageid
) and l.id not in (select id from matched_urls ) and contains(Url, '"barz*"')
union select * from matched_urls
order by Filter
offset 10 rows fetch next 30 rows only
This performs somewhat okay, cutting the execution time down to ~5 seconds. As this is to be used for a table search I would however like to get it down even further. Is there any way to improve this SQL?

You're right to have a look at the generated SQL. In general, I would advise to learn SQL, write a performing SQL query and work your way back (either use a stored procedure or raw SQL, or design your LINQ query with that same philosophy.
I suspect this will be better (not tested):
var page = (
from e in baseQuery
let urlFilter = e.LandingPageUrlFilters.OrderBy(f => f.UrlFilterType).FirstOrDefault(f => f.UrlFilter.GroupId == groupId)
let filterType = urlFilter == null ? UrlFilterType.NotCovered : e.UrlFilter.UrlFilter.UrlFilterType
select new
{
LandingPage = e,
FilterType = filterType
}
).Skip(10).Take(75).ToList();

one of the way to improve the execution time is see execution plan in SSMS (SQL Server Management Studio).
After look on the execution plan you can design some indexes, or if you have no experiences with this, you can see if SSMS recommends some indexes.
Next try to create the indexes and execute the query again and see if execution time was improved.
Note: this is only one of many possible ways to improve execution time...

Linq slow while SQL fast

When I'm trying the next linq query, it is slow (1.5 s):
var rslt = (from t in context.Set<SUB_Transactions>()
where
t.UpdateDate > query.LastUpdate &&
t.TransactionID > query.Index
select new
{
TransactionID = t.TransactionID
}).OrderBy(t => t.TransactionID).Take(query.Amount).ToList();
When converted to SQL, this query is super fast (40 ms):
SELECT TOP (300)
[Project1].[TransactionID] AS [TransactionID]
FROM ( SELECT
[Extent1].[TransactionID] AS [TransactionID]
FROM [dbo].[SUB_Transactions] AS [Extent1]
WHERE ([Extent1].[UpdateDate] > #p__linq__0) AND ([Extent1].[TransactionID] > #p__linq__1)
) AS [Project1]
ORDER BY [Project1].[TransactionID] ASC
What is going on here?
Removing the Take in the first query gives a fast result as well (given the fast that there are no new transactions)
There is a composite index on TransactionID and UpdateDate.

As far as your example goes, this could make your query faster but without some sample data it's not possible to test from our end:
var rslt = (from t in context.Set<SUB_Transactions>()
where t.TransactionID > query.Index // invert order of filter
&& t.UpdateDate > query.LastUpdate
order by t.TransactionID // you can orderby here
select t.TransactionID) // remove anonymous object
.Take(query.Amount)
.AsNoTracking() // you won't be changing IDs so no need to track them
.ToList();
You might also gain some performance if you used the DbSet<SUB_Transactions> instead of calling Set<SUB_Transactions>() which needs to locate the DbSet in the DbContext

Order by descending by row field in query

I want to write a EF query which does order by ascending or descending based on condition. Following is the my pseudo code:
var result= q.OrderByDescending(x => x.StatusId == 3)
if( x.StatusId == 3)
then order by x.ReserveDate
if( x.StatusId != 3 )
then order by descending x.LastUpdateDate
How can i do this?
Update
This is not same as q = condition ? q.OrderBy(..) : q.OrderByDescending(..) as marked in referenced duplicate question, sorting order differs based on value within the row instead of a flag outside query.

You can supply complex expressions in OrderBy...
// you might have to give bounding start,end for
// for this query to work correctly...
var end = DateTime.Now;
var start = end.AddYears(-100);
var result = q.OrderBy(
x => x.StatusId == 3 ?
// this will sort by in ascending order
DbFunctions.DiffDays(x.ReserveDate, start) :
// this will sort in descending order
DbFunctions.DiffDays(end, x.LastUpdateDate) );
SQL Generated will be
SELECT
...
...
FROM ( SELECT CASE
WHEN ([Extent2].[StatusId ] = 3)
THEN DATEDIFF (day, #p__linq__0, [Extent1].[ReserveDate])
ELSE
DATEDIFF (day, [Extent1].[LastUpdateDate], #p__linq__1)
END AS [C1]
FROM [dbo].[Table] AS [Extent1]
) AS [Project1]
ORDER BY [Project1].[C1]

As BoessB's comment has it, this is easiest with two concatenated queries:
var q1 = from x in source
where x.StatusId == 3
order by x.ReserveDate;
var q2 = from x in source
where x.StatusId != 3
order by x.LastUpdateDate descending;
var results = await q1.Concat(q2).ToListAsync();
It might be possible to do in a single expression if you can create a derived field (let clauses help) from ReserveDate and LastUpdateDate that sorts in the right way. But I would suggest splitting the query will be clearer.

Performing two queries in a single round trip to the database

I have the following code to perform a full-text search. It creates a query, gets the total number of rows returned by that query and then retrieves the actual rows for only the current page.
// Create IQueryable
var query = from a in ArticleServerContext.Set<Article>()
where a.Approved
orderby a.UtcDate descending
select a;
// Get total rows (needed for pagination logic)
int totalRows = query.Count()
// Get rows for current page
query = query.Skip((CurrentPage - 1) * RowsPerPage).Take(RowsPerPage);
This works fine, but it requires two round trips to the database. In the interest of optimizing the code, is there any way to rework this query so it only had one round trip to the database?

Yes, you can perform this two operations with the help of the only one query to database:
// Create IQueryable
var query = from a in ArticleServerContext.Set<Article>()
where a.Approved
orderby a.UtcDate descending
select new { a, Total = ArticleServerContext.Set<Article>().Where(x => x.Approved).Count() };
//Get raw rows for current page with Total(Count) field
var result = query.Skip((CurrentPage - 1) * RowsPerPage).Take(RowsPerPage).ToList();
//this data you actually will use with your logic
var actualData = result.Select(x => x.a).ToList();
// Get total rows (needed for pagination logic)
int totalRows = result.First().Total;
If you use MSSQL query wil be look that way:
SELECT
[Extent1].[ID] AS [ID],
[Extent1].[UtcDate] AS [UtcDate],
[Extent1].[Approved] AS [Approved],
[GroupBy1].[A1] AS [C1]
FROM [dbo].[Articles] AS [Extent1]
CROSS JOIN (SELECT
COUNT(1) AS [A1]
FROM [dbo].[Articles] AS [Extent2]
WHERE [Extent2].[Approved] ) AS [GroupBy1]
WHERE [Extent1].[Approved]
ORDER BY [Extent1].[UtcDate] DESC

I'm not sure whether it's worth enough, but it's doable under the following constraints:
(1) CurrentPage and RowsPerPage are not affected by the totalRows value.
(2) The query is materialized after applying the paging parameters.
The trick is to use group by constant value, which is supported by EF. The code looks like this:
var query =
from a in ArticleServerContext.Set<Article>()
where a.Approved
// NOTE: order by goes below
group a by 1 into allRows
select new
{
TotalRows = allRows.Count(),
PageRows = allRows
.OrderByDescending(a => a.UtcDate)
.Skip((CurrentPage - 1) * RowsPerPage).Take(RowsPerPage)
};
var result = query.FirstOrDefault();
var totalRows = result != null ? result.TotalRows : 0;
var pageRows = result != null ? result.PageRows : Enumerable.Empty<Article>();

Linq2Sql: Get every N'th row [duplicate]

Anybody know how to write a LINQ to SQL statement to return every nth row from a table? I'm needing to get the title of the item at the top of each page in a paged data grid back for fast user scanning. So if i wanted the first record, then every 3rd one after that, from the following names:
Amy, Eric, Jason, Joe, John, Josh, Maribel, Paul, Steve, Tom
I'd get Amy, Joe, Maribel, and Tom.
I suspect this can be done... LINQ to SQL statements already invoke the ROW_NUMBER() SQL function in conjunction with sorting and paging. I just don't know how to get back every nth item. The SQL Statement would be something like WHERE ROW_NUMBER MOD 3 = 0, but I don't know the LINQ statement to use to get the right SQL.

Sometimes, TSQL is the way to go. I would use ExecuteQuery<T> here:
var data = db.ExecuteQuery<SomeObjectType>(#"
SELECT * FROM
(SELECT *, ROW_NUMBER() OVER (ORDER BY id) AS [__row]
FROM [YourTable]) x WHERE (x.__row % 25) = 1");
You could also swap out the n:
var data = db.ExecuteQuery<SomeObjectType>(#"
DECLARE #n int = 2
SELECT * FROM
(SELECT *, ROW_NUMBER() OVER (ORDER BY id) AS [__row]
FROM [YourTable]) x WHERE (x.__row % #n) = 1", n);

Once upon a time, there was no such thing as Row_Number, and yet such queries were possible. Behold!
var query =
from c in db.Customers
let i = (
from c2 in db.Customers
where c2.ID < c.ID
select c2).Count()
where i%3 == 0
select c;
This generates the following Sql
SELECT [t2].[ID], [t2]. --(more fields)
FROM (
SELECT [t0].[ID], [t0]. --(more fields)
(
SELECT COUNT(*)
FROM [dbo].[Customer] AS [t1]
WHERE [t1].[ID] < [t0].[ID]
) AS [value]
FROM [dbo].[Customer] AS [t0]
) AS [t2]
WHERE ([t2].[value] % #p0) = #p1

Here's an option that works, but it might be worth checking that it doesn't have any performance issues in practice:
var nth = 3;
var ids = Table
.Select(x => x.Id)
.ToArray()
.Where((x, n) => n % nth == 0)
.ToArray();
var nthRecords = Table
.Where(x => ids.Contains(x.Id));

Just googling around a bit I haven't found (or experienced) an option for Linq to SQL to directly support this.
The only option I can offer is that you write a stored procedure with the appropriate SQL query written out and then calling the sproc via Linq to SQL. Not the best solution, especially if you have any kind of complex filtering going on.

There really doesn't seem to be an easy way to do this:
How do I add ROW_NUMBER to a LINQ query or Entity?
How to find the ROW_NUMBER() of a row with Linq to SQL
But there's always:
peopleToFilter.AsEnumerable().Where((x,i) => i % AmountToSkipBy == 0)
NOTE: This still doesn't execute on the database side of things!

This will do the trick, but it isn't the most efficient query in the world:
var count = query.Count();
var pageSize = 10;
var pageTops = query.Take(1);
for(int i = pageSize; i < count; i += pageSize)
{
pageTops = pageTops.Concat(query.Skip(i - (i % pageSize)).Take(1));
}
return pageTops;
It dynamically constructs a query to pull the (nth, 2*nth, 3*nth, etc) value from the given query. If you use this technique, you'll probably want to create a limit of maybe ten or twenty names, similar to how Google results page (1-10, and Next), in order to avoid getting an expression so large the database refuses to attempt to parse it.
If you need better performance, you'll probably have to use a stored procedure or a view to represent your query, and include the row number as part of the stored proc results or the view's fields.