We Keep Coding

Improving LINQ query for many-to-many relation

I have a database with the following schema:
Now, I'm trying to pull all landingpages for a domain and sort those by the first UrlFilter's FilterType that matches a certain group. This is the LINQ I've come up with so far:
var baseQuery = DbSet.AsNoTracking()
.Where(e => EF.Functions.Contains(EF.Property<string>(e, "Url"), $"\"{searchTerm}*\""))
.Where(e => e.DomainLandingPages.Select(lp => lp.DomainId).Contains(domainId));
var count = baseQuery.Count();
var page = baseQuery
.Select(e => new
{
LandingPage = e,
UrlFilter = e.LandingPageUrlFilters.FirstOrDefault(f => f.UrlFilter.GroupId == groupId)
})
.Select(e => new
{
e.LandingPage,
FilterType = e.UrlFilter == null ? UrlFilterType.NotCovered : e.UrlFilter.UrlFilter.UrlFilterType
})
.OrderBy(e => e.FilterType)
.Skip(10).Take(75).ToList();
Now, while this technically works, it's quite slow with execution times ranging from 10-30 seconds, which is not good enough for the use case. The LINQ is translated to the following SQL:
SELECT [l1].[Id], [l1].[LastUpdated], [l1].[Url], CASE
WHEN (
SELECT TOP(1) [l].[LandingPageId]
FROM [LandingPageUrlFilters] AS [l]
INNER JOIN [UrlFilters] AS [u] ON [l].[UrlFilterId] = [u].[Id]
WHERE ([l1].[Id] = [l].[LandingPageId]) AND ([u].[GroupId] = #__groupId_3)) IS NULL THEN 4
ELSE (
SELECT TOP(1) [u0].[UrlFilterType]
FROM [LandingPageUrlFilters] AS [l0]
INNER JOIN [UrlFilters] AS [u0] ON [l0].[UrlFilterId] = [u0].[Id]
WHERE ([l1].[Id] = [l0].[LandingPageId]) AND ([u0].[GroupId] = #__groupId_3))
END AS [FilterType]
FROM [LandingPages] AS [l1]
WHERE CONTAINS([l1].[Url], #__Format_1) AND #__domainId_2 IN (
SELECT [d].[DomainId]
FROM [DomainLandingPages] AS [d]
WHERE [l1].[Id] = [d].[LandingPageId]
)
ORDER BY CASE
WHEN (
SELECT TOP(1) [l2].[LandingPageId]
FROM [LandingPageUrlFilters] AS [l2]
INNER JOIN [UrlFilters] AS [u1] ON [l2].[UrlFilterId] = [u1].[Id]
WHERE ([l1].[Id] = [l2].[LandingPageId]) AND ([u1].[GroupId] = #__groupId_3)) IS NULL THEN 4
ELSE (
SELECT TOP(1) [u2].[UrlFilterType]
FROM [LandingPageUrlFilters] AS [l3]
INNER JOIN [UrlFilters] AS [u2] ON [l3].[UrlFilterId] = [u2].[Id]
WHERE ([l1].[Id] = [l3].[LandingPageId]) AND ([u2].[GroupId] = #__groupId_3))
END
OFFSET #__p_4 ROWS FETCH NEXT #__p_5 ROWS ONLY
Now my question is, how can I improve the execution time of this? Either by SQL or LINQ
EDIT: So I've been tinkering with some raw SQL and this is what I've come up with:
with matched_urls as (
select l.id, min(f.urlfiltertype) as Filter
from landingpages l
join landingpageurlfilters lpf on lpf.landingpageid = l.id
join urlfilters f on lpf.urlfilterid = f.id
where f.groupid = #groupId
and contains(Url, '"barz*"')
group by l.id
) select l.id, 5 as Filter
from landingpages l
where #domainId in (
select domainid
from domainlandingpages dlp
where l.id = dlp.landingpageid
) and l.id not in (select id from matched_urls ) and contains(Url, '"barz*"')
union select * from matched_urls
order by Filter
offset 10 rows fetch next 30 rows only
This performs somewhat okay, cutting the execution time down to ~5 seconds. As this is to be used for a table search I would however like to get it down even further. Is there any way to improve this SQL?

You're right to have a look at the generated SQL. In general, I would advise to learn SQL, write a performing SQL query and work your way back (either use a stored procedure or raw SQL, or design your LINQ query with that same philosophy.
I suspect this will be better (not tested):
var page = (
from e in baseQuery
let urlFilter = e.LandingPageUrlFilters.OrderBy(f => f.UrlFilterType).FirstOrDefault(f => f.UrlFilter.GroupId == groupId)
let filterType = urlFilter == null ? UrlFilterType.NotCovered : e.UrlFilter.UrlFilter.UrlFilterType
select new
{
LandingPage = e,
FilterType = filterType
}
).Skip(10).Take(75).ToList();

one of the way to improve the execution time is see execution plan in SSMS (SQL Server Management Studio).
After look on the execution plan you can design some indexes, or if you have no experiences with this, you can see if SSMS recommends some indexes.
Next try to create the indexes and execute the query again and see if execution time was improved.
Note: this is only one of many possible ways to improve execution time...

How to write this SQL query as a LINQ statement in .NET Core (C#)?

I'm trying to join three tables together, but I just can't get my LINQ statements to get the data I want to. My SQL query that I want to replicate is like this:
SELECT TOP 5 SUM(Grade) AS 'TotalGrade', COUNT(Restaurants.Name) AS 'NumberOfVisits', Restaurants.Name FROM Lunches
JOIN dbo.LunchRestaurant ON Lunches.LunchId = LunchRestaurant.LunchId
JOIN Restaurants ON Restaurants.RestaurantId = LunchRestaurant.RestaurantId
GROUP BY Restaurants.Name
The LINQ statement I currently have is:
var q = from l in lunchContext.Lunches
join lr in lunchContext.LunchRestaurant on l.LunchId equals lr.LunchId
join r in lunchContext.Restaurants on lr.RestaurantId equals r.RestaurantId
select new { l.Grade, r.RestaurantId};
But with this one, I can't get in my group by statement or the aggregate functions no matter how I try. Do you have any suggestions on what to do? There also doesn't seem to be a way to write raw SQL statements when dealing with several tables, at least not easily done in EF Core if I'm not mistaken. Otherwise that'd be an option too.

I think you're looking for linq grouping. From memory, it would be something like;
var q = from l in lunchContext.Lunches
join lr in lunchContext.LunchRestaurant on l.LunchId equals lr.LunchId
join r in lunchContext.Restaurants on lr.RestaurantId equals r.RestaurantId
group l by lr.RestaurantId into g
select new { Id = g.Key, Grade =g.Sum(x=>x.Grade)};
If you do need to go down the raw SQL route then you can levarage ADO.Net like this
using (var command = lunchContext.Database.GetDbConnection().CreateCommand())
{
command.CommandText = "{Your sql query here}";
context.Database.OpenConnection();
using (var result = command.ExecuteReader())
{
// do something with result
}
}

Cannot make 5 consecutive "joins" in LINQ?

I have this rather complex query in SQL server 2008:
declare #LanguageID as int = 1
select k.datePublish, k.dateEditing, k.dateTables
from TableAreasLevel1 as areaL1
inner join TableAreasLevel2 as areaL2
on areaL1.LanguageID = areaL2.LanguageID and
areaL1.CodeAreaLevel1 = areaL2.CodeAreaLevel1
inner join TableAreasLink as link
on areaL2.CodeAreaLevel1 = link.CodeAreaLevel1 and
areaL2.CodeAreaLevel2 = link.CodeAreaLevel2 and
inner join TableProducts as tblProds
on tblProds.CodeAreaLevel1 = areaL1.CodeAreaLevel1 and
tblProds.CodeAreaLevel2 = areaL2.CodeAreaLevel2
inner join TableSI_Products as prod
on prod.SiAreaCode = link.SiAreaCode
inner join TableCalendar as k
on k.KodTableSI_Products = tblProds.KodTableSI_Products
where areaL1.LanguageID = #LanguageID and
prod.Code = 'some string' and
k.LanguageID = #LanguageID and
tblProds.LanguageID = #LanguageID;
I am trying to develop the same query in LINQ, but I get error when I try join the table TableProducts, i.e the third consecutive join.
Here is my LINQ query:
List<Tuple<DateTime, DateTime, DateTime>> dates = (from areaL1 in gpe.TableAreasLevel1
join areaL2 in gpe.TableAreasLevel2
on new { areaL1.CodeAreaLevel1, areaL1.LanguageID } equals
new { areaL2.CodeAreaLevel1, areaL2.LanguageID }
join link in gpe.TableAreasLink
on new { areaL2.CodeAreaLevel1, areaL2.CodeAreaLevel2, areaL2.RbrOblastNivo2} equals
new {link.CodeAreaLevel1, link.CodeAreaLevel2}
join tblProds in gpe.TableProducts
on tblProds. // The name tblProds is not in the scope of the left side of 'equals'
);
Is the problem connected with how the tables are designed or, something else I should check for?
Any ideas, why tblProds is not visible in the scope of the LINQ query?

You are using Query as your guide something like:
on k.KodTableSI_Products = tblProds.KodTableSI_Products
but take a look at your linq:
on new { areaL1.CodeAreaLevel1, areaL1.LanguageID } equals
it has two fields. I think its not a good idea.
linq join something like
var dates = (from areaL1 in gpe.TableAreasLevel1
join areaL2 in gpe.TableAreasLevel2
on areaL1.PKFields equals areaL2.PKFields
where areaL1.CodeAreaLevel1== areaL2.CodeAreaLevel1 && areaL1.LanguageID = areaL2.LanguageID
Select new YournewClass{YournewClass.Field1=areaL1.fields1, And so on}
)
You can do to join the other tables with aliases.
Sorry need to go for now.
I'm giving you an idea.
Hope it helps.

LINQ Group by and having where clause

Below is the SQL Query I am trying to translate
SELECT dbo.Contracts.Supplier
FROM dbo.Contracts INNER JOIN dbo.Products ON dbo.Contracts.Product = dbo.Products.Product
where dbo.Products.ProductGroup='Crude'
GROUP BY dbo.Contracts.Supplier
Am I doing something wrong because I do not get same results with the following LINQ
var result = from c in context.Contracts
join p in context.Products on c.Product equals p.Product1
where p.Product1.Equals("Crude")
group c by c.Supplier into g
select new { supplier = g.Key };
It is generating a weird statement
SELECT
1 AS [C1],
[Distinct1].[Supplier] AS [Supplier]
FROM ( SELECT DISTINCT
[Extent1].[Supplier] AS [Supplier]
FROM [dbo].[Contracts] AS [Extent1]
WHERE N'Crude' = [Extent1].[Product]
) AS [Distinct1]
Using distinct would work but to get same results, LINQ should be generating a statement like so (it's like it is ignoring the join):
SELECT distinct dbo.Contracts.Supplier
FROM dbo.Contracts INNER JOIN dbo.Products ON dbo.Contracts.Product = dbo.Products.Product
where dbo.Products.ProductGroup='Crude'

I'm assuming that you are using 'EntityFramework' or 'Linq To SQL'. If so, you should be able to use navigation properties to navigate to product and filter invalit results out. This way your query might look something like this:
var result = (from c in context.Contracts
where c.Products.Any(p => p.ProductGroup == "Crude")
select c.Supplier).Distinct();
It will automatically convert into correct query (in this case possibly without join even, just using Exists sql keyword) and return distinct suppliers. This is if I understand your objective correctly - you want to obtain all suppliers assigned to contracts that contain product from 'Crude' product group.
Basically you should try to avoid using joins from linq to sql or linq to entities as much as possible when you can use navigation properties. System will probably be better at converting them into specific sql.

LINQ2SQL - More Questions on when a Cross join with where clause is emitted instead of Inner Join

This is a two part question and for education purposes rather than trying to find a solution to a problem.
I've seen this already and realize that it's very similar to my question
LINQ2SQL - Cross join emitted when I want inner join
But I am hoping for more information from you LINQ and SQL gurus as to why the cross join is created instead of inner join in LINQ2SQL. Additionally, can someone explain how SQL Server decides on the execution plan (or link to further information) since both of these queries generate the same plan? From what I understand, this means that the performance of the queries are the same.
I've created a small example that runs two LINQ expressions on my database that generates these two different SQL queries.
For those who don't want to bother, here's my example db diagram:
http://dl.dropbox.com/u/13256/Screen%20shot%202011-03-16%20at%2011.41.56%20AM.png
Here are the two queries:
Cross Join with Where Clause
var q = from item in context.Items
join i_mem in context.Memberships on new { item_id = item.ID, user_id =
current_user_id.Value } equals new { item_id = i_mem.RelatedItemID, user_id =
i_mem.RelatedUserID } into sq_i_m
from im in sq_i_m.DefaultIfEmpty()
join i_cat in context.Categories on item.RelatedCategoryID equals i_cat.ID
into sq_i_cat
from proj in sq_i_cat
select item;
Inner Join
from item in context.Items
join i_mem in context.Memberships on
new { item_id = item.ID, user_id = current_user_id.Value }
equals
new { item_id = i_mem.RelatedItemID, user_id = i_mem.RelatedUserID }
into sq_i_m
from im in sq_i_m.DefaultIfEmpty()
join i_cat in context.Categories on item.RelatedCategoryID equals i_cat.ID
select item
And here is the test program if you'd like to see for yourself.
Thanks for everyone's help.
Mustafa

They are the same thing, so it does not matter which LINQ2SQL emits.
An inner join is logically equivalent to a cross join with a where clause filter equivalent to the on of the inner join clause.
That's why Sql Server generates the same query plan.
To be clear, the inner join:
Select f1
From T1 inner join T2 on T1.k = T2.k
where T1.f2 like 'X%'
Is the same as the cross join:
Select f1
From T1 cross join T2
where T1.k = T2.k
and T1.f2 like 'X%'
is the same as old-style SQL:
Select f1
From T1, T2
where T1.k = T2.k
and T1.f2 like 'X%'

Lets say you have a datacontext called MyDataContext.
using(MyDataContext db = new MyDataContext())
{
var q = db.Items.Where(x=> x.Categories.Name == "myCategory").Select(x=> x);
}
This is a very simple example, but you didn't need to write out a join or a subquery in TSQL syntax. (I hate writing TSQL).