We Keep Coding

EntityFramework generating poor tsql (linqpad generates much better)

I have a database structure as below
Family(1) ----- (*) FamilyPersons -----(1)Person(1)------() Expenses (1) -----(0..1)GroceriesDetails
Let me explain that relation, Family can have one or more than one person , we have a mapping table FamilyPersons between Family and Persons. Now each person can enter his expenses which go into Expenses Table. Expense Table has a column ExpenseType (groceries, entertainemnet etc)
and details of each of these expenses goes into their own Tables, so we have a GroceriesDetails table (similarly we have other tables), so we have 1 to 0..1 relation between Expense and Groceries.
Now I am writing a query to get Complete GroceriesDetails for a family
GroceriesDetails.Where (g => g.Expenses.Person.FamilyPersons.Any(fp =>
fp.FamilyId == 1) && g.Expenses.ExpenseType == "GC" )
For this the sql generated by EF is
SELECT
[Extent1].[Id] AS [Id],
[Extent1].[Amount] AS [Amount]
FROM [dbo].[GroceriesDetails] AS [Extent1]
INNER JOIN (SELECT [Extent3].[Id] AS [Id1]
FROM [dbo].[Expenses] AS [Extent2]
INNER JOIN [dbo].[GroceriesDetails] AS [Extent3] ON [Extent2].[Id] = [Extent3].[Id]
WHERE N'GC' = [Extent2].[ExpenseType] ) AS [Filter1] ON [Extent1].[Id] = [Filter1].[Id1]
WHERE EXISTS (SELECT
1 AS [C1]
FROM [dbo].[Expenses] AS [Extent4]
INNER JOIN [dbo].[GroceriesDetails] AS [Extent5] ON [Extent4].[Id] = [Extent5].[Id]
INNER JOIN [dbo].[FamilyPerson] AS [Extent6] ON [Extent4].[PersonId] = [Extent6].[PersonId]
WHERE ([Extent1].[Id] = [Extent5].[Id]) AND (1 = [Extent6].[FamilyId])
)
In this query there is a full table join between Expenses and GroceriesDetails tables which is causing performance issues.
Whereas Linqpad generates a much better SQL
SELECT [t0].[Id], [t0].[Amount]
FROM [GroceriesDetails] AS [t0]
INNER JOIN [Expenses] AS [t1] ON [t1].[Id] = [t0].[Id]
WHERE (EXISTS(
SELECT NULL AS [EMPTY]
FROM [Expenses] AS [t2]
INNER JOIN [Person] AS [t3] ON [t3].[Id] = [t2].[PersonId]
CROSS JOIN [FamilyPerson] AS [t4]
WHERE ([t4].[FamilyId] = #p0) AND ([t2].[Id] = [t0].[Id]) AND ([t4].[PersonId] =
[t3].[Id])
)) AND ([t1].[ExpenseType] = #p1)
Please note that we are using WCF data services so this query is written against a WCF data service reference, so I can't traverse from top (family) to bottom (Groceries) as OData allows only one level of select.
Any help on optimizing this code is appreciated.

From the comments I learned that LinqPad uses Linq2SQL while the app uses EF, and that explains the difference.
The thing is that you have zero control on how EF generates SQL.
The only thing you can do is to rewrite your LINQ query to make it "closer" to desired SQL.
For example, instead of
GroceriesDetails.Where (g => g.Expenses.Person.FamilyPersons.Any(fp => fp.FamilyId == 1)
&& g.Expenses.ExpenseType == "GC" )
you can try to write something like (pseudocode):
from g in GrosseriesDetails
join e in Expenses on g.Id = e.GrosseryId
join p in Persons on p.Id = e.PersonId
join f in FamilyPersons on f.PersonId = p.Id
where f.FamilyId == 1 && e.ExpenseType == "GC"
It almost always helps as it tells an ORM a straightforward way to transform it into SQL. The idea is that the expression tree in the "original" case is more complex compare to the proposed scenario, and by simplifying the expression tree we make translator's job easier and more straightforward.
But besides manipulating the LINQ there is no control over how it generates SQL from the expression tree.

How to convert following SQL Query into a LINQ to SQL query

I have following SQL Query and would like to convert to LINQ to SQL which I will use in entity framework 5.0
var internationalDesksList =
from internationalDesks in _context.InternationalDesks
from subsection in
_context.Subsections.Where(
s =>
internationalDesks.EBALocationId == s.LocationId ||
internationalDesks.FELocationId == s.LocationId).DefaultIfEmpty()
where subsection.PublicationId == 1
select new {internationalDesks.Id, subsection.LocationId};
I have referred the following posts and answers. Though no luck.
LINQ to SQL - Left Outer Join with multiple join conditions Linq
left join on multiple (OR) conditions
When I tried this query in LINQPad I got the following answer which is correct.
-- Region Parameters
DECLARE #p0 Int = 1
-- EndRegion
SELECT [t0].[Id], [t1].[Id] AS [Id1]
FROM [InternationalDesks] AS [t0]
LEFT OUTER JOIN [Subsection] AS [t1] ON (([t0].[FELocationId]) = [t1].[LocationId]) OR (([t0].[EBALocationId]) = [t1].[LocationId])
WHERE [t1].[PublicationId] = #p0
However in entity framework 5 ( DBContext ) it is not providing me with the correct query. When I checked in SQL profiler all columns in subsection table is selected. That's it.
Following is the result:
SELECT
[Extent1].[Id] AS [Id],
[Extent1].[Description] AS [Description],
[Extent1].[PracticeAreaId] AS [PracticeAreaId],
[Extent1].[LocationId] AS [LocationId],
...
FROM [dbo].[Subsection] AS [Extent1]
Don't know what could be problem. Please help me.

With LINQ you can't do LEFT OUTER JOIN on some boolean expression, only equijoins are supported. So, you can generate CROSS JOIN this way:
var internationalDesksList =
from internationalDesks in _context.InternationalDesks
from subsection in _context.Subsections
where subsection.PublicationId == 1 &&
(internationalDesks.EBALocationId == subsection.LocationId ||
internationalDesks.FELocationId == subsection.LocationId)
select new {
internationalDesks.Id,
subsection.LocationId
};
EF 5 will generate following SQL:
SELECT
[Extent1].[Id] AS [Id],
[Extent2].[LocationId] AS [LocationId]
FROM [dbo].[InternationalDesks] AS [Extent1]
CROSS JOIN [dbo].[Subsections] AS [Extent2]
WHERE (1 = [Extent2].[PublicationId]) AND
([Extent1].[EBALocationId] = [Extent2].[LocationId] OR
[Extent1].[FELocationId] = [Extent2].[LocationId])
As you can see, only required columns are selected. I also checked this query in LINQ to SQL - following query is generated:
DECLARE #p0 Int = 1
SELECT [t0].[Id], [t1].[LocationId]
FROM [InternationalDesks] AS [t0], [Subsections] AS [t1]
WHERE ([t1].[PublicationId] = #p0) AND
(([t0].[EBALocationId] = [t1].[LocationId]) OR
([t0].[FELocationId] = [t1].[LocationId]))

How can i change the following LINQ query as normal sql query

I am having a query i need this to be as a plain sql query can any one help me. Also is there any alternate way with out using NorthWindDatacontext as per in the code mentioneedMy code is as follows
private void FetchData(int take, int pageSize)
{
using (NorthwindDataContext dc = new NorthwindDataContext())
{
var query = from p in dc.Customers
.OrderBy(o => o.ContactName)
.Take(take)
.Skip(pageSize)
select new
{
ID = p.CustomerID,
Name = p.ContactName,
Count = dc.Customers.Count()
};
PagedDataSource page = new PagedDataSource();
page.AllowCustomPaging = true;
page.AllowPaging = true;
page.DataSource = query;
page.PageSize = 10;
Repeater1.DataSource = page;
Repeater1.DataBind();
if (!IsPostBack)
{
RowCount = query.First().Count;
CreatePagingControl();
}
}
}
I tried as per Jon but i am unable to get the desired result can any one help me
(SELECT [t1].[CustomerID] AS [ID], [t1].[ContactName] AS [Name], (
SELECT COUNT(*)
FROM [dbo].[Customers] AS [t2]
) AS [Count]
FROM (
SELECT TOP (10) [t0].[CustomerID], [t0].[ContactName]
FROM [dbo].[Customers] AS [t0]
ORDER BY [t0].[ContactName]
) AS [t1]
ORDER BY [t1].[ContactName]

The simplest way in my view is to see what LINQ to SQL is doing.
Use DataContext.Log to write out the query information to a log of some kind (e.g. to a StringWriter whose contents you can print out afterwards). It won't always be the cleanest SQL possible, but it's a good starting point IMO.
I realise this doesn't answer the question for your specific query, but it's trying to teach you to fish as opposed to giving you a fish, as it were :)
EDIT: Using LINQPad as suggested by RoccoC5 in comments is another great way of seeing the SQL generated for a LINQ query.

Here's the SQL produced by your LINQ query, per LINQPad, where #p0 is the number of rows to Skip:
SELECT [t3].[CustomerID] AS [ID], [t3].[ContactName] AS [Name], (
SELECT COUNT(*)
FROM [Customers] AS [t4]
) AS [Count]
FROM (
SELECT [t2].[CustomerID], [t2].[ContactName], [t2].[ROW_NUMBER]
FROM (
SELECT ROW_NUMBER() OVER (ORDER BY [t1].[ContactName]) AS [ROW_NUMBER], [t1].[CustomerID], [t1].[ContactName]
FROM (
SELECT TOP (10) [t0].[CustomerID], [t0].[ContactName]
FROM [Customers] AS [t0]
ORDER BY [t0].[ContactName]
) AS [t1]
) AS [t2]
WHERE [t2].[ROW_NUMBER] > #p0
) AS [t3]
ORDER BY [t3].[ROW_NUMBER]

LINQ SQL Statement is returning wrong results

I have a SQL statement which afaik is correct but the response from the SQL server is incorrect. I've debugged this issue and found that if I execute the SQL statement without the wrapping store procedure I get different results. All I have done is replaced the variable with the actual values
Linq generated code:
exec sp_executesql N'SELECT [t0].[RoomId], [t0].[Title], [t0].[Detail], [t0].[ThumbnailPath], [t0].[PageId], [t0].[TypeId], [t0].[LocationId], [t0].[TimeStamp], [t0].[DeleteStamp]
FROM [dbo].[Room] AS [t0]
INNER JOIN [dbo].[RoomType] AS [t1] ON [t1].[RoomTypeId] = [t0].[TypeId]
WHERE ([t1].[Sleeps] >= #p0) AND ([t0].[DeleteStamp] IS NULL) AND (((
SELECT COUNT(*)
FROM [dbo].[Booking] AS [t2]
INNER JOIN [dbo].[Order] AS [t3] ON [t3].[OrderId] = [t2].[OrderId]
WHERE ([t2].[StartStamp] <= #p1)
AND ([t2].[EndStamp] >= #p2)
AND (([t3].[Status] = #p3)
OR ([t3].[Status] = #p4)
OR (([t3].[Status] = #p5) AND ([t3].[CreatedStamp] > #p6)))
AND ([t2].[RoomId] = [t0].[RoomId])
)) = #p7)
',N'#p0 int,#p1 datetime,#p2 datetime,#p3 int,#p4 int,#p5 int,#p6 datetime,#p7 int',
#p0=1,#p1='2011-04-05 00:00:00',#p2='2011-04-04 00:00:00',#p3=3,#p4=5,#p5=0,#p6='2011-04-04 12:36:09.490',#p7=0
Without the SP
SELECT [t0].[RoomId], [t0].[Title], [t0].[Detail], [t0].[ThumbnailPath], [t0].[PageId], [t0].[TypeId], [t0].[LocationId], [t0].[TimeStamp], [t0].[DeleteStamp]
FROM [dbo].[Room] AS [t0]
INNER JOIN [dbo].[RoomType] AS [t1] ON [t1].[RoomTypeId] = [t0].[TypeId]
WHERE ([t1].[Sleeps] >= 1) AND ([t0].[DeleteStamp] IS NULL) AND (((
SELECT COUNT(*)
FROM [dbo].[Booking] AS [t2]
INNER JOIN [dbo].[Order] AS [t3] ON [t3].[OrderId] = [t2].[OrderId]
WHERE ([t2].[StartStamp] <= '2011-04-05 00:00:00')
AND ([t2].[EndStamp] >= '2011-04-04 00:00:00')
AND (([t3].[Status] = 3)
OR ([t3].[Status] = 4)
OR (([t3].[Status] = 5) AND ([t3].[CreatedStamp] > '2011-04-04 12:36:09.490')))
AND ([t2].[RoomId] = [t0].[RoomId])
)) = 0)
The first result set returns 1 row where as the 2nd returns me 21!!
Can anybody spot the difference as its driving me crazy.

You made an error replacing the variables!
You replaced p4 with 4 when you should have replaced it with 5 and p5 with 5 instead of 0.

Well, one difference is #p5=0 while you have [t3].[Status] = 5 in the other.

whats wrong in this LINQ synatx?

I am trying to convert a SQL query to LINQ. Somehow my count(distinct(x)) logic does not seem to be working correctly. The original SQL is quite efficient(or so i think), but the generated SQL is not even returning the correct result.
I am trying to fix this LINQ to do what the original SQL is doing, AND in an efficient way as the original query is doing. Help here would be really apreciated as I am stuck here :(
SQL which is working and I need to make a comparable LINQ of:
SELECT [t1].[PersonID] AS [personid]
FROM [dbo].[Code] AS [t0]
INNER JOIN [dbo].[phonenumbers] AS [t1] ON [t1].[PhoneCode] = [t0].[Code]
INNER JOIN [dbo].[person] ON [t1].[PersonID]= [dbo].[Person].PersonID
WHERE ([t0].[codetype] = 'phone') AND (
([t0].[CodeDescription] = 'Home') AND ([t1].[PhoneNum] = '111')
OR
([t0].[CodeDescription] = 'Work') AND ([t1].[PhoneNum] = '222') )
GROUP BY [t1].[PersonID] HAVING COUNT(DISTINCT([t1].[PhoneNum]))=2
The LINQ which I made is approximately as below:
var ids = context.Code.Where(predicate);
var rs = from r in ids
group r by new { r.phonenumbers.person.PersonID} into g
let matchcount=g.Select(p => p.phonenumbers.PhoneNum).Distinct().Count()
where matchcount ==2
select new
{
personid = g.Key
};
Unfortunately, the above LINQ is NOT generating the correct result, and is actually internally getting generated to the SQL shown below. By the way, this generated query is also reading ALL the rows(about 19592040) around 2 times due to the COUNTS :( Wich is a big performance issue too. Please help/point me to the right direction.
Declare #p0 VarChar(10)='phone'
Declare #p1 VarChar(10)='Home'
Declare #p2 VarChar(10)='111'
Declare #p3 VarChar(10)='Work'
Declare #p4 VarChar(10)='222'
Declare #p5 VarChar(10)='2'
SELECT [t9].[PersonID], (
SELECT COUNT(*)
FROM (
SELECT DISTINCT [t13].[PhoneNum]
FROM [dbo].[Code] AS [t10]
INNER JOIN [dbo].[phonenumbers] AS [t11] ON [t11].[PhoneType] = [t10].[Code]
INNER JOIN [dbo].[Person] AS [t12] ON [t12].[PersonID] = [t11].[PersonID]
INNER JOIN [dbo].[phonenumbers] AS [t13] ON [t13].[PhoneType] = [t10].[Code]
WHERE ([t9].[PersonID] = [t12].[PersonID]) AND ([t10].[codetype] = #p0) AND ((([t10].[codetype] = #p1) AND ([t11].[PhoneNum] = #p2)) OR (([t10].[codetype] = #p3) AND ([t11].[PhoneNum] = #p4)))
) AS [t14]
) AS [cnt]
FROM (
SELECT [t3].[PersonID], (
SELECT COUNT(*)
FROM (
SELECT DISTINCT [t7].[PhoneNum]
FROM [dbo].[Code] AS [t4]
INNER JOIN [dbo].[phonenumbers] AS [t5] ON [t5].[PhoneType] = [t4].[Code]
INNER JOIN [dbo].[Person] AS [t6] ON [t6].[PersonID] = [t5].[PersonID]
INNER JOIN [dbo].[phonenumbers] AS [t7] ON [t7].[PhoneType] = [t4].[Code]
WHERE ([t3].[PersonID] = [t6].[PersonID]) AND ([t4].[codetype] = #p0) AND ((([t4].[codetype] = #p1) AND ([t5].[PhoneNum] = #p2)) OR (([t4].[codetype] = #p3) AND ([t5].[PhoneNum] = #p4)))
) AS [t8]
) AS [value]
FROM (
SELECT [t2].[PersonID]
FROM [dbo].[Code] AS [t0]
INNER JOIN [dbo].[phonenumbers] AS [t1] ON [t1].[PhoneType] = [t0].[Code]
INNER JOIN [dbo].[Person] AS [t2] ON [t2].[PersonID] = [t1].[PersonID]
WHERE ([t0].[codetype] = #p0) AND ((([t0].[codetype] = #p1) AND ([t1].[PhoneNum] = #p2)) OR (([t0].[codetype] = #p3) AND ([t1].[PhoneNum] = #p4)))
GROUP BY [t2].[PersonID]
) AS [t3]
) AS [t9]
WHERE [t9].[value] = #p5
Thanks!

I think the issue might be new { r.phonenumbers.person.PersonID}.
Why are you newing up a new object here rather than just grouping by r.phonenumbers.person directly? new {} is going to be a different object every time which will never group.
After grouping by person I would Select a group => new {person = group.person, phoneNumbers = group.person.phonenumbers} and then perform the check for how many phone numbers they have and then any final projection.

Drats! It looks like the fault was on my side(GIGO principle!)
In my ORM, I had created the associations from right to left, instead of the other way round. I think that was the issue.
Only issue left now is that somehow the LINQ is generating one INNER JOIN twice, and that is keeping the final result to be retrieved correctly. If i comment it out i nthe generated sql, i am getting correct result. That's the only issue now and I guess i'll open a new question for that. Thanks for your time!.