We Keep Coding

SQL Server - DATEDIFF Function taking too long

I've done some extensive research and I've concluded that the DATEDIFF function is making my queries run very slow.
Below is the generated query by Entity Framework and it does look readable enough hopefully.
Here's the Linq that generates the T-SQL:
model.NewTotal1Week = ( from sdo in context.SubscriberDebitOrders
where
(
sdo.CampaignId == campaignId &&
( sdo.Status == ( Int32 ) DebitOrderStatus.New_Faulty ) &&
( SqlFunctions.DateDiff( "week", sdo.Collections.FirstOrDefault( c => c.TxnStatus == "U" ).ProcessDate, DateTime.Now ) <= 1 )
)
select sdo ).Count();
In the query below, I would like to get a COUNT of all Collections which fall within 1 week from the time they were Processed to today's date.
Is there anyone that can help me get rid of the DATEDIFF function? I've seen examples online but I couldn't adapt it to my scenario, forgive me I'm not very genius yet.
exec sp_executesql N'SELECT
[GroupBy1].[A1] AS [C1]
FROM ( SELECT
COUNT(1) AS [A1]
FROM [dbo].[SubscriberDebitOrder] AS [Extent1]
OUTER APPLY (SELECT TOP (1)
[Extent2].[ProcessDate] AS [ProcessDate]
FROM [dbo].[Collections] AS [Extent2]
WHERE ([Extent1].[Id] = [Extent2].[DebitOrderId]) AND (''U'' = [Extent2].[TxnStatus]) ) AS [Limit1]
WHERE ([Extent1].[CampaignId] = #p__linq__0) AND (3 = [Extent1].[Status]) AND ((DATEDIFF(week, [Limit1].[ProcessDate], SysDateTime())) <= 1)
) AS [GroupBy1]',N'#p__linq__0 int',#p__linq__0=3
go
Thanks in advance.

Its not the just DATEDIFF, any function on the column would cause query to do a SCAN on the underlying table/index
DATEDIFF(week, [Limit1].[ProcessDate], SysDateTime())) <=1
Above logic is fetching last week data? You can also write above without putting function around ProcessDate Column.
[Limit1].[ProcessDate] > SysDateTime()-7

This is your query:
SELECT GroupBy1.A1 AS C1
FROM (SELECT COUNT(1) AS[A1
FROM dbo.SubscriberDebitOrder AS Extent1 OUTER APPLY
(SELECT TOP (1) Extent2.ProcessDate
FROM [dbo].Collections Extent2
WHERE (Extent1.Id = Extent2.DebitOrderId AND
'U' = Extent2.TxnStatus
) AS [Limit1]
WHERE (Extent1.CampaignId = #p__linq__0) AND (3 = Extent1.Status) AND
(DATEDIFF(week, Limit1.ProcessDate, SysDateTime()) <= 1)
) GroupBy1;
As mentioned elsewhere, you should change the date logic and get rid of the outer query:
SELECT COUNT(1) AS A1
FROM dbo.SubscriberDebitOrder AS Extent1 OUTER APPLY
(SELECT TOP (1) Extent2.ProcessDate
FROM [dbo].Collections Extent2
WHERE (Extent1.Id = Extent2.DebitOrderId AND
'U' = Extent2.TxnStatus
) AS limit1
WHERE (Extent1.CampaignId = #p__linq__0) AND (3 = Extent1.Status) AND
Limit1.ProcessDate <= DATEADD(-1, week, GETDATE())
Very important note: This is not exactly equivalent to your query. Your original query counted the number of week boundaries between two dates. This depends on datefirst, but it woudld often be the number of Saturday or Sunday nights.
Based on your description, the above is more correct.
Next, you want indexes on Collections(DebitOrderId, TxnStatus, ProcessDate) and SubscriberDebitOrder(CampaignId, Status).

Entity Framework Order By Multiple Related Entity Count - Efficiency

So I have a Topic which has these related entities.
- List<Posts>
- List<Votes>
- List<Views>
I have the following query. Where I want to pull out and order popular Topics based on the count of 3 related entities over a specific date period.
var topics = _context.Topic
.OrderByDescending(x => x.Posts.Count(c => c.DateCreated >= from && c.DateCreated <= to))
.ThenByDescending(x => x.Votes.Count(c => c.DateCreated >= from && c.DateCreated <= to))
.ThenByDescending(x => x.Views.Count(c => c.DateCreated >= from && c.DateCreated <= to))
.Take(amountToShow)
.ToList();
I'm looking for the most efficient query for doing the above? Is what I am doing the best way to do this with EntityFramework? Or am I missing something?
Any help appreciated.

If you put your above code into LINQPad, or check it with the profiler, you will see that it will probably generate something like the following SQL:
SELECT TOP #amountToShow [t0].[id] --and additional columns
FROM [Topic] AS [t0]
ORDER BY (
SELECT COUNT(*)
FROM [Posts] AS [t1]
WHERE ([t1].DateCreated >= #from AND [t1].DateCreated <= #to)
AND ([t1].[topidId] = [t0].[id])
) DESC, (
SELECT COUNT(*)
FROM [Votes] AS [t2]
WHERE ([t2].DateCreated >= #from AND [t2].DateCreated <= #to)
AND ([t2].[topicId] = [t0].[id])
) DESC , (
SELECT COUNT(*)
FROM [Views] AS [t3]
WHERE ([t3].DateCreated >= #from AND [t3].DateCreated <= #to)
AND ([t3].[topicId] = [t0].[id])
) DESC
GO
You could try rewriting the SQL a bit to GROUP the results of the subqueries and LEFT JOIN them to the original table, which does seem to be about 2x faster in the db itself:
SELECT TOP #amountToShow [t0].[id] --etc
FROM [Topic] AS [t0]
LEFT JOIN
(SELECT topicId, COUNT(*) AS num FROM Posts p
WHERE [p].DateCreated >= #from AND .DateCreated <= #to
GROUP BY topicId) [t1]
ON t0.id = t1.topicId
LEFT JOIN
(SELECT topicId, COUNT(*) AS num FROM Votes vo
WHERE [vo].DateCreated >= #from AND [vo].DateCreated <= #to
GROUP BY topidId) [t2]
ON t0.id = t2.topicId
LEFT JOIN
(SELECT topicId, COUNT(*) AS num FROM Views vi
WHERE [vi].DateCreated >= #from AND [vi].DateCreated <= #to
GROUP BY topicId) [t3]
ON t0.id = t3.topicId
ORDER BY t1.num DESC, t2.num DESC, t3.num DESC
But getting LINQ to generate code like this is iffy at best. Doing LEFT JOINs are not exactly its strong suit, and using the techniques that are out there for doing so will probably generate SQL that uses CROSS APPLY and/or OUTER APPLY instead, and will likely be as slow or slower than your current code.
If you are that worried about speed, you might consider putting your fine-tuned SQL into a view so that you know that the query being used is the one you want.
Bear in mind, too, that you or someone else will have to come back to this code and maintain it later. Your current linq statement is very straightforward and easy to understand. A complicated query is going to be harder to maintain and will take more work to alter in the future.

LinqToSQL - ToList() appears to be very slow

I'm newish to LinqToSQL and the project that I am working on cannot be changed to something else. I am translating some old SQL code to Linq. Not being that hot at linq, I used Linqer to do the translation for me. The query took about 90 seconds to run, so I thought it must be the linqToSQL. However, when I copied the query that the LinqToSQL produced and ran an ExecuteQuery on the datacontext it was super quick as I expected. I've copied the full queries, rather than trying to distil it down, but it looks like the issue is with something LinqToSQL is doing behind the scenes.
To summarise, if I copy the T-SQL created by linq and run
var results = DB.ExecuteQuery<InvoiceBalanceCheckDTO.InvoiceBalanceCheck>(#"T-SQL created by Linq - see below").ToList()
it completes with expected results in about 0.5 seconds.
It runs about the same time directly in SSMS. However, if I use the linqToSQL code that creates the T-SQL and do ToList() it takes ages. The result is only 9 records, although without the constraint to check the balance <> 0, there would be around 19,000 records. It's as if it's getting all 19,000 and then checking <> 0 after it's got the records.
I have also changed the Linq to project into the class used above, rather than to an anonymous type, but it makes not difference
This is the original SQL :
SELECT InvoiceNum, Max(AccountCode), Sum(AmountInc) AS Balance
FROM
(SELECT InvoiceNum, AccountCode, AmountInc From TourBookAccount WHERE AccDetailTypeE IN(20,30) AND InvoiceNum >= 1000
UNION ALL
SELECT InvoiceNum, '<no matching invoice>' AS AccountCode, AccountInvoiceDetail.AmountInc
FROM AccountInvoiceDetail
INNER JOIN AccountInvoice ON AccountInvoiceDetail.InvoiceID=AccountInvoice.InvoiceID
WHERE AccDetailTypeE IN(20,30)
AND InvoiceNum >= 1000
) as t
GROUP BY InvoiceNum
HAVING (Sum(t.AmountInc)<>0)
ORDER BY InvoiceNum
and this is the linq
var test = (from t in
(
//this gets the TourBookAccount totals
from tba in DB.TourBookAccount
where
detailTypes.Contains(tba.AccDetailTypeE) &&
tba.InvoiceNum >= dto.CheckInvoiceNumFrom
select new
{
InvoiceNum = tba.InvoiceNum,
AccountCode = tba.AccountCode,
Balance = tba.AmountInc
}
)
.Concat //note that concat, since it's possible that the AccountInvoice record does not actually exist
(
//this gets the Invoice detail totals.
from aid in DB.AccountInvoiceDetail
where
detailTypes.Contains(aid.AccDetailTypeE) &&
aid.AccountInvoice.InvoiceNum >= dto.CheckInvoiceNumFrom &&
select new
{
InvoiceNum = aid.AccountInvoice.InvoiceNum,
AccountCode = "<No Account Records>",
Balance = aid.AmountInc
}
)
group t by t.InvoiceNum into g
where Convert.ToDecimal(g.Sum(p => p.Balance)) != 0m
select new
{
InvoiceNum = g.Key,
AccountCode = g.Max(p => p.AccountCode),
Balance = g.Sum(p => p.Balance)
}).ToList();
and this is the T-SQL that the linq produces
SELECT [t5].[InvoiceNum], [t5].[value2] AS [AccountCode], [t5].[value3] AS [Balance]
FROM (
SELECT SUM([t4].[AmountInc]) AS [value], MAX([t4].[AccountCode]) AS [value2], SUM([t4].[AmountInc]) AS [value3], [t4].[InvoiceNum]
FROM (
SELECT [t3].[InvoiceNum], [t3].[AccountCode], [t3].[AmountInc]
FROM (
SELECT [t0].[InvoiceNum], [t0].[AccountCode], [t0].[AmountInc]
FROM [dbo].[TourBookAccount] AS [t0]
WHERE ([t0].[AccDetailTypeE] IN (20, 30)) AND ([t0].[InvoiceNum] >= 1000)
UNION ALL
SELECT [t2].[InvoiceNum],'<No Account Records>' AS [value], [t1].[AmountInc]
FROM [dbo].[AccountInvoiceDetail] AS [t1]
INNER JOIN [dbo].[AccountInvoice] AS [t2] ON [t2].[InvoiceID] = [t1].[InvoiceID]
WHERE ([t1].[AccDetailTypeE] IN (20, 30)) AND ([t2].[InvoiceNum] >= 1000)
) AS [t3]
) AS [t4]
GROUP BY [t4].[InvoiceNum]
) AS [t5]
WHERE [t5].[value] <> 0

I would bet money, that the problem is in this line:
where Convert.ToDecimal(g.Sum(p => p.Balance)) != 0m
What is probably happening, is that it can't translate this to SQL and silently tries to get all rows from db to memory, and then do filtering on in memory objects (LINQ to objects)
Maybe try to change this to something like:
where g.Sum(p=>.Balance!=0)

Well, the answer turned out not to be LinqToSQL itself (although possibly the way it creates the query could be blamed) , but the way SQL server handles the query. When I was running the query on the database to check speed (and running the created T=SQL in DB.ExecuteQuery) I had all the variables hardcoded. When I changed it to use the exact sql that Linq produces (i.e. with variables that are substituted) it ran just as slow in SSMS.
Looking at the execution plans of the two, they are quite different. A quick search on SO brought me to this page : Why does a parameterized query produces vastly slower query plan vs non-parameterized query which indicated that the problem was SQL server's "Parameter sniffing".
The culprit turned out to be the "No Account Records" string
For completeness, here is the generated T-SQL that Linq creates.
Change #p10 to the actual hardcoded string, and it's back to full speed !
In the end I just removed the line from the linq and set the account code afterwards and all was good.
Thanks #Botis,#Blorgbeard,#ElectricLlama & #Scott for suggestions.
DECLARE #p0 as Int = 20
DECLARE #p1 as Int = 30
DECLARE #p2 as Int = 1000
DECLARE #p3 as Int = 20
DECLARE #p4 as Int = 30
DECLARE #p5 as Int = 1000
DECLARE #p6 as Int = 40
DECLARE #p7 as Int = 10
DECLARE #p8 as Int = 0
DECLARE #p9 as Int = 1
DECLARE #p10 as NVarChar(4000)= '<No Account Records>' /*replace this parameter with the actual text in the SQl and it's way faster.*/
DECLARE #p11 as Decimal(33,4) = 0
SELECT [t5].[InvoiceNum], [t5].[value2] AS [AccountCode], [t5].[value3] AS [Balance]
FROM (
SELECT SUM([t4].[AmountInc]) AS [value], MAX([t4].[AccountCode]) AS [value2], SUM([t4].[AmountInc]) AS [value3], [t4].[InvoiceNum]
FROM (
SELECT [t3].[InvoiceNum], [t3].[AccountCode], [t3].[AmountInc]
FROM (
SELECT [t0].[InvoiceNum], [t0].[AccountCode], [t0].[AmountInc]
FROM [dbo].[TourBookAccount] AS [t0]
WHERE ([t0].[AccDetailTypeE] IN (#p0, #p1)) AND ([t0].[InvoiceNum] >= #p2)
UNION ALL
SELECT [t2].[InvoiceNum], #p10 AS [value], [t1].[AmountInc]
FROM [dbo].[AccountInvoiceDetail] AS [t1]
INNER JOIN [dbo].[AccountInvoice] AS [t2] ON [t2].[InvoiceID] = [t1].[InvoiceID]
WHERE ([t1].[AccDetailTypeE] IN (#p3, #p4)) AND ([t2].[InvoiceNum] >= #p5) AND ([t2].[InvoiceStatusE] <= #p6) AND ([t2].[InvoiceTypeE] = #p7) AND ([t1].[BookNum] <> #p8) AND ([t1].[AccDetailSourceE] = #p9)
) AS [t3]
) AS [t4]
GROUP BY [t4].[InvoiceNum]
) AS [t5]
WHERE [t5].[value] <> #p11
SELECT [t5].[InvoiceNum], [t5].[value2] AS [AccountCode], [t5].[value3] AS [Balance]
FROM (
SELECT SUM([t4].[AmountInc]) AS [value], MAX([t4].[AccountCode]) AS [value2], SUM([t4].[AmountInc]) AS [value3], [t4].[InvoiceNum]
FROM (
SELECT [t3].[InvoiceNum], [t3].[AccountCode], [t3].[AmountInc]
FROM (
SELECT [t0].[InvoiceNum], [t0].[AccountCode], [t0].[AmountInc]
FROM [dbo].[TourBookAccount] AS [t0]
WHERE ([t0].[AccDetailTypeE] IN (20, 30)) AND ([t0].[InvoiceNum] >= 1000)
UNION ALL
SELECT [t2].[InvoiceNum], '<No Account Records>' AS [value], [t1].[AmountInc]
FROM [dbo].[AccountInvoiceDetail] AS [t1]
INNER JOIN [dbo].[AccountInvoice] AS [t2] ON [t2].[InvoiceID] = [t1].[InvoiceID]
WHERE ([t1].[AccDetailTypeE] IN (20, 30)) AND ([t2].[InvoiceNum] >= 0) AND ([t2].[InvoiceStatusE] <= 40) AND ([t2].[InvoiceTypeE] = 10) AND ([t1].[BookNum] <> 0) AND ([t1].[AccDetailSourceE] = 1)
) AS [t3]
) AS [t4]
GROUP BY [t4].[InvoiceNum]
) AS [t5]
WHERE [t5].[value] <> 0

Pulling Data from 1 DataContext to use in another's Method

I have been trying to convert this SQL statement to a LINQ one and am having trouble with the fact that part of the info returned is in a Seperate Database(Datacontext) from the rest. I am pretty sure this can be overcome however I seem to be failing at accomplishing this or finding examples of previous successful attempts.
Can someone offer some guidance on what I do to overcome that hurdle? Thanks
SELECT p.PersonID, p.FirstName, p.MiddleName, p.LastName, cp.EnrollmentID, cp.EnrollmentDate, cp.DisenrollmentDate
FROM [Connect].dbo.tblPerson AS p
INNER JOIN (
SELECT c.ClientID, c.EnrollmentID, c.EnrollmentDate, c.DisenrollmentDate
FROM [CMO].dbo.tblCMOEnrollment AS c
LEFT OUTER JOIN [CMO].dbo.tblWorkerHistory AS wh
ON c.EnrollmentID = wh.EnrollmentID
INNER JOIN [CMO].dbo.tblStaffExtended AS se
ON wh.Worker = se.StaffID
WHERE (wh.EndDate IS NULL OR wh.EndDate >= getdate())
AND wh.Worker = --WorkerGUID Param here
) AS cp
ON p.PersonID = cp.ClientID
ORDER BY p.PersonID
I have asked a similar question here before as was told I would need to create a View in order to accomplish this. Is that still true or was it ever?

I use LINQPad to do a lot of my LINQ to SQL. One of the features it allows is the use of multiple data contexts for one query.
for instance here is some code that I wrote in LINQPad
from template in RateTemplates
where
template.Policies.Any(p =>
Staging_history.Changes.Any(c =>
(c.Policies.Any(cp => cp.PolicyID == p.PolicyID) ||
c.PolicyFees.Any(cpf => cpf.PolicyID == p.PolicyID) ||
c.PolicyOptions.Any(cpo => cpo.PolicyID == p.PolicyID)) &&
c.ChangeTime > new DateTime(2012, 1, 11)
)
)
select new
{
TemplateID = template.ID,
UserID = template.UserID,
PropertyIDs = template.Properties.Select(ppty => ppty.PropertyID)
}
The table "RateTemplates" is a part of my first Data Context (With LINQPad you do not have to define the first data context in your code it is just assumed, but if you do this is C# you would need to specifically say which context to use etc). "Staging_history" is the second Data Context and I am using the table "Changes" from this one.
LINQ to SQL will do all sorts of magic in the background and the resulting SQL that gets executed is ...
-- Region Parameters
DECLARE #p0 DateTime = '2012-01-11 00:00:00.000'
-- EndRegion
SELECT [t0].[ID] AS [TemplateID], [t0].[UserID], [t1].[PropertyID], (
SELECT COUNT(*)
FROM [Property] AS [t7]
WHERE [t7].[RateTemplateID] = [t0].[ID]
) AS [value]
FROM [RateTemplate] AS [t0]
LEFT OUTER JOIN [Property] AS [t1] ON [t1].[RateTemplateID] = [t0].[ID]
WHERE EXISTS(
SELECT NULL AS [EMPTY]
FROM [Policy] AS [t2]
WHERE (EXISTS(
SELECT NULL AS [EMPTY]
FROM [staging_history].[dbo].[Change] AS [t3]
WHERE ((EXISTS(
SELECT NULL AS [EMPTY]
FROM [staging_history].[dbo].[Policy] AS [t4]
WHERE ([t4].[PolicyID] = [t2].[PolicyID]) AND ([t4].[ChangeID] = [t3].[ID])
)) OR (EXISTS(
SELECT NULL AS [EMPTY]
FROM [staging_history].[dbo].[PolicyFee] AS [t5]
WHERE ([t5].[PolicyID] = [t2].[PolicyID]) AND ([t5].[ChangeID] = [t3].[ID])
)) OR (EXISTS(
SELECT NULL AS [EMPTY]
FROM [staging_history].[dbo].[PolicyOption] AS [t6]
WHERE ([t6].[PolicyID] = [t2].[PolicyID]) AND ([t6].[ChangeID] = [t3].[ID])
))) AND ([t3].[ChangeTime] > #p0)
)) AND ([t2].[RateTemplateID] = [t0].[ID])
)
ORDER BY [t0].[ID], [t1].[PropertyID]
So it looks like you would just need to load up one data context for each database that you want to use and then just build up a LINQ query that makes use of both data contexts in one linq statement, like I have up above.
Hopefully this helps you out and gets you the results you are wanting without having to go creating views for each cross context queries that you want to do.

My understanding (I'm no guru on linqtosql) was the same, that it wasn't possible without using a view/sproc.
However, a quick search, found this on MSDN forums with a workaround. Quote from Damien's answer on there:
2.Add one of the tables to the other data context (Copy the DBML over and prefix the name attribute with the name of the database, e.g.
database2.dbo.MyTable)

LINQ SQL Statement is returning wrong results

I have a SQL statement which afaik is correct but the response from the SQL server is incorrect. I've debugged this issue and found that if I execute the SQL statement without the wrapping store procedure I get different results. All I have done is replaced the variable with the actual values
Linq generated code:
exec sp_executesql N'SELECT [t0].[RoomId], [t0].[Title], [t0].[Detail], [t0].[ThumbnailPath], [t0].[PageId], [t0].[TypeId], [t0].[LocationId], [t0].[TimeStamp], [t0].[DeleteStamp]
FROM [dbo].[Room] AS [t0]
INNER JOIN [dbo].[RoomType] AS [t1] ON [t1].[RoomTypeId] = [t0].[TypeId]
WHERE ([t1].[Sleeps] >= #p0) AND ([t0].[DeleteStamp] IS NULL) AND (((
SELECT COUNT(*)
FROM [dbo].[Booking] AS [t2]
INNER JOIN [dbo].[Order] AS [t3] ON [t3].[OrderId] = [t2].[OrderId]
WHERE ([t2].[StartStamp] <= #p1)
AND ([t2].[EndStamp] >= #p2)
AND (([t3].[Status] = #p3)
OR ([t3].[Status] = #p4)
OR (([t3].[Status] = #p5) AND ([t3].[CreatedStamp] > #p6)))
AND ([t2].[RoomId] = [t0].[RoomId])
)) = #p7)
',N'#p0 int,#p1 datetime,#p2 datetime,#p3 int,#p4 int,#p5 int,#p6 datetime,#p7 int',
#p0=1,#p1='2011-04-05 00:00:00',#p2='2011-04-04 00:00:00',#p3=3,#p4=5,#p5=0,#p6='2011-04-04 12:36:09.490',#p7=0
Without the SP
SELECT [t0].[RoomId], [t0].[Title], [t0].[Detail], [t0].[ThumbnailPath], [t0].[PageId], [t0].[TypeId], [t0].[LocationId], [t0].[TimeStamp], [t0].[DeleteStamp]
FROM [dbo].[Room] AS [t0]
INNER JOIN [dbo].[RoomType] AS [t1] ON [t1].[RoomTypeId] = [t0].[TypeId]
WHERE ([t1].[Sleeps] >= 1) AND ([t0].[DeleteStamp] IS NULL) AND (((
SELECT COUNT(*)
FROM [dbo].[Booking] AS [t2]
INNER JOIN [dbo].[Order] AS [t3] ON [t3].[OrderId] = [t2].[OrderId]
WHERE ([t2].[StartStamp] <= '2011-04-05 00:00:00')
AND ([t2].[EndStamp] >= '2011-04-04 00:00:00')
AND (([t3].[Status] = 3)
OR ([t3].[Status] = 4)
OR (([t3].[Status] = 5) AND ([t3].[CreatedStamp] > '2011-04-04 12:36:09.490')))
AND ([t2].[RoomId] = [t0].[RoomId])
)) = 0)
The first result set returns 1 row where as the 2nd returns me 21!!
Can anybody spot the difference as its driving me crazy.

You made an error replacing the variables!
You replaced p4 with 4 when you should have replaced it with 5 and p5 with 5 instead of 0.

Well, one difference is #p5=0 while you have [t3].[Status] = 5 in the other.