We Keep Coding

Simple LINQ to Entities translation

Before anyone jumps on a mark as duplicate, I have looked and everyone is doing something slightly more complicated than I am trying.
So I'm working on a database where there's a lot of data to check and LINQ's Any() extension translated to SQL isn't as fast as SQL's Count(1) > 0, so everywhere I'm writing:
var someBool = Ctx.SomeEntities.Count(x => x.RelatedEntity.Count(y => y.SomeProperty == SomeValue) > 0) > 0;
In Pseudo: Does any of my entities have a relationship with some other entity that has a property with a value of SomeValue.
This works fine and it works fast. However, it's not exactly readable (and I have lots of them, more embedded than that in cases) so what I'd like to do is replace it with:
var someBool = Ctx.SomeEntities.AnyX(x => x.RelatedEntity.AnyX(y => y.SomeProperty == SomeValue));
with:
public static bool AnyX<TSource>(this IEnumerable<TSource> source, Func<TSource, bool> predicate) => source.Count(predicate) > 0;
So you see I'm not doing anything that LINQ can't translate to SQL, I'm not doing anything that LINQ doesn't already translate to SQL, but just by creating an additional extension I get:
LINQ to Entities does not recognize the method Boolean AnyX etc...
There must be some way of writing my extension or some way of telling LINQ just to take a look at the code and you'll see you can do it.

Not an answer to your specific question, but I suggest you rethink how you're approaching the query.
Let's use some descriptive names that make it easier to understand: do any households have a resident with the first name of "Bobby"?
// your way
Ctx.Households.Count( hh => hh.Residents.Count( r => r.FirstName == "Bobby" ) > 0 ) > 0
Yuck, it's backwards. Start with residents:
Ctx.Residents.Count( r =>
r.FirstName == "Bobby"
&& r.Household != null ) // if needed
> 0;
Now, will that generate SQL significantly different than the below?
Ctx.Residents.Any( r => r.FirstName == "Bobby" && r.Household != null)
edit:
Here's a true MCVE that results in the opposite of your conclusion:
/*
create table TestDatum
(
TestValue nchar(36) not null
)
*/
/*
set nocount on
declare #count int
declare #start datetime
declare #end datetime
set #count = 0
set #start = GETDATE()
while #count < 14000000
begin
insert TestDatum values( CONVERT(nchar(36), NEWID()) )
set #count = #count + 1
if (#count % 100000) = 0
begin
print convert(nvarchar, #count)
end
end
set #end = GETDATE()
select CONVERT(nvarchar, DATEDIFF(ms, #start, #end))
*/
/*
-- "Any" test
declare #startdt datetime, #enddt datetime
set #startdt = GETDATE()
DECLARE #p0 NVarChar(1000) = '%abcdef%'
SELECT
(CASE
WHEN EXISTS(
SELECT NULL AS [EMPTY]
FROM TestDatum AS [t0]
WHERE [t0].TestValue LIKE #p0
) THEN 1
ELSE 0
END) AS [value]
set #enddt = GETDATE()
select DATEDIFF(ms, #startdt, #enddt) -- ~7000ms
*/
/*
-- "Count" test
declare #startdt datetime, #enddt datetime
set #startdt = GETDATE()
-- Region Parameters
DECLARE #p0 NVarChar(1000) = '%abcdef%'
-- EndRegion
SELECT COUNT(*) AS [value]
FROM TestDatum AS [t0]
WHERE [t0].TestValue LIKE #p0
set #enddt = GETDATE()
select DATEDIFF(ms, #startdt, #enddt) -- > 48000ms
*/

Have EF Linq Select statement Select a constant or a function

I have a Select statement that is currently formatted like
dbEntity
.GroupBy(x => x.date)
.Select(groupedDate => new {
Calculation1 = doCalculation1 ? x.Sum(groupedDate.Column1) : 0),
Calculation2 = doCalculation2 ? x.Count(groupedDate) : 0)
In the query doCalculation1 and doCalculation2 are bools that are set earlier. This creates a case statement in the Sql being generated, like
DECLARE #p1 int = 1
DECLARE #p2 int = 0
DECLARE #p3 int = 1
DECLARE #p4 int = 0
SELECT (Case When #p1 = 1 THEN Sum(dbEntity.Column1)
Else #p2
End) as Calculation1,
(Case When #p3 = 1 THEN Count(*)
Else #p4
End) as Calculation2
What I want to happen is for the generated sql is to be like this when doCalculation1 is true
SELECT SUM(Column1) as Calculation1, Count(*) as Calculation2
and like this when doCalculation2 is false
SELECT 0 as Calculation1, Count(*) as Calculation2
Is there any way to force a query through EF to act like this?
Edit:
bool doCalculation = true;
bool doCalculation2 = false;
dbEntity
.Where(x => x.FundType == "E")
.GroupBy(x => x.ReportDate)
.Select(dateGroup => new
{
ReportDate = dateGroup.Key,
CountInFlows = doCalculation2 ? dateGroup.Count(x => x.Flow > 0) : 0,
NetAssetEnd = doCalculation ? dateGroup.Sum(x => x.AssetsEnd) : 0
})
.ToList();
generates this sql
-- Region Parameters
DECLARE #p0 VarChar(1000) = 'E'
DECLARE #p1 Int = 0
DECLARE #p2 Decimal(5,4) = 0
DECLARE #p3 Int = 0
DECLARE #p4 Int = 1
DECLARE #p5 Decimal(1,0) = 0
-- EndRegion
SELECT [t1].[ReportDate],
(CASE
WHEN #p1 = 1 THEN (
SELECT COUNT(*)
FROM [dbEntity] AS [t2]
WHERE ([t2].[Flow] > #p2) AND ([t1].[ReportDate] = [t2].[ReportDate]) AND ([t2].[FundType] = #p0)
)
ELSE #p3
END) AS [CountInFlows],
(CASE
WHEN #p4 = 1 THEN CONVERT(Decimal(33,4),[t1].[value])
ELSE CONVERT(Decimal(33,4),#p5)
END) AS [NetAssetEnd]
FROM (
SELECT SUM([t0].[AssetsEnd]) AS [value], [t0].[ReportDate]
FROM [dbEntity] AS [t0]
WHERE [t0].[FundType] = #p0
GROUP BY [t0].[ReportDate]
) AS [t1]
which has many index scans and a spool and a join in the execution plan. It also takes about 20 seconds on average to run on the test set, with the production set going to be much larger.
I want it to run in the same speed as sql like
select reportdate, 1, sum(AssetsEnd)
from vwDailyFundFlowDetail
where fundtype = 'E'
group by reportdate
which runs in about 12 seconds on average and has the majority of the query tied up in a single index seek in the execution plan. What the actual sql output is doesnt matter, but the performance appears to be much worse with the case statements.
As for why I am doing this, I need to generate a dynamic select statements like I asked in Dynamically generate Linq Select. A user may select one or more of a set of calculations to perform and I will not know what is selected until the request comes in. The requests are expensive so we do not want to run them unless they are necessary. I am setting the doCalculation bools based on the user request.
This query is supposed to replace some code that inserts or deletes characters from a hardcoded sql query stored as a string, which is then executed. That runs fairly fast but is a nightmare to maintain

It would technically be possible to pass the Expression in your Select query through an expression tree visitor, which checks for constant values on the left-hand side of ternary operators, and replaces the ternary expression with the appropriate sub-expression.
For example:
public class Simplifier : ExpressionVisitor
{
public static Expression<T> Simplify<T>(Expression<T> expr)
{
return (Expression<T>) new Simplifier().Visit(expr);
}
protected override Expression VisitConditional(ConditionalExpression node)
{
var test = Visit(node.Test);
var ifTrue = Visit(node.IfTrue);
var ifFalse = Visit(node.IfFalse);
var testConst = test as ConstantExpression;
if(testConst != null)
{
var value = (bool) testConst.Value;
return value ? ifTrue : ifFalse;
}
return Expression.Condition(test, ifTrue, ifFalse);
}
protected override Expression VisitMember(MemberExpression node)
{
// Closed-over variables are represented as field accesses to fields on a constant object.
var field = (node.Member as FieldInfo);
var closure = (node.Expression as ConstantExpression);
if(closure != null)
{
var value = field.GetValue(closure.Value);
return VisitConstant(Expression.Constant(value));
}
return base.VisitMember(node);
}
}
Usage example:
void Main()
{
var b = true;
Expression<Func<int, object>> expr = i => b ? i.ToString() : "N/A";
Console.WriteLine(expr.ToString()); // i => IIF(value(UserQuery+<>c__DisplayClass0).b, i.ToString(), "N/A")
Console.WriteLine(Simplifier.Simplify(expr).ToString()); // i => i.ToString()
b = false;
Console.WriteLine(Simplifier.Simplify(expr).ToString()); // i => "N/A"
}
So, you could use this in your code something like this:
Expression<Func<IGrouping<DateTime, MyEntity>>, ClassYouWantToReturn> select =
groupedDate => new {
Calculation1 = doCalculation1 ? x.Sum(groupedDate.Column1) : 0),
Calculation2 = doCalculation2 ? x.Count(groupedDate) : 0
};
var q = dbEntity
.GroupBy(x => x.date)
.Select(Simplifier.Simplify(select))
However, this is probably more trouble than it's worth. SQL Server will almost undoubtedly optimize the "1 == 1" case away, and allowing Entity Framework to produce the less-pretty query shouldn't prove to be a performance problem.
Update
Looking at the updated question, this appears to be one of the few instances where producing the right query really does matter, performance-wise.
Besides my suggested solution, there are a few other choices: you could use raw sql to map to your return type, or you could use LinqKit to choose a different expression based on what you want, and then "Invoke" that expression inside your Select query.

Difference between sql when using Func as a select vs inline select

I'm trying to understand why the sql for these two statements are different (The difference appears to be in determining whether to added SignedXml in the select or use a case statement. But why?).
myTable.Where(p => p.OwnerID.Equals(owner) && p.FolderID == folderId)
.Select(source => new SignedNativeAnalysis
{
ID = source.ID,
Name = source.Name,
Created = source.Created,
FolderID = source.FolderID,
Locked = source.Locked,
Modified = source.Modified,
OwnerID = source.OwnerID,
IsSigned = source.SignedXml != null
});
which generates something like:
exec sp_executesql N'SELECT
[Extent1].[ID] AS [ID],
[Extent1].[Name] AS [Name],
[Extent1].[Created] AS [Created],
[Extent1].[FolderID] AS [FolderID],
[Extent1].[Locked] AS [Locked],
[Extent1].[Modified] AS [Modified],
[Extent1].[OwnerID] AS [OwnerID],
CASE WHEN ([Extent1].[SignedXml] IS NOT NULL) THEN cast(1 as bit) WHEN ([Extent1].[SignedXml] IS NULL) THEN cast(0 as bit) END AS [C1]
FROM [dbo].[NativeAnalyses] AS [Extent1]
WHERE ([Extent1].[OwnerID] = #p__linq__0) AND ([Extent1].[FolderID] = #p__linq__1)',N'#p__linq__0 uniqueidentifier,#p__linq__1
vs when I'm using a function for the select:
myTable.Where(p => p.OwnerID.Equals(owner) && p.FolderID == folderId)
.Select(Select);
and Select is
private SignedNativeAnalysis Select(NativeAnalysis source)
{
return new SignedNativeAnalysis
{
ID = source.ID,
Name = source.Name,
Created = source.Created,
FolderID = source.FolderID,
Locked = source.Locked,
Modified = source.Modified,
OwnerID = source.OwnerID,
IsSigned = source.SignedXml != null
};
}
which generates something like:
exec sp_executesql N'SELECT
[Extent1].[ID] AS [ID],
[Extent1].[Name] AS [Name],
[Extent1].[ModelXml] AS [ModelXml],
[Extent1].[Created] AS [Created],
[Extent1].[Modified] AS [Modified],
[Extent1].[OwnerID] AS [OwnerID],
[Extent1].[SignedXml] AS [SignedXml],
[Extent1].[FolderID] AS [FolderID],
[Extent1].[Locked] AS [Locked]
FROM [dbo].[NativeAnalyses] AS [Extent1]
WHERE ([Extent1].[OwnerID] = #p__linq__0) AND ([Extent1].[FolderID] = #p__linq__1)',N'#p__linq__0 uniqueidentifier,#p__linq__1
And SignedNativeAnalysis just inherits NativeAnalysis with one extra property.
internal class SignedNativeAnalysis : NativeAnalysis
{
public bool IsSigned { get; set; }
}
UPDATE:
As per suggested I tried to use an expression to achieve the result I wanted but I got an exception instead:
Expression<Func<NativeAnalysis, SignedNativeAnalysis>> signed = p => Select(p);
myTable.Where(p => p.OwnerID.Equals(owner) && p.FolderID == folderId)
.Select(signed)
LINQ to Entities does not recognize the method 'SignedNativeAnalysis
Select(OverseerUI.EntityFramework.NativeAnalysis)' method, and this
method cannot be translated into a store expression.

There's a trick. Entity Framework Select doesn't take a Func<T, U>. It takes an Expression<Func<T, U>>.
So when you pass the lambda, the compiler converts your lambda into an Expression, pass it to EF, which converts the expression into SQL code.
When you pass a Func<T, U> there's no way to convert it to an Expression. The Enumerable.Select extension method -- which takes a plain Func, not an Expression -- kicks in instead. So EF execute the first part, up to the Where, in SQL. Then the results are passed to Linq to Object that executes the projection in memory.
You can still do dynamic stuff, but you have to create an instance of Expression, not a delegate.

Optimize LINQ query

I have a report that shows orders made to a determined merchant, and it was working fine until I needed to add a filter for payment status.
This is how I build the query, filter by filter:
var queryOrder = context.Orders.Select(m=>m);
if (viewModel.InitialDate.HasValue)
queryOrder = queryOrder.Where(m => m.CreatedDate.Date >= viewModel.InitialDate.Value);
(...) /* continues building the query, filter by filter */
if (viewModel.SelectedPaymentStatus != null)
queryOrder = queryOrder.Where(m => viewModel.SelectedPaymentStatus.Contains(m.Payments.Select(p => p.PaymentStatusId).Single().ToString()));
queryOrder = queryOrder.Where(m => m.MerchantId == merchantId);
When I run queryOrder, even if it's only a queryOrder.Count(), it takes over 1 minute to execute. Using SQL Server's profiling tool, I extracted the generated query as this:
SELECT [t0].[Id], [t0].[CustomerId], [t0].[MerchantId], [t0].[OrderNumber], [t0].[Amount], [t0].[SoftDescriptor], [t0].[ShippingMethod], [t0].[ShippingPrice], [t0].[IpAddress], [t0].[SellerComment], [t0].[CreatedDate]
FROM [dbo].[Order] AS [t0]
WHERE ([t0].[MerchantId] = #p0)
AND ((CONVERT(NVarChar,(
SELECT [t1].[PaymentStatusId]
FROM [dbo].[Payment] AS [t1]
WHERE [t1].[OrderId] = [t0].[Id]
))) IN (#p1, #p2, #p3, #p4, #p5, #p6, #p7, #p8))
the #p0 parameter is a Guid for merchantId, and the #p1 thru #p8 are numeral strings "1" thru "8", representing the paymentStatusId's.
If I skip the line:
if (viewModel.SelectedPaymentStatus != null)
queryOrder = queryOrder.Where(m => viewModel.SelectedPaymentStatus.Contains(m.Payments.Select(p => p.PaymentStatusId).Single().ToString()));
The query runs in under 1 second. But when I use it, the performance hits the floor. Any tips on how to solve this?

All your queries are deferred and this is both the good/bad part of linq. Try to split the queries and use some in-memory results.
Try removing the first query (doesn't make much sense really, you're returning the same collection) and amend the second query to be like this and see if it makes any difference.
var clause = context.Orders.Payments.Select(p => p.PaymentStatusId).Single().ToString();
if (viewModel.SelectedPaymentStatus != null)
var queryOrder = context.Orders.queryOrder.Where(m => viewModel.SelectedPaymentStatus.Contains(clause));

I've annotated the problem snippet:
if (viewModel.SelectedPaymentStatus != null) {
// Give me only orders from queryOrder where...
queryOrder = queryOrder.Where(
// ...my viewModel's SelectedPaymentStatus collection...
m => viewModel.SelectedPaymentStatus.Contains(
// ...contains the order's payment's PaymentStatusId...
m.Payments.Select(p => p.PaymentStatusId).Single()
// ... represented as a string?!
.ToString()
// Why are database IDs strings?
)
);
}
viewModel.SelectedPaymentStatus appears to be a collection of strings; therefore, you're asking the database to convert PaymentStatusId to an nvarchar and do string comparisons with the elements of SelectedPaymentStatus. Yuck.
Since viewModel.SelectedPaymentStatus is small, it might be better to create a temporary List<int> and use that in your query:
if (viewModel.SelectedPaymentStatus != null) {
// Let's do the conversion once, in C#
List<int> statusIds = viewModel.SelectedPaymentStatus.Select( i => Convert.ToInt32(i) ).ToList();
// Now select the matching orders
queryOrder = queryOrder.Where(
m => statusIds.Contains(
m.Payments.Select(p => p.PaymentStatusId).Single())
)
);
}

Linq-to-Sql: How to specify condition that checks for value contained in some pool but has an extra value associated that needs to be checked

Ok, its difficult to explain it in the title above. So here it is hopefully clearer:
Say I have a pool of document numbers and their associated areas (DocNum, Area):
(AB, 1), (GH, 2), (UI, 3)
Now I want to write a Linq-to-Sql query to retrieve information of all the documents with that combination of values from a database, how should I do that? Because I thought it could be done using a dictionary:
var data = from document in context.documents
where dictionary.Contains(new KeyValuePair(document.DocNum, document.Area))
select new
{
...whatever
}
If I try to do it like this then I get the following error:
System.NotSupportedException: The member
'System.Collections.Generic.KeyValuePair`2[System.String,Area].Key'
has no supported translation to SQL.
Can anyone please help?

The LINQ to SQL has no way of representing the in-memory Dictionary lookup as SQL. One way around that is to use AsEnumerable to ensure you are using LINQ to Objects instead:
var data = from document in context.documents.AsEnumerable()
where dictionary.Contains(new KeyValuePair(document.DocNum, document.Area))
select new
{
...whatever
}
Note that this will effectively mean you are fetching every row from the context.documents. If it's a small table, this shouldn't be a problem, but for a large table you should consider doing this kind of filtering server-side, directly in SQL - e.g. by replacing the client-side Dictionary with a server-side table (that you can JOIN with documents), possibly even a temporary table (depending on your needs).
--- EDIT ---
If the Dictionary is small and the table is large but well indexed, it might be worth executing a separate query for each value being searched. For example:
class Program {
static void Main(string[] args) {
var criteria = new Dictionary<KeyValuePair<string, string>, object> {
{ new KeyValuePair<string, string>("n1", "a1"), null },
{ new KeyValuePair<string, string>("n2", "a2"), null },
{ new KeyValuePair<string, string>("n3", "a3"), null },
};
using (var ctx = new DataClasses1DataContext()) {
ctx.Log = Console.Out;
var rows = new List<Document>();
foreach (var criterion in criteria.Keys) {
var q = from document in ctx.Documents
where document.DocNum == criterion.Key && document.Area == criterion.Value
select document;
rows.AddRange(q);
}
foreach (var row in rows)
Console.WriteLine("{0}, {1}", row.DocNum, row.Area);
}
}
}
This prints the following output:
SELECT [t0].[DocNum], [t0].[Area]
FROM [dbo].[Document] AS [t0]
WHERE ([t0].[DocNum] = #p0) AND ([t0].[Area] = #p1)
-- #p0: Input NVarChar (Size = 4000; Prec = 0; Scale = 0) [n1]
-- #p1: Input NVarChar (Size = 4000; Prec = 0; Scale = 0) [a1]
-- Context: SqlProvider(Sql2008) Model: AttributedMetaModel Build: 4.0.30319.1
SELECT [t0].[DocNum], [t0].[Area]
FROM [dbo].[Document] AS [t0]
WHERE ([t0].[DocNum] = #p0) AND ([t0].[Area] = #p1)
-- #p0: Input NVarChar (Size = 4000; Prec = 0; Scale = 0) [n2]
-- #p1: Input NVarChar (Size = 4000; Prec = 0; Scale = 0) [a2]
-- Context: SqlProvider(Sql2008) Model: AttributedMetaModel Build: 4.0.30319.1
SELECT [t0].[DocNum], [t0].[Area]
FROM [dbo].[Document] AS [t0]
WHERE ([t0].[DocNum] = #p0) AND ([t0].[Area] = #p1)
-- #p0: Input NVarChar (Size = 4000; Prec = 0; Scale = 0) [n3]
-- #p1: Input NVarChar (Size = 4000; Prec = 0; Scale = 0) [a3]
-- Context: SqlProvider(Sql2008) Model: AttributedMetaModel Build: 4.0.30319.1
n1, a1
n2, a2
n3, a3
Note however that this wastes Dictionary's innate search capabilities (it might just as well be any other IEnumerable) and performs very badly if there are many dictionary elements.

Contains can use a simple array of single values and translate that to a Where x In {a,b,c} clause in SQL. I don't think Where...In can support multiple columns in SQL. As a result, you really need a join or complex where (SQL - 82 style). You can join a object collection to a table (but not a table to an object collection) as follows:
var data = from dict in dictionary.Values
join document in context.documents
on new {dict.DocNum, dict.Area} equals new {document.DocNum, document.Area}
select new {};
However, this will require that the entire documents table be pulled into memory and the join then being done via LINQ to Objects.