So, I already know that code like this does everything in memory:
SomeDbObject.SomeNavigationProperty.Where(x => x.IsSomeBool);
In addition to this (still in memory):
SomeDbObject.SomeNavigationProperty.AsQueryable().Where(x => x.IsSomeBool);
So, I came up with a friendly solution that helps me make sure that the whole call will be executed as a SQL command (this is just a property on SomeDbObject):
public IQueryable<AnotherDbObject> QueryableSomeNavigationProperty
{
get
{
return SomeStaticContext.AnotherDbObjects.Where(x => x.ForeignKeyForSomeDbObject == this.Id);
}
}
So, as you can see, this basically does what the Navigation property would have done, but it just creates the command in expression-treeable format, so that subsequent clauses will be built into the SQL command. For example, the same statement as before, would now return an IQueryable upon which we can add a Where clause:
SomeDbObject.QueryableSomeNavigationProperty.Where(x => x.IsSomeBool);
Now, the question is, what happens if I want to query another navigation property in the where clause. For example:
SomeDbObject.QueryableSomeNavigationProperty.Where(x => SomeDbObject.AnotherNavigationProperty.Any());
So, do I need to make another method that returns an IQueryable for SomeDbObject.AnotherNavigationProperty? Or, does EF do the right thing here and build it into a SQL statement?
I can clarify, if needed, but I think this covers the gist of what I am looking for.
Thanks!
Ok, everyone. I ran a bunch of simulations and the verdict is in. See the comments for results in each scenario. Comments denote when the SQL came spitting out! :)
Hope this helps the next poor soul who is confused by what EF6 does at what point!
class Program
{
private static readonly Action<string> DebugWriteLine = s => System.Diagnostics.Debug.WriteLine(s);
private static readonly Action<string> WriteLine = s => { System.Console.WriteLine(s); DebugWriteLine(s); };
static void Main(string[] args)
{
Statics.Entities.Database.Log = WriteLine;
WhereClauseOnSimpleProperty();
WhereClauseOnNavigationProperty();
WhereClauseOnICollection();
WhereClauseOnIQueryable();
WhereClauseOnIQueryableWithIQueryable();
System.Console.ReadKey();
}
static void WhereClauseOnSimpleProperty()
{
WriteLine("Get objects with a where clause (simple property).");
WriteLine(" Calling: var users = entities.Users.Where(u => u.FirstName == \"Julie\");");
var users = Statics.Entities.Users.Where(u => u.FirstName == "Julie");
WriteLine(" Calling: users.ToList();");
var usersList = users.ToList();
// SQL got built and called here (NOTE: SQL call is not made until the data needs to be "realized"):
/* SELECT
[Extent1].[Id] AS [Id],
[Extent1].[Uin] AS [Uin],
[Extent1].[ClientId] AS [ClientId],
[Extent1].[FirstName] AS [FirstName],
[Extent1].[LastName] AS [LastName]
FROM [dbo].[Users] AS [Extent1]
WHERE 'Julie' = [Extent1].[FirstName]
*/
WriteLine(" There are " + usersList.Count + " users.");
}
static void WhereClauseOnNavigationProperty()
{
WriteLine("Get objects with a where clause (1-to-many navigation property).");
WriteLine(" Calling: var users = Entities.Users.Where(u => u.FirstName == \"Julie\" && u.Votes.Any());");
var users = Statics.Entities.Users.Where(u => u.FirstName == "Julie" && u.Votes.Any());
WriteLine(" Calling: users.ToList();");
var usersList = users.ToList();
// SQL got built and called here (NOTE: using the ICollection navigation property on the lambda parameter "u" builds just one SQL statement):
/* SELECT
[Extent1].[Id] AS [Id],
[Extent1].[Uin] AS [Uin],
[Extent1].[ClientId] AS [ClientId],
[Extent1].[FirstName] AS [FirstName],
[Extent1].[LastName] AS [LastName]
FROM [dbo].[Users] AS [Extent1]
WHERE ('Julie' = [Extent1].[FirstName]) AND ( EXISTS (SELECT
1 AS [C1]
FROM [dbo].[Votes] AS [Extent2]
WHERE [Extent1].[Id] = [Extent2].[UserId]
))
*/
WriteLine(" There are " + usersList.Count + " users.");
}
static void WhereClauseOnICollection()
{
WriteLine("Get objects with a where clause (simple property) from an ICollection.");
WriteLine(" Calling: var users = Entities.Users.First(u => u.FirstName == \"Julie\" && u.Votes.Any());");
var user = Statics.Entities.Users.First(u => u.FirstName == "Julie" && u.Votes.Any());
// SQL got built and called here (NOTE: data is realized immediately because we are allocating a single object):
/* SELECT TOP (1)
[Extent1].[Id] AS [Id],
[Extent1].[Uin] AS [Uin],
[Extent1].[ClientId] AS [ClientId],
[Extent1].[FirstName] AS [FirstName],
[Extent1].[LastName] AS [LastName]
FROM [dbo].[Users] AS [Extent1]
WHERE ('Julie' = [Extent1].[FirstName]) AND ( EXISTS (SELECT
1 AS [C1]
FROM [dbo].[Votes] AS [Extent2]
WHERE [Extent1].[Id] = [Extent2].[UserId]
))
*/
WriteLine(" Calling: var votes = user.Votes.AsQueryable().Where(v => v.VoteValue > 0);");
var votes = user.Votes.AsQueryable().Where(v => v.VoteValue > 0);
// SQL got built and called here (NOTE: there "where" clause is executed in app memory/time [it's not in the SQL call]):
/* SELECT
[Extent1].[Id] AS [Id],
[Extent1].[UserId] AS [UserId],
[Extent1].[VoteValue] AS [VoteValue]
FROM [dbo].[Votes] AS [Extent1]
WHERE [Extent1].[UserId] = #EntityKeyValue1
*/
WriteLine(" Calling: votes.ToList();");
var votesList = votes.ToList();
WriteLine(" There are " + votesList.Count + " votes.");
}
static void WhereClauseOnIQueryable()
{
WriteLine("Get objects with a where clause (1-to-many navigation property) from an IQueryable.");
WriteLine(" Calling: var users = Entities.Users.First(u => u.FirstName == \"Julie\" && u.Votes.Any());");
var user = Statics.Entities.Users.First(u => u.FirstName == "Julie" && u.Votes.Any());
// SQL got built and called here:
/* SELECT TOP (1)
[Extent1].[Id] AS [Id],
[Extent1].[Uin] AS [Uin],
[Extent1].[ClientId] AS [ClientId],
[Extent1].[FirstName] AS [FirstName],
[Extent1].[LastName] AS [LastName]
FROM [dbo].[Users] AS [Extent1]
WHERE ('Julie' = [Extent1].[FirstName]) AND ( EXISTS (SELECT
1 AS [C1]
FROM [dbo].[Votes] AS [Extent2]
WHERE [Extent1].[Id] = [Extent2].[UserId]
))
*/
WriteLine(" Calling: var votes = user.QueryableVotes.Where(v => user.Votes.AsQueryable().Contains(v));");
var votes = user.QueryableVotes.Where(v => user.Votes.AsQueryable().Contains(v));
// SQL got built and called here (NOTE: this is just the "user.Votes.AsQueryable().Contains(v)" part of the query):
/* SELECT
[Extent1].[Id] AS [Id],
[Extent1].[UserId] AS [UserId],
[Extent1].[VoteValue] AS [VoteValue]
FROM [dbo].[Votes] AS [Extent1]
WHERE [Extent1].[UserId] = #EntityKeyValue1
*/
WriteLine(" Calling: votes.ToList();");
var votesList = votes.ToList();
// NOTE: EF6 dies here because it had already computed "user.Votes.Contains(v)" (see above), and that can't go into the query.
WriteLine(" There are " + votesList.Count + " votes.");
}
static void WhereClauseOnIQueryableWithIQueryable()
{
WriteLine("Get objects with a where clause (1-to-many navigation property as an IQueryable) from an IQueryable.");
WriteLine(" Calling: var users = Entities.Users.First(u => u.FirstName == \"Julie\" && u.Votes.Any());");
var user = Statics.Entities.Users.First(u => u.FirstName == "Julie" && u.Votes.Any());
// SQL got built and called here:
/* SELECT TOP (1)
[Extent1].[Id] AS [Id],
[Extent1].[Uin] AS [Uin],
[Extent1].[ClientId] AS [ClientId],
[Extent1].[FirstName] AS [FirstName],
[Extent1].[LastName] AS [LastName]
FROM [dbo].[Users] AS [Extent1]
WHERE ('Julie' = [Extent1].[FirstName]) AND ( EXISTS (SELECT
1 AS [C1]
FROM [dbo].[Votes] AS [Extent2]
WHERE [Extent1].[Id] = [Extent2].[UserId]
))
*/
WriteLine(" Calling: var votes = user.QueryableVotes.Where(v => user.QueryableVotes.Contains(v));");
var votes = user.QueryableVotes.Where(v => user.QueryableVotes.Contains(v)); // Yes, I know this is reduntant...just making sure the SQL looks right.
WriteLine(" Calling: votes.ToList();");
var votesList = votes.ToList();
// SQL got built and called here (NOTE: making all expressions true IQueryables will build the "correct" [one call to rule them all] SQL expression):
/* SELECT
[Extent1].[Id] AS [Id],
[Extent1].[UserId] AS [UserId],
[Extent1].[VoteValue] AS [VoteValue]
FROM [dbo].[Votes] AS [Extent1]
WHERE ([Extent1].[UserId] = #p__linq__0) AND ( EXISTS (SELECT
1 AS [C1]
FROM [dbo].[Votes] AS [Extent2]
WHERE ([Extent2].[UserId] = #p__linq__1) AND ([Extent2].[Id] = [Extent1].[Id])
))
*/
WriteLine(" There are " + votesList.Count + " votes.");
}
// SPECIAL NOTE: The clauses should follow these guidelines:
/*
* 1. If the condition operates on the lambda parameter, then use the ICollection navigation property to achieve one statement.
* For example: var user = Statics.Entities.Users.First(u => u.FirstName == "Julie" && u.Votes.Any());
* 2. If the condition operates on a "non-navigation" property of the lambda parameter, then use the IQueryable expression to acheive one statement.
* For example: var votes = user.QueryableVotes.Where(v => user.QueryableVotes.Contains(v));
*/
}
public partial class User
{
public IQueryable<Vote> QueryableVotes
{
get
{
return Statics.Entities.Votes.Where(v => v.UserId == this.Id);
}
}
}
Related
I have two related tables lets say:
Table Products (Main Table)
ID
Name
Type
Table Parts (Child of Products, contains detailled Information)
ID
ProductID
PartName
PartValue
I would like to get Products ordered by the value of a specific part (e.G. Engine)
I came up with the following:
// Code to construct a query to get all desired products
products = products.OrderBy(c => c.Parts.Where(d => d.PartName == "Engine").Select(d => d.Value).FirstOrDefault());
This works but is too slow. Can I improve the query or will I have to redsign my database so I won't be sorting like this in the first place?
Generated SQL-Query:
SELECT
[Project3].[ID] AS [ID],
[Project3].[Name] AS [Name]
FROM ( SELECT
[Project2].[ID] AS [ID],
[Project2].[Name] AS [Name],
[Project2].[C1] AS [C1]
FROM ( SELECT
[Extent1].[ID] AS [ID],
[Extent1].[Name] AS [Name],
(SELECT TOP (1)
[Extent2].[PartValue] AS [PartValue]
FROM [dbo].[Parts] AS [Extent2]
WHERE ([Extent1].[ID] = [Extent2].[ProductID]) AND (N'Engine' = [Extent2].[PartName])) AS [C1]
FROM [dbo].[Products] AS [Extent1]
WHERE (1 = [Extent1].[Type])
) AS [Project2]
) AS [Project3]
ORDER BY [Project3].[C1] ASC
If you want to try a join, it would be a bit like this:
Products
.Join(Parts, pd => pd.ID, pt => pt.ProductID, (pd, pt) => new { pd.Name, pt.PartName, pt.PartValue })
.Where(x => x.PartName == "Engine")
.OrderBy(x => x.PartValue);
This would result in a single select statement without the inner selects from before
I am implementing a search for my application. The search works except for when the user's search cascades by searching related entities. I have debugged the code and tested the SQL generated by Entity Framework. I found the problem is Contains() translates to a '=' in SQL when it should be 'LIKE'. Contains() works as intended for my initial predicate on FirstName, MiddleName etc. but not in the if (cascade) code block.
My C# search logic:
public IList<Individual> Find(string search, bool cascade, bool includeInactive)
{
_context.Database.Log = s => System.Diagnostics.Debug.WriteLine(s);
IQueryable<Individual> query = _context.Individuals;
if (!string.IsNullOrWhiteSpace(search))
{
search = search.Trim();
string[] searchParts = search.Split(' ');
ExpressionStarter<Individual> predicate = PredicateBuilder.New<Individual>(false);
foreach (string searchPart in searchParts)
{
predicate = predicate.Or(c =>
c.FirstName.Contains(searchPart) || c.MiddleNames.Contains(searchPart) ||
c.LastName.Contains(searchPart) || c.PreferredName.Contains(searchPart));
if (cascade)
{
predicate = predicate.Or(c =>
c.IndividualOrganisationGroups.Select(og => og.OrganisationGroup).Select(g => g.Group.Name)
.Contains(searchPart));
}
}
query = query.Where(predicate);
}
if (!includeInactive)
{
query = query.Where(c => c.Active);
}
return query.ToList();
}
The SQL generated from EF:
SELECT
[Extent1].[ID] AS [ID],
[Extent1].[RegistrationTypeID] AS [RegistrationTypeID],
[Extent1].[Title] AS [Title],
[Extent1].[FirstName] AS [FirstName],
[Extent1].[MiddleNames] AS [MiddleNames],
[Extent1].[LastName] AS [LastName],
[Extent1].[PreferredName] AS [PreferredName],
[Extent1].[RegistrationNumber] AS [RegistrationNumber],
[Extent1].[Username] AS [Username],
[Extent1].[AzureID] AS [AzureID],
[Extent1].[Notes] AS [Notes],
[Extent1].[Active] AS [Active],
[Extent1].[CreatedDate] AS [CreatedDate],
[Extent1].[CreatedBy] AS [CreatedBy],
[Extent1].[UpdatedDate] AS [UpdatedDate],
[Extent1].[UpdatedBy] AS [UpdatedBy]
FROM [dbo].[Individual] AS [Extent1]
WHERE ([Extent1].[FirstName] LIKE #p__linq__0 ESCAPE '~') OR ([Extent1].[MiddleNames] LIKE #p__linq__1 ESCAPE '~') OR ([Extent1].[LastName] LIKE #p__linq__2 ESCAPE '~') OR ([Extent1].[PreferredName] LIKE #p__linq__3 ESCAPE '~') OR ( EXISTS (SELECT
1 AS [C1]
FROM [dbo].[IndividualOrganisationGroup] AS [Extent2]
INNER JOIN [dbo].[OrganisationGroup] AS [Extent3] ON [Extent2].[OrganisationGroupID] = [Extent3].[ID]
INNER JOIN [dbo].[Group] AS [Extent4] ON [Extent3].[GroupID] = [Extent4].[ID]
WHERE ([Extent1].[ID] = [Extent2].[IndividualID]) AND (([Extent4].[Name] = #p__linq__4) OR (1 = 0))
))
-- p__linq__0: '%screen%' (Type = AnsiString, Size = 8000)
-- p__linq__1: '%screen%' (Type = AnsiString, Size = 8000)
-- p__linq__2: '%screen%' (Type = AnsiString, Size = 8000)
-- p__linq__3: '%screen%' (Type = AnsiString, Size = 8000)
-- p__linq__4: 'screen' (Type = AnsiString, Size = 8000)
The problematic SQL (last WHERE clause):
WHERE ([Extent1].[ID] = [Extent2].[IndividualID]) AND (([Extent4].[Name] = #p__linq__4) OR (1 = 0))
What it should look like:
WHERE ([Extent1].[ID] = [Extent2].[IndividualID]) AND (([Extent4].[Name] LIKE #p__linq__4) OR (1 = 0))
So my question is, how do I make this code translate to a LIKE in my Entity Framework SQL?
predicate = predicate.Or(c =>
c.IndividualOrganisationGroups.Select(og => og.OrganisationGroup).Select(g => g.Group.Name)
.Contains(searchPart));
It has nothing to do with LINQKit PredicateBuilder. The reason is that the result of
c.IndividualOrganisationGroups.Select(og => og.OrganisationGroup).Select(g => g.Group.Name)
is IEnumerable<string> (or IQueryable<string>), so your are hitting the Enumerable (or Queryable) Contains method instead of the string.Contains as in the other places.
Instead of Select + Contains, what you really need is the Any extension method.
The most concise syntax with your sample is:
c => c.IndividualOrganisationGroups
.Any(og => og.OrganisationGroup.Group.Name.Contains(searchPart))
Of course if you need to check more properties of the Group (or even w/o that), then use Select + Any:
c => c.IndividualOrganisationGroups
.Select(og => og.OrganisationGroup.Group)
.Any(g => g.Name.Contains(searchPart))
or the most natural way of turning collection with filter to single bool by writing the query in a "normal" way. e.g. Select, Where etc. and putting parameterless Any at the end:
c => c.IndividualOrganisationGroups
.Select(og => og.OrganisationGroup.Group)
.Where(g => g.Name.Contains(searchPart))
.Any())
I am trying to understand why is join in my case faster than statement which use navigation property. I have two queries.
First with navigation property :
var result = (from users in context.MetricBloodPreasure
orderby users.User.LastName, users.User.FirstName
select new
{
UserName = users.User.LastName + ", " + users.User.FirstName,
Date = users.DateOfValue,
}).ToList();
Generatet sql :
SELECT
[Project1].[C1] AS [C1],
[Project1].[C2] AS [C2],
[Project1].[DateOfValue] AS [DateOfValue]
FROM ( SELECT
[Extent1].[DateOfValue] AS [DateOfValue],
[Extent2].[FirstName] AS [FirstName],
[Extent2].[LastName] AS [LastName],
1 AS [C1],
CASE WHEN ([Extent2].[LastName] IS NULL) THEN N'' ELSE [Extent2].[LastName] END + N', ' + CASE WHEN ([Extent2].[FirstName] IS NULL) THEN N'' ELSE [Extent2].[FirstName] END AS [C2]
FROM [dbo].[MetricBloodPreasure] AS [Extent1]
INNER JOIN [dbo].[User] AS [Extent2] ON [Extent1].[UserId] = [Extent2].[Id]
) AS [Project1]
ORDER BY [Project1].[LastName] ASC, [Project1].[FirstName] ASC
Second with join:
var result1 = (from u in context.User
orderby u.LastName, u.FirstName
join us in context.MetricBloodPreasure
on u.Id equals us.UserId into users
from s in users
select new
{
UserName = s.User.LastName + ", " + s.User.FirstName,
Date = s.DateOfValue,
}).ToList();
Generated sql:
SELECT
1 AS [C1],
CASE WHEN ([Extent1].[LastName] IS NULL) THEN N'' ELSE [Extent1].[LastName] END + N', ' + CASE WHEN ([Extent1].[FirstName] IS NULL) THEN N'' ELSE [Extent1].[FirstName] END AS [C2],
[Extent2].[DateOfValue] AS [DateOfValue]
FROM [dbo].[User] AS [Extent1]
INNER JOIN [dbo].[MetricBloodPreasure] AS [Extent2] ON ([Extent1].[Id] = [Extent2].[UserId]) AND ([Extent2].[UserId] = [Extent1].[Id])
Before running first query, call var user = context.User.FirstOrDefault(); because I think open connection to database take some time.
Results :
Navigation property query : 00:00:00.6719646
Join query : 00:00:00.4941169
Looking at results it seems that Linq queries that use joins instead of navigation properties are faster. Is that true or I'm doing something wrong ?
To get a better insight into what it is doing you should get the raw SQL and you can check out the execution plan yourself.
To do this, you can either use SQL Profiler to see what query is being run, or you can log the SQL query itself by doing something like this before you run your query:
context.Database.Log = s => System.Diagnostics.Debug.WriteLine(s);
Also doing a simple benchmark like you did by running each once isn't going to necessarily be reliable. You'll want to run it multiple times and average things out. You should also run them in the opposite order to see if that changes things as well.
I have a LINQ query that gets data via Entity Framework Code First from an SQL database. This works, but it works very very slow.
This is the original query:
var tmpResult = from mdv in allMetaDataValues
where mdv.Metadata.InputType == MetadataInputType.String && mdv.Metadata.ShowInFilter && !mdv.Metadata.IsHidden && !string.IsNullOrEmpty(mdv.ValueString)
group mdv by new
{
mdv.ValueString,
mdv.Metadata
} into g
let first = g.FirstOrDefault()
select new
{
MetadataTitle = g.Key.Metadata.Title,
MetadataID = g.Key.Metadata.ID,
CollectionColor = g.Key.Metadata.Collection.Color,
CollectionID = g.Key.Metadata.Collection.ID,
MetadataValueCount = 0,
MetadataValueTitle = g.Key.ValueString,
MetadataValueID = first.ID
};
This is the generated SQL from the original query:
{SELECT
0 AS [C1],
[Project4].[Title] AS [Title],
[Project4].[ID] AS [ID],
[Extent9].[Color] AS [Color],
[Project4].[Collection_ID] AS [Collection_ID],
[Project4].[ValueString] AS [ValueString],
[Project4].[C1] AS [C2]
FROM (SELECT
[Project2].[ValueString] AS [ValueString],
[Project2].[ID] AS [ID],
[Project2].[Title] AS [Title],
[Project2].[Collection_ID] AS [Collection_ID],
(SELECT TOP (1)
[Filter4].[ID1] AS [ID]
FROM ( SELECT [Extent6].[ID] AS [ID1], [Extent6].[ValueString] AS [ValueString], [Extent7].[Collection_ID] AS [Collection_ID1], [Extent8].[ID] AS [ID2], [Extent8].[InputType] AS [InputType], [Extent8].[ShowInFilter] AS [ShowInFilter], [Extent8].[IsHidden] AS [IsHidden1]
FROM [dbo].[MetadataValue] AS [Extent6]
LEFT OUTER JOIN [dbo].[Media] AS [Extent7] ON [Extent6].[Media_ID] = [Extent7].[ID]
INNER JOIN [dbo].[Metadata] AS [Extent8] ON [Extent6].[Metadata_ID] = [Extent8].[ID]
WHERE ( NOT (([Extent6].[ValueString] IS NULL) OR (( CAST(LEN([Extent6].[ValueString]) AS int)) = 0))) AND ([Extent7].[IsHidden] <> cast(1 as bit))
) AS [Filter4]
WHERE (2 = CAST( [Filter4].[InputType] AS int)) AND ([Filter4].[ShowInFilter] = 1) AND ([Filter4].[IsHidden1] <> cast(1 as bit)) AND ([Filter4].[Collection_ID1] = #p__linq__0) AND (([Project2].[ValueString] = [Filter4].[ValueString]) OR (([Project2].[ValueString] IS NULL) AND ([Filter4].[ValueString] IS NULL))) AND (([Project2].[ID] = [Filter4].[ID2]) OR (1 = 0))) AS [C1]
FROM ( SELECT
[Distinct1].[ValueString] AS [ValueString],
[Distinct1].[ID] AS [ID],
[Distinct1].[Title] AS [Title],
[Distinct1].[Collection_ID] AS [Collection_ID]
FROM ( SELECT DISTINCT
[Filter2].[ValueString] AS [ValueString],
[Filter2].[ID3] AS [ID],
[Filter2].[InputType1] AS [InputType],
[Filter2].[Title1] AS [Title],
[Filter2].[ShowInFilter1] AS [ShowInFilter],
[Filter2].[IsHidden2] AS [IsHidden],
[Filter2].[Collection_ID2] AS [Collection_ID]
FROM ( SELECT [Filter1].[ValueString], [Filter1].[Collection_ID3], [Filter1].[IsHidden3], [Filter1].[ID3], [Filter1].[InputType1], [Filter1].[Title1], [Filter1].[ShowInFilter1], [Filter1].[IsHidden2], [Filter1].[Collection_ID2]
FROM ( SELECT [Extent1].[ValueString] AS [ValueString], [Extent2].[Collection_ID] AS [Collection_ID3], [Extent4].[IsHidden] AS [IsHidden3], [Extent5].[ID] AS [ID3], [Extent5].[InputType] AS [InputType1], [Extent5].[Title] AS [Title1], [Extent5].[ShowInFilter] AS [ShowInFilter1], [Extent5].[IsHidden] AS [IsHidden2], [Extent5].[Collection_ID] AS [Collection_ID2]
FROM [dbo].[MetadataValue] AS [Extent1]
LEFT OUTER JOIN [dbo].[Media] AS [Extent2] ON [Extent1].[Media_ID] = [Extent2].[ID]
INNER JOIN [dbo].[Metadata] AS [Extent3] ON [Extent1].[Metadata_ID] = [Extent3].[ID]
LEFT OUTER JOIN [dbo].[Metadata] AS [Extent4] ON [Extent1].[Metadata_ID] = [Extent4].[ID]
LEFT OUTER JOIN [dbo].[Metadata] AS [Extent5] ON [Extent1].[Metadata_ID] = [Extent5].[ID]
WHERE ( NOT (([Extent1].[ValueString] IS NULL) OR (( CAST(LEN([Extent1].[ValueString]) AS int)) = 0))) AND ([Extent2].[IsHidden] <> cast(1 as bit)) AND (2 = CAST( [Extent3].[InputType] AS int)) AND ([Extent3].[ShowInFilter] = 1)
) AS [Filter1]
WHERE [Filter1].[IsHidden3] <> cast(1 as bit)
) AS [Filter2]
WHERE [Filter2].[Collection_ID3] = #p__linq__0
) AS [Distinct1]
) AS [Project2] ) AS [Project4]
LEFT OUTER JOIN [dbo].[Collection] AS [Extent9] ON [Project4].[Collection_ID] = [Extent9].[ID]}
If we remove the "let first = g.FirstOrDefault()" and change "MetadataValueID = first.ID" to "MetadataValueID = 0" so that we just have a fixed ID = 0 for testing purposes, then the data loads very fast and the generated query itself is half the size compared to the original
So it seems that this part is making the query very slow:
let first = g.FirstOrDefault()
...
MetadataValueID = first.ID
};
How can this be rewritten?
If I try to rewrite the code, it is still slow:
MetadataValueID = g.Select(x => x.ID).FirstOrDefault()
or
let first = g.Select(x => x.ID).FirstOrDefault()
...
MetadataValueID = first
};
Any suggestions?
Using EF I have allways felt that it has problems efficiently translating stuff like g.Key.Metadata.Collection, so I try to join more explicitly and to include only fields, that are neccessary for your result. You can use include instead of join using repository pattern.
Then your query would look like this:
from mdv in allMetaDataValues.Include("Metadata").Include("Metadata.Collection")
where mdv.Metadata.InputType == MetadataInputType.String &&
mdv.Metadata.ShowInFilter &&
!mdv.Metadata.IsHidden &&
!string.IsNullOrEmpty(mdv.ValueString)
group mdv by new
{
MetadataID = mdv.Metadata.ID,
CollectionID = mdv.Metadata.Collection.ID,
mdv.Metadata.Title,
mdv.Metadata.Collection.Color,
mdv.ValueString
} into g
let first = g.FirstOrDefault().ID
select new
{
MetadataTitle = g.Key.Title,
MetadataID = g.Key.MetadataID,
CollectionColor = g.Key.Color,
CollectionID = g.Key.CollectionID,
MetadataValueCount = 0,
MetadataValueTitle = g.Key.ValueString,
MetadataValueID = first
}
Good tool for playing with linq is LinqPad.
The problem is also that:
let first = g.FirstOrDefault().ID
cannot be easily translated to SQL see this answer. But this rewrite simplifies the underlying query for it at least. It remains to me unclear, why you need first ID from a set without using orderby.
It could be rewriten like this:
let first = (from f in allMetaDataValues
where f.Metadata.ID == g.Key.MetadataID &&
f.ValuesString == g.Key.ValuesString select f.ID)
.FirstOrDefault()
This way you do not let EF write the query for you and you can specify exactly how to do the select.
To speed up the query you can also consider adding indexes to database according to the generated query - namely index using both colums used in where clause of this let first query.
Try the following solution.
Replace FirstOrDefault() with .Take(1). FirstOrDefault() is not lazy loaded.
var tmpResult = from mdv in allMetaDataValues
where mdv.Metadata.InputType == MetadataInputType.String && mdv.Metadata.ShowInFilter && !mdv.Metadata.IsHidden && !string.IsNullOrEmpty(mdv.ValueString)
group mdv by new
{
mdv.ValueString,
mdv.Metadata
} into g
let first = g.Take(1)
select new
{
MetadataTitle = g.Key.Metadata.Title,
MetadataID = g.Key.Metadata.ID,
CollectionColor = g.Key.Metadata.Collection.Color,
CollectionID = g.Key.Metadata.Collection.ID,
MetadataValueCount = 0,
MetadataValueTitle = g.Key.ValueString,
MetadataValueID = first.ID
};
I'm trying to understand why the sql for these two statements are different (The difference appears to be in determining whether to added SignedXml in the select or use a case statement. But why?).
myTable.Where(p => p.OwnerID.Equals(owner) && p.FolderID == folderId)
.Select(source => new SignedNativeAnalysis
{
ID = source.ID,
Name = source.Name,
Created = source.Created,
FolderID = source.FolderID,
Locked = source.Locked,
Modified = source.Modified,
OwnerID = source.OwnerID,
IsSigned = source.SignedXml != null
});
which generates something like:
exec sp_executesql N'SELECT
[Extent1].[ID] AS [ID],
[Extent1].[Name] AS [Name],
[Extent1].[Created] AS [Created],
[Extent1].[FolderID] AS [FolderID],
[Extent1].[Locked] AS [Locked],
[Extent1].[Modified] AS [Modified],
[Extent1].[OwnerID] AS [OwnerID],
CASE WHEN ([Extent1].[SignedXml] IS NOT NULL) THEN cast(1 as bit) WHEN ([Extent1].[SignedXml] IS NULL) THEN cast(0 as bit) END AS [C1]
FROM [dbo].[NativeAnalyses] AS [Extent1]
WHERE ([Extent1].[OwnerID] = #p__linq__0) AND ([Extent1].[FolderID] = #p__linq__1)',N'#p__linq__0 uniqueidentifier,#p__linq__1
vs when I'm using a function for the select:
myTable.Where(p => p.OwnerID.Equals(owner) && p.FolderID == folderId)
.Select(Select);
and Select is
private SignedNativeAnalysis Select(NativeAnalysis source)
{
return new SignedNativeAnalysis
{
ID = source.ID,
Name = source.Name,
Created = source.Created,
FolderID = source.FolderID,
Locked = source.Locked,
Modified = source.Modified,
OwnerID = source.OwnerID,
IsSigned = source.SignedXml != null
};
}
which generates something like:
exec sp_executesql N'SELECT
[Extent1].[ID] AS [ID],
[Extent1].[Name] AS [Name],
[Extent1].[ModelXml] AS [ModelXml],
[Extent1].[Created] AS [Created],
[Extent1].[Modified] AS [Modified],
[Extent1].[OwnerID] AS [OwnerID],
[Extent1].[SignedXml] AS [SignedXml],
[Extent1].[FolderID] AS [FolderID],
[Extent1].[Locked] AS [Locked]
FROM [dbo].[NativeAnalyses] AS [Extent1]
WHERE ([Extent1].[OwnerID] = #p__linq__0) AND ([Extent1].[FolderID] = #p__linq__1)',N'#p__linq__0 uniqueidentifier,#p__linq__1
And SignedNativeAnalysis just inherits NativeAnalysis with one extra property.
internal class SignedNativeAnalysis : NativeAnalysis
{
public bool IsSigned { get; set; }
}
UPDATE:
As per suggested I tried to use an expression to achieve the result I wanted but I got an exception instead:
Expression<Func<NativeAnalysis, SignedNativeAnalysis>> signed = p => Select(p);
myTable.Where(p => p.OwnerID.Equals(owner) && p.FolderID == folderId)
.Select(signed)
LINQ to Entities does not recognize the method 'SignedNativeAnalysis
Select(OverseerUI.EntityFramework.NativeAnalysis)' method, and this
method cannot be translated into a store expression.
There's a trick. Entity Framework Select doesn't take a Func<T, U>. It takes an Expression<Func<T, U>>.
So when you pass the lambda, the compiler converts your lambda into an Expression, pass it to EF, which converts the expression into SQL code.
When you pass a Func<T, U> there's no way to convert it to an Expression. The Enumerable.Select extension method -- which takes a plain Func, not an Expression -- kicks in instead. So EF execute the first part, up to the Where, in SQL. Then the results are passed to Linq to Object that executes the projection in memory.
You can still do dynamic stuff, but you have to create an instance of Expression, not a delegate.