We are converting from LINQ to SQL to Entity Framework Core 2.2 and are finding that the translation of Contains operations do not become IN clauses in SQL Server. What is happening is that EF Core is pulling back all of the data using the other conditions and then filtering it down locally. This is not acceptable. We can use EF.Functions.Contains but this requires us to enable Full Text Search in SQL Server, which is over kill.
Any idea how to get a statement like the following to translate to an IN clause in SQL Server?
var myValues = new [] { 1, 2, 3, 4, 5 };
var qry = _context.Table.Where(t => myValues.Contains(t.TableProperty));
Okay, maybe I over simplified the code to keep it simple for the question. The actual code looks like:
voterQuery = voterQuery.Where(v => voterFilter.VoterStatus.Select(p => p.Value).Contains(v.VotStatus.ToString()));
What is happening in our code is that we are building up an IQueryable from user selections on a filtering screen. voterFilters contains a collection of these selection criteria. VoterStatus is one of the selection criteria which is a List<CheckedListItem>, which are from a Winforms CheckedListItems control.
The selection of p.Value returns a List of strings. I have tried to project the list of strings to an array, with the same results of the IN clause not being created or a server query. Perhaps, EF Core does not allow strings to be used for the IN clause values. Any insight would be appreciated.
Currently (as of v2.2.3), EF Core requires the expression used for Contains to be a simple IEnumerable<T> variable (no LINQ operators) where the T is primitive type.
Which means you need to move the voterFilter.VoterStatus.Select(p => p.Value) into variable outside the query expression tree and use that variable inside:
var voterStatusFilter = voterFilter.VoterStatus.Select(p => p.Value);
voterQuery = voterQuery.Where(v => voteStatusFilter.Contains(v.VotStatus.ToString()));
Related
I want append a item to column with type int[].
i see PostgreSQL document with array_append can do this.
but how can we do this with EF Core? (i use Npgsql.EntityFrameworkCore.PostgreSQL)
This currently isn't supported (the list of supported array translations can be found here).
In the meantime, as a workaround you can use raw SQL queries to express your query directly in SQL, and possibly compose additional LINQ operators over that.
EDIT: opened https://github.com/npgsql/efcore.pg/issues/2026 to track, this should be doable for 6.0.
raw SQL Query:
var sql= $"UPDATE public.\"TableName\" SET \"ArrayColumnName\" = array_append(\"ArrayColumnName\", {value}) WHERE ... ;";//fill ... with your condition
var result =await _dbContext.Database.ExecuteSqlRawAsync(sql) ;//return int
I've searched through numerous examples on how to filter collection based on another collection and I have found an easy way, like so:
var oShipnotes = await DbContext.ShipNotes.Where(s => oManageRouteDto.ShipNotes.Any(mr => mr.Id == s.Id)).ToListAsync();
however it throws an exception that says it cannot be translated to SQL query.
Could anyone point me the right direction how to solve this?
Thanks!
Replace nested LINQ query to materialized list of identifiers:
// 1) get the list of target ship note identifiers
var ids = oManageRouteDto.ShipNotes.Select(mr => mr.Id).ToList();
// 2) pass this list into Where using Contains
var oShipnotes = await DbContext.ShipNotes.Where(s => ids.Contains(s.Id)).ToListAsync();
EF is aware of this pattern and translates IList<T>.Contains into SQL's IN condition.
Since EF deals with IQueryables, each LINQ query must be translated into valid SQL expression. As a result, EF and underlying provider cannot translate every valid LINQ query (from C# perspective) just because SQL is not C#.
I have a local List with entities, some hundreds, and I have a SQL Server table where I store the ID of the successful processed entities, some millions. I would like to know, which entities form my local set are not yet processed i.e. are not in the SQL Table.
The first approach is to iterate through the local list with the following Linq statement:
Entity entity = db.Entities.FirstOrDefault(m => m.ID == ID);
if (entity == null) { NewList.Add(ID) }
the NewList would then contain all the new entities. However this is very slow.
In LINQ, how would you send the entire local list to the SQL Server with one call and then return the ones not in the SQL table?
Do you really have to create a temporary table with my local list, then left-join on the already processed table and return the ones with a null?
Use .Contains method to retrieve already processed ids
and Except to create list of not yet processed ids.
var localList = new List<int> { 1, 2, 3 };
var processed = db.Entities
.Where(entity => localList.Contains(entity.Id))
.Select(entity => entity.Id)
.ToList();
var notProcessed = localList.Except(processed).ToList();
It will depend on provider, but .Contains should generate sql like:
SELECT Id FROM Entity WHERE Id IN (1, 2, 3)
suggestion:
create a temp table and insert your IDs
select your result on the SQL side
EDIT:
"Can you do that in LINQ?"
TL;DR:
yes* but that's an ugly piece of work, write the SQL yourself
*)depends on what you mean with "in" LINQ, because that is not in the scope of LINQ. In other words: a LINQ expression is one layer too abstract, but if you happen to have an LINQ accessible implementation for this, you can use this in your LINQ statements
on the LINQ expression side you have something like:
List<int> lst = new List<int>() { 1,2,3 };
List<int> result = someQueryable.Where(x=>lst.Contains(x.ID)).Select(x=>x.ID).ToList();
the question now is: what happens on the SQL side (assuming the queryable leads us to a SQL database)?
the queryable provider (e.g. Entity Framework) somehow has to translate that into SQL, execute it and come back with the result
here would be the place to modify the translation...
for example examine the expression tree with regard to the object that is the target for the Contains(...) call and if it is more than just a few elements, go for the temp table approach...
the very same LINQ expression can be translated into different SQL commands. The provider decides how the translation has to be done.
if your provider lacks support for large Contains(...) cases, you will probably experience poor performance... good thing is usually nobody forces you to use it this way ... you can skip linq for performance optimized queries, or you could write a provider extension yourself but then you are not on the "doing something with LINQ"-side but extending the functionality of your LINQ provider
if you are not developing a large scalable product that will be deployed to work with different DB-Backends, it is usually not worth the effort... the easier way to go is to write the sql yourself and just use the raw sql option of your db connection
I have an ASP.NET Core Web API which uses Entity Framework Core (version 2.0.2) to return a paged list of a data model called PhotoAlbum. To do this it builds up an IQueryable<PhotoAlbum> like this:
var query = _context.PhotoAlbums
.Include(album => album.SpotlightPhotoView)
.ApplySecurity(user)
.ApplyFilter(filter)
.Sort(sortInfo);
Where ApplySecurity, ApplyFilter and Sort are my own extensions which apply two Where filters and an OrderBy filter respectively. Finally the code uses Skip and Take to return a specific subset of the matched data.
I was interested to see in my logs the following warnings:
The LINQ expression '"Take(__p_2)"' could not be translated and will be evaluated locally.
The LINQ expression '"Skip(__p_1)"' could not be translated and will be evaluated locally.
The LINQ expression '"where [album].Featured"' could not be translated and will be evaluated locally.
The LINQ expression '"where (([album].Complete OrElse False) OrElse False)"' could not be translated and will be evaluated locally.
Having read up on these warnings I now know that certain Linq methods in EF Core cannot be converted to SQL. These include aggregation functions like Sum and Count as well as Skip and Take.
So my first question is: What is the recommended solution if you want to achive a paged query? Is a stored procedure the option option?
My second question is: Why do I have the warnings around the Where clauses?
To ellaborate on this second question, the code which applies the [album].Featured filter looks like this...
query = query.Where(album => album.Featured);
The code which applies the [album].Complete filter looks like this...
query = query
.Where(album =>
album.Complete || filter.IncludeIncomplete
&& album.Published || filter.IncludeUnpublished);
...where filter is a simple model with a set of boolean properties defining how to filter.
And here's the actual SQL executed according to my logs (with a few columns removed from the SELECT for readability):
SELECT [album].[AlbumID], [album].[AlbumDate], [album.SpotlightPhotoView].[PhotoViewID]
FROM [PhotoAlbum] AS [album]
LEFT JOIN [PhotoView] AS [album.SpotlightPhotoView] ON [album].[SpotlightPhotoViewID] = [album.SpotlightPhotoView].[PhotoViewID]
WHERE ([album].[Complete] = 1) AND ([album].[Featured] = 1)
ORDER BY [album].[AlbumDate] DESC
It seems to have applied the Featured and Complete filters fine in spite of the warnings.
I suspect the custom methods and their inputs generate expressions that can't be translated to SQL. where (([album].Complete OrElse False) OrElse False) definitely can't - there's no OrElse in SQL. OrElse is a VB.NET keyword.
where [album].Featured is another suspicious warning. Filtering by boolean properties is definitely supported in all versions. Is Featured a calculated property without the proper configuration perhaps?
Apart from that, GROUP BY and aggregate functions were added in EF Core 2.1, the latest LTS (Long Term Support) version.
Take and Skip definitely work in EF Core 2.2.6 and EF Core 3.0 Preview 8. Trying this query in LinqPad 6 :
Posts.OrderBy(p=>p.PostId).Skip(100).Take(100)
Generates this SQL query :
SELECT [p].[PostId], [p].[BlogId], [p].[Content], [p].[Title]
FROM [Posts] AS [p]
ORDER BY [p].[PostId]
OFFSET #__p_0 ROWS FETCH NEXT #__p_0 ROWS ONLY
I have few related objects and relation is like
public class Project
{
public List<ProjectEdition> editions;
}
public class ProjectEdition
{
public List<EditionItem> items;
}
public class EditionItem
{
}
I wanted to fetch the EditionItems from Last entries of ProjectEditions only for each Project
Example
Project#1 -> Edition#1 [contains few edition items ] , Edition#2 [contains few edition items]
Project#2 -> Edition#1 ,Edition#2 and Edition#3
My required output contains EditionItems from Edition#2 of Project#1 and Edition#3 of Project#2 only . I mean EditionItems from latest edition of a Project or last edition of a Project only
To get this i tried this query
List<EditionItem> master_list = context.Projects.Select(x => x.ProjectEditions.LastOrDefault())
.SelectMany(x => x.EditionItems).ToList();
But its returns error at LatsOrDefault() section
An exception of type 'System.NotSupportedException' occurred in EntityFramework.SqlServer.dll but was not handled in user code
Additional information: LINQ to Entities does not recognize the method '---------.Models.ProjectEdition LastOrDefault[ProjectEdition](System.Collections.Generic.IEnumerable`1
so how can i filter for last edition of a project and then get the list of EditionItems from it in a single LINQ call
Granit got the answer right, so I won't repeat his code. I would like to add the reasons for this behaviour.
Entity Framework is magic (sometimes too much magic) but it yet translates your LINQ queries into SQL and there are limitations to that of what your underlying database can do (SQL Server in this case).
When you call context.Projects.FirstOrDefault() it is translated into something like Select TOP 1 * from Projects. Note the TOP 1 part - this is SQL Server operator that limits number of rows returned. This is part of query optimisation in SQL Server. SQL Server does not have any operators that will give you LAST 1 - because it needs to run the query, return all the results, take the last one and dump the rest - this is not very efficient, think of a table with a couple (bi)million records.
So you need to apply whatever required sort order to your query and limit number of rows you return. If you need last record from the query - apply reverse sort order. You do need to sort because SQL Server does not guarantee order of records returned if no Order By is applied to the query - this is due to the way the data is stored internally.
When you write LINQ queries with EF I do recommend keep an eye on what SQL is generated by your queries - sometimes you'll see how complex they come out and you can easily simplify the query. And sometimes with lazy-loading enabled you introduce N+1 problem with a stroke of a key (literally). I use ExpressProfiler to watch generated SQL, LinqPad can also show you the SQL queries and there are other tools.
You cannot use method LastOrDefault() or Last() as discussed here.
Insetad, you can use OrderByDescending() in conjunction with FirstOrDefault() but first you need to have a property in you ProjectEdition with which you want to order the entities. E.g. if ProjectEdition has a property Id (which there is a good chance it does), you can use the following LINQ query:
List<EditionItem> master_list = context.Projects.Select(
x => x.ProjectEditions
.OrderByDescending(pe => pe.Id)
.FirstOrDefault())
.SelectMany(x => x.EditionItems).ToList();
List<EditionItem> master_list = context.Projects
.Select(p => p.editions.LastOrDefault())
.SelectMany(pe => pe.items).ToList();
IF LastOrDefault not supported you can try using OrderByDescending
List<EditionItem> master_list = context.Projects
.Select(p => p.editions.OrderByDescending(e => e.somefield).FirstOrDefault())
.SelectMany(pe => pe.items).ToList();
from p in context.project
from e in p.projectEdition.LastOrDefault()
select new EditionItem
{
item1 = e.item1
}
Please try this