We are using Entity Framework Core 3 with SqlServer Database. Business program needs to create many columns which are not in the Database, due to storage, high querying cost etc. Currently, the team is Copying the whole Database Layer, and Creating whole another layer adding computed members in new entities. Currently taking database layer and applying AutoMapper to new layer. For some reason, this does not seem like optimal method.
In this example, we require computed members called
FullName => FirstName + LastName
AccountValue => Quantity * StockPrice
Entity Framework 3 does not allow Client Side Evaluation anymore, https://devblogs.microsoft.com/dotnet/announcing-ef-core-3-0-and-ef-6-3-general-availability/
so curious what is standardized way for computed members in Entity Framework Core 3?
Reading this article, curious wondering what is up to date, and can be used?
Or does Entity Framework Core 3 Offer New Syntax?
https://daveaglick.com/posts/computed-properties-and-entity-framework
1) We could Materialize the entities. Seems okay, however this forces developer to remember utilize ToList(), had issues where developers forget, causing long db scanning queries, or clientside evaluation caused error.
var result = ctx.Customers
.ToList()
.Select(c => new
{
FullName = c.FullName,
AccountValue = c.AccountValue
});
2) Create Queryable Extension. This only extracts the computed columns, or forces developers to create computed members all in one class (breaks SRP single responsibility idea). Unless there is an alternative modification which address this. This also brings composition chain issues, and possible performance problems like option 1.
public static IQueryable<CustomerData> SelectCustomerData(this IQueryable<Customer> customers) { return customers.Select(c => new CustomerData {
FullName = c.FirstName + " " + c.LastName,
AccountValue = c.Holdings.Sum(h => h.Quantity * h.Stock.Price) }); }
3) Expression Projection, does not allow assignment in Select without Linq Expression Project. Company does not allow this third party tool, unless built from Microsoft.
public readonly Expression<Func<Customer, decimal>> AccountValueExpression = c => c.Holdings.Sum(h => h.Quantity * h.Stock.Price);
Or does Entity Framework Core 3 offer newer syntax?
Solution needs to be where, (a) person can extract some or All of the existing members of original DBEntity, (b) and some or all of New Members,
Example, need FirstName (Existing) and AccountValue (New Member)
Or FullName, FirstName, LastName, StockPrice,
Or Everything, FirstName, LastName, FullName ,Quantity, StockPrice, AccountValue, etc, etc
Any mix or match from entities.
Actually migrating from 2.2 to Core 3, however 2.2 has ClientSide Evaluation Disabled. Cannot utilize third party tools, like Linq.Translations, or DelegateCompiler unless they are created from Microsoft vendor .
Prefer not to use SqlServer Computed columns, as we are relying on DBA team. Additionally there are more intricate calculations.
Client side evaluation is evil, thus developers of EF Core 3 has made a good decision to forbid it. The code which could evaluates on client often leads to annoying performance issues. So I wouldn't recommend you to use computed properties in EF Core 2.* as well.
what is standardized way for computed members in Entity Framework Core
If you want to do a computation, sorting, modification, etc. as a part of your query, you should project your entity into DTO at first. In such a case, the query will be compiled into SQL query (and not evaluated on client).
For this task you can use AutoMapper library. It automatically maps properties with the same name. Other properties (computed properties) can be mapped using custom expression.
var config = new MapperConfiguration(cfg =>
{
cfg.CreateMap<Customer, CustomerDto>()
.ForMember(x => x.FullName, x => x.MapFrom(z => z.FirstName + " " + z.LastName))
.ForMember(x => x.AccountValue, x => x.MapFrom(z => z.Quantity * z.StockPrice));
});
var mapper = config.CreateMapper();
Then, you can use ProjectTo extension method. ProjectTo internally call Select so it doesn't materialize entity. Hence, Where statement is parsed into SQL query.
var customers = await context.Customers
.ProjectTo<CustomerDto>(mapper.ConfigurationProvider)
.Where(x => x.FullName == "full name" && x.AccountValue > 4)
.ToListAsync();
Projection of entities is often a good practice. It allows you to select just a few columns from DB and offers you other stuff that is not possible when you are returning just plain entities (e.g. sorting):
var config = new MapperConfiguration(cfg =>
{
cfg.CreateMap<Invoice, InvoiceDto>();
cfg.CreateMap<Customer, CustomerDto>()
.ForMember(x => x.Invoices, x => x.MapFrom(z => z.Invoices.OrderBy(x => x.Date)));
});
var mapper = config.CreateMapper();
// Customers with invoices sorted by date
var customers = await context.Customers
.ProjectTo<CustomerDto>(mapper.ConfigurationProvider)
.ToListAsync();
AutoMapper can be also used with DI. However, it is 3rd party library. If your company doesn't permit it, you can create your own mapping layer by hand. Which includes a lot of monkey work ..
Assuming it's supported by your backend, you can mirror your computed properties with computed database columns.
public string FullName => FirstName + LastName;
entity.Property(e => e.FullName).HasComputedColumnSql("FirstName + LastName");
Then you can trivially filter, order by, project etc those properties.
I haven't tried this, but it is just a thought - How about creating a custom Extension method of DbFunctions?
I have used EF.Functions.Like method which is an implementation of the SQL LIKE operation. On relational databases this is usually directly translated to SQL.
Check these links -
DbFunctions Class (Microsoft.EntityFrameworkCore)
src/EFCore/DbFunctionsExtensions.cs
Stack Overflow - Entity framework EF.Functions.Like vs string.Contains
First of all, let me quote the EF 3 breaking changes notice here:
Starting with 3.0, EF Core only allows expressions in the top-level projection (the last Select() call in the query) to be evaluated on the client. When expressions in any other part of the query can't be converted to either SQL or a parameter, an exception is thrown.
I have just successfully tested the following query:
var list = context.Customers.Include(c => c.Stocks).Select(c => new
{
FullName = c.FirstName + " " + c.LastName,
TotalInvestment = c.Stocks.Sum(s => s.Price*s.Quantity)
});
list.ToList();
/*
SELECT ([c].[FirstName] + N' ') + [c].[LastName] AS [FullName], (
SELECT SUM([s].[Price] * [s].[Quantity])
FROM [Stocks] AS [s]
WHERE [c].[Id] = [s].[CustomerId]) AS [TotalInvestment]
FROM [Customers] AS [c]
*/
But let's explore the topic a bit further and say you want to query your table on a computed field without bringing all evaluation to client side.
var list = context.Customers.Include(c => c.Stocks)
.Where(c => string.Concat(string.Concat(c.FirstName, " "), c.LastName) == "John Doe") // notice how we have to do string.Concat twice. observe what hppens if you use the next line instead
//.Where(c => string.Concat(c.FirstName, " ", c.LastName) == "John Doe"); // this query will fail to evaluate on SQL and will throw an error, because EF's default concat translator implementation only caters for two parameters
;
list.ToList();
/* the following SQL has been generated
SELECT [c].[Id], [c].[FirstName], [c].[LastName], [s].[Id], [s].[CustomerId], [s].[Price], [s].[Quantity]
FROM [Customers] AS [c]
LEFT JOIN [Stocks] AS [s] ON [c].[Id] = [s].[CustomerId]
WHERE (([c].[FirstName] + N' ') + [c].[LastName]) = N'John Doe'
ORDER BY [c].[Id], [s].[Id]
*/
Apparently, EF3 comes with a few pre-built functions that allow you to do that: see IMethodCallTranslator implementations (such as StringMethodTranslator for example) to get a bit more insight on how it's doing that (and other translators available to you out of the box).
Okay, what if your translator isn't implemented, I hear you ask. This is where things get a bit more exciting. I have successfully done this for EF 2.2 as outlined in this SO answer (which I invite you to check out). The code unfortunately doesn't directly translate to EF3 1:1 but I'm fairly confident the same approach will work.
UPD: see my github repo for PoC of custom DB functions working with EF Core 3.1
Related
I have a question about Entity Framework Core and using LINQ. I would like to get the other table details while accessing the Clients table. I can get them using below code. There are a total of around 10 tables I need to join, in this case is the below approach is good or any other, better approach? ClientId is the foreign key for all tables.
Actually I am getting a warning as below
[09:34:33 Warning] Microsoft.EntityFrameworkCore.Query
Compiling a query which loads related collections for more than one collection navigation either via 'Include' or through projection but no 'QuerySplittingBehavior' has been configured. By default Entity Framework will use 'QuerySplittingBehavior.SingleQuery' which can potentially result in slow query performance. See https://go.microsoft.com/fwlink/?linkid=2134277 for more information. To identify the query that's triggering this warning call 'ConfigureWarnings(w => w.Throw(RelationalEventId.MultipleCollectionIncludeWarning))'
Code:
var client = await _context.Clients
.Include(x => x.Address)
.Include(x => x.Properties)
.Include(x => x.ClientDetails)
-------------------
-------------------
-------------------
-------------------
.Where(x => x.Enabled == activeOnly && x.Id == Id).FirstOrDefaultAsync();
Actually when you use Eager loading (using include()) It uses left join (all needed queries in one query) to fetch data. Its default the ef behavior in ef 5.
You can set AsSplitQuery() in your query for split all includes in separated queries. like:
var client = await _context.Clients
.Include(x => x.Address)
.Include(x => x.Properties)
.Include(x => x.ClientDetails)
-------------------
-------------------
-------------------
-------------------
.Where(x =>x.Id == Id).AsSplitQuery().FirstOrDefaultAsync()
This approach needs more database connection, but it's nothing really important.
and for the final recommendation, I advise using AsNoTracking() for queries to high performance.
I have 3 different approaches depending on the version of EF Core you're using
EF Core 5 - as some have mentioned in previous answers there is new call which will simply break up the query into smaller subqueries and map all the relations in the end.
/*rest of the query here*/.AsSplitQuery();
If you are not able to just migrate your EF version you could still split the query manually
var client = await _context.Clients.FirstOrDefaultAsync(t => t.Enabled /*other conditions*/);
var Address = await _context.Addresses.FirstOrDefaultAsync(t => t.ClientId == client.Id);
/// Because they are tracked EF's entitytracker can under the hood
/// map the sub queries to their correct relations
/// in this case you should not use .AsNoTracking()
/// unless you would want to stitch relations together yourself
Another alternative is to write your query as a Select statement. This greatly improves performance but is a bit more of a hassle to construct.
var clientResult = await _context.Clients.Where(x => x.Id == id).Select(x => new
{
client = x,
x.Address,
Properties = x.Properties.Select(property => new
{
property.Name /*sub query for one to many related*/
}).ToList(),
x.ClientDetails
}).ToListAsync();
it doesn't take many includes to create cartesian explosion
you can read more up on the problem at hand in this article here
cartesian explosion in EF Core
and referral link to optimizing performance through EF Core can be found here
Maximizing Entity Framework Core Query Performance
I am currently trying to convert an existing SQL server query to EF Core. The goal is to get all users and get their latest order date-time and latest support request date-time. I want to ensure users are returned even if they don't have an order yet or a support request yet. If they have not placed an order, the column for "latest order date-time" should be NULL. If they have not filed a support request, the column for "latest support request date-time" should be NULL.
The outputted columns should be: Id, Name, Email, LatestOrderDateTime, LatestSupportRequestDateTime
Here is my working SQL server query:
SELECT [User].[Id], [User].[Name], [User].[Email], MAX([Order].[DateTime]) as LatestOrderDateTime, MAX([SupportRequest].[DateTime]) as LatestSupportRequestDateTime FROM [User]
LEFT JOIN [Order] on [User].[Id] = [Order].[UserId]
LEFT JOIN [SupportRequest] on [User].[Id] = [SupportRequest].[ConsumerId]
GROUP BY [User].[Id], [User].[Name], [User].[Email]
ORDER BY [User].[Id]
This is what I've tried, however it does not evaluate on the server:
await this.context.User
.GroupBy(u => new { u.Id, u.Name, u.Email })
.Select(g => new
{
id = g.Key.Id,
name = g.Key.Name,
email = g.Key.Email,
lastOrderDateTime = g.Max(o => o.Orders.Select(o => o.DateTime)),
lastSupportRequestDateTime = g.Max(o => o.SupportRequests.Select(s => s.DateTime)),
})
.OrderBy(c => c.id)
.ToListAsync();
I just want to convert this query to EF core (where the query DOES NOT get evaluated locally).
If you could do it in method syntax, that'd be great, but no worries if not since I can convert it with JetBrains Rider.
Thank you so much for your help!
I just want to convert this query to EF core (where the query DOES NOT get evaluated
locally).
Can not be done, use EntityFramework 6.4, not core, if you want this.
The SQL generation in current EntityFramework (and I mean current up to the nightly builds of veryion 5) is EXTREMELY Limited in the SQL it can generate, combined with what looks like utter ignorance to even accept that fact from the team (which reminds me of the times of EntityFramework 2 and 3 until that team started being serious about LINQ in their version 4).
If it tells you it can not generate this as SQL then your only 2 choises are:
Use EntityFramework 6.4 (which works in dotnetcore 3.1) and get server side execution
Open a bug report, HOPE someone picks it up and then either wait until November for the release of version 5 OR - once it possibly is fixed - work with nightly builds until then.
This is not a syntax issue. They deactivated client evaluation of SQL and their SQL generator is not able to handle a LOT of standard cases. Given you do not want the first (which is what we do at the moment), their feature set just means it can not be done.
You could try to explicitly spell out the left joins in Linq (left join syntax is a bit un-intuitive iirc so it may take some doing to sort it out).
You can find more information at:
https://learn.microsoft.com/en-us/dotnet/csharp/linq/perform-left-outer-joins
The way your Linq is set up specifically asks for object linking, which is why it happens client side. I believe what you're trying to do has a solution in EF Core.
I have a .NetCore 3.1 project. I know that there are breaking changes from EF Core 2 to 3 but searching for the solution to this is leading me places that make no sense.
The following works in .NetCore 2.2.
I have a list of user-names that is generated from other queries. I now want to find those user-names in our personnel database with the goal of returning the associated email address for each user-name. A person may elect to use a company email address or supply a different address. If the person.EmailAddress field is empty then the address I need is the username with the company domain appended.
private static List<string> GetEmailAddrsFromBp(PersonnelContext personnelContext, IEnumerable<string> userNames) {
try {
var personEmail = (
from person in personnelContext.Persons
join userName in userNames
on person.userName.Trim().ToLower() equals userName.Trim().ToLower()
where person.ActualEndDate == null
select person.EmailAddress.Trim().Equals("")
? person.userName.Trim().ToLower() + "#myCompany.com"
: person.EmailAddress.Trim().ToLower()
).Distinct().OrderBy(a => a).ToList();
return personEmail;
} catch (Exception e) {
throw new Exception("GetEmailAddrsFromBp: " + e.Message);
}
}
in 3.1 I get the exception:
Processing of the LINQ expression 'DbSet<Persons>
.Join(
outer: __p_0,
inner: person => person.userName.Trim().ToLower(),
outerKeySelector: userName => userName.Trim().ToLower(),
innerKeySelector: (person, userName) => new {
person = person,
userName = userName
})' by 'NavigationExpandingExpressionVisitor' failed. This may indicate either a bug or a limitation in EF Core. See https://go.microsoft.com/fwlink/?linkid=2101433 for more detailed information.
I do not understand this error. Going to the suggested Microsoft site is not helpful. Other googling has proven unhelpful. What is going on? How do you do "simple" joins now?
I do not understand this error.
The error message of course is not user friendly. The only relevant part is
This may indicate either a bug or a limitation in EF Core.
which can safely be read as "This is either a bug or a limitation in EF Core."
What is going on? How do you do "simple" joins now?
You can do "simple" joins, but not joins to memory collections. In fact joins to memory collections were never really supported. Just EF Core 1.x / 2.x used the so called client evaluation for the things it cannot translate. But implicit client evaluation has been removed in 3.0, and now you are supposed to either find a translatable construct, or switch explicitly to client evaluation through LINQ to Objects (or System.Linq.Async).
Since specifically for joins switching to client evaluation is not efficient, it's better to find/use a translatable query construct. If you use non-equi or multi-key join, you basically have no option. But for single key equi-join there is a construct which is supported in all EF / EF Core version, and it is Enumerable.Contains which translates to SQL IN (val1, val2, ..., valN).
So the solution for you concrete case would be something like this:
userNames = userNames.Select(userName => userName.Trim().ToLower()).Distinct();
var personEmail = (
from person in personnelContext.Persons
where userNames.Contains(person.userName.Trim().ToLower())
// the rest of the query unchanged...
I'am using wrapper to get some data from table User
IQueryable<StarGuestWrapper> WhereQuery =
session.Linq<User>().Where(u => u.HomeClub.Id == clubId && u.IsActive).Select(
u =>
new StarGuestWrapper()
{
FullName = u.Name + " " + u.LastName,
LoginTime = u.SomeDateTime,
MonthsAsMember = u.SomeIntergerValue,
StarRating = u.SomeOtherInteregValue,
UserPicture = u.Photo.PhotoData,
InstructorFullName = u.SomeInstructorName,
TalkInteractionDuringSession = u.SomeBoolValue,
GoalInteractionDuringSession = u.SomeOtherBoolValue
});
I use this without a problem as a IQueryable so I can do useful things before actually running the query. Like :
WhereQuery.Skip(startRowIndex).Take(maximumRows).ToList();
and so on.
The problem occurs using 'where' statement on query.
For example:
WhereQuery.Where(s => s.StarRating == 1)
will throw an exception in runtime that 'StarRating' doesn't exist in User table - of course it doesn't it's a wrappers property. It will work if I materialize query by
WhereQuery.AsEnumerable().Where(s => s.StarRating == 1)
but then it loses all the sens of using IQueryable and I don't want to do this.
What is strange and interesting that not all properties from wrapper throw error, all the bool values can be used in where statement. Example :
WhereQuery.Where(s => s.TalkInteractionDuringSession)
It works in EntityFramework , why do I get this error in NHibernate and how to get it working the way I want it to ?
Keep in mind the older nHibernate Linq provider is only a partial implementation and is no longer being actively worked on. A new and more complete linq provider is being developed now and will be part of NH3.0 (you can checkout the trunk and build it to see if it addresses this problem).
My recommendation is to change your code to call ToList() at the point when you explicitly wish to hit the database. You are passing a future valued query back from your repository at which point anything could technically happen to the query. Even EF and LINQ2SQL cannot translate any possible linq query into SQL.
I do realize this isn't what you want to be able to do, but I think you are trying to bend the framework to do something in a way this isn't very natural at all.
I'm having trouble building an Entity Framework LINQ query whose select clause contains method calls to non-EF objects.
The code below is part of an app used to transform data from one DBMS into a different schema on another DBMS. In the code below, Role is my custom class unrelated to the DBMS, and the other classes are all generated by Entity Framework from my DB schema:
// set up ObjectContext's for Old and new DB schemas
var New = new NewModel.NewEntities();
var Old = new OldModel.OldEntities();
// cache all Role names and IDs in the new-schema roles table into a dictionary
var newRoles = New.roles.ToDictionary(row => row.rolename, row => row.roleid);
// create a list or Role objects where Name is name in the old DB, while
// ID is the ID corresponding to that name in the new DB
var roles = from rl in Old.userrolelinks
join r in Old.roles on rl.RoleID equals r.RoleID
where rl.UserID == userId
select new Role { Name = r.RoleName, ID = newRoles[r.RoleName] };
var list = roles.ToList();
But calling ToList gives me this NotSupportedException:
LINQ to Entities does not recognize
the method 'Int32
get_Item(System.String)' method, and
this method cannot be translated into
a store expression
Sounds like LINQ-to-Entities is barfing on my call to pull the value out of the dictionary given the name as a key. I admittedly don't understand enough about EF to know why this is a problem.
I'm using devart's dotConnect for PostgreSQL entity framework provider, although I assume at this point that this is not a DBMS-specific issue.
I know I can make it work by splitting up my query into two queries, like this:
var roles = from rl in Old.userrolelinks
join r in Old.roles on rl.RoleID equals r.RoleID
where rl.UserID == userId
select r;
var roles2 = from r in roles.AsEnumerable()
select new Role { Name = r.RoleName, ID = newRoles[r.RoleName] };
var list = roles2.ToList();
But I was wondering if there was a more elegant and/or more efficient way to solve this problem, ideally without splitting it in two queries.
Anyway, my question is two parts:
First, can I transform this LINQ query into something that Entity Framework will accept, ideally without splitting into two pieces?
Second, I'd also love to understand a little about EF so I can understand why EF can't layer my custom .NET code on top of the DB access. My DBMS has no idea how to call a method on a Dictionary class, but why can't EF simply make those Dictionary method calls after it's already pulled data from the DB? Sure, if I wanted to compose multiple EF queries together and put custom .NET code in the middle, I'd expect that to fail, but in this case the .NET code is only at the end, so why is this a problem for EF? I assume the answer is something like "that feature didn't make it into EF 1.0" but I am looking for a bit more explanation about why this is hard enough to justify leaving it out of EF 1.0.
The problem is that in using Linq's delayed execution, you really have to decide where you want the processing and what data you want to traverse the pipe to your client application. In the first instance, Linq resolves the expression and pulls all of the role data as a precursor to
New.roles.ToDictionary(row => row.rolename, row => row.roleid);
At that point, the data moves from the DB into the client and is transformed into your dictionary. So far, so good.
The problem is that your second Linq expression is asking Linq to do the transform on the second DB using the dictionary on the DB to do so. In other words, it is trying to figure out a way to pass the entire dictionary structure to the DB so that it can select the correct ID value as part of the delayed execution of the query. I suspect that it would resolve just fine if you altered the second half to
var roles = from rl in Old.userrolelinks
join r in Old.roles on rl.RoleID equals r.RoleID
where rl.UserID == userId
select r.RoleName;
var list = roles.ToDictionary(roleName => roleName, newRoles[roleName]);
That way, it resolves your select on the DB (selecting just the rolename) as a precursor to processing the ToDictionary call (which it should do on the client as you'd expect). This is essentially exactly what you are doing in your second example because AsEnumerable is pulling the data to the client before using it in the ToList call. You could as easily change it to something like
var roles = from rl in Old.userrolelinks
join r in Old.roles on rl.RoleID equals r.RoleID
where rl.UserID == userId
select r;
var list = roles.AsEnumerable().Select(r => new Role { Name = r.RoleName, ID = newRoles[r.RoleName] });
and it'd work out the same. The call to AsEnumerable() resolves the query, pulling the data to the client for use in the Select that follows it.
Note that I haven't tested this, but as far as I understand Entity Framework, that's my best explanation for what's going on under the hood.
Jacob is totally right.
You can not transform the desired query without splitting it in two parts, because Entity Framework is unable to translate the get_Item call into the SQL query.
The only way is to write the LINQ to Entities query and then write a LINQ to Objects query to its result, just as Jacob advised.
The problem is Entity-Framework-specific one, it does not arise from our implementation of the Entity Framework support.