I'm having trouble building an Entity Framework LINQ query whose select clause contains method calls to non-EF objects.
The code below is part of an app used to transform data from one DBMS into a different schema on another DBMS. In the code below, Role is my custom class unrelated to the DBMS, and the other classes are all generated by Entity Framework from my DB schema:
// set up ObjectContext's for Old and new DB schemas
var New = new NewModel.NewEntities();
var Old = new OldModel.OldEntities();
// cache all Role names and IDs in the new-schema roles table into a dictionary
var newRoles = New.roles.ToDictionary(row => row.rolename, row => row.roleid);
// create a list or Role objects where Name is name in the old DB, while
// ID is the ID corresponding to that name in the new DB
var roles = from rl in Old.userrolelinks
join r in Old.roles on rl.RoleID equals r.RoleID
where rl.UserID == userId
select new Role { Name = r.RoleName, ID = newRoles[r.RoleName] };
var list = roles.ToList();
But calling ToList gives me this NotSupportedException:
LINQ to Entities does not recognize
the method 'Int32
get_Item(System.String)' method, and
this method cannot be translated into
a store expression
Sounds like LINQ-to-Entities is barfing on my call to pull the value out of the dictionary given the name as a key. I admittedly don't understand enough about EF to know why this is a problem.
I'm using devart's dotConnect for PostgreSQL entity framework provider, although I assume at this point that this is not a DBMS-specific issue.
I know I can make it work by splitting up my query into two queries, like this:
var roles = from rl in Old.userrolelinks
join r in Old.roles on rl.RoleID equals r.RoleID
where rl.UserID == userId
select r;
var roles2 = from r in roles.AsEnumerable()
select new Role { Name = r.RoleName, ID = newRoles[r.RoleName] };
var list = roles2.ToList();
But I was wondering if there was a more elegant and/or more efficient way to solve this problem, ideally without splitting it in two queries.
Anyway, my question is two parts:
First, can I transform this LINQ query into something that Entity Framework will accept, ideally without splitting into two pieces?
Second, I'd also love to understand a little about EF so I can understand why EF can't layer my custom .NET code on top of the DB access. My DBMS has no idea how to call a method on a Dictionary class, but why can't EF simply make those Dictionary method calls after it's already pulled data from the DB? Sure, if I wanted to compose multiple EF queries together and put custom .NET code in the middle, I'd expect that to fail, but in this case the .NET code is only at the end, so why is this a problem for EF? I assume the answer is something like "that feature didn't make it into EF 1.0" but I am looking for a bit more explanation about why this is hard enough to justify leaving it out of EF 1.0.
The problem is that in using Linq's delayed execution, you really have to decide where you want the processing and what data you want to traverse the pipe to your client application. In the first instance, Linq resolves the expression and pulls all of the role data as a precursor to
New.roles.ToDictionary(row => row.rolename, row => row.roleid);
At that point, the data moves from the DB into the client and is transformed into your dictionary. So far, so good.
The problem is that your second Linq expression is asking Linq to do the transform on the second DB using the dictionary on the DB to do so. In other words, it is trying to figure out a way to pass the entire dictionary structure to the DB so that it can select the correct ID value as part of the delayed execution of the query. I suspect that it would resolve just fine if you altered the second half to
var roles = from rl in Old.userrolelinks
join r in Old.roles on rl.RoleID equals r.RoleID
where rl.UserID == userId
select r.RoleName;
var list = roles.ToDictionary(roleName => roleName, newRoles[roleName]);
That way, it resolves your select on the DB (selecting just the rolename) as a precursor to processing the ToDictionary call (which it should do on the client as you'd expect). This is essentially exactly what you are doing in your second example because AsEnumerable is pulling the data to the client before using it in the ToList call. You could as easily change it to something like
var roles = from rl in Old.userrolelinks
join r in Old.roles on rl.RoleID equals r.RoleID
where rl.UserID == userId
select r;
var list = roles.AsEnumerable().Select(r => new Role { Name = r.RoleName, ID = newRoles[r.RoleName] });
and it'd work out the same. The call to AsEnumerable() resolves the query, pulling the data to the client for use in the Select that follows it.
Note that I haven't tested this, but as far as I understand Entity Framework, that's my best explanation for what's going on under the hood.
Jacob is totally right.
You can not transform the desired query without splitting it in two parts, because Entity Framework is unable to translate the get_Item call into the SQL query.
The only way is to write the LINQ to Entities query and then write a LINQ to Objects query to its result, just as Jacob advised.
The problem is Entity-Framework-specific one, it does not arise from our implementation of the Entity Framework support.
Related
I have a local List with entities, some hundreds, and I have a SQL Server table where I store the ID of the successful processed entities, some millions. I would like to know, which entities form my local set are not yet processed i.e. are not in the SQL Table.
The first approach is to iterate through the local list with the following Linq statement:
Entity entity = db.Entities.FirstOrDefault(m => m.ID == ID);
if (entity == null) { NewList.Add(ID) }
the NewList would then contain all the new entities. However this is very slow.
In LINQ, how would you send the entire local list to the SQL Server with one call and then return the ones not in the SQL table?
Do you really have to create a temporary table with my local list, then left-join on the already processed table and return the ones with a null?
Use .Contains method to retrieve already processed ids
and Except to create list of not yet processed ids.
var localList = new List<int> { 1, 2, 3 };
var processed = db.Entities
.Where(entity => localList.Contains(entity.Id))
.Select(entity => entity.Id)
.ToList();
var notProcessed = localList.Except(processed).ToList();
It will depend on provider, but .Contains should generate sql like:
SELECT Id FROM Entity WHERE Id IN (1, 2, 3)
suggestion:
create a temp table and insert your IDs
select your result on the SQL side
EDIT:
"Can you do that in LINQ?"
TL;DR:
yes* but that's an ugly piece of work, write the SQL yourself
*)depends on what you mean with "in" LINQ, because that is not in the scope of LINQ. In other words: a LINQ expression is one layer too abstract, but if you happen to have an LINQ accessible implementation for this, you can use this in your LINQ statements
on the LINQ expression side you have something like:
List<int> lst = new List<int>() { 1,2,3 };
List<int> result = someQueryable.Where(x=>lst.Contains(x.ID)).Select(x=>x.ID).ToList();
the question now is: what happens on the SQL side (assuming the queryable leads us to a SQL database)?
the queryable provider (e.g. Entity Framework) somehow has to translate that into SQL, execute it and come back with the result
here would be the place to modify the translation...
for example examine the expression tree with regard to the object that is the target for the Contains(...) call and if it is more than just a few elements, go for the temp table approach...
the very same LINQ expression can be translated into different SQL commands. The provider decides how the translation has to be done.
if your provider lacks support for large Contains(...) cases, you will probably experience poor performance... good thing is usually nobody forces you to use it this way ... you can skip linq for performance optimized queries, or you could write a provider extension yourself but then you are not on the "doing something with LINQ"-side but extending the functionality of your LINQ provider
if you are not developing a large scalable product that will be deployed to work with different DB-Backends, it is usually not worth the effort... the easier way to go is to write the sql yourself and just use the raw sql option of your db connection
I am using LINQ to retrieve data from my EF context as well as from Asp .Net Identity 2.0 - both located in the same MS SQL Server database.
My problem is that LINQ sees them as 2 different cases of data context and is unable to process the query.
"The specified LINQ expression contains references to queries that are associated with different contexts."
What I want to achieve is a simple return of 10 top items (I skip this in the code extract) from EF table, previously sorted by the UserName from ASP .NET Identity table.
I have seen a few cases of this problem on StackOverflow but I was unable to apply any solution in my case.
The preferred solution would be of course not to download all of the table data and do the sorting on the server.
The query in question:
var dbDataSorted = from entry in dbData
join user in this.UserManager.Users
on entry.UserId equals new Guid(user.Id)
orderby user.UserName ascending
select entry;
return dbDataSorted;
I was able to get this to work in my case by using AsEnumerable(). YMMV.
In your case:
var dbDataSorted = from entry in dbData.AsEnumerable()
join user in this.UserManager.Users
on entry.UserId equals new Guid(user.Id)
orderby user.UserName ascending
select entry;
return dbDataSorted;
LINQ and EF are pretty cool. But sometimes, its abstractions don't offer what you need.
Just fall back to base, write the query by hand, put it in a string, run it against yourcontext.YourDbSet with the SqlQuery method, and be done with it.
var query = #"SELECT * FROM dbData as entry
INNER JOIN Users
ON entry.UserId = Users.Id
ORDER BY Users.Username";
yourcontext.dbData.SqlQuery(query);
If the abstractions offered to you don't work with what you need, abusing the abstractions to do something weird is far less clear than using the lower level interface.
You can't use Linq to Entities and Linq to object in one query. It's because when you make query to instance of IQueryable interface, the query will be transformed to SQL, and in SQL you can't use any IEnumerable collection. So you should get data from database (for example dbData.AsEnumerable()).
But if you want do all job on the SQL Server side, the easiest way is to create sql procedure and pass the users table as a parameter. The easiest way to pass users table as xml and than parse it in the procedure on the server side.
If you're using ASP.NET Identity 2 code-first, then your DbContext presumably inherits from IdentityDbContext<>, and so you can access the Users table directly from the context. You don't need to use UserManager to access the users.
I am new to Entity Framework and was looking for some help.
I have the following C# code for my entity data model. My question is: Will payables.Vendors get vendors from the database twice or only once? These statements actually appear exactly in the same order in my code. I am not sure if the first call to payables.Vendors will be cached so the second call to payables.Vendors does not go to the database.
PayablesEntities payables = new PayablesEntities();
var selectedVendor =(from vendor in payables.Vendors
where vendor.VendorID == vendorID
select vendor).First();
var pendingVendor = (from vendor in payables.Vendors
where vendor.IsPending == true
select vendor).First();
Since you have two separate queries then there will be two SQL queries to execute.LINQ to Entities doesn't magically combine your queries.If you want to execute one query then you need to create one query and combine them.
I believe the answer to this question may be to use Linq to Sql, but wanted to see if this is something which is possible using QueryExpressions:-
I create a query expression which queries against Entity A, it also links to Entity B (via LinkEntity) and imposes additional criteria. It is possible to retrieve columns from Entity B by adding the appropriate attribute names. However, it will only retrieve the linked entity (inner join).
Is it possible using QueryExpression to retrieve all related records (and required columns) from Entity B related to Entity A (e.g. all cases associated with contact where contact passes specified criteria). Normally I would consider inverting the query and searching for Entity B relatig to Entity A with the appropriate LinkEntity Conditions, but there are a number of linked entities which I would like to retrieve for the same contact query.
So I'm left with some options:-
(1) Perform a second query (not ideal when iterating over a large number of results from the initial query),
(2) Perform a query using Linq to CRM on the filtered views,
(3) A different method entirely?
Any thoughts would be appreciated.
EDIT:
I ended up using Linq-to-Sql to complete this task and the code used is similar to that below (albeit with a few more joins for the actual query!):-
var dataCollection = (from eA in xrmServiceContext.EntityASet
join eB in xrmServiceContext.EntityBSet on new EntityReference(EntityA.EntityLogicalName, eA.Id) equals (EntityReference)eB.EntityBLookupToEntityA
select new
{
Id = eA.Id,
EntityBInterestingAttribute = eB.InterestingAttributeName
}
So this will bring back a row per Entity A, per Entity B. To make things easier I then defined a custom class "MyEntityAClass" which had properties which were Lists so I could return one object for filling of GridView etc. This is more to do with the processing of these results though so I haven't posted that code here.
I hope that makes sense. Essentially, it is getting the multiple rows per record a la SQL which makes this method work.
QueryExpression can only return fields from one type of entity, the one specified in QueryExpression.EntityName.
You can use FetchXML which allows you to also get the fields of any link entities, which would be an option 3 for you, unfortunately it returns the data as XML which you would then have to parse yourself.
It might be quicker to run the FetchXML, but it will take longet to write and test, and its not the easiest thing to maintain either.
Sample Code, this gets the first 101 of all Cases that are active for all accounts that are active
string fetch = "<fetch count='101' mapping='logical'><entity name='account'><filter type='and'><condition attribute='statecode' operator='eq' value='1'/></filter><link-entity name='incident' from='customerid' to='accountid'><all-attributes/><filter type='and'><condition attribute='statecode' operator='eq' value='1'/></filter></link-entity></entity></fetch>";
string data = yourCrmServiceObject.Fetch(fetch);
Using the Entity Framework, when one executes a query on lets say 2000 records requiring a groupby and some other calculations, does the query get executed on the server and only the results sent over to the client or is it all sent over to the client and then executed?
This using SQL Server.
I'm looking into this, as I'm going to be starting a project where there will be loads of queries required on a huge database and want to know if this will produce a significant load on the network, if using the Entity Framework.
I would think all database querying is done on the server side (where the database is!) and the results are passed over. However, in Linq you have what's known as Delayed Execution (lazily loaded) so your information isn't actually retrieved until you try to access it e.g. calling ToList() or accessing a property (related table).
You have the option to use the LoadWith to do eager loading if you require it.
So in terms of performance if you only really want to make 1 trip to the Database for your query (which has related tables) I would advise using the LoadWith options. However, it does really depend on the particular situation.
It's always executed on SQL Server. This also means sometimes you have to change this:
from q in ctx.Bar
where q.Id == new Guid(someString)
select q
to
Guid g = new Guid(someString);
from q in ctx.Bar
where q.Id == g
select q
This is because the constructor call cannot be translated to SQL.
Sql's groupby and linq's groupby return differently shaped results.
Sql's groupby returns keys and aggregates (no group members)
Linq's groupby returns keys and group members.
If you use those group members, they must be (re-)fetched by the grouping key. This can result in +1 database roundtrip per group.
well, i had the same question some time ago.
basically: your linq-statement is converted to a sql-statement. however: some groups will get translated, others not - depending on how you write your statement.
so yes - both is possible
example:
var a = (from entity in myTable where entity.Property == 1 select entity).ToList();
versus
var a = (from entity in myTable.ToList() where entity.Property == 1 select entity).ToList();