what is the problem with the following code?
var results = db.Departments.Select(x => customfunction(x.DepartmentID));
results.toList();
Why this is this not caught in compile time but gives an exception in runtime?
There are many different LINQ providers that allow you to use LINQ against a variety of data sources (LINQ to Entities, LINQ to Objects, LINQ to XML among others).
Although LINQ to Entities does not know how to invoke your custom method (at least, not the providers for common databases), some LINQ providers might well understand how to execute myfunction. The compiler is not integrated with all of the many LINQ providers, so the information as to whether your custom method can be included is only available at runtime.
In fact, LINQ to Objects can execute it
var results = db.Departments
.AsEnumerable()
.Select(x => myfunction(x.DepartmentID));
The conversion of
db.Departments.Select(x => myfunction(x.DepartmentID));
to sql does not happen until very late in the process at runtime.
So at compilation time Entity Framework does not know that myFunction is not something it is not going to know how to convert to sql yet.
of course you could do
db.Departments.Select(x => (x * 2));
And EF will happily be able to translate to sql but if you try to use a custom property on Departments object that doesn't map to a field in the db or use a custom method that that the Linq data provider does not know how to translate to sql you will get the same error.
The idea is simple: .Select(x => myfunction(x.DepartmentID)) is never executed in C#. Instead, Entity Framework parses this expression and translates it to SQL.
For example, if you call .Count() in C# EF will translate this to SELECT COUNT(...)
This is all very powerful, except for a few cases such as yours.
EF analyzes your code through Expression Trees.
If you call myFunction() in an expression tree, it is impossible to "expand" that call to get the body of your function. This is a limitation of how C# is implemented. That's why EF is unable to generate sensible SQL, because it has no clue what your method does.
However, there is a reasonable alternative: LinqKit
LinqKit will allow you to write "functions" that are reusable in your expressions.
Example:
static string[] QueryCustomers (Expression<Func<Purchase, bool>> purchaseCriteria)
{
var data = new MyDataContext();
var query = from c in data.Customers.AsExpandable()
where c.Purchases.Any (purchaseCriteria.Compile())
select c.Name;
return query.ToArray();
}
Related
I am getting an InvalidOperationException when using a projection into a record type (.Net 7). The query works fine if the where clause is chained. It also works if where clause is not chained and the projection is not into a record (class, anonymous type both ok).
public record TestDto(
int CellLocationId,
string AreaCode
);
This works - Chained:
var query = _context.myTable.Where(x => x.AreaCode == areaCode).Select(s => new TestDto(s.CellLocationId, s.AreaCode));
This fails - Unchained with record:
var query = _context.myTable.Select(s => new TestDto(s.CellLocationId, s.AreaCode));
query = query.Where(x => x.AreaCode == areaCode);
This works - Unchained with anonymous type:
var query = _context.myTable.Select(s => new {s.CellLocationId, s.AreaCode});
query = query.Where(x => x.AreaCode == areaCode);
When projecting into a record the sql shown in the error appears as:
System.InvalidOperationException: The LINQ expression 'DbSet().Where(v => new TestDto(v.CellLocationId, v.AreaCode).AreaCode == __areaCode_0)' could not be translated.
Why can I not use a record when my IQueryable is using an unchained where clause?
15/11/2022 Update.
Github issue
Possible workarounds - project into class, anonymous type or as per Bagus Tesa answer (and my preferred option) project into record type at end of statement.
About IEnumerable and IQueryable
IQueryable extends IEnumerable
IEnumerable most of the time works in memory, with some exceptions
IQueryable will be translated into SQL Query
Both IQueryable and IEnumerable execution are delayed until they are materialized (e.g. using ToList, AsEnumerable on IQueryable, etc.)
The API is designed this way for convenience in working with database queries. It allows (somewhat) interchangeable linq chains between the two. However, there are some caveats that need to be kept in mind.
IQueryable can only infer whats within its Expression. You can't for example do something like this:
public int GetAge(DateTime date)
{
...
}
var firstUserAge = _context.users.First().Select(q => GetDate(q.DoY));
It won't work, see QA. Linq is simply unable to translate the GetDate method into SQL Query.
The Problem
Let's take a look with the third case (anonymous class).
var query = _context.myTable
.Select(s => new {s.CellLocationId, s.AreaCode});
query = query.Where(x => x.AreaCode == areaCode);
Up to _context.MyTable the only known type is simply DbSet<myTable>. However, as it reach the Select(s => ...) it became aware of the anonymous type. Linq the consider that query should be an IQueryable<..__AnonymousType..>. Given the complete information is given to it. Linq can translate it into (roughly) sql query shown below and the result snuggly fits the anonymous type.
select s.cellLocationId, s.AreaCode
from myTable as s
where s.AreaCode = ....
Now, let's see what happens in the first case which surprisingly works.
var query = _context.myTable
.Select(s => new TestDto(s.CellLocationId, s.AreaCode))
.Where(x => x.AreaCode == areaCode);
Hang on a second! Did not I previously mentioned that you can't call a method on your program's side from within an IQueryable?
Yes, your code above won't work in the past, see discussion. You can find numerous examples of similar problem. You can also see examples on EF Core 7 documentation that they used anonymous types all the time. The microsoft's official tutorial on DTO also assigns properties manually one-by-one without using constructor.
However, EF Core 7 is pretty new. It is said to have better performance compared to earlier EF Core. The changes may involve reworks on how it materialize IQueryable. There are plenty of issues need to be sorted out.
You should run query intercepts and see the queries being feed into the database to see the difference between the two cases above. I'd bet that the first case actually fetch everything in myTable.
What We Should Do
We can simply map into DTO at a later part of the code (at the very end of the chain).
var query = _context.MyTable;
if(someFlag)
{
query = query.Where(s => s.AreaCode = areaCode);
}
//... a bunch more filters
return query.AsEnumerable()
.Select(s => new TestDto(s.CellLocationId, s.AreaCode));
This way, you can avoid the bug altogether while keeping most of the heavy-lifting on the database.
You may complain "I will have duplicate code for computing AreaCode." You have to remember, DTO in the first place does not contain business logic.
Regarding what Bagus Tesa wrote, the most important parts there are:
However, EF Core 7 is pretty new. It is said to have better performance compared to earlier EF Core. The changes may involve reworks on how it materialize IQueryable. There are plenty of issues need to be sorted out.
and
We can simply map into DTO at a later part of the code (at the very end of the chain).
I do not like how that answer is written, and I started writing my own, and then, frankly, I ended up with my own wall of text, and I don't like it just the same as their answer. Oh well. Maybe someone finds it useful. I mark it with community-wiki, because I don't really want to compete. It's more like, IDK, addendum?
Regarding the latter point, there have been some issues with early-selects since the dawn of EF. Select-as-custom-class is a special construct designed to allow your query to return your custom entity types (instead of anon bags of fields), especially in cases where you Join several tables, where the result cannot be simply the context.table you start the query from.
Thanks to how IQueryable is designed, you can moreless put such a .Select anywhere in your query, but there's a catch: once IQueryable is now <of-you-custom-type>, your custom properties are exposed instead of original tables columns, and it gets increasingly harder to guarantee that your code doesn't try to map/execute something untranslatable.
For example, imagine:
// case1:
var result = context.myTable
.Where(x => x.AreaCode == areaCode)
.Select<MyClass>(...)
.ToList();
// case2:
var result = context.myTable
.Select<MyClass>(...)
.Where(x => x.AreaCode == areaCode)
.ToList();
In the first case, it's obvious. AreaCode is a column from MyTable, defined by whatever type of MyTable.
In second case, it's AreaCode but from MyClass, and since someone wanted to introduce a new type, we can assume it's not mapped in EF mappings to the same table as MyTable. So if EF hanles that, it may have to so some guessing what MyClass.AreaCode really is.
Furthermore, what if MyClass.AreaCode was:
private string theCode = "5";
public string AreaCode {
set { theCode = value; }
get { return theCode + "1"; }
}
In that case, case1 would still function properly. And case2 can never function properly. Not even mentioning, custom class constructors, we write .Select(.. => new MyClas(...)) and it's very tempting to hide some custom logic into the ctor, even if it's as simple as if/throw/ ArgumentNullException.
If EF handles such cases it in ANY way (like: assume props/ctors are transparent, ignoring logic inside), results will be unintuitive to the programmer: MyClass used in Query will behave differently than MyClass used in the runtime. EF shouldn't handle it, it should throw, but determining that a getter or constructor is pass-through without custom logic isn't that easy.
That's why anon-types new {foo=, bar=} are always preferred (2) as the intermediate types - they are 100% compiler-generated, the compiler knows everything about them, the compiler knows all fields/properties are plain and transparent, and that's why it's not a huge problem if you put .Select<anon> even early in the LINQ query. (1)
But let's get back to your code.
In that record-related code samples you provided:
// first:
var query = _context.myTable
.Select(s => new TestDto(s.CellLocationId, s.AreaCode))
.Where(x => x.AreaCode == areaCode);
// second:
var query = _context.myTable
.Select(s => new TestDto(s.CellLocationId, s.AreaCode));
query = query.Where(x => x.AreaCode == areaCode);
These pieces of code, if there is really nothing else between context:get_myTable and IQueryable<record>:Where, something that you'd for some reason omit, then these two pieces of code are essentially the same. Or should be.
When we write return " mom ".ToUpper().Trim(); we don't expect it to have different results than if we write var x = " mom ".ToUpper(); x = x.Trim(); return x;, right? That's all complete basics of operation/result/variable semantics.
The only difference here is splitting such unmaterialized IQueryable 'result' of a .Select() to a variable. LINQ call chain is exactly the same. Names are exactly the same. Difference is purely syntactical. So if you see that one of them works, and the other fails, this means there's a bug in the compiler, and you should:
file a bug issue at roslyn github
get away from whatever this bug can be related to as far as possible (meaning not use records yet, or at least not early in the query)
or, at least test the hell out of it, so you know how to use it safely even bugged, and get as much of those tests automated
There's really not much else to be advised here.
(1) actually sometimes it is problematic. Every ".Select" in the chain changes the "shape of the query", as they call it sometimes in MSDN articles, and it usually breaks any .Include setups that were done on the original shape. But YMMV there.
(2) anon-types are perfect here from the LINQ point of view, but are often a nightmare for the developer who wants to do anything a bit more reusable or structured. You cannot return them easily from your methods, since the type is anonymous. And so on. This severely limits reuse, and I can easily imagine, that's why RecordTypes are introduced - to provide a plain type with guaranteed mechanics, a type that can be named and easily referred to.
I want to define a Func<ProductItemVendor, bool> filter expression named CompareProductItemVendorIds, which can be used throughout my application, mostly within Entity Framework/LINQ queries.
I've learned that in order to be able to use this filter in a LINQ query, I must declare it as Expression<Func<>> instead of just Func<>. I understand the reason for this, and it's easy for me to do this.
But I'm having the following problems using that expression in my queries.
First, code such as:
ProductItem.ProductItemVendors.FirstOrDefault(CompareProductItemVendorIds)
Note: ProductItem is a database entity, and its ProductItemVendors property is a navigation collection.
Produces the error:
`Instance argument: cannot convert from 'System.Collections.Generic.ICollection' to 'System.Linq.IQueryable'
And second, code such as:
var results = from v in Repository.Query<ProductItemVendor>()
where CompareProductItemVendorIds(v)
select v;
Produces the error:
'CompareProductItemVendorIds' is a 'variable' but is used like a 'method'
So I have my nice shiny new Expression<Func<>>. How can I use it in my LINQ queries?
ProductItem is already an Entity, so you can't use your Expression, you need to use Compile() to get the Func<> from your Expression<Func<>> since ProductItemVendors is no longer an IQueryable
ProductItem.ProductItemVendors.FirstOrDefault(CompareProductItemVendorIds.Compile())
You would have to use your Expression on the ProductItemVendorsContext like this:
var item = Context.ProductItemVendors.FirstOrDefault(CompareProductItemVendorIds);
You cant use an Expression inside query syntax, you need to use method sytanx
var results = from v in Repository.Query<ProductItemVendor>()
.Where(CompareProductItemVendorIds)
select v;
The first case;
ProductItemVendor productItemVendor = ProductItem.ProductItemVendors.FirstOrDefault(CompareProductItemVendorIds);
Produces the error:
`Instance argument: cannot convert from 'System.Collections.Generic.ICollection' to 'System.Linq.IQueryable'
Happens because it is an ICollection, the entity has already been loaded. Given dbContext or objectContext lazy loading vs explicit loading is implemented a bit differently, I assume you are using explicit loading though. If you change the loading to lazy the type of ProductItemVendors will be IQueryable and what you are trying will succeed.
Given the second case, the expression must be compilable to SQL, else you get a lot of weird errors, probably it's possible that that is the case here.
It's hard to give you more explicit help given the information in the question, I cannot recreate it easily. If you can create a MWE-solution and upload it somewhere I can have a look, but I'm afraid I can't help more here.
I don't think composing expressions like that is supported by default.
You could try LINQKit
Here's an example from the LINQKit page; one Expression<Func<>> inside another:
Call Invoke to call the inner expression Call Expand on the final
result. For example:
Expression<Func<Purchase,bool>> criteria1 = p => p.Price > 1000;
Expression<Func<Purchase,bool>> criteria2 = p => criteria1.Invoke (p)
|| p.Description.Contains ("a");
Console.WriteLine (criteria2.Expand().ToString());
(Invoke and Expand are extension methods in LINQKit.) Here's the output:
p => ((p.Price > 1000) || p.Description.Contains("a"))
Notice that we have a nice clean expression: the call to Invoke has
been stripped away.
I have a question related to this previous question of mine. In an existing bit of LINQ which involves a number of joins, I'm trying to take each separate method comprising the join and convert it to a CompiledQuery.
First, the normal LINQ method:
private IQueryable<Widget> GetWidgetQuery()
{
return db.Widgets.Where(u => (!u.SomeField.HasValue || !u.SomeField.Value));
}
And here, a delegate (field) definition for a CompiledQuery along these lines:
private static readonly Func<DBDataContext, IQueryable<Widget>> GetWidgetQuery =
CompiledQuery.Compile((DBDataContext db) =>
db.Widgets.Where(u => (!u.SomeField.HasValue || !u.SomeField.Value)));
If I hover over the normal LINQ statement for the method GetWidgetQuery(), I see that it's a method as below:
(method) IQueryable<Widget> GetWidgetQuery()
However, the compiled query delegate (field) differs as follows:
(field) Func<DBDataContext, IQueryable<Widget>> GetWidgetQuery
Upon executing the latter as part of the LINQ statement, the syntax differs as follows. First, the normal LINQ's participation in the join:
var myquery =
from wxr in GetWidgetXRQuery()
join w in GetWidgetQuery() on wxr.WidgetID equals w.ID
select new DTO.WidgetList
{
...
}
And here, the invocation of the CompiledQuery in the form of the delegate:
var myquery =
from wxr in GetWidgetXRQuery()
join w in GetWidgetQuery.Invoke(myContext) on wxr.WidgetID equals w.ID
select new DTO.WidgetList
{
...
}
The former returns the expected result set; the latter, when I attempt myquery.ToList(), yields a stackoverflow exception, in part related to this limitation of .NET 3.5, I think.
Can someone please help me understand how the compiled statement existing as a field (or I guess I should say a delegate) rather than a method is killing my query? In short I know what I'm doing is wrong, but I'm not sure I understand what I misunderstand.
I tried doing roughly the same thing you're doing on EF 4, and everything seems to work fine. So it's either an EF 3.5 issue, or it has something to do with your implementation of GetWidgetXRQuery, or some combination of the two.
But the real point I'd like to make is that, as Roy Goode stated in an answer to your previous question, you lose all the advantages of a precompiled query once you extend that query in any way. By trying to perform a Join on your query, you are converting it to just a plain old query. So you might as well just use the non-compiled version which appears to work for you.
Update
Realized you were talking about LINQ to SQL. This sort of query does appear to have support in Entity Framework, but not LINQ to SQL. In .NET 4, I'm getting the following error:
An IQueryable that returns a self-referencing Constant expression is not supported.
That doesn't mean much to me, but I'm guessing that it has something to do with the way the compiled query is represented internally. I still get the same error if I evaluate the query into a variable and use that variable in the query later, so it clearly has nothing to do with the difference between a delegate and a function. I still maintain that a compiled query is not appropriate to use here. Either you need to create one big compiled query to represent the whole query you want to perform, or you need to use regular queries if you want to piece them together this way.
I just came across this same error while doing db integration testing, and to jump straight to the point without trying to explain my specific issue. Linq to Sql will create the sql query internally when using IQueryable and the moment you execute a method on that IQueryable, i.e. ToList() it executes that query on the database. So in my case I am joining to a method that returns IQueryable but is mocked to return a result, it is trying to compile that to a sql query but the IQueryable I created does not have an internal SQL query
This is probably a stupid question, but here goes. I would like to be able to dynamically construct a predicate < T > from a string parsed from a database VARCHAR column, or any string, for that matter. For example, say the column in the database contained the following string:
return e.SomeStringProperty.Contains("foo");
These code/string values would be stored in the database knowing what the possible properties of the generic "e" is, and knowing that they had to return a boolean. Then, in a magical, wonderful, fantasy world, the code could execute without knowing what the predicate was, like:
string predicateCode = GetCodeFromDatabase();
var allItems = new List<SomeObject>{....};
var filteredItems = allItems.FindAll(delegate(SomeObject e) { predicateCode });
or Lambda-ized:
var filteredItems = allItems.FindAll(e => [predicateCode]);
I know it can probably never be this simple, but is there a way, maybe using Reflection.Emit, to create the delegate code dynamically from text and give it to the FindAll < T > (or any other anonymous/extension) method?
The C# and VB compilers are available from within the .NET Framework:
C# CodeDom Provider
Be aware though, that this way you end up with a separate assembly (which can only be unloaded if it's in a separate AppDomain). This approach is only feasible if you can compile all the predicates you are going to need at once. Otherwise there is too much overhead involved.
System.Reflection.Emit is a great API for dynamically emitting code for the CLR. It is, however, a bit cumbersome to use and you must learn CIL.
LINQ expression trees are an easy to use back-end (compilation to CIL) but you would have to write your own parser.
I suggest you have a look at one of the "dynamic languages" that run on the CLR (or DLR) such as IronPython. It's the most efficient way to implement this feature, if you ask me.
Check out the Dynamic Linq project it does all this and more!
http://weblogs.asp.net/scottgu/archive/2008/01/07/dynamic-linq-part-1-using-the-linq-dynamic-query-library.aspx
Great for simple stuff like user selected orderby's or where clauses
It is possible using emit, but you'd be building your own parser.
EDIT
I remember that in ScottGu's PDC keynote, he showed a feature using the CLI version of the .net framework that resembled Ruby's eval, but I can't find a URL that can corroborate this. I'm making this a commnity wiki so that anyone who has a good link can add it.
I stepped off the dynamic linq because it's limited in ways I want to search a collection, unless you prove me wrong.
My filter needs to be: in a list of orders, filter the list so that I have only the orders with in the collection of items in that order, an item with the name "coca cola".
So that will result to a method of: orders.Findall(o => o.Items.Exists(i => i.Name == "coca cola"))
In dynamic linq I didn't find any way to do that, so I started with CodeDomProvicer.
I created a new Type with a method which contains my dynamically built FindAll Method:
public static IList Filter(list, searchString)
{
// this will by dynamically built code
return orders.Findall(o => o.Items.Exists(i => i.Name == "coca cola"));
}
when I try to build this assembly:
CompilerResults results = provider.CompileAssemblyFromSource(parameters, sb.ToString());
I'm getting the error:
Invalid expression term ">"
Why isn't the compiler able to compile the predicate?
I am developing Web Custom control which able to apply filters to the LinqDataSource object. It is generic control since it should operate with object of certain type.
The control knows what field of object it should operate by following field
/// <summary>
/// Method to get compared column from object
/// </summary>
public Expression<Func<T, int>> GetColumnMethod;
(I transfer to it method which gets appropriate field from object type)
We perform the filtering with code like that
... if (selectedValue == "<=")
predicate = predicate.And(c => method(c) <= val);
if (selectedValue == "<")
predicate = predicate.And(c => method(c) < val);
All proceeds OK until LINQ to SQL transformation occurs.
Then error "Method '.....' has no supported translation to SQL.
sure, CLR doesn't know how to make SQL for delegates.
If only C# could compile the expression before translation to SQL, but I have no idea how to make it to do it.
Perversions like Expression.Compile (whatever tricky ways I tried the whole day - I already cannot remember them all...nothing helped)
But ... at the runtime CLR already knows the type of my object, so it could to manage to build SQL expression having compiked delegate values. But how to do it ? God knows.
Help highly appreciates.
The LINQ to SQL provider is responsible for translation your expression tree into a valid T-SQL statement. As there is not a 1-to-1 relationship between C# and T-SQL it is highly probable that the more sophisticated your expression tree, the less likely it is that LINQ to SQL will be able to translate.
When you use a lambda expression you have to decide if you want to compile it or use it as an expression tree. If you want to use the lambda as an expression then you are responsible for ensuring that the expression contains functions and syntax that your provider supports.
It won't work. Basically your LINQ query is valid C# code so it compiles fine but it fails during runtime on anything outside of scope of LINQ to SQL provider.
Read more here:
“Cannot call methods on DateTime”, and other limitations
If it's just a matter of selecting a particular field to use in your query, you should be able to do this using the Expression.Compile "perversion". But that's likely to be a lot of work. You can probably get a head start by compiling what you have now and using Reflector against the output to see what the code generated by the C# compiler looks like.
Can you split the query into two parts -- one that gets translated to SQL and runs on the server and one that uses your GetColumnMethod and runs in memory against the data output by the first part?
There is no SQL for arbitrary delegates. If method(c) can be expressed as a lambda, you can invoke the lambda as a sub-expression using Expression.Invoke, but you'd need to build the expression tree yourself, like so.