If I have a method that returns a List of Dogs, does Dapper convert the Where predicate into a SQL Where? or does it get all the dogs first and then filter the list?
public IList<Dog> GetDogsBy(Func<Dog, bool> predicate)
{
return db.Query<Dog>("SELECT * FROM Dog",null).Where(predicate).ToList();
}
It depends on what overload resolution does when resolving the Where.
If the result of the call to Query is a List<Dog> and you have a using System.Linq; directive, and the argument is a delegate, then the Where will resolve to the LINQ-to-objects version of Where. So the select query will run on the server, and then the Where will run on the client.
If the result of Query is IQueryable<Dog> and the argument is an Expression then overload resolution will choose the Where which combines with the existing query. When that query is executed, the Where runs on whatever side the query providers chooses to run it on, which is typically the server.
When it comes to SQL trnalsation, while an expression tree can be translated, a compiled method cannot. This is because once the method is compiled, it's a matter of reading IL to perform the translation, which is not exposed through the .NET API, whereas Expression Trees are explicitly designed to be navigated using metacode for the sake of translation.
Because you're passing in a Func<...> instead of an Expression<Func<...>>, you're passing that compiled method, and thus the code cannot get the expression tree, and thus the above code either will not work or will execute on your local machine.
If you were to replace the method signature with
public IList<Dog> GetDogsBy(Expression<Func<Dog, bool>> predicate)
Then it might be able to perform that translation, depending on the semantics of Dapper.
Related
In one of my projects I have an ExpressionVisitor to translate provided expression into some query string. But before translating it I need to evaluate all refferences in the expression to real values. To do that I use Evaluator.PartialEval method from EntityFramework Project.
Assuming I have this query:
var page = 100;
var query = myService.AsQueryable<Product>()
//.Where(x=>x.ProductId.StartsWith(p.ProductId))
.Skip(page)
.Take(page);
var evaluatedQueryExpr = Evaluator.PartialEval(query.Expression);
As you can see I have commented Where method. In this case evaluatedQueryExpr will not contain the methods Take and Skip.
However, if I use any other method with Expression before Take or Skip everything works, Evaluator evaluates an expression correctly and return it fully.
I found out that the problem occurs in the line 80 of the Evaluator class:
return Expression.Constant(fn.DynamicInvoke(null), e.Type);
Could you explain why this happens and suggest a workaround?
Update
here is a project on github
LinqToSolrQueriable inherited from IOrderedQueryable
LinqToSolrProvider inherited from IQueryProvider including line range causing the issue
The good news are that the expression is not really reduced (Skip and Take are still there :), but is simply converted from MethodCallExpression to ConstantExpression containing the original expression:
query.Expression:
.Call System.Linq.Queryable.Take(
.Call System.Linq.Queryable.Skip(
.Constant<LinqToSolr.Query.LinqToSolrQueriable`1[LinqToSolrTest.Product]>(LinqToSolr.Query.LinqToSolrQueriable`1[LinqToSolrTest.Product]),
100),
100)
evaluatedQueryExpr:
.Constant<System.Linq.IQueryable`1[LinqToSolrTest.Product]>(LinqToSolr.Query.LinqToSolrQueriable`1[LinqToSolrTest.Product])
Here the debug display is giving you a wrong impression. If you take the ConstaintExpression.Value, you'll see that it's a IQueryable<Product> with Expression property being exactly the same as the original query.Expression.
The bad news are that this is not what you expect from PartialEval - in fact it doesn't do anything useful in this case (except potentially breaking your query translation logic).
So why is this happening?
The method you are using from EntityFramework.Extended library is in turn taken (as indicated in the comments) from MSDN Sample Walkthrough: Creating an IQueryable LINQ Provider. It can be noticed that the PartialEval method has two overloads - one with Func<Expression, bool> fnCanBeEvaluated parameter used to identify whether a given expression node can be part of the local function (in other words, to be partially evaluated or not), and one without such parameter (used by you) which simply calls the first passing the following predicate:
private static bool CanBeEvaluatedLocally(Expression expression)
{
return expression.NodeType != ExpressionType.Parameter;
}
The effect is that it stops evaluation of ParameterExpression type expressions and any expressions containing directly or indirectly ParameterExpression. The last should explain the behavior you are observing. When the query contains Where (and basically any LINQ operator) with parametrized lambda expression (hence parameter) before the Skip / Take calls, it would stop evaluation of the containing methods (which you can see from the above query.Expression debug view - the Where call will be inside the Skip).
Now, this overload is used by the MSDN example to evaluate a concrete nested Where method lambda expression and is not generally applicable for any type of expression like IQueryable.Expression. In fact the linked project is using the PartialEval method in a single place inside QueryCache class, and also calling the other overload passing a different predicate which in addition to ParameterExpressions stops the evaluation of any expression with result type of IQueryable.
Which I think is the solution of your problem as well:
var evaluatedQueryExpr = Evaluator.PartialEval(query.Expression,
// can't evaluate parameters or queries
e => e.NodeType != ExpressionType.Parameter &&
!typeof(IQueryable).IsAssignableFrom(e.Type)
);
I'm starting to look under the LINQ hood a little more, and I'm having trouble understanding some of the LINQ extension method overloads.
For example, lets say I'm querying a DbContext using .Where(). I'm always passing a standard Func<>, as opposed to an Expression<> of said Func<>. Example query below:
var db = new MyContext();
var foo = db.products.Where(p => p.Category == "books");
Here is where I'm confused. When I look at the available method signatures, I would assume the overload I'm using above would be returning me an IEnumerable...but it's actually returning an IQueryable. How is this possible, if the IQueryable overload is expecting an Expression as opposed to just a Func? It feels like the compiler is somehow helping me out by (in this case) building the Expression for me, but I can't find a resource that explains if this is the case. Thanks!
It feels like the compiler is somehow helping me out by (in this case) building the Expression for me
That is exactly what is happening. The compiler can interpret lambdas (as long as they don't have a "statement body") as either delegates or expression trees. The Queryable.Where<T>(this IQuerayble<T>, ...) extension method takes precedence over the Enumerable.Where<T>(this IEnumerable<T>, ...) extension method, so the compiler chooses to interpret your predicate as an expression tree.
It feels like the compiler is somehow helping me out by (in this case) building the Expression for me
Correct, the compiler is compiling the lambda into an Expression rather than a Func.
I can't find a resource that explains if this is the case
From MSDN:
When a lambda expression is assigned to a variable of type Expression<TDelegate>, the compiler emits code to build an expression tree that represents the lambda expression.
You're not passing a Func in that example. You're passing an Expression. A lambda can compile down into either a delegate or an expression, based on the context around it. In this case the context around it is expecting an expression, so that is what the lambda compiles into. If you actually did pass in a Func and not a lambda (which could be either) then you would get an IEnumerable for your result, not an IQueryable.
The compiler can automatically build an expression tree for you from a lamba expression, and that is what's happening here.
http://msdn.microsoft.com/en-gb/library/bb397951.aspx
I have the following code to get a collection of Types from an Assembly where the Type is a Class:
Assembly assembly = Assembly.LoadFile(DLLFile);
var types = assembly.GetTypes().AsEnumerable().Where(x => x.IsClass);
This works fine and as expected. However I wanted to pull out the lambda expression to a Linq Expression variable (as later on it will be used in a parameter of this method). So I did the following:
private Expression<Func<Type, bool>> _standardFilter = (x => x.IsClass);
Assembly assembly = Assembly.LoadFile(DLLFile);
var types = assembly.GetTypes().AsEnumerable().Where(_standardFilter);
However this won't compile with the error:
System.Collections.Generic.IEnumerable<System.Type>' does not contain a
definition for 'Where' and the best extension method overload
'System.Linq.Enumerable.Where<TSource>(System.Collections.Generic.IEnumerable<TSource>, System.Func<TSource,int,bool>)' has some invalid arguments
I understand that my expression doesn't conform to the predicate System.Func<TSource,int,bool>, however the Where function has an overload that takes a predicate of System.Func<TSource,bool>, which as far as I can tell should work.
I have tried converting the result of assembly.GetTypes() (which is an array) to a List in several ways without it helping the issue.
I have also made sure that I have got all the correct using statements for this class as that seems to be an issue several people run into during my Googling.
In the past I have managed to use this same technique on a IQueryable collection, but I don't understand why this won't work when the Where function is available on the IEnumerable collection and should accept the predicate I am providing.
Thanks very much for any assistance.
Compile your expression into executable code (delegate):
var types = assembly.GetTypes().AsEnumerable().Where(_standardFilter.Compile());
As #Kirk stated, it's better not to use expression tree, if you are not going to analyze it. Simply use filter of type Func<Type, bool>.
Note that the Enumerable extension method takes a Func, not an Expression. So your argument is incorrectly typed.
This is in contrast with Entity Framework, which takes Expression<Func> rather than plain Func.
So be attentive to method signatures, they are similar, can both be called with the same lambda, but are actually different!
The reason behind that is that EF inspects the Expression and converts it to SQL code, whereas Linq to Object simply executes the predicate in memory.
You could try this instead:
private Func<Type, bool> standardFilter;
standardFilter = (type) => {
return type.IsClass;
};
var types = assembly.GetTypes().AsEnumerable().Any(x => standardFilter(x));
Then you're free to change your implementation of standardFilter. As long as it takes in a Type and returns bool for whether or not that should be included it will remain modular and accomplish what I think you're going for.
I know what a Lambda Expression is.
But I am not sure if that is the same thing as an Expression. There seems to be more to know here than I know.
I am looking at wrapping IQueryable and that uses Expressions a lot. So, for example, is there more to the 'Expression' parameter here than can be thought of in a Lambda?
public InterceptedQuery(InterceptingProvider provider, Expression expression)
{
this._provider = provider;
this._expression = expression;
}
A lambda expression is a compiler feature that is compiled, depending on context, into one of two things:
A (hidden) function and a delegate to it
An Expression
Once the application has been compiled, the concept of a lambda expression doesn't exist, as it's been turned into one of the above two options.
I'm not sure what you mean by
is there more to the 'Expression' parameter here than can be thought of in a Lambda
An expression encapsulates and expressed application logic in an inspectable form (in other words, it lets you see what the developer wrote in terms of properties and functions invoked, constants included, comparisons, etc.). This is how query providers (for the most part, object-relational mappers like the Entity Framework) take code and turn it into SQL.
Lambdas are generally compiled code, where Expression represents "abstract syntax trees" (AST), ie. a data structure that represents code, and can be compiled to code. IQueryable generally operates on Expression because it's supposed to compile the AST to code that runs in different environments, like an SQL server, instead of just the host machine. There are IQueryableProviders that compile to SQL (Linq2Sql), to JavaScript, to OpenGL shaders (Bling), and more.
The C# compiler can sometimes turn a lambda into an expression, if the method parameter expects an expression of the right type:
void Foo(Expression<Func<int>>) { ... }
...
Foo(() => 3);
I am developing Web Custom control which able to apply filters to the LinqDataSource object. It is generic control since it should operate with object of certain type.
The control knows what field of object it should operate by following field
/// <summary>
/// Method to get compared column from object
/// </summary>
public Expression<Func<T, int>> GetColumnMethod;
(I transfer to it method which gets appropriate field from object type)
We perform the filtering with code like that
... if (selectedValue == "<=")
predicate = predicate.And(c => method(c) <= val);
if (selectedValue == "<")
predicate = predicate.And(c => method(c) < val);
All proceeds OK until LINQ to SQL transformation occurs.
Then error "Method '.....' has no supported translation to SQL.
sure, CLR doesn't know how to make SQL for delegates.
If only C# could compile the expression before translation to SQL, but I have no idea how to make it to do it.
Perversions like Expression.Compile (whatever tricky ways I tried the whole day - I already cannot remember them all...nothing helped)
But ... at the runtime CLR already knows the type of my object, so it could to manage to build SQL expression having compiked delegate values. But how to do it ? God knows.
Help highly appreciates.
The LINQ to SQL provider is responsible for translation your expression tree into a valid T-SQL statement. As there is not a 1-to-1 relationship between C# and T-SQL it is highly probable that the more sophisticated your expression tree, the less likely it is that LINQ to SQL will be able to translate.
When you use a lambda expression you have to decide if you want to compile it or use it as an expression tree. If you want to use the lambda as an expression then you are responsible for ensuring that the expression contains functions and syntax that your provider supports.
It won't work. Basically your LINQ query is valid C# code so it compiles fine but it fails during runtime on anything outside of scope of LINQ to SQL provider.
Read more here:
“Cannot call methods on DateTime”, and other limitations
If it's just a matter of selecting a particular field to use in your query, you should be able to do this using the Expression.Compile "perversion". But that's likely to be a lot of work. You can probably get a head start by compiling what you have now and using Reflector against the output to see what the code generated by the C# compiler looks like.
Can you split the query into two parts -- one that gets translated to SQL and runs on the server and one that uses your GetColumnMethod and runs in memory against the data output by the first part?
There is no SQL for arbitrary delegates. If method(c) can be expressed as a lambda, you can invoke the lambda as a sub-expression using Expression.Invoke, but you'd need to build the expression tree yourself, like so.