What happens behind the curtains when I include a function into my compiled query, like I do with DataConvert.ToThema() here to convert a table object into my custom business object:
public static class Queries
{
public static Func<MyDataContext, string, Thema> GetThemaByTitle
{
get
{
var func = CompiledQuery.Compile(
(MyDataContext db, string title) =>
(from th in elan.tbl_Thema
where th.Titel == title
select DataConvert.ToThema(th)).Single()
);
return func;
}
}
}
public static class DataConvert
{
public static Thema ToThema(tbl_Thema tblThema)
{
Thema thema = new Thema();
thema.ID = tblThema.ThemaID;
thema.Titel = tblThema.Titel;
// and some other stuff
return thema;
}
}
and call it like this
Thema th = Queries.GetThemaByTitle.Invoke(db, "someTitle");
Apparently the function is not translated in to SQL or something (how could it), but it also does not hold when I set a breakpoint there in VS2010.
It works without problems, but I don't understand how or why. What exactly happens there?
Your DataConvert.ToThema() static method is simply creating an instance of a type which has a default constructor, and setting various properties, is that correct? If so, it's not terribly different from:
(from th in elan.tbl_Thema
where th.Titel == title
select new Thema{ID=th.ThemaID, Titel=th.Titel, etc...}
).Single());
When you call Queries.GetThemaByTitle, a query is being compiled. (The way you are calling this, by the way, may or may not actually be giving you any benefits from pre-compiling). That 'Query' is actually a code expression tree, only part of which is intended to generate SQL code that is sent to the database.
Other parts of it will generate IL code which is grabbing what is returned from the database and putting it into some form for your consumption. LINQ (EF or L2S) is smart enough to be able to take your static method call and generate the IL from it to do what you want - and maybe it's doing so with an internal delegate or some such. But ultimately, it doesn't need to be (much) different from what would be generated from I substituted above.
But note that this happens regardless what the type is that you get back; somewhere, IL code is being generated that puts DB values into a CLR object. That is the other part of those expression trees.
If you want a more detailed look at those expression trees and what they involved, I'd have to dig for ya, but I'm not sure from your question if that's what you are looking for.
Let me start by pointing out, that whether you compile your query or not does not matter. You would observe the very same results even if you did not pre-compile.
Technically, as Andrew has pointed out, making this work is not that complicated. When your LINQ expression is evaluated an expression tree is constructed internally. Your function appears as a node in this expression tree. No magic here. You'll be able to write this expression both in L2S and L2E and it will compile and run fine. That is until you try to actually execute the actual SQL query against the database. This is where difference begins. L2S seems to happily execute this task, whereas L2E fails with NotSupportedException, and reporting that it does not know how to convert ToThema into store query.
So what's happening inside? In L2S, as Andrew has explained, the query compiler understands that your function can be run separately from the store query has been executed. So it emits calls to your function into the object reading pipeline (where data read from SQL is transformed to the objects that are returned as the result of your call).
Once thing Andrew was not quite right, is that it matters what's inside your static method. I don't think it does.
If you put a break point in the debugger to your function, you will see that it's called once per returned row. In the stack trace you will see "Lightweight Function", which, in reality, means that the method was emitted at run time. So this is how it works for Linq to Sql.
Linq to Entity team seemed to go different route. I do not know, what was the reasoning, why they decided to ban all InvocationExpressions from L2E queries. Perhaps these were performance reason, or may be the fact that they need to support all kind of providers, not SQL Server only, so that data readers might behave differently. Or they simply thought that most people wouldn't realize that some of those are executed per returned row and preferred to keep this option closed.
Just my thoughts. If anyone has any more insight, please chime in!
Related
I've written some code that takes an Expression and acts as a proxy to make certain calls. The main bit of code that makes this work is such
private static IEnumerable<object> GetArguments(MethodCallExpression body)
{
IEnumerable<object> arguments =
body.Arguments.Select(
expression =>
{
MemberExpression member = expression as MemberExpression;
return ((dynamic) member.Member).GetValue(((ConstantExpression) member.Expression).Value);
});
return arguments;
}
This gets the values of the parameters present in the Expression and returns them in an IEnumerable. That's the behavior I get when I wrote this in the initially consuming project and I can see the resulting argument values being output correctly. I even moved this to a test bed project to test the code and it worked fine there as well. However, when I put the .cs files in another project, updating the namespaces, and reference it to be able to use it widely, I get exceptions such as
System.InvalidCastException: Unable to cast object of type 'System.Linq.Expressions.PropertyExpression' to type 'System.Linq.Expressions.ConstantExpression'.
and I can't even use my application.
I've rolled my extraction back to the "it works locally" state to ensure it, in fact, worked, and it did. However, I have noticed I still get some exceptions in my logging even though it didn't throw up on me like it does when it's external to the project. I can still use my application in spite of this.
Repeating the move to an external project and using it by way of a reference reintroduces those errors and I can't even use my app.
New question: how do I handle PropertyExpressions? They don't expose a Value property and are internal so I there's no cast and check. MemberExpression also doesn't have a Value property.
The simplest solution would be not to rely on the shape of the expressions at all and let the Expression Tree library evaluate each subexpression for you by building a LambdaExpression and then using Compile() on it and executing the returned delegate:
private static IEnumerable<object> GetArguments(MethodCallExpression body)
{
return body.Arguments.Select(
expression => Expression.Lambda<Func<object>>(
Expression.Convert(expression, typeof(object))).Compile()());
}
Note that compiling each subexpression will take some time, so this approach is going to be relatively slow.
You should evaluate the expression and not rely on it's shape. The easiest way is to call .Compile().Invoke(), but that is slow and creates a memory-leak. You can avoid that by interpreting the expression (stepping recursively through it with a visitor and perform the operations in it manually) or cache your compilation result. I wrote a library that can do both, is fast and used in production for quite some time now: https://github.com/Miaplaza/expression-utils
I have a question related to this previous question of mine. In an existing bit of LINQ which involves a number of joins, I'm trying to take each separate method comprising the join and convert it to a CompiledQuery.
First, the normal LINQ method:
private IQueryable<Widget> GetWidgetQuery()
{
return db.Widgets.Where(u => (!u.SomeField.HasValue || !u.SomeField.Value));
}
And here, a delegate (field) definition for a CompiledQuery along these lines:
private static readonly Func<DBDataContext, IQueryable<Widget>> GetWidgetQuery =
CompiledQuery.Compile((DBDataContext db) =>
db.Widgets.Where(u => (!u.SomeField.HasValue || !u.SomeField.Value)));
If I hover over the normal LINQ statement for the method GetWidgetQuery(), I see that it's a method as below:
(method) IQueryable<Widget> GetWidgetQuery()
However, the compiled query delegate (field) differs as follows:
(field) Func<DBDataContext, IQueryable<Widget>> GetWidgetQuery
Upon executing the latter as part of the LINQ statement, the syntax differs as follows. First, the normal LINQ's participation in the join:
var myquery =
from wxr in GetWidgetXRQuery()
join w in GetWidgetQuery() on wxr.WidgetID equals w.ID
select new DTO.WidgetList
{
...
}
And here, the invocation of the CompiledQuery in the form of the delegate:
var myquery =
from wxr in GetWidgetXRQuery()
join w in GetWidgetQuery.Invoke(myContext) on wxr.WidgetID equals w.ID
select new DTO.WidgetList
{
...
}
The former returns the expected result set; the latter, when I attempt myquery.ToList(), yields a stackoverflow exception, in part related to this limitation of .NET 3.5, I think.
Can someone please help me understand how the compiled statement existing as a field (or I guess I should say a delegate) rather than a method is killing my query? In short I know what I'm doing is wrong, but I'm not sure I understand what I misunderstand.
I tried doing roughly the same thing you're doing on EF 4, and everything seems to work fine. So it's either an EF 3.5 issue, or it has something to do with your implementation of GetWidgetXRQuery, or some combination of the two.
But the real point I'd like to make is that, as Roy Goode stated in an answer to your previous question, you lose all the advantages of a precompiled query once you extend that query in any way. By trying to perform a Join on your query, you are converting it to just a plain old query. So you might as well just use the non-compiled version which appears to work for you.
Update
Realized you were talking about LINQ to SQL. This sort of query does appear to have support in Entity Framework, but not LINQ to SQL. In .NET 4, I'm getting the following error:
An IQueryable that returns a self-referencing Constant expression is not supported.
That doesn't mean much to me, but I'm guessing that it has something to do with the way the compiled query is represented internally. I still get the same error if I evaluate the query into a variable and use that variable in the query later, so it clearly has nothing to do with the difference between a delegate and a function. I still maintain that a compiled query is not appropriate to use here. Either you need to create one big compiled query to represent the whole query you want to perform, or you need to use regular queries if you want to piece them together this way.
I just came across this same error while doing db integration testing, and to jump straight to the point without trying to explain my specific issue. Linq to Sql will create the sql query internally when using IQueryable and the moment you execute a method on that IQueryable, i.e. ToList() it executes that query on the database. So in my case I am joining to a method that returns IQueryable but is mocked to return a result, it is trying to compile that to a sql query but the IQueryable I created does not have an internal SQL query
I use LINQ-SQL as my DAL, I then have a project called DB which acts as my BLL. Various applications then access the BLL to read / write data from the SQL Database.
I have these methods in my BLL for one particular table:
public IEnumerable<SystemSalesTaxList> Get_SystemSalesTaxList()
{
return from s in db.SystemSalesTaxLists
select s;
}
public SystemSalesTaxList Get_SystemSalesTaxList(string strSalesTaxID)
{
return Get_SystemSalesTaxList().Where(s => s.SalesTaxID == strSalesTaxID).FirstOrDefault();
}
public SystemSalesTaxList Get_SystemSalesTaxListByZipCode(string strZipCode)
{
return Get_SystemSalesTaxList().Where(s => s.ZipCode == strZipCode).FirstOrDefault();
}
All pretty straight forward I thought.
Get_SystemSalesTaxListByZipCode is always returning a null value though, even when it has a ZIP Code that exists in that table.
If I write the method like this, it returns the row I want:
public SystemSalesTaxList Get_SystemSalesTaxListByZipCode(string strZipCode)
{
var salesTax = from s in db.SystemSalesTaxLists
where s.ZipCode == strZipCode
select s;
return salesTax.FirstOrDefault();
}
Why does the other method not return the same, as the query should be identical ?
Note that, the overloaded Get_SystemSalesTaxList(string strSalesTaxID) returns a record just fine when I give it a valid SalesTaxID.
Is there a more efficient way to write these "helper" type classes ?
Thanks!
This is probably down to the different ways LINQ handles IEnumerable<T> and IQueryable<T>.
You have declared Get_SystemSalesTaxList as returning IEnumerable<SystemSalesTaxList>. That means that when, in your first code sample, you apply the Where operator to the results of Get_SystemSalesTaxList, it gets resolved to the Enumerable.Where extension method. (Note that what matters is the declared type. Yes, at runtime Get_SystemSalesTaxList is returning an IQueryable<SystemSalesTaxList>, but its declared type -- what the compiler sees -- is IEnumerable<SystemSalesTaxList>.) Enumerable.Where runs the specified .NET predicate over the target sequence. In this case, it iterates over all the SystemSalesTaxList objects returned by Get_SystemSalesTaxList, yielding the ones where the ZipCode property equals the specified zip code string (using the .NET String == operator).
But in your last code sample, you apply the Where operator to db.SystemSalesTaxList, which is declared as being of type IQueryable<SystemSalesTaxList>. So the Where operator in that sample gets resolved to Queryable.Where, which translates the specified predicate expression to SQL and runs it on the database.
So what's different in the zip code methods is that the first one runs the C# s.ZipCode == strZipCode test in .NET, and the second translates that into a SQL query WHERE ZipCode = 'CA 12345' (parameterised SQL really but you get the idea). Why do these give different results? Hard to be sure, but the C# == predicate is case-sensitive, and depending on your collation settings the SQL may or may not be case-sensitive. So my suspicion is that strZipCode doesn't match the database zip codes in case, but in the second version SQL Server collation is smoothing this over.
The best solution is probably to change the declaration of Get_SystemSalesTaxList to return IQueryable<SystemSalesTaxList>. The major benefit of this is that it means queries built on Get_SystemSalesTaxList will be executed database side. At the moment, your methods are pulling back EVERYTHING in the database table and filtering it client side. Changing the declaration will get your queries translated to SQL and they will run much more efficiently, and hopefully will solve your zip code issue into the bargain.
The real issue here is the use of IEnumerable<T>, which breaks "composition" of queries; this has two effects:
you are reading all (or at least, more than you need) of your table each time, even if you ask for a single row
you are running LINQ-to-Objects rules, so case-sensitivity applies
Instead, you want to be using IQueryable<T> inside your data layer, allowing you to combine multiple queries with additional Where, OrderBy, Skip, Take, etc as needed and have it build the TSQL to match (and use your db's case-sensitivity rules).
Is there a more efficient way to write these "helper" type classes ?
For more efficient (less code to debug, doesn't stream the entire table, better use of the identity-map to short-circuit additional lookups (via FirstOrDefault etc)):
public IEnumerable<SystemSalesTaxList> Get_SystemSalesTaxList()
{
return db.SystemSalesTaxLists;
}
public SystemSalesTaxList Get_SystemSalesTaxList(string salesTaxID)
{
return db.SystemSalesTaxLists.FirstOrDefault(s => s.SalesTaxID==salesTaxID);
}
public SystemSalesTaxList Get_SystemSalesTaxListByZipCode(string zipCode)
{
return db.SystemSalesTaxLists.FirstOrDefault(s => s.ZipCode == zipCode);
}
I am developing Web Custom control which able to apply filters to the LinqDataSource object. It is generic control since it should operate with object of certain type.
The control knows what field of object it should operate by following field
/// <summary>
/// Method to get compared column from object
/// </summary>
public Expression<Func<T, int>> GetColumnMethod;
(I transfer to it method which gets appropriate field from object type)
We perform the filtering with code like that
... if (selectedValue == "<=")
predicate = predicate.And(c => method(c) <= val);
if (selectedValue == "<")
predicate = predicate.And(c => method(c) < val);
All proceeds OK until LINQ to SQL transformation occurs.
Then error "Method '.....' has no supported translation to SQL.
sure, CLR doesn't know how to make SQL for delegates.
If only C# could compile the expression before translation to SQL, but I have no idea how to make it to do it.
Perversions like Expression.Compile (whatever tricky ways I tried the whole day - I already cannot remember them all...nothing helped)
But ... at the runtime CLR already knows the type of my object, so it could to manage to build SQL expression having compiked delegate values. But how to do it ? God knows.
Help highly appreciates.
The LINQ to SQL provider is responsible for translation your expression tree into a valid T-SQL statement. As there is not a 1-to-1 relationship between C# and T-SQL it is highly probable that the more sophisticated your expression tree, the less likely it is that LINQ to SQL will be able to translate.
When you use a lambda expression you have to decide if you want to compile it or use it as an expression tree. If you want to use the lambda as an expression then you are responsible for ensuring that the expression contains functions and syntax that your provider supports.
It won't work. Basically your LINQ query is valid C# code so it compiles fine but it fails during runtime on anything outside of scope of LINQ to SQL provider.
Read more here:
“Cannot call methods on DateTime”, and other limitations
If it's just a matter of selecting a particular field to use in your query, you should be able to do this using the Expression.Compile "perversion". But that's likely to be a lot of work. You can probably get a head start by compiling what you have now and using Reflector against the output to see what the code generated by the C# compiler looks like.
Can you split the query into two parts -- one that gets translated to SQL and runs on the server and one that uses your GetColumnMethod and runs in memory against the data output by the first part?
There is no SQL for arbitrary delegates. If method(c) can be expressed as a lambda, you can invoke the lambda as a sub-expression using Expression.Invoke, but you'd need to build the expression tree yourself, like so.
In a previous question I asked how to make "Computed properties" in a linq to sql object. The answer supplied there was sufficient for that specific case but now I've hit a similar snag in another case.
I have a database with Items that have to pass through a number of Steps. I want to have a function in my database that retrieves the Current step of the item that I can then build on. For example:
var x = db.Items.Where(item => item.Steps.CurrentStep().Completed == null);
The code to get the current step is:
Steps.OrderByDescending(step => step.Created).First();
So I tried to add an extension method to the EntitySet<Step> that returned a single Step like so:
public static OrderFlowItemStep CurrentStep(this EntitySet<OrderFlowItemStep> steps)
{
return steps.OrderByDescending(o => o.Created).First();
}
But when I try to execute the query at the top I get an error saying that the CurrentStep() function has no translation to SQL. Is there a way to add this functionality to Linq-to-SQL in any way or do I have to manually write the query every time? I tried to write the entire query out first but it's very long and if I ever change the way to get the active step of an item I have to go over all the code again.
I'm guessing that the CurrentStep() method has to return a Linq expression of some kind but I'm stuck as to how to implement it.
The problem is that CurrentStep is a normal method. Hence, the Expression contains a call to that method, and naturally SQL cannot execute arbitrary .NET methods.
You will need to represent the code as an Expression. I have one in depth example here: http://www.atrevido.net/blog/2007/09/06/Complicated+Functions+In+LINQ+To+SQL.aspx
Unfortunately, the C# 3.0 compiler has a huge omission and you cannot generate calls to Expressions. (i.e., you can't write "x => MyExpression(x)"). Working around it either requires you to write the Expression manually, or to use a delegate as a placeholder. Jomo Fisher has an interesting post about manipulating Expression trees in general.
Without actually having done it, the way I'd probably approach it is by making the CurrentStep function take the predicate you want to add ("Completed == null"). Then you can create a full Expression> predicate to hand off to Where. I'm lazy, so I'm going to do an example using String and Char (String contains Chars, just like Item contains Steps):
using System;
using System.Linq;
using System.Linq.Expressions;
class Program {
static void Main(string[] args) {
Console.WriteLine(StringPredicate(c => Char.IsDigit(c)));
var func = StringPredicate(c => Char.IsDigit(c)).Compile();
Console.WriteLine(func("h2ello"));
Console.WriteLine(func("2ello"));
}
public static Expression<Func<string,bool>> StringPredicate(Expression<Func<char,bool>> pred) {
Expression<Func<string, char>> get = s => s.First();
var p = Expression.Parameter(typeof(string), "s");
return Expression.Lambda<Func<string, bool>>(
Expression.Invoke(pred, Expression.Invoke(get, p)),
p);
}
}
So "func" is created by using StringPredicate to create an Expression. For the example, we compile it to execute it locally. In your case, you'd pass the whole predicate to "Where" so it gets translated to SQL.
The "get" expression is where you put your "extension" stuff (OrderByWhatever, First, etc.). This is then passed in to the predicate that's given to you.
Don't worry if it looks complicated; it sorta is at first. If you haven't done this kinda stuff before, it'll take a bit of time (the first time I did this kinda stuff, it took hours to get it right :|.. now it comes slightly easier). Also, as I mentioned, you can write a helper method to do this re-writing for you (so you don't directly need to use the Expression.Whatever methods), but I haven't seen any examples and haven't really needed it yet.
Check out my answer to "switch statement in linq" and see if that points you in the right direction...
The technique i demonstrate there is the one that got me past the scary "no translation to SQL" error.