I have users who will generate business rules via simple statements that result in true or false
Example: Result > 10 AND Result < 100
I am using Linq.Dynamic to convert these statements I am finding the ParseLambda is taking too long (simple example took 21ms). So I would like to save the Lambda Expression to the database to pull and use later. From my testing I have found the call to ParseLambda is taking the most time, therefore I am wondering if it is possible to serialize this Lambda Expression and persist to a database pulling it when needed. The idea is to have it saved to the database already parsed.
Maybe this is the wrong approach all together. I thought about using CodeDom and scripting but found I would have to create an assembly for every rule, which would pollute the AppDomain, I could have over 1000 rules.
I am open to other options, maybe there is a better library out there.
ParameterExpression P = Expression.Parameter(typeof(InternalEvalData), _ParameterHeading);
LambdaExpression X = System.Linq.Dynamic.DynamicExpression.ParseLambda(new[] { P }, null, parsedExpression);
result = (bool)d.DynamicInvoke(testData);
Related
I need to do a query on my database that might be something like this where there could realistically be 100 or more search terms.
public IQueryable<Address> GetAddressesWithTown(string[] towns)
{
IQueryable<Address> addressQuery = DbContext.Addresses;
addressQuery.Where( x => towns.Any( y=> x.Town == y ) );
return addressQuery;
}
However when it contains more than about 15 terms it throws and exception on execution because the SQL generated is too long.
Can this kind of query be done through Entity Framework?
What other options are there available to complete a query like this?
Sorry, are we talking about THIS EXACT SQL?
In that case it is a very simple "open your eyes thing".
There is a way (contains) to map that string into an IN Clause, that results in ONE sql condition (town in ('','',''))
Let me see whether I get this right:
addressQuery.Where( x => towns.Any( y=> x.Town == y ) );
should be
addressQuery.Where ( x => towns.Contains (x.Town)
The resulting SQL will be a LOT smaller. 100 items is still taxing it - I would dare saying you may have a db or app design issue here and that requires a business side analysis, I have not me this requirement in 20 years I work with databases.
This looks like a scenario where you'd want to use the PredicateBuilder as this will help you create an Or based predicate and construct your dynamic lambda expression.
This is part of a library called LinqKit by Joseph Albahari who created LinqPad.
public IQueryable<Address> GetAddressesWithTown(string[] towns)
{
var predicate = PredicateBuilder.False<Address>();
foreach (string town in towns)
{
string temp = town;
predicate = predicate.Or (p => p.Town.Equals(temp));
}
return DbContext.Addresses.Where (predicate);
}
You've broadly got two options:
You can replace .Any with a .Contains alternative.
You can use plain SQL with table-valued-parameters.
Using .Contains is easier to implement and will help performance because it translated to an inline sql IN clause; so 100 towns shouldn't be a problem. However, it also means that the exact sql depends on the exact number of towns: you're forcing sql-server to recompile the query for each number of towns. These recompilations can be expensive when the query is complex; and they can evict other query plans from the cache as well.
Using table-valued-parameters is the more general solution, but it's more work to implement, particularly because it means you'll need to write the SQL query yourself and cannot rely on the entity framework. (Using ObjectContext.Translate you can still unpack the query results into strongly-typed objects, despite writing sql). Unfortunately, you cannot use the entity framework yet to pass a lot of data to sql server efficiently. The entity framework doesn't support table-valued-parameters, nor temporary tables (it's a commonly requested feature, however).
A bit of TVP sql would look like this select ... from ... join #townTableArg townArg on townArg.town = address.town or select ... from ... where address.town in (select town from #townTableArg).
You probably can work around the EF restriction, but it's not going to be fast and will probably be tricky. A workaround would be to insert your values into some intermediate table, then join with that - that's still 100 inserts, but those are separate statements. If a future version of EF supports batch CUD statements, this might actually work reasonably.
Almost equivalent to table-valued paramters would be to bulk-insert into a temporary table and join with that in your query. Mostly that just means you're table name will start with '#' rather than '#' :-). The temp table has a little more overhead, but you can put indexes on it and in some cases that means the subsequent query will be much faster (for really huge data-quantities).
Unfortunately, using either temporary tables or bulk insert from C# is a hassle. The simplest solution here is to make a DataTable; this can be passed to either. However, datatables are relatively slow; the over might be relevant once you start adding millions of rows. The fastest (general) solution is to implement a custom IDataReader, almost as fast is an IEnumerable<SqlDataRecord>.
By the way, to use a table-valued-parameter, the shape ("type") of the table parameter needs to be declared on the server; if you use a temporary table you'll need to create it too.
Some pointers to get you started:
http://lennilobel.wordpress.com/2009/07/29/sql-server-2008-table-valued-parameters-and-c-custom-iterators-a-match-made-in-heaven/
SqlBulkCopy from a List<>
http://geekswithblogs.net/Martinez/archive/2009/06/29/understanding-expression-trees.aspx
By reading this article that expline what are Expression trees and what they are used for I have some question thats the article those not answer me clearly.
A. What is the difference between Data Structure and 'compiled piece of code'.
* I know that data structure is Array,List,.. But this is not the answer I am looking for.
B. Why is it better to use data structure in Entity Framework than 'compiled piece of code' in terms of efficiency.
Answer to both A and B.
The compiled piece of code can not be reverse engineered into something that you can translate to SQL. There are to many details that made the original concepts you want to translate to SQL go away. The expression trees can be analyzed and understood at run time, when the compiled code can not.
Another meta representation of the queries would be possible to use, but the expression trees gives the advantage that existing plumbing in the compiler can be used for a lot of the heavy work.
An Expression is code representing code. It is not an executable piece of code that you can run until you call Compile().
An Expression contains an object graph that describes some code. So the example function:
x => x + 1
Is representing a function that takes 1 parameter, and returns the expression (x + 1). As an object graph, it's something like this:
Expression
AdditionExpression
RightValue
VariableExpression (x)
LeftValue
LiteralExpression (1)
You could parse through the expression and figure out the name of the parts of the expression, or what the literal value is, or what the operation is. Once it is compiled, it is a logical sequence of operations, and all of that metadata is lost.
In terms of Entity Framework, given a sample expression like so:
myObjects.Where(x => x.Id > 10).ToList()
The LINQ to SQL IQueryable provider is inspecting this expression, finding the name of the property Id and then finding the literal value of 10 to convert that into a SQL statement. If this was a function (a compiled unit of code), there is no metadata to inspect, and conversion to SQL would be impossible. It also allows you to deconstruct things into supported and unsupported expressions for efficiency in making sure your SQL query returns the requested data only, instead of a large data set.
I was able to create a LINQ statement that I thought was strange and wanted to see if anyone else had experience with it.
I've simplified it to this:
var x = db.Test
.Where(a => a.Field1 == Utils.CreateHash(Preferences.getValue(a.Field2)))
.FirstOrDefault();
Now how does this translate to database code? Wouldn't LINQ need to do a double query for every single row, i.e. for row a:
1) Query a.Field2
2) Return value to run Utils.CreateHash(Preferences.getValue(a.Field2))
3) Take that value from step 2 and compare it against a.Field1
4) Repeat 1-3 until I've gone through all the rows or returned a matching row
Wouldn't this be extremely inefficient? Or is LINQ smart enough to run this in a better way? Note, I haven't actually run this code so another possibility is a runtime error. Why wouldn't LINQ be smart enough to detect a conflict then and not let me compile it?
The query as is will not work since have a call to Utils.CreateHash in your lambda that you are trying to execute on the DB - in that context you cannot execute that method since there simply is no equivalent on the DB side hence the query will fail.
In general the ability of 3rd party Linq IQuerable providers (e.g. Linq to SQL, Linq to Entities) to access in memory constructs such as methods or classes is very limited, as a rule of thumb at most accessing primitive values or collections of primitives will work.
Just to add fast...
A good example to know how this works would be to write (extreme case I agree, but best :) or go through the source code for a custom (open source) LINQ provider (e.g. http://relinq.codeplex.com/ has one etc.).
Basically (I'm simplifying things here a bit), a LINQ provider can only 'map' to Db (supported SQL, functions) what he 'knows' about.
i.e. it has a standard set it can work with, other than that, and with your custom methods (that do not translate to constants etc.) in the frame, there is no way to resolve that on the 'Db/SQL side'.
E.g. with your 'custom' linq provider (not the case here) you could add a specific extension call e.g. .MyCalc() - which would be properly resolved and translated into SQL equivalent - and then you'd be able to use it.
Other than that, I think if I recall correct, provider will leave that as an expression, to resolve when it returns from the Db 'fetch', query operation. Or complain about it in certain cases.
Linq is based on IQueryable - and you can take a look at extension methods provided there for SQL equivalents supported.
hope this helps
EDIT: whether things 'work' or not doesn't matter - it still doesn't mean it'd execute on the Db context - i.e. it'd be unacceptable performance wise in most cases. IQueryable works with expressions (and if you look at the interface) - and linq is executed when you invoke or enumerate usually. At that point some of the expressions may evaluate to a const value that can be worked into a SQL, but not in your case.
Best way to test is to test back the SQL generated by query (possibly this one I think Translate LINQ to sql statement).
No.
The LINQ provider will run a single SELECT query that selects both fields, then execute your lambda expression with the two values for each returned row.
The fact that I expected this to work and it didn't leads me to search for the piece of the picture that I don't see.
Imagine that query being remoted over to a database. How is the database engine supposed to reach over the internet as it is executing the query and tell the "count" variable on your machine to update itself? There is no standard mechanism for doing so, and therefore anything that would mutate a variable on the local machine cannot be put into an expression tree that would run on a remote machine.
More generally, queries that cause side effects when they are executed are very, very bad queries. Never, ever put a side effect in a query, even in the cases where doing so is legal. It can be very confusing. Remember, a query is not a for loop. A query results in an object represents the query, not the results of the query. A query which has side effects when executed will execute those side effects twice when asked for its results twice, and execute them zero times when asked for the results zero times. A query where the first part mutates a variable will mutate that variable before the second clause executes, not during the execution of the second clause. As a result, many query clauses give totally bizarre results when they depend on side effect execution.
For more thoughts on this, see Bill Wagner's MSDN article on the subject:
http://msdn.microsoft.com/en-us/vcsharp/hh264182
If you were writing a LINQ-to-Objects query you could do what you want by zipping:
var zipped = model.DirectTrackCampaigns.Top("10").Zip(Enumerable.Range(0, 10000), (first, second)=>new { first, second });
var res = zipped.Select(c=> new Campaign { IsSelected = c.second % 2 == 0, Name = c.first.CampaignName };
That is, make a sequence of pairs of numbers and campaigns, then manipulate the pairs.
How you'd do that in LINQ-to-whatever-you're-using, I don't know.
When you pass anything to a method that expects an Expression, nothing is actually evaluated at that time. All it does is, it breaks apart the code and creates an Expression tree out of it.
Now, in .Net 4.0, alot of things were added to the Expression API (including Expression.Increment and Expression.Assign which would actually be what you're doing), however the compiler (I think it's a limitation of the C# compiler) hasn't been updated yet to take advantage of the new 4.0 Expression stuff. Therefore, we're limited to method calls, and assignment calls will not work. Hypothetically, this could be supported in the future.
IsSelected = (count++) % 2 == 0
Pretty much means:
IsSelected = (count = count + 1) % 2 == 0
That's where the assignment is happening and the source of the error.
According to the different answers to this question, this should be possible with .NET 4.0 using expressions, though not in lambdas.
I am having a lot of fun with Linq2Sql. Expression Trees have been great, and just the standard Linq2Sql syntax has been a lot of fun.
I am now down to part of my application where I have to somehow store queries in a database, that are custom for different customers that use the same database and same tables (well, view, but you know what I mean). Basically, I cant hard-code anything, and I have to leave the query language clear text so someone can write a new where-clause type query.
So, if that description was harsh, let me clarify:
In a previous version of our application, we used to do direct SQL calls to the db using raw SQL. Yea. it was fun, dirty, and it worked. We would have a database table fulled of different criteria like
(EventType = 6 and Total > 0)
or a subquery style
(EventType = 7
AND Exists (
select *
from events as e1
where events.EventType = e1.EventType
and e1.objectNumber = 89)
)
(sql injection anyone?)
In Linq2Sql, this is a little more challenging. I can make all these queries no problem in the CLR, but being able to pass dynamic where criterias to Linq is a little more challenging, especially if I want to perform a sub query (like the above example).
Some ideas I had:
Get the raw expression, and store it --- but I have no idea how to take the raw text expression and reverse it back to executable to object expression.
Write a SQl like language, and have it parse the code and generate Linq Expression -- wow, that could be a lot of fun
I am quite sure there is no SomeIqueryable.Where("EventType = 6 and Total > 54"). I was reading that it was available in beta1, but I don't see how you can do that now.
var exp2 = context.POSDataEventView.Where("EmployeeNumber == #0", 8310);
This would be the easiest way for me to deploy.. I think.
Store serialized Expressions -- wow.. that would be confusing to a user trying to write a query --- hell, I'm not sure I could even type it all out.
So, I am looking for some ideas on how I can store a query in some kind of clear text, and then execute it against my Linq2Sql objects in some fashion without calling the ExecuteSQL. I want to use the LinqObjects.
P.S. I am using pLinqo for this application if that helps. Its still linq2sql though.
Thanks in advance!
Perhaps the Dynamic LINQ Library (in the MSDN samples) would help?
In particular, usage like:
This should work with any IQueryable<T> source - including LINQ-to-Objects simply by calling .AsQueryable() on the sequence (typically IEnumerable<T>).