We are in process of preparing dynamic filters which will be used for filtering records. We just should be able to select an entity (Lets say Person) then define filters according to that entity selected.
The design is something similar to this.
So how to design such a filtering system which can be used with any entity and any property?
We need to use
C#
MVC 5
I have created a dynamic filter feature using WPF, the way I did it was using LINQ To SQL ORM, using my DataContext I get all the tables and its corresponding fields (MetaDataMember) and used it build the UI
IEnumerable<MetaTable> mappedTables = DBContext.Mapping.GetTables();
IEnumerable<MetaDataMember> tablesColumns = mappedTables.SelectMany(t => t.RowType.DataMembers);
Then I build Expression Trees on the fly for each criteria the user created using the MetaDataMember and its property.
Type entityType = metaDataMember.DeclaringType.Type;
ParameterExpression parameterExpression = Expression.Parameter(entityType, "t"); // t => t; t is the parameterExpression representing the Table type
MemberExpression leftStringExpression = Expression.PropertyOrField(parameterExpression, metaDataMember.Name); // t => t.columnName
And then build the conditions i.e, Expression.LessThan (etc) and keep building the tree for each criteria using the conjunction expressions( Expression.AndAlso or Expression.OrElse) for each criteria and then finally create an lambda expression Expressions.Lambda and pass it to .Where() Linq Extension method to get my IQueryable<> filtered values back and use it to filter the UI.
IQueryable<SomeTable> results = DataContext.SomeTable.Where(whereLambdaExpression).Select(selectLambdaExpression);
The reason I went with this approach was because the query doesn't get executed until the enumeration (results.ToList()) occurs, and the query is executed in the DB server and not in the current application. LINQ converts the query expression to pure SQL statement and only the results are returned. It is pretty fast as everything happens in the DB Server.
I am not able to paste all my code as it pretty huge, but you get the gist of it.
I had come across this kind of problem and I solved it by compiling text as LINQ query and executing it on the data (collection) and collecting the result. It will not give you best performance as each time it needs to dynamically compile an assembly (I have used it with collection as big as 100K rows and works well). However, it is very flexible. You can perf tune it further. Code is pretty long so here is the github link.
Here is the main idea:
create a valid linq query expression as string
compile it as an assembly
load the assembly into memory and pass the input collection to the assembly
collect the output and use it.
The github link I shared has a simple working application with samples and projections.
Related
In entity framework 6 is it possible to mix raw SQL generated dynamically with IQueryable like this:
IQueryable<Tree> tree_query = context.Trees.Where(t=>t.Height> 2);
IEnumerable<int> tree_additional_filter = context.Database.SqlQuery<int>("SELECT Id FROM TREE_VIEW WHERE Width < 1");
IQueryable<Tree> final_query = from tree in tree_query
join filtering_tree in tree_additional_filter on filtering_tree.id equals tree.id
select tree;
This produces a result as is, but the "tree_additional_filter" is executed in the database in order to construct the final_query. How can I make entity framework construct only one query from this?
I need this to create dynamic filter fields which work together with static ones.
I also tried creating TREE_VIEW entity with only Id column, which I know to always be there.
Used System.Linq.Dynamic to construct "where" clause dynamically on a TREE_VIEW entity which has only Id property, but apparently expressions can't be built if properties do not exist in the Type.
In entity framework 6 is it possible to mix raw SQL generated dynamically with IQueryable like this:
No. Notice that Database.SqlQuery returns an IEnumerable<T>, not an IQueryable<T>. Therefore any additional query expressions will be executed against the query results using LINQ to Objects.
Query Composition with raw SQL queries was introduced in EF Core.
I managed to do it.
Using Dynamic type generation to create a type (NewDynamicType) from fields which I got selecting top 1 from my TREE_VIEW. Attached the NewDynamicType to the db context via DbModelBuilder.RegisterEntityType in OnModelCreating.
With System.Linq.Dynamic then I could construct IQueryable selecting from context.Set(NewDynamicType) any fields the user wants to filter by.
Join into final_query like in my question.
Now my HTML form gets fields from the database view and in each distibution I can have different filters defined without having to write any c#.
We use Entity Framework, and we need some runtime build queries on our objects. Building expression trees from scratch seems like a lot of work, so we want to use "System.Linq.Dynamic"
Working through the samples I got this to work:
dbModel.As.Where("AStuff.Contains(#0) OR AStuff.Contains(#1)","ac","bc")
But if I try to build the expressions seperately like this:
Expression<Func<A, bool>> predicateA =
DynamicExpression.ParseLambda<A, bool>(
"AStuff.Contains(#0)",
"ac"
);
Expression<Func<A,bool>> predicateB =
DynamicExpression.ParseLambda<A, bool>(
"AStuff.Contains(#0)",
"bc"
);
dbModel.As.Where("#0(it) OR #1(it)", predicateA, predicateB);
it explodes with an exception:
NotSupportedException>>The LINQ expression node type 'Invoke' is not supported in LINQ to Entities.
It may be possible to build the entire query in the first form, but the later would be more useful in our scenario. Is there a way to make that work?
Just use Predicate Builder to join (Or/And) multiple predicates.
I want to dynamically query an object with System.Linq.Dynamic.
var selectData = (from i in data
select i).AsQueryable().Where("Name = #0","Bob1");//This works fine with a non-entity object
I know that we cannot project onto a mapped entity. I believe that is the reason this code fails
foreach (var item in rawQuery.ObsDataResultList)
{
var propertyData = (from i in item
select i).AsQueryable().Where("PropertyName = #0", "blah");
}//item is a Entity Complex Type
Error
Could not find an implementation of the query pattern for
source type 'ClassLibrary1.Model.bhcs_ObsData_2_Result'. 'Select' not
found.
Given the fact that I need to specify the PropertyName at runtime, I don't see any way to project with an anonymous type or a DTO.
I don't need to retain any of the Entity functionality at this point, I just need the data. Copying the data onto something that is queryable is a valid solution. So, is it possible to query entity framework with dynamic LINQ?
And here is the entity class header (the thing I'm trying to query, aka the item object)
[EdmComplexTypeAttribute(NamespaceName="MyDbModel", Name="blah_myQuery_2_Result")]
[DataContractAttribute(IsReference=true)]
[Serializable()]
public partial class blah_myQuery_2_Result : ComplexObject
{
First of all, let me clarify that System.Linq.Dynamic is not a full fledged Microsoft product. It is just a sample we release some time ago, and we don't thoroughly test different LINQ implementations to work correctly with it. If you are looking for a fully supported text-based query language for EF ObjectContext API you should take a look at Entity SQL instead.
Besides that, if you want to use System.Linq.Dynamic and you are ok with testing yourself that you don't hit anything that will block your application from working, then I'll try to see if I can help. I am going to need additional information since I am not sure I understand everything in your code snippets.
First of all I would like to understand, in your first example what is "data" and where did it come from? In your second snippet, what is "rawQuery" and where did it come from? Besdies, what is rawQuery.DataResultList and what is rawQuery.ObsDataResultList?
Also regarding your second snippet, it seems that you are trying to compose with query operators on top of an object that is not actually of a query type (although that doesn't explain the error you are getting given that you are calling AsQueryable the compiler should have complained before that bhcs_ObsData_2_Result is not an IEnumerable nor a non-generic IEnumerable).
In your propposed answer you are saying that you tried with ObjectResult and that seemed to help. Just be aware that ObjectResult is not a query object and therefore it won't allow you to build queries that get send to the server. In other words, any query operators that you apply to ObjectResult will be evaluated in memory and if you don't keep this in mind you may end up bringing all the data from that table into memory before you apply any filtering.
Query ObjectResult<blah_myQuery_2_Result> directly instead of the item blah_myQuery_2_Result. For example
var result = (from i in rawQuery.DataResultList
select i).AsQueryable().Where("CreatedDTM > #0", DateTime.Now.Subtract(new TimeSpan(30, 0, 0, 0)));
So I have the following scenario and I am not sure how to approach it.
In the app that we are building we have a ReferenceDataService which is used to load data via RIA serices. A query is built up on the client as an IQuerable and submitted to the server for retrieval. Now most of the reference queries that are submitted are identical and called multiple times, therefore I would like to service these queries from the entity cache instead.
Below is an example of the query expression that is generated and I have highlighted the area where the query changes. There are some scenarios where the query has additonal criteria but the majority probably dont.
Chime.DataModel.classification[]
.Where(c => ((value(Chime.Modules.Reference.Client.Agent.ReferenceDataLoader+<>c__DisplayClass20).includeSchema
*AndAlso c.class_code.StartsWith(value(Chime.Modules.Reference.Client.Agent.ReferenceDataLoader+<>c__DisplayClass20).schema))*
OrElse (Not(value(Chime.Modules.Reference.Client.Agent.ReferenceDataLoader+<>c__DisplayClass20).includeSchema)
AndAlso *c.parent_code.StartsWith(value(Chime.Modules.Reference.Client.Agent.ReferenceDataLoader<>c__DisplayClass20).schema))))*
.OrderByDescending(c => c.value_scheme_ind)
.ThenBy(c => c.sequence_number)
.ThenBy(c => c.class_label)
So what I would like to do is actually cache the query and if another comes in with identical criteria I can hit the cache. The problem I have is that the literals used in the query do not seem to be available therefore I cannot determin if one query is different from another. The schema code for example is nowhere to be found. It must be sent the the server at some point but I am unsure how I can get at it.
Does anyone know a way to do this or come across this before?
Dave
If you were using WCF Data Services, I would just say call ToString() on your IQueryable and it would return an exact URI of the query to be executed against the Data Service...and you could simply cache based on that string value.
As far as I know, RIA doesn't expose such hooks, so you'll have to rely on more traditional methods.
Implement an ExpressionVisitor that visits your IQueryable.Expression and computes a hash code for it.
You mentioned you can't see your constant string values in your expression tree. That's because they are not inlined yet into the expression tree and thus still exist as MemberAccessExpressions, not ConstantExpressions. To fix this, you can use a partial evaluating ExpressionVisitor. The source for such is given here
http://msdn.microsoft.com/en-us/library/bb546158.aspx
Look at the Evaluator class.
So basically I have an application that works with just one user, but I'd like to make it multi-user. This would involve me logging a user in, and keeping the user ID in session, plus adding a user id column to my database tables. no biggy.
I'd like to change my dbml, or use it's partial, so that any query I throw through it also gets
.WHERE(user=>user.id=MYPARAM)
added to it, to return just records for the logged in user.
Is there a quick win to be had? A single place where I can specify this in the .dbml?
I would perhaps create a lambda expression dynamically.
The Where clause takes an object of type Expression>
You can then create an expression using the following.
Expression<Func<User,bool>> func = null;
Then
ParameterExpression xParam = Expression.Parameter(typeof(User), "u");
You would then create an expression for u.id = "test" by using a binary expression.
BinaryExpression uidTest = Expression.Equal(Expression.Property(xParam, "id"),
Expression.Constant("test");
and then attaching it to the Expression as follows:
func = Expression.Lambda<Func<User, bool>>(uidTest, xParam)
In effect this is building a lambda expression u=> u.id = "test"
The func object can then be used in the .Where as follows:
.Where(func)
Obviously, you can dynamically build this to any criteria you need at any time in your application.
I am not sure if there is something like that.
If there isn't, you can expose already filtered properties for all the tables in the partial class. You would then receive the user id as a parameter to the data context.
You can combine that with a dynamic expression so that those properties are as simple as possible to define.
If you are still having trouble implementing this functionality using LINQ to SQL, then you may want to solve the problem at the database level using row level security. Essentially you could wrap each table with a view which implements a dynamic predicate:
WHERE user_id = SUSER_SNAME()
This requires you to dynamically set each user's connection properties before a connection is made to the database (i.e. each database user has a specific SQL username and password). If this not a viable solution, you may want to consider building a framework around the L2S context so that you can inject your own logic. If you would like further inspiration, havea read here: http://www.west-wind.com/WebLog/posts/160237.aspx