We Keep Coding

Prepare dynamic filters

We are in process of preparing dynamic filters which will be used for filtering records. We just should be able to select an entity (Lets say Person) then define filters according to that entity selected.
The design is something similar to this.
So how to design such a filtering system which can be used with any entity and any property?
We need to use
C#
MVC 5

I have created a dynamic filter feature using WPF, the way I did it was using LINQ To SQL ORM, using my DataContext I get all the tables and its corresponding fields (MetaDataMember) and used it build the UI
IEnumerable<MetaTable> mappedTables = DBContext.Mapping.GetTables();
IEnumerable<MetaDataMember> tablesColumns = mappedTables.SelectMany(t => t.RowType.DataMembers);
Then I build Expression Trees on the fly for each criteria the user created using the MetaDataMember and its property.
Type entityType = metaDataMember.DeclaringType.Type;
ParameterExpression parameterExpression = Expression.Parameter(entityType, "t"); // t => t; t is the parameterExpression representing the Table type
MemberExpression leftStringExpression = Expression.PropertyOrField(parameterExpression, metaDataMember.Name); // t => t.columnName
And then build the conditions i.e, Expression.LessThan (etc) and keep building the tree for each criteria using the conjunction expressions( Expression.AndAlso or Expression.OrElse) for each criteria and then finally create an lambda expression Expressions.Lambda and pass it to .Where() Linq Extension method to get my IQueryable<> filtered values back and use it to filter the UI.
IQueryable<SomeTable> results = DataContext.SomeTable.Where(whereLambdaExpression).Select(selectLambdaExpression);
The reason I went with this approach was because the query doesn't get executed until the enumeration (results.ToList()) occurs, and the query is executed in the DB server and not in the current application. LINQ converts the query expression to pure SQL statement and only the results are returned. It is pretty fast as everything happens in the DB Server.
I am not able to paste all my code as it pretty huge, but you get the gist of it.

I had come across this kind of problem and I solved it by compiling text as LINQ query and executing it on the data (collection) and collecting the result. It will not give you best performance as each time it needs to dynamically compile an assembly (I have used it with collection as big as 100K rows and works well). However, it is very flexible. You can perf tune it further. Code is pretty long so here is the github link.
Here is the main idea:
create a valid linq query expression as string
compile it as an assembly
load the assembly into memory and pass the input collection to the assembly
collect the output and use it.
The github link I shared has a simple working application with samples and projections.

Can Dynamic LINQ be made compatible with Entity Complex Types?

I want to dynamically query an object with System.Linq.Dynamic.
var selectData = (from i in data
select i).AsQueryable().Where("Name = #0","Bob1");//This works fine with a non-entity object
I know that we cannot project onto a mapped entity. I believe that is the reason this code fails
foreach (var item in rawQuery.ObsDataResultList)
{
var propertyData = (from i in item
select i).AsQueryable().Where("PropertyName = #0", "blah");
}//item is a Entity Complex Type
Error
Could not find an implementation of the query pattern for
source type 'ClassLibrary1.Model.bhcs_ObsData_2_Result'. 'Select' not
found.
Given the fact that I need to specify the PropertyName at runtime, I don't see any way to project with an anonymous type or a DTO.
I don't need to retain any of the Entity functionality at this point, I just need the data. Copying the data onto something that is queryable is a valid solution. So, is it possible to query entity framework with dynamic LINQ?
And here is the entity class header (the thing I'm trying to query, aka the item object)
[EdmComplexTypeAttribute(NamespaceName="MyDbModel", Name="blah_myQuery_2_Result")]
[DataContractAttribute(IsReference=true)]
[Serializable()]
public partial class blah_myQuery_2_Result : ComplexObject
{

First of all, let me clarify that System.Linq.Dynamic is not a full fledged Microsoft product. It is just a sample we release some time ago, and we don't thoroughly test different LINQ implementations to work correctly with it. If you are looking for a fully supported text-based query language for EF ObjectContext API you should take a look at Entity SQL instead.
Besides that, if you want to use System.Linq.Dynamic and you are ok with testing yourself that you don't hit anything that will block your application from working, then I'll try to see if I can help. I am going to need additional information since I am not sure I understand everything in your code snippets.
First of all I would like to understand, in your first example what is "data" and where did it come from? In your second snippet, what is "rawQuery" and where did it come from? Besdies, what is rawQuery.DataResultList and what is rawQuery.ObsDataResultList?
Also regarding your second snippet, it seems that you are trying to compose with query operators on top of an object that is not actually of a query type (although that doesn't explain the error you are getting given that you are calling AsQueryable the compiler should have complained before that bhcs_ObsData_2_Result is not an IEnumerable nor a non-generic IEnumerable).
In your propposed answer you are saying that you tried with ObjectResult and that seemed to help. Just be aware that ObjectResult is not a query object and therefore it won't allow you to build queries that get send to the server. In other words, any query operators that you apply to ObjectResult will be evaluated in memory and if you don't keep this in mind you may end up bringing all the data from that table into memory before you apply any filtering.

Query ObjectResult<blah_myQuery_2_Result> directly instead of the item blah_myQuery_2_Result. For example
var result = (from i in rawQuery.DataResultList
select i).AsQueryable().Where("CreatedDTM > #0", DateTime.Now.Subtract(new TimeSpan(30, 0, 0, 0)));

Send a linq query to server as an object

I have some linq query like this :
from x in from dbcontext
join y in dbcontext on x.id equals y.id select new {x,y}
my problem is how I can send these kinds of query to server as an object here is my linq query is direct data-access I need to send them to my data-server but I need an object. how I can implement this ?

Badly. Linq query contains references to the context. You don't have the context on the client and even if you have it will be different instance on the server. Doing anything like that would require damn big effort. Also you are returning projection to anonymous type - another very complicated feature when doing it remotely.
This can be of course handled - WCF Data Services already did this. They have client side "proxy" of context, query is serialized as OData query string and expression tree is built from that string on the server side. They also supports projections but they don't support STEs. Try WCF Data Services.
If you don't like the idea of WCF Data Services you should expose a method for each such query on the server side and client must call that method remotely.

firstly, IIRC you can't serialize Expressions, so firstly take a look at this post
then something along the lines of the following:
public List<ContactDetails> GetAllContactDetails( List<Guid> ContactIDs, Func<List<ContactDetails>, IEnumerable<ContactDetails>> filter )
{
List<ContactDetails> result = new List<ContactDetails>();
// get contacts as usual, fill result
// ...
return filter.Invoke( result ).ToList();
}
Usage:
GetAllContactDetails( idList, ( result ) =>
from contact in result
where contact.DateOfBirth > DateTime.Now.AddYears( -10 )
select contact );
D.R

Since Expression Trees are not serializable, it cannot be easily done.
My data access layer is not built with EF, so my solution may or may not be helpful to you, but here it is anyway:
I parse the expression on client side to a self-made serializable expression (which has properties like "columns", "joins", "filters", etc), and on the serverside I create sql based on these expressions.
You can create expressions from them of course, and send them against an EF linq provider!

Designing an OO and Unit Test Friendly Query System

I'm working on an application that allows dentists to capture information about certain clinical activities. While the application is not highly customizable (no custom workflows or forms) it does offer some rudimentary customization capabilities; clients can choose to augment the predefined form fields with their own custom ones. There are about half a dozen different field types that admins can create (i.e. Text, Date, Numeric, DropDown, etc). We're using Entity-Attribute-Value (EAV) on the persistence side to model this functionality.
One of the other key features of the application is the ability to create custom queries against these custom fields. This is accomplished via a UI in which any number of rules (Date <= (Now - 5 Days), Text Like '444', DropDown == 'ICU') can be created. All rules are AND'ed together to produce a query.
The current implementation (which I "inherited") is neither object oriented nor unit testable. Essentially, there is a single "God" class that compiles all the myriad rule types directly into a complex dynamic SQL statement (i.e. inner joins, outer joins, and subselects). This approach is troublesome for several reasons:
Unit testing individual rules in isolation
is nearly impossible
That last point also means adding additional rule types in the
future will most definitely violate
the Open Closed Principle.
Business logic and persistence concerns are being co-mingled.
Slow running unit tests since a real database is required (SQLLite can't parse T-SQL and mocking out a parser would be uhh...hard)
I'm trying to come up with a replacement design that is flexible, maintainable and testable, while still keeping query performance fairly snappy. This last point is key since I imagine an OOAD based implementation will move at least some of the data filtering logic from the database server to the (.NET) application server.
I'm considering a combination of the Command and Chain-of-Responsibility patterns:
The Query class contains a collection of abstract Rule classes (DateRule, TextRule, etc). and holds a reference to a DataSet class that contains an unfiltered set of data. DataSet is modeled in a persistence agnostic fashion (i.e no references or hooks into database types)
Rule has a single Filter() method which takes in an DataSet, filters it appropriately, and then returns it to the caller. The Query class than simply iterates over each Rule, allowing each Rule to filter the DataSet as it sees fit. Execution would stop once all rules have been executed or once the DataSet has been filtered down to nothing.
The one thing that worries me about this approach are the performance implications of parsing a potentially large unfiltered data set in .NET. Surely there are some tried and true approaches to solving just this kind of problem that offer a good balance between maintainability and performance?
One final note: management won't allow the use of NHibernate. Linq to SQL might be possible, but I'm not sure how applicable that technology would be to the task at hand.
Many thanks and I look forward to everyone's feedback!
Update: Still looking for a solution on this.

I think that LINQ to SQL would be an ideal solution coupled, perhaps, with Dynamic LINQ from the VS2008 samples. Using LINQ, particularly with extension methods on IEnumerable/IQueryable, you can build up your queries using your standard and custom logic depending on the inputs that you get. I use this technique heavily to implement filters on many of my MVC actions to great effect. Since it actually builds an expression tree then uses it to generate the SQL at the point where the query needs to be materialized, I think it would be ideal for your scenario since most of the heavy lifting is still done by the SQL server. In cases where LINQ proves to generate non-optimal queries you can always use table-valued functions or stored procedures added to your LINQ data context as methods to take advantage of optimized queries.
Updated: You might also try using PredicateBuilder from C# 3.0 in a Nutshell.
Example: find all Books where the Title contains one of a set of search terms and the publisher is O'Reilly.
var predicate = PredicateBuilder.True<Book>();
predicate = predicate.And( b => b.Publisher == "O'Reilly" );
var titlePredicate = PredicateBuilder.False<Book>();
foreach (var term in searchTerms)
{
titlePredicate = titlePredicate.Or( b => b.Title.Contains( term ) );
}
predicate = predicate.And( titlePredicate );
var books = dc.Book.Where( predicate );

The way I've seen it done is by creating objects that model each of the conditions you want the user to build their query from, and build up a tree of objects using those.
From the tree of objects you should be able to recursively build up an SQL statement that satisfies the query.
The basic ones you'll need will be AND and OR objects, as well as objects to model comparison, like EQUALS, LESSTHAN etc. You'll probably want to use an interface for these objects to make chaining them together in different ways easier.
A trivial example:
public interface IQueryItem
{
public String GenerateSQL();
}
public class AndQueryItem : IQueryItem
{
private IQueryItem _FirstItem;
private IQueryItem _SecondItem;
// Properties and the like
public String GenerateSQL()
{
StringBuilder builder = new StringBuilder();
builder.Append(_FirstItem.GenerateSQL());
builder.Append(" AND ");
builder.Append(_SecondItem.GenerateSQL());
return builder.ToString();
}
}
Implementing it this way should allow you to Unit Test the rules pretty easily.
On the negative side, this solution still leaves the database to do a lot of the work, which it sounds like you don't really want to do.

Is it possible to use/access scalar functions with LINQ to SQL?

We have scalar functions in our database for returning things like "number of tasks for a customer" or "total invoice amount for a customer".
We are experimenting and looking to try to do this w/o stored procedures ... normally we would just call this function in our stored procedure and return it as a single value.
Is there a way to use or access scalar functions with LINQ to SQL? If so, I would be interested in see an example of how to ... if not, how would it be best to handle this type of situation ... if it is even doable.

LINQ-to-SQL supports use with UDFs, if that is what you mean. Just drag the UDF onto the designer surface and you're done. This creates a matching method on the data-context, marked [Function(..., IsComposable=true)] or similar, telling LINQ-to-SQL that it can use this in queries (note that EF doesn't support this usage).
You would then use it in your query like:
var qry = from cust in ctx.Custs
select new {Id = cust.Id, Value = ctx.GetTotalValue(cust.Id)};
which will become TSQL something like:
SELECT t1.Id, dbo.MyUdf(t1.Id)
FROM CUSTOMER t1
(or there-abouts).
The fact that it is composable means that you can use the value in queries - for example in a Where()/WHERE - and so reduce the data brought back from the server (although obviously the UDF will still need to be executed in some way).
Here's a similar example, showing a pseudo-UDF at use on a data-context, illustrating that the C# version of the method is not used.
Actually, I'm currently looking at such UDFs to provide "out of model" data in a composable way - i.e. a particular part of the system needs access to some data (that happens to be in the same database) that isn't really part of the same model, but which I want to JOIN in interesting ways. I also have existing SPs for this purpose... so I'm looking at porting those SPs to tabular UDFs, which provides a level of contract/abstraction surrounding the out-of-model data. So because it isn't part of my model, I can only get it via the UDF - yet I retain the ability to compose this with my regular model.

I believe this MSDN documentation is what you're after (as part of this wider topic of calling user-defined functions in LINQ to SQL). Can't say I've done it myself, but it sounds right...