We Keep Coding

Linq query takes too long when using Func & OrderByDescending

Consider this code:
public List<Clients> GetFilteredClients(DateTime? FromDate = null,
DateTime? ToDate = null,
int? fromLocationType = null,
int? toLocationType = null)
{
Func<Clients, bool> fromDateFilter = f => true;
if (FromDate.HasValue)
{
fromDateFilter = z => z.Insert_Date.Value.Date >= FromDate.Value.Date;
}
Func<Clients, bool> toDateFilter = f => true;
if (ToDate.HasValue)
{
toDateFilter = z => z.Insert_Date.Value.Date <= ToDate.Value.Date;
}
Func<Clients, bool> fromLocationTypeFilter = f => true;
if (fromLocationType.HasValue)
{
fromOrgFilter = z => z.LocationTypeId >= fromLocationType.Value;
}
Func<Clients, bool> toLocationTypeFilter = f => true;
if (toLocationType.HasValue)
{
toLocationTypeFilter = z => z.LocationTypeId <= toLocationType.Value;
}
var filtered = DB_Context.Clients
.Where(fromDateFilter)
.Where(toDateFilter)
.Where(fromLocationTypeFilter)
.Where(toLocationTypeFilter)
.OrderByDescending(k => k.Id)
.Take(1000)
.ToList();
return filtered;
}
I have something like 100K records in the DB, I need only the top 1000 that answer to the requirements of:
.Where(fromDateFilter)
.Where(toDateFilter)
.Where(fromLocationTypeFilter)
.Where(toLocationTypeFilter)
However the execution time still takes something like 10 seconds.
Any idea why?

You must use Expression<Func<...>> rather than Func<...>. When you use Func, only the enumerable methods can be used on the queryable, which in this case means you first download everything to memory, and then do the filtering. If you switch over to Expression<...>, the O/RM will do the filtering on the DB server, rather than in your application.
Also, there's better ways to do what you're doing. For example, you can build the conditions like so:
var query = DB_Context.Clients.AsQueryable();
if (FromDate.HasValue) query = query.Where(...);
if (ToDate.HasValue) query = query.Where(...);
...
return query.OrderByDescending(k => k.Id).Take(1000).ToList();
Of course, this means that whatever DB provider you're using must be able to support the kind of filtering you're trying to do - you'll need to consult the documentation.

You are using delegates instead LINQ expressions. That leads to processing a data by your application and not by SQL Server.
LINQ expressions look like lambda expressions thanks for the syntax, but they are not same thing. The compiler takes a decision what to create (delegates or LINQ expressions) depending on the situation.
If an object implements the IQueriable interface, then the compiler uses the Queryable class and generates LINQ expression trees, which later can be translated into a SQL query or other form by the specific IQueryProvider.
Otherwise, the compiler uses extensions from the Enumerable class, which create iterators over source collection (all records from the table in your case).
As an example. The code bellow will be compilled into LINQ expressions.
// Source code
IQueryable<Clients> source = null;
IQueryable<Clients> result = source.Where(c => c.LocationTypeId >= 1);
// Compiller generated code
IQueryable<Clients> source = null;
Expression parameterC = Expression.Parameter(typeof(Clients), "c");
IQueryable<Clients> result = Queryable.Where<Clients>(
source,
Expression.Lambda<Func<Clients, bool>>(
Expression.LessThanOrEqual(
Expression.Property(
parameterC ,
typeof(Clients).GetProperty("LocationTypeId").GetGetMethod()
),
Expression.Constant(1, typeof(int))
),
new ParameterExpression[]
{
parameterC
}
);
And this code uses delegates:
// Source code
IQueryable<Clients> source = null;
Func<Clients, bool> filter = c => c.LocationTypeId >= 1;
IEnumerable<Clients> result = source.Where(filter );
// Compiller generated code
IQueryable<Clients> source = null;
Func<Clients, bool> filter = c => c.LocationTypeId >= 1;
IEnumerable<Clients> result = Enumerable.Where(source, filter);
So, to solve you problem use Expression<Func<Clients, bool>> instead of Func<Clients, bool>:
IQueryable<Clients> result = DB_Context.Clients;
if (someFilter.HasValue)
result = result.Where(c => c.SomeProperty == someFilter.Value);
// other filters
return query
.OrderByDescending(k => k.Id)
.Take(1000)
.ToList();

Using custom methods in linq to entities

I have a Person table in my database that has NationalId field.
Is there any way to load all Persons with even NationalId, using Ef code first and Linq to entities, without loading all Persons to memory?
Somethings like:
public bool IsEven(int number)
{
return number % 2 == 0;
}
var context = new MyContext();
var personsWithEvenNationalId = context.Persons
.Where(x=> IsEven(x.NationalId))
.ToList();

You would have to do your check inline
var personsWithEvenNationalId = context.Persons
.Where(x=> x.NationalId%2 == 0)
.ToList();
Linq to Entities doesn't know how to translate your custom method into SQL basically. If you do need to use a custom method you would have to get Persons as enumerable and then use your custom method, i.e.
var personsWithEvenNationalId = context.Persons
.AsEnumerable()
.Where(x=> IsEven(x.NationalId))
.ToList();
But that's hardly ideal as it would load all Persons and then filter on IsEven
Edit: Thinking about it you could also create an extension method for IQueryable<Person> if you don't want to have to write it inline every time. Something like this where you build an Expression
public static IQueryable<Person> WhereEven(this IQueryable<Person> source, Expression<Func<Person, int>> property)
{
var expression = Expression.Equal(
Expression.Modulo(
property.Body,
Expression.Constant(2)),
Expression.Constant(0));
var methodCallExpression = Expression.Call(typeof (Queryable),
"where",
new Type[] {source.ElementType},
source.Expression,
Expression.Lambda<Func<Person, bool>>(expression, property.Parameters));
return source.Provider.CreateQuery<Person>(methodCallExpression);
}
And to use it:
context.Persons.WhereEven(x => x.NationalId).ToList();

Rather than a function that does what you want, you'll need to have an function (or property, or field) that provides an Expression that does the projection that you want:
public static Expression<Func<int, bool>> IsEven()
{
return number => number % 2 == 0;
}
You can now write:
using(var context = new MyContext())
{
var personsWithEvenNationalId = context.Persons
.Select(x=> x.NationalId)
.Where(IsEven())
.ToList();
}

Generate LINQ or Lambda Expression for Multiple columns based on the user selection of checkboxes

I have a few checkboxes on top of my webpage which correspond to the columns of my Table. Like there is a column studentId then there will be a checkbox studentId and so on. I want to write such a linq/lamda expression for List which will filter the on the basis of the checkboxes selected. For example if selects studentId and studentType checkbox then linq/lambda expression should bring all the rows matching the selection.
example:
If studentId and studentType checked then:
foreach (Child c in SomeList)
{
if (chkStudentId.checked)
{
List.FindAll (h=> h.StudentId == c.studentId);
}
if (chkStudentType.checked)
{
List.FindAll (h => h.StudentType == c.studentType)
}
}
}
I can't figure out how am I going to write such a code that if user selects multiple checkboxes, the query should compare to all the columns and bring the values based only on the checkboxes checked. The above is only static and does not help. please help. Thanks.

Expression trees are a great help if you want your query to be totally dynamic. But if the number of checkboxes is static you can also choose for the following solution:
var students = <your list of students>.AsQueryable();
if ( chkStudentId.checked)
{
students = students.Where(s => s.StudentId == c.StudentId);
}
if (chkStudentType.checked))
{
students = students.Where(s => s.StudentType== h.StudentType);
}
In such a way you can combine the where clauses in a dynamic way.

Disclaimer: I'm fairly new to this and there is probably much better ways to do this. Any feedback is higlhy appreciated.
Note that this method has no error/null checking. Since it uses reflection, you should profile it if you plan to use it in production.
public static IEnumerable<T> Filter<T>(IEnumerable<T> collection, Dictionary<string, object> filters) {
var type = typeof (T);
var properties = type.GetProperties(BindingFlags.Public | BindingFlags.Instance);
var queryable = collection.AsQueryable();
var instance = Expression.Parameter(type, "instance");
var expressions = new Stack<Expression>();
foreach (var filter in filters) {
var propertyName = filter.Key;
var property = properties.FirstOrDefault(x => x.Name == propertyName);
if (property == null)
continue;
var left = Expression.Property(instance, property);
var right = Expression.Constant(filter.Value, property.PropertyType);
var expr = Expression.Equal(left, right);
expressions.Push(expr);
}
Expression call = null;
Expression previousExpression = null;
while(expressions.Count > 0) {
var expr = expressions.Pop();
if(previousExpression == null) {
previousExpression = expr;
call = expr;
} else {
var and = Expression.AndAlso(previousExpression, expr);
call = and;
previousExpression = and;
}
}
var whereCallExpression = Expression.Call(
typeof(Queryable),
"Where",
new[] { queryable.ElementType },
queryable.Expression,
Expression.Lambda<Func<T, bool>>(call, new[] { instance }));
return queryable.Provider.CreateQuery<T>(whereCallExpression);
}
It overates all filters and tries to find a matching property. If it does find a property, it creates a EqualExpression which compares the actual value and the value you want to filter by. It then creates a MethodCallExpression which is passed to the query provider.
Theese expressions are then combined. I think the part with the stack is wrong, and that there is a better way to do it.
Usage:
var persons = new List<Person> {new Person {Name = "Alex", Age = 22}, new Person {Name = "Jesper", Age = 30}};
var filters = new Dictionary<string, object>();
filters.Add("Name", "Alexander Nyquist");
var results = Filter(persons, filters);
Since it's building expressions, it does works with Linq 2 Sql (tested) and probably Entity Framework. Linq 2 sql produces the following query:
SELECT [t0].[Id], [t0].[Name], [t0].[Email]
FROM [dbo].[Persons] AS [t0]
WHERE [t0].[Name] = #p0
-- #p0: Input VarChar (Size = 8000; Prec = 0; Scale = 0) [Alexander Nyquist]
-- Context: SqlProvider(Sql2008) Model: AttributedMetaModel Build: 4.0.30319.1
Hope this helps.

From your description of your problem it seemed fairly straight forward that you have a list of predicates, that are enabled or disable based on the check boxes, and then you want to aggregate them to create a single composite filter. Easy!
Your predicates look like this:
h => h.StudentId == c.studentId
h => h.StudentType == c.studentType
But because of the check boxes, you really want them to look like this:
h => chkStudentId.Checked ? h.StudentId == c.studentId : true
h => chkStudentId.Checked ? h.StudentType == c.studentType : true
You're effectively extending the predicates to include the check box. Here's a function that does that:
Func<CheckBox, Func<Student, bool>, Func<Student, bool>> extend =
(cb, p) =>
s => cb.Checked ? p(s) : true;
Now you can write your list of predicates like this:
var predicates = new Func<Student, bool>[]
{
extend(chkStudentId, h => h.StudentId == c.studentId),
extend(chkStudentType, h => h.StudentType == c.studentType),
// etc
};
Next LINQ has an easy way to turn a list of somethings into a single something:
Func<Student, bool>
predicate =
predicates.Aggregate((a, p) => s => a(s) && p(s));
Now you just have to execute this code to get your filtered list:
var filtered = SomeList.Where(predicate);
Now anytime any of the check boxes change you just need to enumerate the filtered collection and you'll get your filtered results.

Expression Trees can do what you want, but they are rather complicated. They allow you to dynamically build the linq query.
http://msdn.microsoft.com/en-us/library/bb882637.aspx
http://msdn.microsoft.com/en-us/library/bb397951.aspx
The Dynamic Linq Library may also be of use. I haven't personally used it though.
http://weblogs.asp.net/scottgu/archive/2008/01/07/dynamic-linq-part-1-using-the-linq-dynamic-query-library.aspx

The solution that worked for me was a bit modification of what Wouter de Kort
suggest and was like this:
var students = StudentList.AsQueryable();
foreach (Student sc in GroupingList)
{
students = students.Where(s => (chkStudentId.Checked? h.StudentId.Trim() == sc.StudentId.Trim() : true) && (chkStudentType.Checked? h.StudentType.Trim() == sc.StudentType.Trim() : true) );
}

How to refactor multiple similar Linq-To-Sql queries?

Suppose I have the two following Linq-To-SQL queries I want to refactor:
var someValue1 = 0;
var someValue2= 0;
var query1 = db.TableAs.Where( a => a.TableBs.Count() > someValue1 )
.Take( 10 );
var query2 = db.TableAs.Where( a => a.TableBs.First().item1 == someValue2)
.Take( 10 );
Note that only the Where parameter changes. There is any way to put the query inside a method and pass the Where parameter as an argument?
All the solutions posted in the previous question have been tried and failed in runtime when I try to enumerate the result.
The exception thrown was: "Unsupported overload used for query operator 'Where'"

Absolutely. You'd write:
public IQueryable<A> First10(Expression<Func<A,bool>> predicate)
{
return db.TableAs.Where(predicate).Take(10);
}
(That's assuming that TableA is IQueryable<A>.)
Call it with:
var someValue1 = 0;
var someValue2= 0;
var query1 = First10(a => a.TableBs.Count() > someValue1);
var query2 = First10(a => a.TableBs.First().item1 == someValue2);
I believe that will work...
The difference between this and the answers to your previous question is basically that this method takes Expression<Func<T,bool>> instead of just Func<T,bool> so it ends up using Queryable.Where instead of Enumerable.Where.

If you really want reusability you can try to write your own operators. E.g. instead of repeatedly writing:
var query =
Products
.Where(p => p.Description.Contains(description))
.Where(p => p.Discontinued == discontinued);
you can write simple methods:
public static IEnumerable<Product> ByName(this IEnumerable<Product> products, string description)
{
return products.Where(p => p.Description.Contains(description));
}
public static IEnumerable<Product> AreDiscontinued(IEnumerable<Product> products, bool isDiscontinued)
{
return products.Where(p => p.Discontinued == discontinued);
}
and then use it like this:
var query = Products.ByName("widget").AreDiscontinued(false);

How can I conditionally apply a Linq operator?

We're working on a Log Viewer. The use will have the option to filter by user, severity, etc. In the Sql days I'd add to the query string, but I want to do it with Linq. How can I conditionally add where-clauses?

if you want to only filter if certain criteria is passed, do something like this
var logs = from log in context.Logs
select log;
if (filterBySeverity)
logs = logs.Where(p => p.Severity == severity);
if (filterByUser)
logs = logs.Where(p => p.User == user);
Doing so this way will allow your Expression tree to be exactly what you want. That way the SQL created will be exactly what you need and nothing less.

If you need to filter base on a List / Array use the following:
public List<Data> GetData(List<string> Numbers, List<string> Letters)
{
if (Numbers == null)
Numbers = new List<string>();
if (Letters == null)
Letters = new List<string>();
var q = from d in database.table
where (Numbers.Count == 0 || Numbers.Contains(d.Number))
where (Letters.Count == 0 || Letters.Contains(d.Letter))
select new Data
{
Number = d.Number,
Letter = d.Letter,
};
return q.ToList();
}

I ended using an answer similar to Daren's, but with an IQueryable interface:
IQueryable<Log> matches = m_Locator.Logs;
// Users filter
if (usersFilter)
matches = matches.Where(l => l.UserName == comboBoxUsers.Text);
// Severity filter
if (severityFilter)
matches = matches.Where(l => l.Severity == comboBoxSeverity.Text);
Logs = (from log in matches
orderby log.EventTime descending
select log).ToList();
That builds up the query before hitting the database. The command won't run until .ToList() at the end.

I solved this with an extension method to allow LINQ to be conditionally enabled in the middle of a fluent expression. This removes the need to break up the expression with if statements.
.If() extension method:
public static IQueryable<TSource> If<TSource>(
this IQueryable<TSource> source,
bool condition,
Func<IQueryable<TSource>, IQueryable<TSource>> branch)
{
return condition ? branch(source) : source;
}
This allows you to do this:
return context.Logs
.If(filterBySeverity, q => q.Where(p => p.Severity == severity))
.If(filterByUser, q => q.Where(p => p.User == user))
.ToList();
Here's also an IEnumerable<T> version which will handle most other LINQ expressions:
public static IEnumerable<TSource> If<TSource>(
this IEnumerable<TSource> source,
bool condition,
Func<IEnumerable<TSource>, IEnumerable<TSource>> branch)
{
return condition ? branch(source) : source;
}

When it comes to conditional linq, I am very fond of the filters and pipes pattern.
http://blog.wekeroad.com/mvc-storefront/mvcstore-part-3/
Basically you create an extension method for each filter case that takes in the IQueryable and a parameter.
public static IQueryable<Type> HasID(this IQueryable<Type> query, long? id)
{
return id.HasValue ? query.Where(o => i.ID.Equals(id.Value)) : query;
}

Doing this:
bool lastNameSearch = true/false; // depending if they want to search by last name,
having this in the where statement:
where (lastNameSearch && name.LastNameSearch == "smith")
means that when the final query is created, if lastNameSearch is false the query will completely omit any SQL for the last name search.

Another option would be to use something like the PredicateBuilder discussed here.
It allows you to write code like the following:
var newKids = Product.ContainsInDescription ("BlackBerry", "iPhone");
var classics = Product.ContainsInDescription ("Nokia", "Ericsson")
.And (Product.IsSelling());
var query = from p in Data.Products.Where (newKids.Or (classics))
select p;
Note that I've only got this to work with Linq 2 SQL. EntityFramework does not implement Expression.Invoke, which is required for this method to work. I have a question regarding this issue here.

It isn't the prettiest thing but you can use a lambda expression and pass your conditions optionally. In TSQL I do a lot of the following to make parameters optional:
WHERE Field = #FieldVar OR #FieldVar IS NULL
You could duplicate the same style with a the following lambda (an example of checking authentication):
MyDataContext db = new MyDataContext();
void RunQuery(string param1, string param2, int? param3){
Func checkUser = user =>
((param1.Length > 0)? user.Param1 == param1 : 1 == 1) &&
((param2.Length > 0)? user.Param2 == param2 : 1 == 1) &&
((param3 != null)? user.Param3 == param3 : 1 == 1);
User foundUser = db.Users.SingleOrDefault(checkUser);
}

I had a similar requirement recently and eventually found this in he MSDN.
CSharp Samples for Visual Studio 2008
The classes included in the DynamicQuery sample of the download allow you to create dynamic queries at runtime in the following format:
var query =
db.Customers.
Where("City = #0 and Orders.Count >= #1", "London", 10).
OrderBy("CompanyName").
Select("new(CompanyName as Name, Phone)");
Using this you can build a query string dynamically at runtime and pass it into the Where() method:
string dynamicQueryString = "City = \"London\" and Order.Count >= 10";
var q = from c in db.Customers.Where(queryString, null)
orderby c.CompanyName
select c;

You can create and use this extension method
public static IQueryable<TSource> WhereIf<TSource>(this IQueryable<TSource> source, bool isToExecute, Expression<Func<TSource, bool>> predicate)
{
return isToExecute ? source.Where(predicate) : source;
}

Just use C#'s && operator:
var items = dc.Users.Where(l => l.Date == DateTime.Today && l.Severity == "Critical")
Edit: Ah, need to read more carefully. You wanted to know how to conditionally add additional clauses. In that case, I have no idea. :) What I'd probably do is just prepare several queries, and execute the right one, depending on what I ended up needing.

You could use an external method:
var results =
from rec in GetSomeRecs()
where ConditionalCheck(rec)
select rec;
...
bool ConditionalCheck( typeofRec input ) {
...
}
This would work, but can't be broken down into expression trees, which means Linq to SQL would run the check code against every record.
Alternatively:
var results =
from rec in GetSomeRecs()
where
(!filterBySeverity || rec.Severity == severity) &&
(!filterByUser|| rec.User == user)
select rec;
That might work in expression trees, meaning Linq to SQL would be optimised.

Well, what I thought was you could put the filter conditions into a generic list of Predicates:
var list = new List<string> { "me", "you", "meyou", "mow" };
var predicates = new List<Predicate<string>>();
predicates.Add(i => i.Contains("me"));
predicates.Add(i => i.EndsWith("w"));
var results = new List<string>();
foreach (var p in predicates)
results.AddRange(from i in list where p.Invoke(i) select i);
That results in a list containing "me", "meyou", and "mow".
You could optimize that by doing the foreach with the predicates in a totally different function that ORs all the predicates.