We Keep Coding

Replace OrderBy with Where in LINQ query

There is a known issue with CosmosDb where if you use an ORDER BY clause it excludes documents that do not have this property defined
To work round this I'm trying to create functionality that takes a LINQ query and replaces the Order clause with a check for documents where the property is not defined so we can then run both queries and combine the results.
So:
ordersDb.Where(x => x.Name == customerName).OrderBy(x => x.CompanyName)
would become:
ordersDb.Where(x => x.Name == customerName)
.Where(x => !x.CompanyName.IsDefined()) // IsDefined is a built in CosmosDb function
Using Expression Builder I've created the following. However, I'm having issues trying to call my expression as a Where method - :
private sealed class OrderByToIsNotDefinedVisitor : ExpressionVisitor
{
protected override Expression VisitMethodCall(MethodCallExpression node)
{
if (node.Method.DeclaringType == typeof(Queryable) &&
(node.Method.Name == "OrderBy" || node.Method.Name == "OrderByDescending"))
{
// Get the IsDefined method
var methodIsDefined = typeof(TypeCheckFunctionsExtensions).GetMethod("IsDefined",
BindingFlags.Static | BindingFlags.NonPublic | BindingFlags.Public, null,
new Type[] { typeof(object) }, null);
// Apply the IsDefined method to the property that was being used for OrderBy
var isDefinedItem = Expression.Call(methodIsDefined, node.Arguments[1]);
// Alter the expression to check for !IsDefined()
var isNotDefinedItem = Expression.Not(isDefinedItem);
var entityType = node.Method.GetGenericArguments()[0];
var genericWhere = BuildGenericWhere();
var methodWhere = genericWhere.MakeGenericMethod(entityType);
var param = Expression.Parameter(entityType);
Expression newExpression =
Expression.Call(
methodWhere,
node.Arguments[0],
Expression.Lambda(
typeof(Func<,>).MakeGenericType(entityType, typeof(bool)),
isNotDefinedItem,
param));
return newExpression;
}
return base.VisitMethodCall(node);
}
}
private static MethodInfo BuildGenericWhere()
{
var genericWhereMethod = typeof(Enumerable).GetMethods(BindingFlags.Public | BindingFlags.Static)
.Where(x => x.Name == "Where" && x.GetGenericArguments().Length == 1)
.Select(x => new { Method = x, Parameters = x.GetParameters() })
.Where(x => x.Parameters.Length == 2 &&
x.Parameters[0].ParameterType.IsGenericType &&
x.Parameters[0].ParameterType.GetGenericTypeDefinition() == typeof(IEnumerable<>) &&
x.Parameters[1].ParameterType.IsGenericType &&
x.Parameters[1].ParameterType.GetGenericTypeDefinition() == typeof(Func<,>))
.Select(x => x.Method)
.Single();
return genericWhereMethod;
}
This compiles OK, but when I execute the query I get:
Microsoft.Azure.Documents.Linq.DocumentQueryException: Expression with
NodeType Lambda is not supported.
...as documentDb cannot handle the lambda clause
I've also tried replacing the lambda clause with a direct call to the where method, so the call becomes:
var updatedQueryExpression = Expression.Call(node.Arguments[0], methodWhere, isNotDefinedItem);
return updatedQueryExpression;
...however, this results in:
System.ArgumentException: Static method requires null instance,
non-static method requires non-null instance

Firstly,we are told that we can only sort with properties of document, not derived values.
So,for me,i also recommand you following the suggestion which is mentioned by #Paul in the comment: Querying all the matching data from cosmos db,then try to sort the result list and take the top elements.I believe that you already know the expression How to get first N elements of a list in C#?:
var firstFiveArrivals = myList.OrderBy(i => i.ArrivalTime).Take(5);
Since you have to get the top elements and if your matching data set is large enough, whether you use LINQ or SQL, you will encounter thresholds which could be solved by Continuation Token.

Construct expression for where through lambdas

The situation
I have a method that takes in a POCO. This POCO is like the following
private class SearchCriteria
{
public string name get;set;
public string age get;set;
public string talent get;set;
..
....
}
The method basically has a query to the db , that uses the above criteria.
public void query(SearchCriteria crit)
{
if(crit.name!=null && crit.age!=null && crit.talent !=null)
{
dbContext.Students.Where(c=>c.name ==crit.name && c.age==crit.age...)
}
else if(crit.name !=null && crit.age!=null)
{
}
else if(....
{
}
As you can see there is a definite problem above , where in, in case of large number of criteria, I will have to write a lot of if-elses to drop out specific arguments from the where clause .
The possible solution ?
I am actually new to the lambda expressions world but I believe we must be having a facility which would allow us to do something like below.
dbContext.Students.Where(processCriteria(searchCriteriaPOCO)).
Can you folks lead me to the proper direction ?. Thanks

Get a queryable and then keep adding where clauses to it. That way you only need to test each possible criteria the once and also only generate the number of where clauses that are absolutely needed.
IQueryable<Student> q = dbContext.Students.AsQueryable();
if (crit.name != null)
q = q.Where(c => c.name == crit.name);
if (crit.age != null)
q = q.Where(c => c.age== crit.age);

Let me start by saying that this answer uses the same basic idea as #PhilWright's answer. It just wraps it up in an extension method that applies this pattern for you, and allows you to have a syntax that reads nice.
public static class SearchExtensions
{
public static IQueryable<Student> Query(this SearchCriteria criteria, IQueryable<Student> studentQuery)
{
return studentQuery
.Match(criteria.name, (student) => student.name == criteria.name)
.Match(criteria.age, (student) => student.age == criteria.age)
.Match(criteria.talent, (student) => student.talent == criteria.talent);
// add expressions for other fields if needed.
}
private static IQueryable<Student> Match<T>(
this IQueryable<Student> studentQuery,
T criterionValue,
Expression<Func<Student, bool>> whereClause) where T : class
{
// only use the expression if the criterion value is non-null.
return criterionValue == null ? studentQuery : studentQuery.Where(whereClause);
}
}
You can then use it in your code like this:
var criteria = new SearchCriteria() {
name = "Alex",
talent = "Nosepicking"
};
var results = criteria.Query(dbContext.Students);

Maybe I'm missing something, as the code example is not the clearest I've seen, but for your specific example, I would think the following should be fine:
dbContext.Students.Where(c => (crit.name == null || crit.name == c.name) &&
(crit.age == null || crit.age == c.age) &&
(crit.talent == null || crit.talent == c.talent));
No need to chain a bunch of if statements.
For more complicated scenarios, you might prefer something like PredicateBuilder

You can use a pattern like this:
dbContext.Students.Where(c=>(crit.name == null || c.name ==crit.name) && ...)
A search criterion which is null will give a subexpression which is always true.

Filtering EF LINQ based on dynamic condition

Using Entity Framework 6, Repository pattern with domain model
I want to filter products (a model) based upon if the user has selected a currency (which is linked into products via a FK)
The filter would apply to a separate object (a table) for the currency
I was thinking of creating an IQueryable extension method, but as it's a custom type, and only in the Product object due to a FK, I don't think this would work
What I'd prefer to avoid (as I have quite a few methods that query Products)
public bool DoesUniqueHaveChildProducts(string productName, int brandId)
{
if (UserCurrency.userCurrency == -1)
{
//don't filter
return _db.Products.Any(x => x.BrandID == brandId && x.Name.Contains(productName));
}
return _db.Products.Any(x => x.BrandID == brandId && x.Name.Contains(productName) && x.MasterSite.CurrencyID == UserCurrency.userCurrency);
}
Looking for ideas & suggestions
Thanks!

This should work for you:
Func<Product, bool> predicate = x => x.BrandID == brandId && x.Name.Contains(productName);
if (UserCurrency.userCurrency != -1)
{
predicate = x => x.BrandID == brandId &&
x.Name.Contains(productName) &&
x.MasterSite.CurrencyID == UserCurrency.userCurrency;
}
return _db.Products.Any(predicate);

In each case Queryable<TModel>.Any takes an argument of type Expression<Func<TModel, bool>>.
So you could just put the condition in a local:
Expression<Func<TModel, bool>> pred;
if (whatever) {
pred = m => Condition(m)
} else {
pred = m => OtherCondition(m)
}
return db.Products.Any(pred);
which might be be cleaner (depends on the sitution).
It is also possible to build an Expression<whatever> dynamically, but like reflection that can be a lot of code... making use of the types in System.Linq.Expressions.

Custom function in Entity Framework query sometimes translates properly, sometimes doesn't

I have this function:
public static IQueryable<Article> WhereArticleIsLive(this IQueryable<Article> q)
{
return q.Where(x =>
x != null
&& DateTime.UtcNow >= x.PublishTime
&& x.IsPublished
&& !x.IsDeleted);
}
And it works just fine in this query:
from a in Articles.WhereArticleIsLive()
where a.Id == 5
select new { a.Title }
But it doesn't work in this only slightly more complex query:
from s in Series
from a in Articles.WhereArticleIsLive()
where s.Id == a.SeriesId
select new { s, a }
I get this error message:
NotSupportedException: LINQ to Entities does not recognize the method 'System.Linq.IQueryable1[TheFraser.Data.Articles.Article] WhereArticleIsLive(System.Linq.IQueryable1[TheFraser.Data.Articles.Article])' method, and this method cannot be translated into a store expression.
Any idea why? Is there another way to consolidate query parameters like this?
Thanks in advance.

EDIT: corrections by Craig.
I'm leaving this here, because I think it's a valuable tool: Use linqkit! But not for solving this question though :-)
Instead of returning IQueryable, use Expression to factor out predicates. E.g. you could define the following static method on Article:
public static Expression<Func<Article,bool>> IsLive()
{
return x =>
x != null
&& DateTime.UtcNow >= x.PublishTime
&& x.IsPublished
&& !x.IsDeleted
}
Then, ensure to store a reference to this expression when building your query, something along the lines of (not tested):
var isLive = Article.IsLive();
from s in Series
from a in Articles.Where(isLive)
where s.Id == a.SeriesId
select new { s, a }

How can I conditionally apply a Linq operator?

We're working on a Log Viewer. The use will have the option to filter by user, severity, etc. In the Sql days I'd add to the query string, but I want to do it with Linq. How can I conditionally add where-clauses?

if you want to only filter if certain criteria is passed, do something like this
var logs = from log in context.Logs
select log;
if (filterBySeverity)
logs = logs.Where(p => p.Severity == severity);
if (filterByUser)
logs = logs.Where(p => p.User == user);
Doing so this way will allow your Expression tree to be exactly what you want. That way the SQL created will be exactly what you need and nothing less.

If you need to filter base on a List / Array use the following:
public List<Data> GetData(List<string> Numbers, List<string> Letters)
{
if (Numbers == null)
Numbers = new List<string>();
if (Letters == null)
Letters = new List<string>();
var q = from d in database.table
where (Numbers.Count == 0 || Numbers.Contains(d.Number))
where (Letters.Count == 0 || Letters.Contains(d.Letter))
select new Data
{
Number = d.Number,
Letter = d.Letter,
};
return q.ToList();
}

I ended using an answer similar to Daren's, but with an IQueryable interface:
IQueryable<Log> matches = m_Locator.Logs;
// Users filter
if (usersFilter)
matches = matches.Where(l => l.UserName == comboBoxUsers.Text);
// Severity filter
if (severityFilter)
matches = matches.Where(l => l.Severity == comboBoxSeverity.Text);
Logs = (from log in matches
orderby log.EventTime descending
select log).ToList();
That builds up the query before hitting the database. The command won't run until .ToList() at the end.

I solved this with an extension method to allow LINQ to be conditionally enabled in the middle of a fluent expression. This removes the need to break up the expression with if statements.
.If() extension method:
public static IQueryable<TSource> If<TSource>(
this IQueryable<TSource> source,
bool condition,
Func<IQueryable<TSource>, IQueryable<TSource>> branch)
{
return condition ? branch(source) : source;
}
This allows you to do this:
return context.Logs
.If(filterBySeverity, q => q.Where(p => p.Severity == severity))
.If(filterByUser, q => q.Where(p => p.User == user))
.ToList();
Here's also an IEnumerable<T> version which will handle most other LINQ expressions:
public static IEnumerable<TSource> If<TSource>(
this IEnumerable<TSource> source,
bool condition,
Func<IEnumerable<TSource>, IEnumerable<TSource>> branch)
{
return condition ? branch(source) : source;
}

When it comes to conditional linq, I am very fond of the filters and pipes pattern.
http://blog.wekeroad.com/mvc-storefront/mvcstore-part-3/
Basically you create an extension method for each filter case that takes in the IQueryable and a parameter.
public static IQueryable<Type> HasID(this IQueryable<Type> query, long? id)
{
return id.HasValue ? query.Where(o => i.ID.Equals(id.Value)) : query;
}

Doing this:
bool lastNameSearch = true/false; // depending if they want to search by last name,
having this in the where statement:
where (lastNameSearch && name.LastNameSearch == "smith")
means that when the final query is created, if lastNameSearch is false the query will completely omit any SQL for the last name search.

Another option would be to use something like the PredicateBuilder discussed here.
It allows you to write code like the following:
var newKids = Product.ContainsInDescription ("BlackBerry", "iPhone");
var classics = Product.ContainsInDescription ("Nokia", "Ericsson")
.And (Product.IsSelling());
var query = from p in Data.Products.Where (newKids.Or (classics))
select p;
Note that I've only got this to work with Linq 2 SQL. EntityFramework does not implement Expression.Invoke, which is required for this method to work. I have a question regarding this issue here.

It isn't the prettiest thing but you can use a lambda expression and pass your conditions optionally. In TSQL I do a lot of the following to make parameters optional:
WHERE Field = #FieldVar OR #FieldVar IS NULL
You could duplicate the same style with a the following lambda (an example of checking authentication):
MyDataContext db = new MyDataContext();
void RunQuery(string param1, string param2, int? param3){
Func checkUser = user =>
((param1.Length > 0)? user.Param1 == param1 : 1 == 1) &&
((param2.Length > 0)? user.Param2 == param2 : 1 == 1) &&
((param3 != null)? user.Param3 == param3 : 1 == 1);
User foundUser = db.Users.SingleOrDefault(checkUser);
}

I had a similar requirement recently and eventually found this in he MSDN.
CSharp Samples for Visual Studio 2008
The classes included in the DynamicQuery sample of the download allow you to create dynamic queries at runtime in the following format:
var query =
db.Customers.
Where("City = #0 and Orders.Count >= #1", "London", 10).
OrderBy("CompanyName").
Select("new(CompanyName as Name, Phone)");
Using this you can build a query string dynamically at runtime and pass it into the Where() method:
string dynamicQueryString = "City = \"London\" and Order.Count >= 10";
var q = from c in db.Customers.Where(queryString, null)
orderby c.CompanyName
select c;

You can create and use this extension method
public static IQueryable<TSource> WhereIf<TSource>(this IQueryable<TSource> source, bool isToExecute, Expression<Func<TSource, bool>> predicate)
{
return isToExecute ? source.Where(predicate) : source;
}

Just use C#'s && operator:
var items = dc.Users.Where(l => l.Date == DateTime.Today && l.Severity == "Critical")
Edit: Ah, need to read more carefully. You wanted to know how to conditionally add additional clauses. In that case, I have no idea. :) What I'd probably do is just prepare several queries, and execute the right one, depending on what I ended up needing.

You could use an external method:
var results =
from rec in GetSomeRecs()
where ConditionalCheck(rec)
select rec;
...
bool ConditionalCheck( typeofRec input ) {
...
}
This would work, but can't be broken down into expression trees, which means Linq to SQL would run the check code against every record.
Alternatively:
var results =
from rec in GetSomeRecs()
where
(!filterBySeverity || rec.Severity == severity) &&
(!filterByUser|| rec.User == user)
select rec;
That might work in expression trees, meaning Linq to SQL would be optimised.

Well, what I thought was you could put the filter conditions into a generic list of Predicates:
var list = new List<string> { "me", "you", "meyou", "mow" };
var predicates = new List<Predicate<string>>();
predicates.Add(i => i.Contains("me"));
predicates.Add(i => i.EndsWith("w"));
var results = new List<string>();
foreach (var p in predicates)
results.AddRange(from i in list where p.Invoke(i) select i);
That results in a list containing "me", "meyou", and "mow".
You could optimize that by doing the foreach with the predicates in a totally different function that ORs all the predicates.