Let's say I have a list of ids passed by some other part in my program. I need to perform a linq-to-sql query on the DB (to lookup some other data). I want to do it based on the ids that identify my items in scope.
If id list was small enough, I would use "contains" in the linq expression. But it isn't. The problem is that the resulting SQL uses a "where in (id1,id2...)" clause which is getting too large.
My question is: how could I avoid that?
Of course I don't want an in-memory query, that would be easy - I want to let the DB do the job for performane reasons :)
Update: An example linq:
//this is passed in my program from somewhere
int[] myList = new int[]{0,1,2,3,4}; is a list of ids.
//this linq-to-sql will end up in a "where in" clause
myDataTable.Where(a => myList.Contains(a.ID));
We use this extension:
public static IQueryable<T> WhereValueIn<T>(this IQueryable<T> query,
Expression<Func<T, long>> propertyAccessor, ICollection<long> valuesList)
{
return query.Where(PrepareContainsPredicate(propertyAccessor, valuesList));
}
private static Expression<Func<T1, T3>> Combine<T1, T2, T3>(
Expression<Func<T1, T2>> first,
Expression<Func<T2, T3>> second)
{
var param = Expression.Parameter(typeof(T1), #"param");
var newFirst = new ReplaceVisitor(first.Parameters.First(), param).Visit(first.Body);
var newSecond = new ReplaceVisitor(second.Parameters.First(), newFirst).Visit(second.Body);
return Expression.Lambda<Func<T1, T3>>(newSecond, param);
}
public static Expression<Func<T, bool>> PrepareContainsPredicate<T>(Expression<Func<T, long>> idAccessor, ICollection<long> entityIds)
{
if (entityIds.Count == 1)
{
var cardId = entityIds.First();
return Combine(idAccessor, x => x == cardId);
}
if (entityIds.Count == 2)
{
var card1Id = entityIds.ElementAt(0);
var card2Id = entityIds.ElementAt(1);
return Combine(idAccessor, x => x == card1Id || x == card2Id);
}
if (entityIds.Count == 3)
{
var card1Id = entityIds.ElementAt(0);
var card2Id = entityIds.ElementAt(1);
var card3Id = entityIds.ElementAt(2);
return Combine(idAccessor, x => x == card1Id || x == card2Id || x == card3Id);
}
return Combine(idAccessor, x => entityIds.Contains(x));
}
private sealed class ReplaceVisitor : ExpressionVisitor
{
private readonly Expression _from;
private readonly Expression _to;
public ReplaceVisitor(Expression from, Expression to)
{
_from = from;
_to = to;
}
public override Expression Visit(Expression node)
{
return node == _from ? _to : base.Visit(node);
}
}
And call this:
var ids = new long[] {1L, 2L};
var result = query.WhereValueIn(o => o.Id, ids);
So, if you have small collection, LINQ will generate code with Id = 1 OR Id = 2, other way it will generate IN statement
Related
I'm trying to do a LinQ request with a group by where a parameter is parametrizable by an Expression ( (Expression<Func<CompanyModel,TKey>> myGroupingProperty) and the other one is hard coded. But even if my code compile I get an error that linq does not support lambdas. Would you have any idea how to do this request?
Here is the code sample:
public List<timelineResult> getTimelinebyCompany<TKey>(Expression<Func<CompanyModel,TKey>> myGroupingProperty, Filter item)
{
int cntToBeSureThatTheQueryExecuteAtLeastOneTime = 0;
List<timelineResult> toto = new List<timelineResult>();
using (var db = new fintechDbContext())
{
while (cntToBeSureThatTheQueryExecuteAtLeastOneTime == 0)
{
toto = (from p in db.companyDBSET
select p).GroupBy(p=> new {p.Founded_Year, myGroupingProperty})
.Select(o => new timelineResult{ year = o.Key.Founded_Year, cluster = o.myGroupingProperty.ToString(), count = o.Count() })
.OrderBy(o => o.year).ToList();
cntToBeSureThatTheQueryExecuteAtLeastOneTime++;
}
}
return toto;
}
What you are seeking for is doable, but not the way you tried because the passed lambda expression cannot be used directly inside another lambda expression.
You should start first by creating a generic class to hold the grouping key. It's similar to system provided Tuple, but has parameterless constructor and simple property get/setters to conform to the EF projection rules:
public class GroupKey<K1, K2>
{
public K1 Key1 { get; set; }
public K2 Key2 { get; set; }
}
Then you need to build dynamically lambda expression like this
Expression<Func<T, K1, K2>> keySelector = x =>
new GroupKey<K1, K2> { Key1 = x.Prop1, Key2 = x.Prop2 };
In order to do that, you'll need some Expression helpers:
public static class ExpressionUtils
{
public static Expression ReplaceParameter(this Expression expression, ParameterExpression source, Expression target)
{
return new ParameterReplacer { Source = source, Target = target }.Visit(expression);
}
class ParameterReplacer : ExpressionVisitor
{
public ParameterExpression Source;
public Expression Target;
protected override Expression VisitParameter(ParameterExpression node)
{
return node == Source ? Target : base.VisitParameter(node);
}
}
}
and you can encapsulate the grouping part in a custom extension method:
public static class QueryableExtensions
{
public static IQueryable<IGrouping<GroupKey<K1, K2>, T>> GroupByPair<T, K1, K2>(this IQueryable<T> source, Expression<Func<T, K1>> keySelector1, Expression<Func<T, K2>> keySelector2)
{
var parameter = keySelector1.Parameters[0];
var key1 = keySelector1.Body;
var key2 = keySelector2.Body.ReplaceParameter(keySelector2.Parameters[0], parameter);
var keyType = typeof(GroupKey<K1, K2>);
var keySelector = Expression.Lambda<Func<T, GroupKey<K1, K2>>>(
Expression.MemberInit(
Expression.New(keyType),
Expression.Bind(keyType.GetProperty("Key1"), key1),
Expression.Bind(keyType.GetProperty("Key2"), key2)),
parameter);
return source.GroupBy(keySelector);
}
}
Finally, the essential part of your method becomes like this:
toto = db.companyDBSET
.GroupByPair(p => p.Founded_Year, myGroupingProperty)
.Select(g => new timelineResult
{
year = g.Key.Key1,
cluster = g.Key.Key2.ToString(),
count = g.Count()
})
.OrderBy(o => o.year)
.ToList();
public List<DbSet> Get(String q = null)
{
List<DbSet> objs = new List<DbSet>();
if (!string.IsNullOrEmpty(q) && q != "undefined")
{
objs = from dealer in db.Dealers
where dealer.Contains
......(?????)
}
}
I have some 4 DbSets in my dbcontext class. I am able to run a search like this
objs = from dealer in db.Dealers
where dealer.Name.Contains(q)
However I want to be able to do something similar except do the search against all the fields in dealer, and not just name. All the fields are strings
EDIT
Okay so I am starting to think this is not the best way to do what I'm trying to achieve. I'm looking into something called "Full Text Search". Can someone either explain to me how this works in entity or give me a link to a good resource
You could write a linq extension method:
Checkout my blog post.
http://jnye.co/Posts/7/generic-iqueryable-or-search-on-multiple-properties-using-expression-trees
(classes are also in github: https://github.com/ninjanye/SearchExtensions)
public static IQueryable<T> Search<T>(this IQueryable<T> source, string searchTerm, params Expression<Func<T, string>>[] stringProperties)
{
if (String.IsNullOrEmpty(searchTerm))
{
return source;
}
var searchTermExpression = Expression.Constant(searchTerm);
//Variable to hold merged 'OR' expression
Expression orExpression = null;
//Retrieve first parameter to use accross all expressions
var singleParameter = stringProperties[0].Parameters.Single();
//Build a contains expression for each property
foreach (var stringProperty in stringProperties)
{
//Syncronise single parameter accross each property
var swappedParamExpression = SwapExpressionVisitor.Swap(stringProperty, stringProperty.Parameters.Single(), singleParameter);
//Build expression to represent x.[propertyX].Contains(searchTerm)
var containsExpression = BuildContainsExpression(swappedParamExpression, searchTermExpression);
orExpression = BuildOrExpression(orExpression, containsExpression);
}
var completeExpression = Expression.Lambda<Func<T, bool>>(orExpression, singleParameter);
return source.Where(completeExpression);
}
private static Expression BuildOrExpression(Expression existingExpression, Expression expressionToAdd)
{
if (existingExpression == null)
{
return expressionToAdd;
}
//Build 'OR' expression for each property
return Expression.OrElse(existingExpression, expressionToAdd);
}
private static MethodCallExpression BuildContainsExpression<T>(Expression<Func<T, string>> stringProperty, ConstantExpression searchTermExpression)
{
return Expression.Call(stringProperty.Body, typeof(string).GetMethod("Contains"), searchTermExpression);
}
You will also need this class:
//Create SwapVisitor to merge the parameters from each property expression into one
public class SwapVisitor : ExpressionVisitor
{
private readonly Expression from, to;
public SwapVisitor(Expression from, Expression to)
{
this.from = from;
this.to = to;
}
public override Expression Visit(Expression node)
{
return node == from ? to : base.Visit(node);
}
public static Expression Swap(Expression body, Expression from, Expression to)
{
return new SwapVisitor(from, to).Visit(body);
}
}
You can then write something like:
db.Dealers.Search(q, x => x.Field1,
x => x.Field2,
...
x => x.Field20)
Sorry, no short cuts here:
objs = from dealer in db.Dealers
where dealer.Name.Contains(q) ||
dealer.Field2.Contains(q) ||
...
dealer.Field20.Contains(q)
select dealer;
You have to specify which fields you are going to search the value in.
You can manually write conditions for all fields:
objs = from dealer in db.Dealers
where dealer.Name.Contains(q) ||
dealer.Foo.Contains(q) ||
// etc
dealer.Bar.Contains(q)
select dealer;
There is no simple way to tell Entity Framework to check all properties of entity for some condition.
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
LINQ To SQL exception: Local sequence cannot be used in LINQ to SQL implementation of query operators except the Contains operator
I am trying the following query:
var data = (from bk in DataContext.Book
where ((searchArray.Count() == 0 || searchArray.ToList().Any(x => bk.Name.Contains(x))) ||
(searchArray.Count() == 0 || searchArray.ToList().Any(x => bk.Genre.Contains(x)))))
where searchArray is a Array containing the individual words that I want to search for, I split the string the user entered and put the results in this array. Whenever I try to run this I get the following error:
"Local sequence cannot be used in LINQ to SQL implementations of query operators except the Contains operator."
Can anyone tell me what I am doing wrong and what is the correct way to perform this search?
In a nutshell, I am trying to allow a user to enter a string like "Hello World" and for a query to be generated that will look for either hello or world or both. But, a user can enter any number of words.
The simplest option is probably to build the lambda expression by hand:
static class ContainsAny
{
private static readonly MethodInfo StringContains
= typeof(string).GetMethod("Contains", new[] { typeof(string) });
public static Builder<T> Words<T>(IEnumerable<string> words)
{
return new Builder<T>(words);
}
public static Builder<T> Words<T>(params string[] words)
{
return new Builder<T>(words);
}
public sealed class Builder<T>
{
private static readonly ParameterExpression Parameter
= Expression.Parameter(typeof(T), "obj");
private readonly List<Expression> _properties = new List<Expression>();
private readonly List<ConstantExpression> _words;
internal Builder(IEnumerable<string> words)
{
_words = words
.Where(word => !string.IsNullOrEmpty(word))
.Select(word => Expression.Constant(word))
.ToList();
}
public Builder<T> WithProperty(Expression<Func<T, string>> property)
{
if (_words.Count != 0)
{
_properties.Add(ReplacementVisitor.Transform(
property, property.Parameters[0], Parameter));
}
return this;
}
private Expression BuildProperty(Expression prop)
{
return _words
.Select(w => (Expression)Expression.Call(prop, StringContains, w))
.Aggregate(Expression.OrElse);
}
public Expression<Func<T, bool>> Build()
{
if (_words.Count == 0) return (T obj) => true;
var body = _properties
.Select(BuildProperty)
.Aggregate(Expression.OrElse);
return Expression.Lambda<Func<T, bool>>(body, Parameter);
}
}
private sealed class ReplacementVisitor : ExpressionVisitor
{
private ICollection<ParameterExpression> Parameters { get; set; }
private Expression Find { get; set; }
private Expression Replace { get; set; }
public static Expression Transform(
LambdaExpression source,
Expression find,
Expression replace)
{
var visitor = new ReplacementVisitor
{
Parameters = source.Parameters,
Find = find,
Replace = replace,
};
return visitor.Visit(source.Body);
}
private Expression ReplaceNode(Expression node)
{
return (node == Find) ? Replace : node;
}
protected override Expression VisitConstant(ConstantExpression node)
{
return ReplaceNode(node);
}
protected override Expression VisitBinary(BinaryExpression node)
{
var result = ReplaceNode(node);
if (result == node) result = base.VisitBinary(node);
return result;
}
protected override Expression VisitParameter(ParameterExpression node)
{
if (Parameters.Contains(node)) return ReplaceNode(node);
return Parameters.FirstOrDefault(p => p.Name == node.Name) ?? node;
}
}
}
With this code in place, you can call:
Expression<Func<Book, bool>> filter = ContainsAny
.Words<Book>(searchArray)
.WithProperty(book => book.Name)
.WithProperty(book => book.Genre)
.Build();
var data = DataContext.Book.Where(filter);
For example, if the searchArray contains { "Hello", "World" }, the generated lambda will be:
obj => (obj.Name.Contains("Hello") || obj.Name.Contains("World"))
|| (obj.Genre.Contains("Hello") || obj.Genre.Contains("World")))
If I understand what you're trying to do correctly, you should be able to condense your query down to:
from bk in DataContext.Book
where searchArray.Contains(bk.Name) || searchArray.Contains(bk.Genre)
select bk
This is basically equivalent to the SQL:
select bk.*
from Book bk
where bk.Name in (...) or bk.Genre in (...)
In your case you must combine interpreted and local queries which can hurt performance or use SQL CLR integration by creating CLR function on database.
I have a variable number of OR conditions that I want to put together into one Linq query.
How do I do this in a loop? Basically, the final query is to be:
IQueryable<MyObject> Q;
Q = Q.Where(q => (condition1) || (condition2) || ..... || (condition N));
Something like:
For (int i = 0; i < someNumber; i++) {
Q = Q.Where(q => (existing conditions) || (q.Value == i));
}
What statement can I use to replace (existing condition) in example above without having the final expression (Q) have nested Q's inside them?
Thanks.
You'd need to build an expression tree representing all the conditions you were interested in, combined with Expression.OrElse, and then call Where a single time at the end.
This may be somewhat tricky if your current source is an anonymous type, but it shouldn't be too bad otherwise. Here's a sample - there may be a simpler way of doing the parameter replacement, but this isn't too bad. (Although ExpressionVisitor only works in .NET 4... you'd have to implement something similar yourself if you wanted to use this in .NET 3.5.)
using System;
using System.Linq;
using System.Linq.Expressions;
public class Test
{
static void Main()
{
IQueryable<string> strings = (new[] { "Jon", "Tom", "Holly",
"Robin", "William" }).AsQueryable();
Expression<Func<string, bool>> firstPredicate = p => p.Contains("ll");
Expression<Func<string, bool>> secondPredicate = p => p.Length == 3;
Expression combined = Expression.OrElse(firstPredicate.Body,
secondPredicate.Body);
ParameterExpression param = Expression.Parameter(typeof(string), "p");
ParameterReplacer replacer = new ParameterReplacer(param);
combined = replacer.Visit(combined);
var lambda = Expression.Lambda<Func<string, bool>>(combined, param);
var query = strings.Where(lambda);
foreach (string x in query)
{
Console.WriteLine(x);
}
}
// Helper class to replace all parameters with the specified one
class ParameterReplacer : ExpressionVisitor
{
private readonly ParameterExpression parameter;
internal ParameterReplacer(ParameterExpression parameter)
{
this.parameter = parameter;
}
protected override Expression VisitParameter
(ParameterExpression node)
{
return parameter;
}
}
}
A less-than-optimized version (pray that the backend will do the necessary lifting and optimization).
public static IQueryable<T> Any<T>(this IQueryable<T> q,
params Expression<Func<T, bool>>[] preds)
{
var par = Expression.Parameter(typeof(T), "x");
Expression body = Expression.Constant(false);
foreach (var pred in preds)
{
body = Expression.OrElse(body, Expression.Invoke(pred, par));
}
var ff = Expression.Lambda(body, par) as Expression<Func<T, bool>>;
return q.Where(ff);
}
static void Main(string[] args)
{
var q = new[] { "jim", "bob", "Jon", "leppie" }.AsQueryable();
Expression<Func<string, bool>>[] preds =
{
x => x == "Jon",
x => x == "Skeet",
x => x == "leppie"
};
var result = q.Any(preds).ToArray();
}
public static IEnumerable<T> GetItemsThatMatchAny<T> (this IEnumerable<T> source, IEnumerable<Func<T,bool>> predicates)
{
return source.Where(t => predicates.Any(predicate => predicate(t)));
}
An example of a predicate generator:
private static IEnumerable<Func<MyClass, bool>> GetPredicates (int num)
{
var predicates = new Func<MyClass, bool>[] {m => m.Foo == 3, m => m.Bar =="x", m => DateTime.Now.DayOfWeek == DayOfWeek.Sunday};
return predicates.Take (num);
}
I have a query of IQueryable and want to apply sorting to it dynamically, sorting can be on many columns (asc or desc). I've written the following generic function:
private IQueryable<T> ApplySorting<T,U>(IQueryable<T> query, Expression<Func<T, U>> predicate, SortOrder order)
{
if (order == SortOrder.Ascending)
{
{
return query.OrderBy<T, U>(predicate);
}
}
else
{
{
return query.OrderByDescending<T, U>(predicate);
}
}
}
SortOrder is my simple enum with 2 values: Ascending and Descending
Then I call this function in a loop, for each column that user requested sorting. However I've noticed it fails because it always sorts on the last column used, ignoring the other ones.
Then I found there's a 'ThenBy' method on IOrderedQueryable so the valid usage is:
var q = db.MyType.OrderBy(x=>x.Col1).ThenBy(y=>y.Col2); //etc.
But how can I make it generic? I tried to test if the query is IOrderedQueryable but it seems always to be true even if it's simplest var q = from x in db.MyType select x
I have no clue why it was designed like this. What's wrong with:
var q = db.MyType.OrderBy(x=>x.Col1).OrderBy(y=>y.Col2); //etc.
it's so much intuitive
You just need to check if the query is already ordered :
private IQueryable<T> ApplySorting<T,U>(IQueryable<T> query, Expression<Func<T, U>> predicate, SortOrder order)
{
var ordered = query as IOrderedQueryable<T>;
if (order == SortOrder.Ascending)
{
if (ordered != null)
return ordered.ThenBy(predicate);
return query.OrderBy(predicate);
}
else
{
if (ordered != null)
return ordered.ThenByDescending(predicate);
return query.OrderByDescending(predicate);
}
}
How about just making the first OrderBy static, then always ThenBy?
OrderColumn[] columnsToOrderby = getColumnsToOrderby();
IQueryable<T> data = getData();
if(!columnToOrderBy.Any()) { }
else
{
OrderColumn firstColumn = columnsToOrderBy[0];
IOrderedEnumerable<T> orderedData =
firstColumn.Ascending
? data.OrderBy(predicate)
: data.OrderByDescending(predicate);
for (int i = 1; i < columnsToOrderBy.Length; i++)
{
OrderColumn column = columnsToOrderBy[i];
orderedData =
column.Ascending
? orderedData.ThenBy(predicate)
: orderedData.ThenByDescending(predicate);
}
}
Total guess, but can you do something like this?
query.OrderBy(x => 1).ThenBy<T,U>(predicate)
Any syntax errors aside, the idea is to do an OrderBy() that doesn't affect anything, then do the real work in the .ThenBy() method call
I would write a wrapper around and internally use linq extension methods.
var resultList = presentList
.MyOrderBy(x => x.Something)
.MyOrderBY(y => y.SomethingElse)
.MyOrderByDesc(z => z.AnotherThing)
public IQueryable<T> MyOrderBy(IQueryable<T> prevList, Expression<Func<T, U>> predicate) {
return (prevList is IOrderedQueryable<T>)
? query.ThenBy(predicate)
: query.OrderBy(predicate);
}
public IQueryable<T> MyOrderByDesc(IQueryable<T> prevList, Expression<Func<T, U>> predicate) {
return (prevList is IOrderedQueryable<T>)
? query.ThenByDescending(predicate)
: query.OrderByDescending(predicate);
}
PS: I didn't test the code
Extension to dynamic multi-order:
public static class DynamicExtentions
{
public static IEnumerable<T> DynamicOrder<T>(this IEnumerable<T> data, string[] orderings) where T : class
{
var orderedData = data.OrderBy(x => x.GetPropertyDynamic(orderings.First()));
foreach (var nextOrder in orderings.Skip(1))
{
orderedData = orderedData.ThenBy(x => x.GetPropertyDynamic(nextOrder));
}
return orderedData;
}
public static object GetPropertyDynamic<Tobj>(this Tobj self, string propertyName) where Tobj : class
{
var param = Expression.Parameter(typeof(Tobj), "value");
var getter = Expression.Property(param, propertyName);
var boxer = Expression.TypeAs(getter, typeof(object));
var getPropValue = Expression.Lambda<Func<Tobj, object>>(boxer, param).Compile();
return getPropValue(self);
}
}
Example:
var q =(myItemsToSort.Order(["Col1","Col2"]);
Note: not sure any IQueryable provider can translate this