Consider this code:
public List<Clients> GetFilteredClients(DateTime? FromDate = null,
DateTime? ToDate = null,
int? fromLocationType = null,
int? toLocationType = null)
{
Func<Clients, bool> fromDateFilter = f => true;
if (FromDate.HasValue)
{
fromDateFilter = z => z.Insert_Date.Value.Date >= FromDate.Value.Date;
}
Func<Clients, bool> toDateFilter = f => true;
if (ToDate.HasValue)
{
toDateFilter = z => z.Insert_Date.Value.Date <= ToDate.Value.Date;
}
Func<Clients, bool> fromLocationTypeFilter = f => true;
if (fromLocationType.HasValue)
{
fromOrgFilter = z => z.LocationTypeId >= fromLocationType.Value;
}
Func<Clients, bool> toLocationTypeFilter = f => true;
if (toLocationType.HasValue)
{
toLocationTypeFilter = z => z.LocationTypeId <= toLocationType.Value;
}
var filtered = DB_Context.Clients
.Where(fromDateFilter)
.Where(toDateFilter)
.Where(fromLocationTypeFilter)
.Where(toLocationTypeFilter)
.OrderByDescending(k => k.Id)
.Take(1000)
.ToList();
return filtered;
}
I have something like 100K records in the DB, I need only the top 1000 that answer to the requirements of:
.Where(fromDateFilter)
.Where(toDateFilter)
.Where(fromLocationTypeFilter)
.Where(toLocationTypeFilter)
However the execution time still takes something like 10 seconds.
Any idea why?
You must use Expression<Func<...>> rather than Func<...>. When you use Func, only the enumerable methods can be used on the queryable, which in this case means you first download everything to memory, and then do the filtering. If you switch over to Expression<...>, the O/RM will do the filtering on the DB server, rather than in your application.
Also, there's better ways to do what you're doing. For example, you can build the conditions like so:
var query = DB_Context.Clients.AsQueryable();
if (FromDate.HasValue) query = query.Where(...);
if (ToDate.HasValue) query = query.Where(...);
...
return query.OrderByDescending(k => k.Id).Take(1000).ToList();
Of course, this means that whatever DB provider you're using must be able to support the kind of filtering you're trying to do - you'll need to consult the documentation.
You are using delegates instead LINQ expressions. That leads to processing a data by your application and not by SQL Server.
LINQ expressions look like lambda expressions thanks for the syntax, but they are not same thing. The compiler takes a decision what to create (delegates or LINQ expressions) depending on the situation.
If an object implements the IQueriable interface, then the compiler uses the Queryable class and generates LINQ expression trees, which later can be translated into a SQL query or other form by the specific IQueryProvider.
Otherwise, the compiler uses extensions from the Enumerable class, which create iterators over source collection (all records from the table in your case).
As an example. The code bellow will be compilled into LINQ expressions.
// Source code
IQueryable<Clients> source = null;
IQueryable<Clients> result = source.Where(c => c.LocationTypeId >= 1);
// Compiller generated code
IQueryable<Clients> source = null;
Expression parameterC = Expression.Parameter(typeof(Clients), "c");
IQueryable<Clients> result = Queryable.Where<Clients>(
source,
Expression.Lambda<Func<Clients, bool>>(
Expression.LessThanOrEqual(
Expression.Property(
parameterC ,
typeof(Clients).GetProperty("LocationTypeId").GetGetMethod()
),
Expression.Constant(1, typeof(int))
),
new ParameterExpression[]
{
parameterC
}
);
And this code uses delegates:
// Source code
IQueryable<Clients> source = null;
Func<Clients, bool> filter = c => c.LocationTypeId >= 1;
IEnumerable<Clients> result = source.Where(filter );
// Compiller generated code
IQueryable<Clients> source = null;
Func<Clients, bool> filter = c => c.LocationTypeId >= 1;
IEnumerable<Clients> result = Enumerable.Where(source, filter);
So, to solve you problem use Expression<Func<Clients, bool>> instead of Func<Clients, bool>:
IQueryable<Clients> result = DB_Context.Clients;
if (someFilter.HasValue)
result = result.Where(c => c.SomeProperty == someFilter.Value);
// other filters
return query
.OrderByDescending(k => k.Id)
.Take(1000)
.ToList();
Related
(In theory) We have the following query:
var variableDate = DateTime.Parse("Nov 2, 2021")
var results = (from x in db.FooBar
where x.Date == variableDate
select x).ToList();
We can simply modify the variableDate to 11/3/21, 11/4/21, etc. to return results for different dates.
The question is can we use this same query to return all results after variableDate with some modification to the operator(s)/variable(s)? For example including the operator (> or ==) within variableDate (let's call it variableDateFormula):
> 11/2/2021
or
== 11/2/2021
This way we can simply call the same query with the same variable to return results for different operators:
var results = (from x in db.FooBar
where x.Date variableDateFormula
select x).ToList();
I know the suggestion above will not work, it is just for visualization. I have a query that takes up 9 lines and has 8 different iterations depending on 3 values passed into the function. I would love to avoid writing basically the same query 8 times and taking up ~70 lines, and instead dynamically change the operators/criteria to have only 1 query.
Yes, the operator is just syntactic sugar for a function call and you can change that function, for example with a lambda:
var equalFunc = ((d1,d2) => d1 == d2);
var greaterFunc = ((d1,d2) => DateTime.Compare(d1, d2) > 0);
var actualFunc = greaterFunc; // Insert logic here to choose the appropriate function
Then your select becomes
from x in db.FooBar
where actualFunc(x.Date, variableDate)
select x
Yes. The LINQ is just building an ExpressionTree that Entity Framework translates in to SQL, so you can use other expressions or lambdas as parameters that get inserted into the tree, like in this LINQpad example:
void Main()
{
var aDate = new DateTime(2002, 1, 1);
GetResults(d => d.TheDate == aDate).Dump();
GetResults(d => d.TheDate <= aDate).Dump();
GetResults(d => d.TheDate > aDate).Dump();
}
IEnumerable<Data> GetResults(Func<Data, bool> op)
{
var data = new List<Data>
{
new Data{TheDate = new DateTime(2000,1,1)},
new Data{TheDate = new DateTime(2001,1,1)},
new Data{TheDate = new DateTime(2002,1,1)},
new Data{TheDate = new DateTime(2003,1,1)},
};
return data.Where(d => op(d));
}
public class Data
{
public DateTime TheDate { get; set; }
}
Which produces the following results:
If you are doing more complex logic in your expressions, the EF Database Provider may not be able to translate the expression to SQL though. There will be limitations. You couldn't for example, do this:
GetResults(d => d.TheDate.ToString().Reverse() == "1234");
because it wouldn't understand the custom Reverse extension method.
I would suggest to use LINQKit for such task. EF Core cannot translate local variable as expression function.
var variableDate = DateTime.Parse("Nov 2, 2021");
Expression<Func<DateTime, DateTime, bool>> compareFunc = (d1, d2) => d1 > d2;
var results = (from x in db.FooBar
where compareFunc.Invoke(x.Date, variableDate)
select x).ToList();
For enabling LINQKit for EF Core add the following to the options:
builder
.UseSqlServer(connectionString)
.WithExpressionExpanding(); // enabling LINQKit extension
Or if you use other LINQ provider, add AsExpandable() at the top of the query:
var results = (from x in db.FooBar.AsExpandable()
where compareFunc.Invoke(x.Date, variableDate)
select x).ToList();
Task: I need to give some expression with parameters into LINQ's where to get some data from database, but have an error above
This example of working expression:
var shopExp = GetPersonForShop(PersonTypeIds.Director, new Guid("adda423f-8c38-40e0-9f39-6deceb787bc0")); // id
Where(shopExp)
But i need assign id dynamically, but got error above :
_repository.Persons
.Where(GetPersonForShop(PersonTypeIds.Director, person.PersonId)
And got error:
{"Unable to cast object of type 'System.Linq.Expressions.InstanceMethodCallExpression2' to type 'System.Linq.Expressions.LambdaExpression'."}
How does function for where(linq) look:
private Expression<Func<Person, bool>> GetPersonForShop(PersonTypeIds personTypeId, Guid personId)
{
return person => person .PeronTypeId== (int) personTypeId && person .PersonId == personId;
}
This is approximate look like out production, just change names of parametrs code
How can I add expression with parameters to Where clause??
Lambda expressions use => notation. Try something like this:
var idToFind = new Guid("adda423f-8c38-40e0-9f39-6deceb787bc0");
var result = _repository.Persons
.Where(p => p.TypeId == PersonTypeIds.Director && p.PersonId == idToFind);
In this expression, p represents each Person record in the Persons table, compared one-by-one using the boolean expression that follows it.
Depending on your datasource, the comparison for each p will either be done by .NET in memory, or it will happen inside your database using a SQL WHERE clause which is constructed from the boolean expression. The last would be optimal because it would mean that not the entire Persons table has to be transferred into .NET memory before comparison can take place.
Update - To apply the same condition multiple times without repeating it in your code, while still keeping the advantages of LINQ to SQL translation intact, you can put the condition in an Expression<Func<Person, bool>> object and then use that multiple times:
Expression<Func<Person, bool>> expression =
p => p.TypeId == PersonTypeIds.Director && p.PersonId == idToFind;
var result1 = datasource1.Where(expression);
var result2 = datasource2.Where(expression);
var result3 = datasource3.Where(expression);
Or through a method that produces the Expression object:
var result1 = datasource1.Where(GetExpression(idToFind));
var result2 = datasource2.Where(GetExpression(idToFind));
var result3 = datasource3.Where(GetExpression(idToFind));
public Expression<Func<Person, bool>> GetExpression(Guid idToFind)
{
return p => p.TypeId == PersonTypeIds.Director && p.PersonId == idToFind;
}
Or alternatively you can use a helper method:
var result1 = FilterByTypeAndId(datasource1, idToFind);
var result2 = FilterByTypeAndId(datasource2, idToFind);
var result3 = FilterByTypeAndId(datasource3, idToFind);
public IQueryable<Person> FilterByTypeAndId(IQueryable<Person> datasource, Guid idToFind)
{
return datasource.Where(p => p.TypeId == PersonTypeIds.Director && p.PersonId == idToFind);
}
based on the previous response, I am going to give you a few alternatives and suggestions.
var idToFind = new Guid("adda423f-8c38-40e0-9f39-6deceb787bc0");
var result = _repository
.Persons
.Where(p => p.TypeId == PersonTypeIds.Director)
.Where(p => p.PersonId == idToFind)
.ToList();
First is doing the where clause in 2 steps and then, adding the ToList(), with the ToList(), you will deal with collections and LINQ that is pretty useful. And by doing the where clause in 2 steps, is more for readable purposes.
So I have a function like below which takes in a list of objects and keeps including them into my query string. This works great.
query = includeExpressions.Aggregate(query, (current, include) => current.Include(include));
But what I am wanting to do is to Aggregate the OrderBys as well. This is where I am running into issues. The issue I have is that one must use OrderBy then use ThenBy on the same line. I've tried using a for loop first item use order by then on use thenby. The issue I am running in is that in order for thenby to be activated the order by needs to proceed it on the same object line.
query.OrderBy(orderBy).ThenBy
I can not do
var usedOrderBy = true;
foreach (var orderBy in orderBys)
{
if (usedOrderBy)
{
query = query.OrderBy(orderBy);
usedOrderBy = true;
}
else
{
query = query.ThenBy(orderBy); // <-- Can not locate thenby
}
}
any ides.
I want to pass in like
orderItems(x => x.Item1, x => x.Item2)
The problem is in the type of query variable. ThenBy is the extension method that can be applied to System.Linq.IOrderedEnumerable<TSource> or System.Linq.IOrderedQueryable<TSource> so you can't do just
IEnumerable<Point> points = GetPoints();
points = points.OrderBy(p => p.X);
points = points.ThenBy(p => p.Y);
You should have an extra variable for preordered result to solve your scenario like in an example below:
IEnumerable<Point> points = GetPoints();
var orderedPoints = points.OrderBy(p => p.X);
orderedPoints = orderedPoints.ThenBy(p => p.Y);
Check this
var orderByList = new List<Expression<Func<TEntity, object>>>();
Expression<Func<TEntity,object>> orderBy1 = x => x.Id;
Expression<Func<TEntity, object>> orderBy2 = x => x.ToString();
Expression<Func<TEntity, object>> orderBy3 = x => x.Id;
orderByList.Add(orderBy1);
orderByList.Add(orderBy2);
orderByList.Add(orderBy3);
var resultOrderedQueryable = orderByList.Aggregate<Expression<Func<TEntity, object>>, IOrderedQueryable<TEntity>>(null, (current, orderBy) => current != null ? current.ThenBy(orderBy) : query.OrderBy(orderBy));
Your query will work if can have initial seed of OrderBy. Then you can chain ThenBy's. As mentioned, ThenBy requires an IOrderedEnumerable (or IOrderedQueryable) first before attaching ThenBy clauses:
Your code will look like below:
IOrderedQueryable<T> orderedQuery = null;
for(int temp = 0; temp < orderBys.Count; temp++)
{
if(temp == 0)
orderedQuery = query.OrderBy(orderBys[temp]);
else orderedQuery = orderedQuery.ThenBy(orderBys[temp]);
}
query = orderedQuery ?? query;
Reference: How to generate a dynamic number of ThenBy clauses in a Specification
I have a Person table in my database that has NationalId field.
Is there any way to load all Persons with even NationalId, using Ef code first and Linq to entities, without loading all Persons to memory?
Somethings like:
public bool IsEven(int number)
{
return number % 2 == 0;
}
var context = new MyContext();
var personsWithEvenNationalId = context.Persons
.Where(x=> IsEven(x.NationalId))
.ToList();
You would have to do your check inline
var personsWithEvenNationalId = context.Persons
.Where(x=> x.NationalId%2 == 0)
.ToList();
Linq to Entities doesn't know how to translate your custom method into SQL basically. If you do need to use a custom method you would have to get Persons as enumerable and then use your custom method, i.e.
var personsWithEvenNationalId = context.Persons
.AsEnumerable()
.Where(x=> IsEven(x.NationalId))
.ToList();
But that's hardly ideal as it would load all Persons and then filter on IsEven
Edit: Thinking about it you could also create an extension method for IQueryable<Person> if you don't want to have to write it inline every time. Something like this where you build an Expression
public static IQueryable<Person> WhereEven(this IQueryable<Person> source, Expression<Func<Person, int>> property)
{
var expression = Expression.Equal(
Expression.Modulo(
property.Body,
Expression.Constant(2)),
Expression.Constant(0));
var methodCallExpression = Expression.Call(typeof (Queryable),
"where",
new Type[] {source.ElementType},
source.Expression,
Expression.Lambda<Func<Person, bool>>(expression, property.Parameters));
return source.Provider.CreateQuery<Person>(methodCallExpression);
}
And to use it:
context.Persons.WhereEven(x => x.NationalId).ToList();
Rather than a function that does what you want, you'll need to have an function (or property, or field) that provides an Expression that does the projection that you want:
public static Expression<Func<int, bool>> IsEven()
{
return number => number % 2 == 0;
}
You can now write:
using(var context = new MyContext())
{
var personsWithEvenNationalId = context.Persons
.Select(x=> x.NationalId)
.Where(IsEven())
.ToList();
}
EDIT: Let's try this again. This time I've used the AdventureWorks sample database so you can all play along. This will rule out anything crazy I've done in my own database. Here's a new example demonstrating what works and what I would expect to work (but doesn't). Can anyone explain why it doesn't work or suggest a different way of achieving my goal (refactoring out the common expression so it can be reused elsewhere)?
using (AdventureWorksDataContext db = new AdventureWorksDataContext())
{
// For simplicity's sake we'll just grab the first result.
// The result should have the name of the SubCategory and an array of Products with ListPrice greater than zero.
var result = db.ProductSubcategories.Select(subCategory => new
{
Name = subCategory.Name,
ProductArray = subCategory.Products.Where(product => product.ListPrice > 0).ToArray()
}).First();
Console.WriteLine("There are {0} products in SubCategory {1} with ListPrice > 0.", result.ProductArray.Length, result.Name);
// Output should say: There are 3 products in SubCategory Bib-Shorts with ListPrice > 0.
// This won't work. I want to pull the expression out so that I can reuse it in several other places.
Expression<Func<Product, bool>> expression = product => product.ListPrice > 0;
result = db.ProductSubcategories.Select(subCategory => new
{
Name = subCategory.Name,
ProductArray = subCategory.Products.Where(expression).ToArray() // This won't compile because Products is an EntitySet<Product> and that doesn't have an overload of Where that accepts an Expression.
}).First();
Console.WriteLine("There are {0} products in SubCategory {1} with ListPrice > 0.", result.ProductArray.Length, result.Name);
}
</Edit>
The following LINQ to SQL works fine:
var result = from subAccount in db.SubAccounts
select new ServiceTicket
{
MaintenancePlans = subAccount.Maintenances.Where(plan => plan.CancelDate == null && plan.UpgradeDate == null).Select(plan => plan.ToString()).ToArray()
// Set other properties...
};
However, I want to break out the predicate passed to the Where since it's used throughout the code. But if I try and pass a defined predicate into the Where it fails, such as:
Func<DatabaseAccess.Maintenance, bool> activePlanPredicate = plan => plan.CancelDate == null && plan.UpgradeDate == null;
var result = from subAccount in db.SubAccounts
select new ServiceTicket
{
MaintenancePlans = subAccount.Maintenances.Where(activePlanPredicate).Select(plan => plan.ToString()).ToArray()
// Set other properties...
};
This makes no sense to me. Can anyone explain what's going on? Maintenances is of type EntitySet<DatabaseAccess.Maintenance>. The error I get is:
System.NotSupportedException:
Unsupported overload used for query
operator 'Where'..
EDIT: For those interested, here's what Reflector has for the first (working) example with Optimization set to .NET 2.0:
using (BugsDatabaseDataContext db = new BugsDatabaseDataContext())
{
ParameterExpression CS$0$0001;
ParameterExpression CS$0$0006;
ParameterExpression CS$0$0010;
return db.SubAccounts.Select<SubAccount, ServiceTicket>(Expression.Lambda<Func<SubAccount, ServiceTicket>>(
Expression.MemberInit(
Expression.New(
(ConstructorInfo) methodof(ServiceTicket..ctor),
new Expression[0]),
new MemberBinding[]
{
Expression.Bind(
(MethodInfo) methodof(ServiceTicket.set_MaintenancePlans),
Expression.Call(
null,
(MethodInfo) methodof(Enumerable.ToArray),
new Expression[]
{
Expression.Call(
null,
(MethodInfo) methodof(Enumerable.Select),
new Expression[]
{
Expression.Call(
null,
(MethodInfo) methodof(Enumerable.Where),
new Expression[]
{
Expression.Property(CS$0$0001 = Expression.Parameter(typeof(SubAccount), "subAccount"), (MethodInfo) methodof(SubAccount.get_Maintenances)),
Expression.Lambda<Func<Maintenance, bool>>(
Expression.AndAlso(
Expression.Equal(
Expression.Property(CS$0$0006 = Expression.Parameter(typeof(Maintenance), "plan"), (MethodInfo) methodof(Maintenance.get_CancelDate)),
Expression.Convert(Expression.Constant(null, typeof(DateTime?)), typeof(DateTime?)), false, (MethodInfo) methodof(DateTime.op_Equality)
),
Expression.Equal(
Expression.Property(CS$0$0006, (MethodInfo) methodof(Maintenance.get_UpgradeDate)),
Expression.Convert(Expression.Constant(null, typeof(DateTime?)), typeof(DateTime?)), false, (MethodInfo) methodof(DateTime.op_Equality)
)
),
new ParameterExpression[] { CS$0$0006 }
)
}
),
Expression.Lambda<Func<Maintenance, string>>(
Expression.Call(
CS$0$0010 = Expression.Parameter(typeof(Maintenance), "plan"),
(MethodInfo) methodof(object.ToString),
new Expression[0]
),
new ParameterExpression[] { CS$0$0010 }
)
}
)
}
)
)
}
),
new ParameterExpression[] { CS$0$0001 }
)
).ToList<ServiceTicket>();
}
EDIT: The Reflector output for the second example (using a predicate) is mostly similar. The biggest difference being that, in the call to Enumerable.Where, rather than passing an Expression.Lambda it passes Expression.Constant(activePlanPredicate).
I don't fully understand the guts of Linq to Entities, but there is an Open Source (usable in proprietary software) toolkit specifically designed to help solve this problem, called LinqKit, linked off this O'Reilly-related article:
http://www.albahari.com/nutshell/predicatebuilder.aspx
Since I don't fully understand the guts, I'll just quote them:
Entity Framework's query processing pipeline cannot handle invocation expressions, which is why you need to call AsExpandable on the first object in the query. By calling AsExpandable, you activate LINQKit's expression visitor class which substitutes invocation expressions with simpler constructs that Entity Framework can understand.
Here is a direct link to LinqKit.
And here is the type of code that this project enables:
using LinqKit;
// ...
Expression<Func<Product, bool>> expression = product => product.ListPrice > 0;
var result = db.ProductSubcategories
.AsExpandable() // This is the magic that makes it all work
.Select(
subCategory => new
{
Name = subCategory.Name,
ProductArray = subCategory.Products
// Products isn't IQueryable, so we must call expression.Compile
.Where(expression.Compile())
})
.First();
Console.WriteLine("There are {0} products in SubCategory {1} with ListPrice > 0."
, result.ProductArray.Count()
, result.Name
);
The result is:
There are 3 products in SubCategory Bib-Shorts with ListPrice > 0.
Yay, no exception, and we can extract the predicate!
I'd refactor the original like this
private bool IsYourPredicateSatisfied(Maintenance plan)
{
return plan.CancelDate == null && plan.UpgradeDate == null;
}
Then your Where clause is Where(m => IsYourPredicateSatisfied(m))
Try this:
Expression<Func<DatabaseAccess.Maintenance, bool>> activePlanPredicate = plan => plan.CancelDate == null && plan.UpgradeDate == null;
var result = from subAccount in db.SubAccounts
select new ServiceTicket
{
MaintenancePlans = subAccount.Maintenances.Where(activePlanPredicate).Select(plan => plan.ToString()).ToArray()
// Set other properties...
};
I don't have VisualStudio in front of me, so that may require some tweaking. The issue you're running into is that you want to access the IQueryable extension of Where, but just having a Func<T,bool> gives you the IEnumerable extension.