This is my Custom filter(Func) to pass in where clause
Func<Project,bool> filter = f =>
{
bool filteredContent = true;
if (!CreatorId.Equals(0))
filteredContent = f.CreatedBy.Equals(CreatorId);
if (filteredContent && !VerticalMarketId.Equals(0))
filteredContent = f.VerticalMarketsId.Equals(VerticalMarketId);
if (filteredContent && !ProductCategoryId.Equals(0))
filteredContent = f.ProductCategoriesId.Equals(ProductCategoryId);
return filteredContent;
};
This is my code where I get all the projects based on the conditions created in filter expression
getProjects = await _context.Projects.Where(x => x.IsDeleted == false && filter.Invoke(x))// Here I'm getting the exception
.Include(PC => PC.ProjectComments.Where(x => x.IsDeleted == false))
.Include(SP => SP.SharedProjects)
.AsNoTracking().ToListAsync();
Exception:The LINQ expression (DbSet......) could not be
translated. Either rewrite the query in a form that can be translated,
or switch to client evaluation explicitly by inserting a call to
'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or 'ToListAsync'.
Can someone tell how can I filter the data using expression in this?
NOTE: I can do ToListAsync() before applying the filter, but it'll get all the records from DB then filter on client side. But I want to filter the data on server side.
IF you were using Linq To Objects that should work but you are doing Linq To SQL and in this case you must think on how you would translate this function into a valid SQL statement. Question yourself: How could I pass this function call in a SQL Statement? Depending what you do on the body of your expression, you cannot translate it to SQL, you must be simpler sometimes.
Candidate solution
Add PredicateBuilder class on your project. It will give you easily logical operators to you handle expressions.
http://www.albahari.com/nutshell/predicatebuilder.aspx
Try to define an expression and pass it as argument on Where method of your query method chain. For sample (read the comments):
// define a expression with default condition
Expression<Func<Project, bool>> filter = f => !f.IsDeleted;
// check conditions to add new filtes with `And` logical operator
if (!CreatorId.Equals(0))
filter = filter.And(f => f.CreatedBy.Equals(CreatorId));
else if (!VerticalMarketId.Equals(0))
filter = filter.And(f => f.VerticalMarketsId.Equals(VerticalMarketId));
else if (!ProductCategoryId.Equals(0))
filter = filter.And(f => f.ProductCategoriesId.Equals(ProductCategoryId));
// apply the filter on the query and execute it
getProjects = await _context.Projects.Where(filter)
.Include(PC => PC.ProjectComments.Where(x => !x.IsDeleted))
.Include(SP => SP.SharedProjects)
.AsNoTracking()
.ToListAsync();
Note: I didn't test this code and it probably should be fixed somehow!
Important tips on Linq To SQL:
Logical operators are ok and tend to be translated fine to sql;
Where(x => x.Children.Any(j => j.Children.Any())), each Any call generates a subquery on query scope, be careful with it given it can compromise your database performance.
If you just need to check the existence of an item, use queryable.Any(expression).
If you need to check and then do something, prefer using queryable.FirstOrDefault(expression) and check if the result is null before using.
Use paging with .Take(int) and .Skip(int).
Always concrete your queries by calling .ToList(), .ToArray() or async versions of these methods. Avoid passing queryable in the top layers (query can be executed out of the scope you want).
I figured it out by creating a simple Expression as fololows:
private static Expression<Func<Project, bool>> ProjectFilterExpression(
int creatorId,
int verticalMarketId,
int productCategoryId)
{
Expression<Func<Project, bool>> projectFilterExpression = pfe =>
!pfe.IsDeleted
//CreatorId Filter
&& (creatorId.Equals(0) || pfe.CreatedBy.Equals(creatorId))
//Vertical Market Filter
&& (verticalMarketId.Equals(0) || pfe.VerticalMarketsId.Equals(verticalMarketId))
// Product Category Filter
&& (productCategoryId.Equals(0) || pfe.ProductCategoriesId.Equals(productCategoryId));
return projectFilterExpression;
}
Then I call this static method inside my filter method.
var filter = ProjectFilterExpression(CreatorId, VerticalMarketId, ProductCategoryId);
And finally I applied this filter in my LINQ where clause
getProjects = await _context.Projects.Where(filter).AsNoTracking().ToListAsync();
It's working totally fine.
Related
I'm trying to implement a global query filter for tenancy within my application. I've got AssessmentModels that can have multiple owners, the owners coming from a 3rd party that I don't have complete control over, but can adjust as needed. (And can manipulate before I save to my database) Right now, the Owners are stored as a semicolon delimited string (.e.g, team1;team2).
I've come up with the following which works for selecting data, but does not work for a global query filter:
private Expression<Func<AssessmentModel, bool>> GetAssessmentFilter()
{
// The lambda parameter.
var assessmentParameter = Expression.Parameter(typeof(AssessmentModel), "a");
// Build the individual conditions to check against.
var orConditions = _adminTeamNames
.Select(keyword => (Expression<Func<AssessmentModel, bool>>)(a => EF.Functions.Like(a.Owners, $"%{keyword}%")))
.Select(lambda => (Expression)Expression.Invoke(lambda, assessmentParameter))
.ToList();
// Combine the individual conditions to an expression tree of nested ORs.
var orExpressionTree = orConditions
.Skip(1)
.Aggregate(
orConditions.First(),
(current, expression) => Expression.OrElse(expression, current));
// Build the final predicate (a lambda expression), so we can use it inside of `.Where()`.
var predicateExpression = (Expression<Func<AssessmentModel, bool>>)Expression.Lambda(
orExpressionTree,
assessmentParameter);
return predicateExpression;
}
So,
var result = db.Assessments.Where(predicateExpression).ToList(); works, but modelBuilder.Entity<AssessmentModel>().HasQueryFilter(predicateExpression); gives the error:
System.InvalidOperationException : The LINQ expression 'DbSet<AssessmentModel>()
.Where(a => Invoke(a => value(DbFunctions).Like(a.Owners, "%Los%"), a)
|| Invoke(a => value(DbFunctions).Like(a.Owners, "%Atl%"), a)
)' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or 'ToListAsync'. See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
AFAIK EF Core is not very good at translating invoke expressions. Try manually replacing parameter of build lambda with your custom parameter expression (i.e. assessmentParameter) :
var orConditions = _adminTeamNames
.Select(keyword => (Expression<Func<AssessmentModel, bool>>)(a => EF.Functions.Like(a.Owners, $"%{keyword}%")))
.Select(lambda => new ReplacingExpressionVisitor(lambda.Parameters, new []{assessmentParameter}).Visit(lambda.Body))
.ToList();
I have the following database query where I am trying to check if there exists an item with a particular barcode that is linked to a particular mailbag. The query is as follows:
var exists = await dbcontext.Items
.Include(t => t.MailBagItems)
.ThenInclude(mt => mt.MailBag)
.AnyAsync(t => t.Barcode.Equals(barcode) &&
t.MailBagItems.FirstOrDefault() != null &&
t.MailBagItems.FirstOrDefault().MailBag.Number.ToLower().Equals(mailbagNumber.ToLower()));
For some reason, I'm getting the following exception:
System.InvalidOperationException: The LINQ expression could not be
translated. Either rewrite the query in a form that can be translated,
or switch to client evaluation explicitly by inserting a call to
either AsEnumerable(), AsAsyncEnumerable(), ToList(), or
ToListAsync().
I know for a fact from removing parts of the boolean expression that the issue is in the last boolean condition where I'm checking the mailbag number. However, I get the same error if I remove the calls to ToLower(). Can someone indicate what is wrong with my expression and how to fix it? Please note I'm using .NET core 3 and SQL Server.
Managed to make the query work by changing it to the following:
var exists = dbcontext.Items
.AnyAsync(t => t.Barcode.Equals(barcode) &&
t.MailBagItems.Any(t => t.MailBag.Number.ToLower().Equals(mailbagNumber.ToLower())));
Seems it wasn't enjoying the .FirstOrDefault().MailBag before.
Your AnyAsync is to complex for EF to transform to SQL, if you want to still use that query you will have to materialize the entities first, like this:
var exists = dbcontext.Items
.Include(t => t.MailBagItems)
.ThenInclude(mt => mt.MailBag)
.ToListAsync()
.AnyAsync(t => t.Barcode.Equals(barcode) &&
t.MailBagItems.FirstOrDefault() != null &&
t.MailBagItems.FirstOrDefault().MailBag.Number.ToLower().Equals(mailbagNumber.ToLower()));
Also you are missing the await keyword, or was that intended?
we are Entity Framework in our project. Need to know the performance impact between .ANY() and Expressions to form Where clause.
In the below function i used two approach to get result:
APPROACH 1 - Form Lambda expression query using ANY()
From my observation using .Any() is not adding where clause when query is executed(checked in sql profiler) what EF does is gets all matched inner joined records store in-memory and then apply condition specified in .ANY()
APPROACH 2 - Form Expression Query Starts
With Expressions i'm explicitly forming where clause and executing.checked the same in SQL query Profiler i'm able to see where clause.
Note: To form expression where clause i'm doing additional loops and "CombinePredicates".
Now, my doubts are:
which approach will improve performance. Do i need to go with Lambda
with .ANY() or Expressions?
what is the right way to from where clause to improve performance?
If not the two approach suggest me the right way to do it
private bool GetClientNotifications(int clientId, IList<ClientNotification> clientNotifications)
{
IList<string> clientNotificationList = null;
var clientNotificationsExists = clientNotifications?.Select(x => new { x.Name, x.notificationId
}).ToList();
if (clientNotificationsExists?.Count > 0)
{
//**Approach 1 => Form Lamada Query starts**
notificationList = this._clientNotificationRepository?.FindBy(x => clientNotificationsExists.Any(x1 => x.notificationId == x1.notificationId && x.clientId ==
clientId)).Select(x => x.Name).ToList();
**//Form Lamada Query Ends**
//**Approach 2 =>Form Expression Query Starts**
var filterExpressions = new List<Expression<Func<DbModel.ClientNotification, bool>>>();
Expression<Func<DbModel.ClientNotification, bool>> predicate = null;
foreach (var clientNotification in clientNotificationsExists)
{
predicate = a => a.clientId == clientId && a.notificationId == clientNotification .notificationId;
filterExpressions.Add(predicate);
}
predicate = filterExpressions.CombinePredicates<DbModel.ClientNotification>(Expression.OrElse);
clientNotificationList = this._clientNotificationRepository?.FindBy(predicate).Select(x => x.Name).ToList();
//**Form Expression Query Ends**
}
return clientNotificationList;
}
If non of the approaches were good please suggest me the right way to do.
I noticed the clientId being used on both approaches is used on conditions linked to clientNotificationsExists but there is no actual relationship with its objects so this condition can be moved up one level. This might bring a minuscule benefit as the overall sql will be smaller without the duplicate clauses.
From the described behavior the first approach filter is not being translated to sql so it is being resolved at the client side however, if it could be translated the performance would be similar or identical.
If you want to generate an IN sql clause you can use an array and use the Contains method within your filter expression. That could perhaps improve a little bit but don't get your hopes too high.
Try the following:
private bool GetClientNotifications(int clientId, IList<ClientNotification> clientNotifications)
{
IList<string> clientNotificationList = null;
var clientNotificationsExists = clientNotifications?
.Select(x => new { x.Name, x.notificationId }).ToList();
if (clientNotificationsExists?.Count > 0)
{
var notificationIds = clientNotificationsExists.Select(x => x.notificationId).ToArray();
clientNotificationList = this._clientNotificationRepository?
.FindBy(x => x.clientId == clientId && notificationIds.Contains(x.notificationId));
}
return clientNotificationList;
}
At the moment I have a linq query with a method residing inside of it. I'm getting the error LINQ to Entities does not recognize the method. So I found I can convert the method to be an expression LINQ to Entities does not recognize the method but I'm wondering is there a clean and easy way to add the expression to my linq query.
Original Method
public bool IsAvailable()
{
return Eligibility.ProgramType == InteractionProgramTypes.Available;
}
Changed to
public System.Linq.Expressions.Expression<Func<InteractionProgram, bool>> IsAvailable()
{
return i => i.Eligibility.ProgramType == InteractionProgramTypes.Available;
}
Linq query without expression
x => x.ActivityDate <= endDate && x.IsAvailable()
Linq query with expression
x => x.ActivityDate <= endDate && x.IsAvailable().Compile()
When doing that I get the compiler error, && operator cannot be applied to operands.
May I ask how do I append the expression to my current linq query.
Since you already converted IsAvailable to return Expression<Func<InteractionProgram,bool>>, all you need to do is to pass the result of calling this method to the Where method of IQueryable<T>:
var res = ctx.InteractionPrograms.Where(InteractionProgram.IsAvailable());
Note that in order for this to compile your IsAvailable method needs to be static. Moreover, you could make it a property for an even better readability:
class InteractionProgram {
public static Expression<Func<InteractionProgram,bool>> IsAvailable {get;} =
i => i.Eligibility.ProgramType == InteractionProgramTypes.Available;
... // other members of the class
}
...
var res = ctx.InteractionPrograms.Where(InteractionProgram.IsAvailable);
what about the other condition x.ActivityDate <= endDate. Where would that go?
Other conditions go into separate Where clauses either immediately before or immediately after IsAvailable condition. EF driver will combine the two expressions for you, resulting in a single query on RDBMS side.
Another alternative to sharing this expression would be creating an extension method on the EF context that returns IQueryable<InteractionProgram> pre-filtered for availability:
public static IQueryable<InteractionProgram> AvailableInteractionPrograms(this MyDbContext dbCtx) =>
dbXtx.InteractionPrograms.Where(i =>
i.Eligibility.ProgramType == InteractionProgramTypes.Available
);
This hides the function behind a shared method.
I have a list that I need to query based on about 10 different parameters that user pass to my program.
What s the best way to do it ?
ex:
List<Users>
Query params can be : username and/or user id and/or age etc.
What you are trying to achieve is composing a dynamic query in LINQ. You could do this in two ways:
using dynamic LINQ;
using PredicateBuilder by Joseph Albahari (recommended in this case);
Briefly, this is how to use PredicateBuilder in your case:
var predicate = PredicateBuilder.True<User>();
if (!string.IsNullOrWhitespace(username))
predicate = predicate.And(a => a.Username == username);
if (!string.IsNullOrWhitespace(whatever))
predicate = predicate.And(a => a.Whatever == whatever);
/* etc. etc. */
var filteredUsers = myUsers.Where(predicate);
I guess you're talking about filtering objects based on an initially unknown number of conditions.
LINQ does not evaluate your query until you explicitly ask so by enumerating it (using foreach, ToList(), ToArray()...). And when you do that, depending on the implementation of LINQ (SQL, objects, XML, etc), the query will be optimized.
AFAIK, all implementation are able to optimize "where condition1 where condition2" into "where condition1 && condition2".
Which means you just have to add your filtering conditions one by one. For instance:
private List<User> FilterUsers(string username, string userid, int? minAge, int? maxAge)
{
IEnumerable<User> query = GetUsers();
if (!string.IsNullOrEmpty(username)) query = query.Where(u => u.Username.StartsWith(username);
if (!string.IsNullOrEmpty(userid)) query = query.Where(u => u.Userid == userid);
if (minAge != null) query = query.Where(u => u.Age >= minAge.Value);
if (maxAge != null) query = query.Where(u => u.Age <= maxAge.Value);
return query.ToList();
}
Here GetUsers() is supposed to return an IEnumerable. Can be a Table if you're using LINQ-to-SQL, ChildNodes() if you're using LINQ-to-XML. Try to enumerable the least possible before applying your where clauses.
I assume that you want to build the where part of your Linq query dynamically - I don't recommend that via a string (see link below) because it could introduce something similar to what is known as SQL injection to LINQ i.e. change of behaviour of the LINQ query with user-supplied params in ways you don't want...
That said check this link out http://weblogs.asp.net/scottgu/archive/2008/01/07/dynamic-linq-part-1-using-the-linq-dynamic-query-library.aspx - Scott Gu demonstrates something that comes close to what you describe including a library and some sample code...
What you want to do is have a method with the folowing signature
public void Filter(Func<Account, bool> filterExpression)
{
list.Where(filterExpression).ToList();
}
this way you can support filtering for various scenarios on the UI.
Not breaking the encapsulation of your (I assume) repository, that's why I do the ToList() so no IQueryable is sent to the client.