We Keep Coding

Using dynamic filters with Entity Framework Core

I'm developing an application (.Net Core 3.1, C# 8) that is using Entity Framework Core.
I would like to filter a table with several filtering options.
I'm getting the filter conditions in JSON and I'm deserializing it into an object. I want to write a where LINQ query which will filter the table based on those dynamic filtering options.
The twist is I need to manage the filtering with many options and combinations.
You can filter for the market, country, vendor and the rest of the filter options will be null.
You would like to filter for country and vendor then the market will be null and also the rest of the filter options.
I'm querying a huge table, so it is important to write a query which translates fully into SQL.
The following code is not working properly. I'm looking for something similar that can solve this issue:
var filters = new demoFilterEntity()
{
Market = new List<string>() { "LAT", "NAM" }
};
var filteredData = demoMainRepository.GetAll().Where(x =>
x.Market != null && (filters.Market != null ? filters.Market.Contains(x.Market) : false) &&
x.Country != null && (filters.Country != null ? filters.Country.Contains(x.Market) : false)).ToList();
I would appreciate suggestions on how I can get through this and manage the filtering dynamically.

If you have only AND conditions you can do it with just chaining Where clauses:
var query = demoMainRepository.GetAll().Where(x => x.Market != null);
if(filters.Market != null)
{
query = query.Where(x => filters.Market.Contains(x.Market));
}
...
var filteredData = query.ToList();
Also as #Lajos Arpad said maybe you need to consider combining fields null checks (i.e. x.Market != null) with the filters checks:
var query = demoMainRepository.GetAll();
if(filters.Market != null)
{
query = query.Where(x => x.Market != null && filters.Market.Contains(x.Market));
}
...
var filteredData = query.ToList();

I'm currently researching this topic and have found dynamic lambda expressions is a thing
The .Where method accepts Expression<Func<Entity, bool>> type which you can create dynamically.
Kind of cumbersome but it works good once you wrap your head around it
I recommend this article in Codemag Dynamic Lambda Expressions

What will be the equivalent sql query for this code?

This code is giving inaccurate results and I need to change it but I do not understand what the whole code is doing.
I read about defaultifempty and its implementation. I read the documentation but I could not find the answer to the question.
var records = from entity in _db.InventoryDetails
from am in _db.AccountMeters
.Where(am => entity.SerialNumber == (am.SerialNumber ?? am.RemoteId.ToString())).DefaultIfEmpty()
from ac in _db.Accounts
.Where(ac => ac.AccountId == am.AccountId).DefaultIfEmpty()
from i in _db.Inventories
.Where(idd => idd.ProjectId == projectid)
.Where(idd => idd.InventoryId == entity.InventoryId)
from u in _db.Users
.Where(e => e.Id == (entity.InstallerId ?? entity.PrevInstaller)).DefaultIfEmpty()
It does not give error just the query is giving wrong result. If I can find sql equivalent for the code then I can find out what he was trying to do and then find out business requirement and re code.

You can get SQL code which will be generated by your code through calling ToString() method:
var records = from entity in _db.InventoryDetails
// the other code is omitted for the brevity
var sqlCode = records.ToString(); // here your generated SQL code

With EF 6 and beyond, you can use
context.Database.Log = Console.Write;
to write the SQL to the console.

Entity Framework - Linq to Entities - Optional filter

I am struggling to figure out how to get a LINQ statement to produce a specific WHERE clause in SQL in a single statement.
I am after it producing something like this:
SELECT ColA, ColB, ColC, ColN...
FROM Orders
WHERE Client = #ClientId
AND (#CompanyId IS NULL OR #CompanyId = CompanyId)
My (failing) LINQ statement looks like this:
var includeAllCompanies = company == null;
var data = context.Orders.Where(o => o.Client.Id == clientId
&& (includeAllCompanies
|| (c.Company != null && c.Company.Id == company.Id)).ToList();
However, it always throws an exception when the variable company is NULL (it works fine when it has been initialised). The exception being:
Non-static method requires a target.
My current fix is to split my LINQ statement into two. One using an Expression<Func<>> (to be transformed to a SQL statement with partial filtering). Then another that uses Func<> to perform the remaining filters on the returned list.
Expression<Func<>> to let SQL do some of the work (excluding nullable objects)
var data = context.Orders.Where(o => o.Client.Id == clientId).ToList();
Func<> to then filter out the nullable objects
data = data.Where(c => (territory == null
|| (c.Territory != null && c.Territory.Id == territory.Id))).ToList();
This works, however, I want SQL to be performing this query.

The problem is that, company is server-side variable. Regardles includeAllCompanies value, EF has to translate whole LINQ query to SQL - and in this case SQL doesn't know what is company.Id - so EF has to always get company.Id value in order to put into SQL query. Even if company is null (so that is why you get exception). I hope you see my point, if not - I'll try to give some sample.
In order get rid of exception you can do the following:
var companyId = company == null ? null : (int?)company.Id;
var data = context.Orders.Where(o => o.Client.Id == clientId
&& (companyId == null
|| (c.Company != null && c.Company.Id == companyId)).ToList();

LINQ to Entities does not recognize the method IsNullOrWhiteSpace

I have the below code:
var countries = from c in db.Countries
where (string.IsNullOrWhiteSpace(searchAlpha2) || (c.Alpha2 ?? string.Empty).ToUpper().Contains(searchAlpha2.ToUpper()))
&& (string.IsNullOrWhiteSpace(searchAlpha2) || (c.Alpha3 ?? string.Empty).ToUpper().Contains(searchAlpha3.ToUpper()))
&& (string.IsNullOrWhiteSpace(searchName) || (c.Name ?? string.Empty).ToUpper().Contains(searchName.ToUpper()))
select c;
This code uses Entity Framework v6 Code First over a SQL database.
Aside from performance, if I don't include the IsNullOrWhitespace I get no results when the filter criteria are blank (I've tested both null and blank values); however when a value is present this works as expected.
I'm getting the error:
LINQ to Entities does not recognize the method 'Boolean IsNullOrWhiteSpace(System.String)' method, and this method cannot be translated into a store expression.
I'm trying to use the searchXXX strings to filter on columns. I've tried using RegEx.IsMatch, SqlMethods.Like, and the code below, but all give me errors saying those functions are not allowed (errors come from either EntityFramework.SqlServer or from Linq to Entities). I've seen numerous posts on here where this has been done successfully though - so wonder if I'm missing something fundamental?

If you want to use your statement in current form you might want to replace
string.IsNullOrWhiteSpace(searchAlpha2)
to
!(searchAlpha2 == null || searchAlpha2.Trim() == string.Empty)
and all the other values too, for it to get translated to working SQL.
Update: Copied from comment by #DavidKempfner
As of EntityFramework.6.2.0 it generated SQL that checked for
!(searchAlpha2.Trim() == string.Empty),
I.E. It ignored the searchAlpha2 == null || part.
Use this instead:
!string.IsNullOrEmpty(entity.searchAlpha2.Trim())

I would suggest a different approach - use the ability to build queries up on the fly, and thus avoid passing optional query parameters to the expressions altogether - this will result in improved query plans when parsed to sql and executed on the database.
Also, if your database (?SqlServer) is not set to case sensitive collation (i.e. xx_CI_xx), you can avoid the casing conversion as well, as it is redundant:
var myQueryable = db.Countries.AsQueryable();
if (!string.IsNullOrWhiteSpace(searchAlpha2))
{
myQueryable = myQueryable.Where(c => c.Alpha2.Contains(searchAlpha2));
}
...
var countries = myQueryable.ToList();
You can get this and a bunch more functionality using PredicateBuilder
Update
JB: based on StuartLC's answer, here's the code amended to use PredicateBuilder:
var predicate = PredicateBuilder.True<Country>();
if (!string.IsNullOrWhiteSpace(searchAlpha2))
predicate = predicate.And(c => c.Alpha2 != null ? c.Alpha2.Contains(searchAlpha2) : false);
if (!string.IsNullOrWhiteSpace(searchAlpha3))
predicate = predicate.And(c => c.Alpha3 != null ? c.Alpha3.Contains(searchAlpha3) : false);
if (!string.IsNullOrWhiteSpace(searchName))
predicate = predicate.And(c => c.Name != null ? c.Name.Contains(searchName) : false);
IQueryable<Country> countries = db.Countries.AsExpandable().Where(predicate);

when you use linq in Entity Framework to get data from DataBase you need to use only with functions that the Entity Framework can convert to sql query.

I know that there an already accepted answer for this question but I got an idea to share.
Instead of putting multiple checks in the LINQ or using if statement to apply where clause on list result, you can use a simple trick. Just declare a bool variable and assign IsNullOrWhitespace of add to linq like:
bool isNull = string.IsNullOrWhiteSpace(searchAlpha2);
var countries = db.Countries.Where(c => isNull || c.Alpha2.Contains(searchAlpha2)).ToList();

Conditional OR has no effect in Linq-to-entities?

I stumbled across a null ref exception today in one of my Linq-To-Entitites queries and wondered how that could be possible. It seems an conditional OR used as an OR-gate has no effect in Linq-To-Entities. A simplified example of my query is this:
from a in db.Articles
where a.Author == "John Doe"
&& (tag == null || a.Tags.Any(t => t.TagName == tag.TagName))
select a;
Now when tag is NULL, the right side of the where query still gets executed and a NULL reference exception occurs on tag.TagName. Maybe this is because Linq-To-Entities always translates the complete statement into SQL?
Anyhow ... how to get around this issue?
Thanks very much :)

var query = db.Articles.Where(x => x.Author == "John Doe");
query = tag == null
? query
: query.Where(x => x.TagName == tag.TagName);
or:
var query = from a in db.Articles
where a.Author == "John Doe"
select a;
query = tag == null
? query
: from a in query
where a.Tags.Any(t => t.TagName == tag.TagName)
select a;
Personally, I find the first one cleaner

Think about sql. Linq converts your code as a whole into sql query passing 'outside' objects as parameters and evaluating them. That's why it fails on evaluation of your null object.
It's a good practice to construct a linq query bit by bit based on conditions to reduce number of unnecessary code in resulted query, so it's better to split your query:
var query = db.Articles.Where(x => x.Author == "John Doe");
if( tag != null)
query = query.Where(x => x.TagName == tag.TagName);
Because your query will be evaluated and executed on selection you are welcome to add more conditions without worrying about multiple requests.