We Keep Coding

Dynamic query execution in Entity Framework Core

I am developing a hospitality domain application (.Net Core 2.2) in which i am developing a reporting module. From a dashboard few filters are available to fetch records from Data Base.
Below is the DTO i am using to contains filters
public class SearchDto
{
public DateTime DateForm {get;set;}
public DateTime DateTo {get;set;}
public DateSearchType SearchType {get;set;}
public string RegionId {get;set;}
public OrdersStatus status {get;set;}
public string PaymentModeTypes {get;set;}
public string channel {get;set;}
}
Here DateSearchType is a enum with value
Start // service Start Date
End // service End Date
Creation // Order Creation Date
Also OrdersStatus (an enum) with values like All , Confirmed , Cancelled , PaymnetFailed etc
PaymentModeTypes can be a single string or comma seprated string for ex : "NetBanking, CreditCard, DebitCard, Cash"
RegionId is also a single string or comma seprated string as "101, 102, 102"
Same for Channel either "Web" or "Web, Mobile"
Curently ef core expressssion i am using is as follow
var v = Database.Orders.List(
x => ((SearchType == DateSearchType.Start) ? x.Services.Any(y => (y.MinStartTime >= DateForm.Date && y.MinStartTime <= DateTo.Date)) || DateForm.Date.Equals(DateTime.MinValue) : true)
&& ((SearchType == DateSearchType.End) ? x.Services.Any(y => y.MaxEndTime >= DateForm.Date && y.MaxEndTime <= DateTo.Date) : true)
&& ((SearchType == DateSearchType.Creation) ? x.BookingDate.Date >= DateForm.Date && x.BookingDate.Date <= DateTo.Date : true)
&& (RegionId.Length == 0 || RegionId.Contains(x.RegionId))
&& (status == OrdersStatus.All || x.Status == status)
&& (string.IsNullOrEmpty(PaymentModeTypes) || PaymentTypes.Contains(x.PaymentType))
&& (string.IsNullOrEmpty(channel) || channels.Contains(x.ChannelName)), //Where
x => x.Guests,
x => x.Services
);
Here Guests and Services are two another tables set as navigation property in Orders Model
This expression is working fine but taking too much time to execute , any good approach to optimize it or right way to rewrite this code ?
Also what is the best practice to exclude any filter if its value is not provided.
Few Filters are not mandatory, they can be supplied or not, so query execution has to be of dynamic nature. current implementation if a filter value is not supplied
&& (string.IsNullOrEmpty(PaymentModeTypes) || PaymentTypes.Contains(x.PaymentType))
Can anybody suggest some good material or any piece of code regarding this so i can optimize it.
Please ignore Typo as i am not habitual to use dark theme
Edit 1:Client Evaluation set as off
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder
.ConfigureWarnings(warnings => warnings.Throw(RelationalEventId.QueryClientEvaluationWarning));
}

There is some inefficiency in the generated SQL which might be causing the performance issue.
Before EF Core one is expected to conditionally chain multiple Where calls or use some predicate builder utility to build conditionally Where predicate having only the necessary conditions.
This normally is not needed in EF Core, because it tries to automatically eliminate such conditions. It does that for logical expressions (|| , &&), but failing to do so for the conditional expressions (? :). So the solution is to replace the later with the equivalent logical expressions.
You are doing that for most of your conditions, but not for the first 3. So replace
x => ((SearchType == DateSearchType.Start) ? x.Services.Any(y => (y.MinStartTime >= DateForm.Date && y.MinStartTime <= DateTo.Date)) || DateForm.Date.Equals(DateTime.MinValue) : true)
&& ((SearchType == DateSearchType.End) ? x.Services.Any(y => y.MaxEndTime >= DateForm.Date && y.MaxEndTime <= DateTo.Date) : true)
&& ((SearchType == DateSearchType.Creation) ? x.BookingDate.Date >= DateForm.Date && x.BookingDate.Date <= DateTo.Date : true)
with
x => (SearchType != DateSearchType.Start || x.Services.Any(y => y.MinStartTime >= DateForm.Date && y.MinStartTime <= DateTo.Date) || DateForm.Date.Equals(DateTime.MinValue))
&& (SearchType != DateSearchType.End || x.Services.Any(y => y.MaxEndTime >= DateForm.Date && y.MaxEndTime <= DateTo.Date))
&& (SearchType != DateSearchType.Creation || x.BookingDate.Date >= DateForm.Date && x.BookingDate.Date <= DateTo.Date)
and see if that helps. For sure the generated SQL will be optimal.
Noticed that you are using different variable names for comma separated strings and Contains filters. So I'm assuming you have something like this
public IEnumerable<string> PaymentTypes => (PaymentModeTypes ?? "").Split(", ");
public IEnumerable<string> channels => (channel ?? "").Split(", ");
which is good and will generate SQL IN (...) conditions when necessary.
You might consider doing the same for regions, e.g. add
public IEnumerable<string> RegionIds => (RegionId ?? "").Split(", ");
and replace
&& (RegionId.Length == 0 || RegionId.Contains(x.RegionId))
with
&& (string.IsNullOrEmpty(RegionId) || RegionIds.Contains(x.RegionId))

Filtering results with Linq

I have two dropdowns on my page. First dropdown shows Authors for books and the other dropdown shows status's i.e Overdue or All.
If they choose Overdue then I need to return all books that have been borrowed more than a week ago so the dueback date (Datetime variable) will be taken into consideration.
I currently have this working correctly filtering on the Author as shown here:
model.ListBooks = (from x in tempModel
where ((x.BookAuthor == model.ListAuthors.SelectedAuthor || model.ListAuthors.SelectedAuthor == null))
select x).ToList(); // Filter the results
But as soon as I pass in the additional search filter I.e status it fails to shows me any books that match the selected Author even though I haven't chosen a status this is what it currently looks like.
model.ListBooks = (from x in tempModel
where (
(x.BookAuthor == model.ListAuthors.SelectedAuthor || model.ListAuthors.SelectedAuthor == null)
&&
(model.BookStatus.SelectedStatusId == (int)Enums.Registration.OverDue && x.DueBack < DateTime.Now.)
)
select x).ToList(); // Filter the results
Can someone see what I'm doing wrong here?

I think your query fails if selected status is not OverDue. In that case you have
where authorFilter && (false && dateFilter)
that gives you false for all books. Thus you have only two statuses, you can just add status.SelectedStatusId != (int)Enums.Registration.OverDue check just as you did with null-check for selected author:
var authors = model.ListAuthors;
var status = model.BookStatus;
model.ListBooks = (from x in tempModel
where (authors.SelectedAuthor == null || x.BookAuthor == authors.SelectedAuthor) &&
(status.SelectedStatusId != (int)Enums.Registration.OverDue || x.DueBack < DateTime.Now)
select x).ToList();
I would use method syntax here to make query more readable:
var books = tempModel; // probably you will need IEnumerable<T> or IQueryable<T> here
if (model.ListAuthors.SelectedAuthor != null)
books = books.Where(b => b.BookAuthor == model.ListAuthors.SelectedAuthor);
if (model.BookStatus.SelectedStatusId == (int)Enums.Registration.OverDue)
books = books.Where(b => b.DueBack < DateTime.Now);
model.ListBooks = books.ToList();

Here:
(model.BookStatus.SelectedStatusId == (int)Enums.Registration.OverDue && x.DueBack < DateTime.Now.)
Should be instead:
(x.BookStatus.SelectedStatusId == (int)Enums.Registration.OverDue && x.DueBack < DateTime.Now.)
Because You like to compare element of LINQ query, not the model.

How to select count of the record including other fields in LINQ

I have following LINQ query. I want to be able to select Count of the record selected as well. For example, select w.EndDate, countOfRecordsSelected.
How do I go about it ?
var ConferenceOrderedByDate = (from c in participant.Conference
where c.Status == Status.Completed
&& c.EndDate.HasValue
&& c.EndDate <=Deadline
&& c.EndDate > MinimumDate
orderby c.EndDate ascending
select c.EndDate);
Thanks in advance!!

var ConferenceOrderedByDate = (from c in participant.Conference
where c.Status == Status.Completed
&& c.EndDate.HasValue
&& c.EndDate <=Deadline
&& c.EndDate > MinimumDate
orderby c.EndDate ascending
select new {
EndDate = c.Detail.EndDate,
Count = c.countOfRecordsSelected
});
this will create an anonymous type and you can access them by simply using ConferenceOrderedByDate.EndDate and ConferenceOrderedByDate.Count.
If you want the amount of records selected by the query, because I'm not sure which count you're referring to, just use
ConferenceOrderedByDate.Count()

Using Linq to filter by List<ListItem>

I am trying to extend my linq query with additional search criteria to filter the data by sending also a List<Listitem> to the function for processing. The List can contain 1 or more items and the objective is to retreive all items which match any criteria.
Since i am sending several search criteria to the function the goal is to make a more accurate filter result the more information i am sending to the filter. If one or several criterias are empty then the filter will get less accurate results.
Exception is raised every time i execute following code, and I cant figure out how to solve the using statement to include the List<ListItem>. Appreciate all the help in advance!
Exception: Unable to create a constant value of type 'System.Web.UI.WebControls.ListItem'. Only primitive types or enumeration types are supported in this context.
using (var db = new DL.ENTS())
{
List<DL.PRODUCTS> products =
(from a in db.PRODUCTS
where (description == null || description == "" ||
a.DESCRIPTION.Contains(description)) &&
(active == null || active == "" || a.ACTIVE.Equals(active, StringComparison.CurrentCultureIgnoreCase)) &&
(mID == null || mID == "" || a.MEDIA_ID == mID) &&
(mID == null || objTypes.Any(s => s.Value == a.OBJECTS)) //Exception here!
select a).ToList<DL.PRODUCTS>();
return products;
}

Pass collection of primitive values to expression:
using (var db = new DL.ENTS())
{
var values = objTypes.Select(s => s.Value).ToArray();
List<DL.PRODUCTS> products =
(from a in db.PRODUCTS
where (description == null || description == "" || a.DESCRIPTION.Contains(description)) &&
(active == null || active == "" || a.ACTIVE.Equals(active, StringComparison.CurrentCultureIgnoreCase)) &&
(mID == null || mID == "" || a.MEDIA_ID == mID) &&
(mID == null || values.Contains(a.OBJECTS))
select a).ToList<DL.PRODUCTS>();
return products;
}
That will generate SQL IN clause.
Note - you can use lambda syntax to compose query by adding filters based on some conditions:
var products = db.PRODUCTS;
if (!String.IsNullOrEmpty(description))
products = products.Where(p => p.DESCRIPTION.Contains(description));
if (!String.IsNullOrEmpty(active))
products = products.Where(p => p.ACTIVE.Equals(active, StringComparison.CurrentCultureIgnoreCase)));
if (!String.IsNullOrEmpty(mID))
products = products.Where(p => p.MEDIA_ID == mID);
if (mID != null)
products = products.Where(p => values.Contains(p.OBJECTS));
return products.ToList();

Linq isn't able to convert the predicate on ListItem to something useful to Sql.
I would suggest that you pre-project the values of the ListItems into a simple List<string> before using this with Contains (which is converted to IN)
var listValues = objTypes.Select(_ => _.Value).ToList();
List<DL.PRODUCTS> products = ...
listValues.Contains(a.OBJECTS))

Linq parser issue?

Well,
I am not sure if I am wrong or if the linq parsed mistakes, but the following linq query returns what I DON'T want if I don't use additional parenthesis.
int? userLevel;
Guid? supervisorId;
List<int> levelsIds = new List<int>();
string hierarchyPath;
// init the vars above
// ...
var qry = from f in items
where userLevel.HasValue
? (f.IsMinLevelIdNull() || (levelsIds.Contains(f.MinLevelId)))
: true
&& supervisorId.HasValue
? (f.IsSupervisorIdNull() || (!f.ChildrenVisibility && (f.SupervisorId == supervisorId.Value))
|| (f.ChildrenVisibility && (hierarchyPath.IndexOf(f.SupervisorId.ToString()) >= 0)))
: true
select f;
The idea here is to run a query in AND between two blocks of conditions 'activated' by the presence of the variables 'userLevel' and 'supervisorId'.
For example, if both userLevel and supervisoId are null the query becomes:
var qry = from f in items
where true && true
select f;
Well, this query works well only if I enclose in additional parentheses the 2 trigraphs:
var qry = from f in items
where (userLevel.HasValue
? (f.IsMinLevelIdNull() || (levelsIds.Contains(f.MinLevelId)))
: true)
&& (supervisorId.HasValue
? (f.IsSupervisorIdNull() || (!f.ChildrenVisibility && (f.SupervisorId == supervisorId.Value))
|| (f.ChildrenVisibility && (hierarchyPath.IndexOf(f.SupervisorId.ToString()) >= 0)))
: true)
select f;
The question is: why the extra parenthesis are required? My opinion is that there is a problem in the linq parser.

From 7.2.1 Operator precedence and associativity && gets evaluated before ? :

The additional parentheses would be required as the precedence is evaluated in the wrong order for example your code would be evaluated as the following because there is no break between true && supervisorId.HasValue...
var qry = from f in items
where
1st: userLevel.HasValue
?
2nd: (f.IsMinLevelIdNull() || (levelsIds.Contains(f.MinLevelId)))
:
3rd: true && supervisorId.HasValue
? (f.IsSupervisorIdNull() || (!f.ChildrenVisibility && (f.SupervisorId == supervisorId.Value))
|| (f.ChildrenVisibility && (hierarchyPath.IndexOf(f.SupervisorId.ToString()) >= 0))) **)**
: true
select f;