We Keep Coding

C# Linq to EF Separate Subquery into Expression

I have a complex where clause in my EF linq statement which repeats a subquery expression, on _db.OPESRRecoveryElements, but with different parameters, one of which is depending on records from the main entity, OPCases/OPCaseDto.
The query as it is works, but its hard for people to read. Ideally I'd like to be able to create an expression which could be re-used at the 3 necessary points and would still allow it to execute as a single, server-side SQL statement.
Is there a way to create an Expression / IQueryable definition which can be used for a subquery like this?
List<OPCaseDto> opCases = await _db.OPCases
.ProjectTo<OPCaseDto>(_autoMapperConfig, null, requestedExpands)
.Where(c =>
c.OPStatusId == OPStatusIds.AwaitingRecoveryElement
&& (
(c.OPCategoryLetter == "B"
// Only need a gross pensionable element if there is an outstanding gross pensionable figure
&& (c.GrossOverpaidPensionable - c.GrossRecoveredPensionable == 0
|| _db.OPESRRecoveryElements.Any(e => !e.NonPensionable && e.OPRecoveryMethod.OPTypeLetter == "G"
&& !e.OPRecoveryPlans.Any(rp
=> (rp.RecoveryStatus == OPRecoveryStatuses.NotStarted || rp.RecoveryStatus == OPRecoveryStatuses.InRecovery)
&& rp.AssignmentNo == c.RecoveryAssignmentNo)))
// Only need a gross non-pensionable element if there is an outstanding gross non-pensionable figure
&& (c.GrossOverpaidNonPensionable - c.GrossRecoveredNonPensionable == 0
|| _db.OPESRRecoveryElements.Any(e => e.NonPensionable && e.OPRecoveryMethod.OPTypeLetter == "G"
&& !e.OPRecoveryPlans.Any(rp
=> (rp.RecoveryStatus == OPRecoveryStatuses.NotStarted || rp.RecoveryStatus == OPRecoveryStatuses.InRecovery)
&& rp.AssignmentNo == c.RecoveryAssignmentNo))))
|| (c.OPCategoryLetter == "D"
// Don't need to check for an outstanding net figure - if the case is net and isn't complete, there will be one!
&& _db.OPESRRecoveryElements.Any(e => e.OPRecoveryMethod.OPTypeLetter == "N"
&& !e.OPRecoveryPlans.Any(rp
=> (rp.RecoveryStatus == OPRecoveryStatuses.NotStarted || rp.RecoveryStatus == OPRecoveryStatuses.InRecovery)
&& rp.AssignmentNo == c.RecoveryAssignmentNo)))
)
)
.AsNoTracking()
.ToListAsync();
If it wasn't for the c.RecoveryAssignmentNo part, I could easily create an expression like:
public Expression<Func<OPESRRecoveryElement, bool>> NoActiveRecoveryPlans(string opType, bool nonPen)
{
return e => e.OPRecoveryMethod.OPTypeLetter == opType
&& e.NonPensionable == nonPen
&& !e.OPRecoveryPlans.Any(rp
=> (rp.RecoveryStatus == OPRecoveryStatuses.NotStarted || rp.RecoveryStatus == OPRecoveryStatuses.InRecovery));
}
and use it like:
(c.OPCategoryLetter == "B"
// Only need a gross pensionable element if there is an outstanding gross pensionable figure
&& (c.GrossOverpaidPensionable - c.GrossRecoveredPensionable == 0
|| _db.OPESRRecoveryElements.Any(NoActiveRecoveryPlans("G", false)))
and it would get executed before the query to get the OPCases.
I could also fetch all the OPCaseDto records and OPESRRecoveryElements as separate queries and filter in memory, but I don't want to do that.
If I add a parameter to the function, string assignmentNo, I (unsurprisingly) get an error - "Unable to cast object of type 'System.Linq.Expressions.InstanceMethodCallExpression3' to type 'System.Linq.Expressions.LambdaExpression'"

LINQ equivalent of SQL NOT IN statement

I have a query that in T-SQL it is
SELECT *
FROM rpm_scrty_rpm_usr ru
WHERE ru.inact_ind = 'N'
AND email_id IS NOT NULL
AND wwid IS NULL
AND LTRIM(RTRIM (email_id)) <> ''
AND dflt_ste_id NOT IN (25,346,350,352,353,354,355,357,358,366,372,411)
When I have been converting it to LINQ, I have everything except the "NOT IN"
var querynonSystem = (from ru in Rpm_scrty_rpm_usrs
where ru.Inact_ind == "N" && ru.Email_id != null && ru.Wwid == null && ru.Email_id.Trim() != ""
&& ru.Dflt_ste_id != 25
select ru).Count();
I did temporarily put in this line && ru.Dflt_ste_id != 25
However I need to have AND dflt_ste_id NOT IN (25,346,350,352,353,354,355,357,358,366,372,411)
I am seeing a lot of different code like
this lambda where !(list2.Any(item2 => item2.Email == item1.Email))
Then var otherObjects = context.ItemList.Where(x => !itemIds.Contains(x.Id));
For my linq query, how can I do this Not In in simple manner?

You can use Contains with !. In addition, if you just want to count rows, you can use Count.
var ids = new List<int> {25, 346, 350, 352, 353, 354, 355, 357, 358, 366, 372, 411};
var querynonSystem = XXXcontext.Rpm_scrty_rpm_usrs.Count(x =>
x.Inact_ind == "N" &&
x.Email_id != null &&
x.Wwid == null &&
x.Email_id.Trim() != "" &&
!ids.Contains(x.Dflt_ste_id));
From comment: if you want to retrieve all, you can still use Where and Select.
var querynonSystem = XXXcontext.Rpm_scrty_rpm_usrs.Where(x =>
x.Inact_ind == "N" &&
x.Email_id != null &&
x.Wwid == null &&
x.Email_id.Trim() != "" &&
!ids.Contains(x.Dflt_ste_id)).Select(x => x).ToList();
FYI: you cannot call Rpm_scrty_rpm_usrs table class to query. Instead, you need DbContext or some other repository.

There is no "not in" operator unless the type of the query is the same as the type you want to filter against (in which case you could use except). Here it is not. You're working on an IEnumerable and you want to filter on it's ID so a list of int. The where with lambda and contains is your best bet and WILL be translated to an in on the SQL side by most providers.
var FilterIds = new List<int>{1,2,3,4,344,3423525};
var querynonSystem = (from ru in Rpm_scrty_rpm_usrs
where ru.Inact_ind == "N" && ru.Email_id != null && ru.Wwid == null && ru.Email_id.Trim() != ""
&& ru.Dflt_ste_id != 25
select ru)
// Use this
.Where(ru=>!FilterIds.Any(id=>ru.dflt_ste_id ==id))
// Or this
.Where(ru=>!FilterIds.Contains(ru.dflt_ste_id))
.Count();

Entity Framework causing Timeout Error

I am working with the following Entity Framework query. I know there's a lot going on here but am hoping it's clear enough that someone might be able to spot the issue.
var lineItems = from li in Repository.Query<CostingLineItem>()
let cid = (li.ParentCostingPackage != null) ?
li.ParentCostingPackage.ParentCostingEvent.ProposalSection.Proposal.Costing.Id :
li.ParentCostingEvent.ProposalSection.Proposal.Costing.Id
where cid == costingId &&
li.OriginalProductId.HasValue &&
(li.Quantity.HasValue && li.Quantity.Value > 0) && // li.QuantityUnitMultiplier
Classifications.Contains(li.OriginalProduct.ClassificationEnumIndex)
let selectedChoiceId = li.OriginalPackageOptionId.HasValue ?
(from c in li.OriginalPackageOption.CostingLineItems
orderby (c.IsIncluded ?? false) ? -2 : (c.IsDefaultItem ?? false) ? -1 : c.Id
select (int)c.OriginalPackageOptionChoiceId).FirstOrDefault() :
0
where selectedChoiceId == 0 || (li.OriginalPackageOptionChoiceId.HasValue && li.OriginalPackageOptionId.Value == selectedChoiceId)
let hasProviderAvailable = li.OriginalProductItem.ProductItemVendors.Any(
piv => piv.ProductPricings.Any(pp => pp.ProductItemVendor.CompanyId != null || pp.ProductItemVendor.HotelId != null))
select new
{
LineItem = li,
ProductItem = li.OriginalProductItem,
Product = li.OriginalProduct,
Vendors = li.CostingLineItemVendors,
HasProviderAvailable = hasProviderAvailable
};
As is, this query generates the following run-time error:
The wait operation timed out
If I change the section that declares selectedChoiceId to the following, the error goes away:
let selectedChoiceId = 0
Can anyone see how that code is consistently causing a time-out error?
(Note: This code is part of a large application that has been running for several years. So I really don't think this has anything to do with the connection string or anything like that. If I make the change above, it works consistently.)

The query can be simplified in a number of ways, which should make it easier to optimize by the database engine.
Firstly, you can remove a number of null checks (HasValue), because they're not relevant in SQL, but they do bloat the generated SQL.
Secondly, I think this check involving selectedChoiceId can be greatly simplified. This is what I think the statement could look like:
from li in Repository.Query<CostingLineItem>()
let cid = (li.ParentCostingPackage != null) ?
li.ParentCostingPackage.ParentCostingEvent.ProposalSection.Proposal.Costing.Id :
li.ParentCostingEvent.ProposalSection.Proposal.Costing.Id
where cid == costingId &&
li.OriginalProductId.HasValue &&
li.Quantity > 0 && // no null check
Classifications.Contains(li.OriginalProduct.ClassificationEnumIndex)
let selectedChoiceId = (from c in li.OriginalPackageOption.CostingLineItems
orderby c.IsIncluded ? -2 : c.IsDefaultItem ? -1 : c.Id // no null checks
select (int)c.OriginalPackageOptionChoiceId).FirstOrDefault()
where !li.OriginalPackageOptionId.HasValue || li.OriginalPackageOptionId == selectedChoiceId
let hasProviderAvailable = li.OriginalProductItem.ProductItemVendors.Any(
piv => piv.ProductPricings.Any(pp => pp.ProductItemVendor.CompanyId != null || pp.ProductItemVendor.HotelId != null))
select new
{
LineItem = li,
ProductItem = li.OriginalProductItem,
Product = li.OriginalProduct,
Vendors = li.CostingLineItemVendors,
HasProviderAvailable = hasProviderAvailable
}
For the rest, of course there are the usual suspects. Better indexes may become more important as the database volume increases. Checking for (and fixing) database fragmentation can also have a significant impact.

I think this will give you a better performance but not sure if it'll fix the problem :
let selectedChoiceId = li.OriginalPackageOptionId.HasValue
? (from c in li.OriginalPackageOption.CostingLineItems
let cOrder = (c.IsIncluded ?? false) ? -2 : (c.IsDefaultItem ?? false) ? -1 : c.Id
orderby cOrder
select (int) c.OriginalPackageOptionChoiceId).FirstOrDefault()
: 0

How to use array in where condition in Linq queries

This is my linq query.
var data =
(from obj in lstEventsDataForGrid
where obj.strDateCode.Contains(thisWeekend[0] == null ? "" : thisWeekend[0])
|| obj.strDateCode.Contains(thisWeekend[1] == null ? "$" : thisWeekend[1])
|| obj.strDateCode.Contains(thisWeekend[2] == null ? "$" : thisWeekend[2])
&& strArr.Contains(obj.strEventType.ToString().ToUpper())
orderby obj.dtmStartDate ascending
select obj).GroupBy(x => x.strEventCode)
.Select(y => y.First()).ToArray();
Expected Result
It should not come whether the strEventType is not in the strArr.
But it is coming even that type is not in the array.
Issue I noticed is if I remove one where condition i.e that obj.strDateCode.Contains(...) the other condition is working.
Where am I going wrong? Please suggest something!

I rewrote your query using null-coalesce operators to make it more readable. I also added line numbers to point out what I think is wrong here:
1. var data = (
2. from obj in lstEventsDataForGrid
3. where obj.strDateCode.Contains(thisWeekend[0] ?? "") ||
4. obj.strDateCode.Contains(thisWeekend[1] ?? "$") ||
5. obj.strDateCode.Contains(thisWeekend[2] ?? "$") &&
6. strArr.Contains(obj.strEventType.ToString().ToUpper())
7. orderby obj.dtmStartDate ascending
8. select obj
9. ).GroupBy(x => x.strEventCode).Select(y => y.First()).ToArray();
You need to change the following lines:
3. where (obj.strDateCode ... // add '('
5. ... thisWeekend[2] ?? "$")) && // add ')'
This way, your && will overpower the rest of the conditions.

I think you're missing some parentheses. Do you mean to treat the three || conditions as one option? As in
where (A || B || C) && D
Try put one after where here:
where (obj.strDateCode.Contains(thisWeekend[0]
and a second one here:
: thisWeekend[2]))

Your where predicate contains an error:
obj.strDateCode.Contains(thisWeekend[0] == null ? "" : thisWeekend[0])
|| obj.strDateCode.Contains(thisWeekend[1] == null ? "$" : thisWeekend[1])
|| obj.strDateCode.Contains(thisWeekend[2] == null ? "$" : thisWeekend[2])
&& strArr.Contains(obj.strEventType.ToString().ToUpper())
This is an expression of the form:
where Condition1 || Condition2 || Condition3 && Condition4.
In C#, the && operator takes precedence over the || operator, so this is equivalent to
where Condition1 || Condition2 || (Condition3 && Condition4).
In this situation, Condition4 is only evaluated if Condition3 is true. If Condition1 or Condition2 are true, the entire predicate will return true and the remainder of the expression will short-circuit.
What you probably intended was:
where (Condition1 || Condition2 || Condition3) && Condition4
Or, extended to your example:
(obj.strDateCode.Contains(thisWeekend[0] == null ? "" : thisWeekend[0])
|| obj.strDateCode.Contains(thisWeekend[1] == null ? "$" : thisWeekend[1])
|| obj.strDateCode.Contains(thisWeekend[2] == null ? "$" : thisWeekend[2])
) && strArr.Contains(obj.strEventType.ToString().ToUpper())
This will ensure that an obj will not be returned if it is not contained within strArr,
which your question indicates is the required result.

Conditional LINQ where statement?

I have a linq statement that I want to add an additional where clause to if a drop down index is not 0.
people.Where(n.surname == "surname" || n.forename == "forename" && (dropdown.SelectedIndex > 0) ? n.id = dropdown.SelectedValue : n.id > 0).Select(n => n);
I am not even sure if what I am trying is possible??
I would like to do this rather than having to write two different statements.
Any ideas?
Thanks

Fortunately, this is easy because queries compose:
var query = people.Where(n.surname == "surname" || n.forename == "forename");
if (dropdown.SelectedIndex > 0)
{
query = query.Where(n => n.id.ToString() == dropdown.SelectedValue);
}