We Keep Coding

How can I pull a repeated where clause expression from linq into a function?

I have a large project where I have dozens of linq statements where I am looking for a matching record by checking several fields to see if they match or both field and compared field are null.
var testRecord = new { firstField = "bob", secondField = (string)null, thirdField = "ross" };
var matchRecord = dataContext.RecordsTable.FirstOrDefault(vi =>
(vi.first == testRecord.firstField || ((vi.first == null || vi.first == string.Empty) && testRecord.firstField == null))
&& (vi.second == testRecord.secondField || ((vi.second == null || vi.second == string.Empty) && testRecord.secondField == null))
&& (vi.third == testRecord.thirdField || ((vi.third == null || vi.third == string.Empty) && testRecord.thirdField == null)));
//do stuff with matchRecord
Ideally I would replace all that duplicated code (used around 50 times across the system I'm working on) with something like the following
Expression<Func<string, string, bool>> MatchesOrBothNull = (infoItem, matchItem) => (
infoItem == matchItem || ((infoItem == null || infoItem == string.Empty) && matchItem == null));
var matchRecord = dataContext.RecordsTable.FirstOrDefault(vi =>
MatchesOrBothNull(vi.first, testRecord.firstField)
&& MatchesOrBothNull(vi.second, testRecord.secondField)
&& MatchesOrBothNull(vi.third, testRecord.thirdField));
//do stuff with matchRecord
My question is two-fold: First, is there a matched or both null function already available? (I've looked without luck).
Second, the code block above compiles, but throws a "no supported translation to sql" error, is there a way to have a function in the where clause? I know that there is a translation because it works if I don't pull it into the function. How can I get that translated?

First of all you can check whether string is null or empty with single code which is called : String.IsNullOrEmpty(vi.first). You need a method like this one :
public bool MatchesOrBothNull(string first,string second){
if(first==second||String.IsNullOrEmpty(first)||String.IsNullOrEmpty(second))
return true;
else return false;
}
You can use it in where clause
var matchRecord = dataContext.RecordsTable.Where(vi =>
MatchesOrBothNull(vi.first, testRecord.firstField)
&& MatchesOrBothNull(vi.second, testRecord.secondField)
&& MatchesOrBothNull(vi.third, testRecord.thirdField)
).FirstOrDefault();

Entity Framework: Count() very slow on large DbSet and complex WHERE clause

I need to perform a count operation on this Entity Framework (EF6) data set using a relatively complex expression as a WHERE clause and expecting it to return about 100k records.
The count operation is obviously where records are materialized and therefore the slowest of operations to take place. The count operation is taking about 10 seconds in our production environment which is unacceptable.
Note that the operation is performed on the DbSet directly (db being the Context class), so no lazy loading should be taking place.
How can I further optimize this query in order to speed up the process?
The main use case is displaying an index page with multiple filter criteria but the the function is also used for writing generic queries to the ParcelOrderstable as required for other operations in the service classes which might be a bad idea resulting in very complex queries resulting from laziness and might potentially be a future problem.
The count is later used for pagination, and a much smaller number of records (e.g. 500) is actually displayed. This is a database-first project using SQL Server.
ParcelOrderSearchModel is a C#-class that serves to encapsualte query parameters and is used exclusively by service classes in order to call the GetMatchingOrdersfunction.
Note that on the majority of calls, the majority of the parameters of ParcelOrderSearchModel will be null.
public List<ParcelOrderDto> GetMatchingOrders(ParcelOrderSearchModel searchModel)
{
// cryptic id known --> allow public access without login
if (String.IsNullOrEmpty(searchModel.KeyApplicationUserId) && searchModel.ExactKey_CrypticID == null)
throw new UnableToCheckPrivilegesException();
Func<ParcelOrder, bool> userPrivilegeValidation = (x => false);
if (searchModel.ExactKey_CrypticID != null)
{
userPrivilegeValidation = (x => true);
}
else if (searchModel.KeyApplicationUserId != null)
userPrivilegeValidation = privilegeService.UserPrivilegeValdationExpression(searchModel.KeyApplicationUserId);
var criteriaMatchValidation = CriteriaMatchValidationExpression(searchModel);
var parcelOrdersWithNoteHistoryPoints = db.HistoryPoint.Where(hp => hp.Type == (int)HistoryPointType.Note)
.Select(hp => hp.ParcelOrderID)
.Distinct();
Func<ParcelOrder, bool> completeExpression = order => userPrivilegeValidation(order) && criteriaMatchValidation(order);
searchModel.PaginationTotalCount = db.ParcelOrder.Count(completeExpression);
// todo: use this count for pagination
}
public Func<ParcelOrder, bool> CriteriaMatchValidationExpression(ParcelOrderSearchModel searchModel)
{
Func<ParcelOrder, bool> expression =
po => po.ID == 1;
expression =
po =>
(searchModel.KeyUploadID == null || po.UploadID == searchModel.KeyUploadID)
&& (searchModel.KeyCustomerID == null || po.CustomerID == searchModel.KeyCustomerID)
&& (searchModel.KeyContainingVendorProvidedId == null || (po.VendorProvidedID != null && searchModel.KeyContainingVendorProvidedId.Contains(po.VendorProvidedID)))
&& (searchModel.ExactKeyReferenceNumber == null || (po.CustomerID + "-" + po.ReferenceNumber) == searchModel.ExactKeyReferenceNumber)
&& (searchModel.ExactKey_CrypticID == null || po.CrypticID == searchModel.ExactKey_CrypticID)
&& (searchModel.ContainsKey_ReferenceNumber == null || (po.CustomerID + "-" + po.ReferenceNumber).Contains(searchModel.ContainsKey_ReferenceNumber))
&& (searchModel.OrKey_Referencenumber_ConsignmentID == null ||
((po.CustomerID + "-" + po.ReferenceNumber).Contains(searchModel.OrKey_Referencenumber_ConsignmentID)
|| (po.VendorProvidedID != null && po.VendorProvidedID.Contains(searchModel.OrKey_Referencenumber_ConsignmentID))))
&& (searchModel.KeyClientName == null || po.Parcel.Name.ToUpper().Contains(searchModel.KeyClientName.ToUpper()))
&& (searchModel.KeyCountries == null || searchModel.KeyCountries.Contains(po.Parcel.City.Country))
&& (searchModel.KeyOrderStates == null || searchModel.KeyOrderStates.Contains(po.State.Value))
&& (searchModel.KeyFromDateRegisteredToOTS == null || po.DateRegisteredToOTS > searchModel.KeyFromDateRegisteredToOTS)
&& (searchModel.KeyToDateRegisteredToOTS == null || po.DateRegisteredToOTS < searchModel.KeyToDateRegisteredToOTS)
&& (searchModel.KeyFromDateDeliveredToVendor == null || po.DateRegisteredToVendor > searchModel.KeyFromDateDeliveredToVendor)
&& (searchModel.KeyToDateDeliveredToVendor == null || po.DateRegisteredToVendor < searchModel.KeyToDateDeliveredToVendor);
return expression;
}
public Func<ParcelOrder, bool> UserPrivilegeValdationExpression(string userId)
{
var roles = GetRolesForUser(userId);
Func<ParcelOrder, bool> expression =
po => po.ID == 1;
if (roles != null)
{
if (roles.Contains("ParcelAdministrator"))
expression =
po => true;
else if (roles.Contains("RegionalAdministrator"))
{
var user = db.AspNetUsers.First(u => u.Id == userId);
if (user.RegionalAdministrator != null)
{
expression =
po => po.HubID == user.RegionalAdministrator.HubID;
}
}
else if (roles.Contains("Customer"))
{
var customerID = db.AspNetUsers.First(u => u.Id == userId).CustomerID;
expression =
po => po.CustomerID == customerID;
}
else
{
expression =
po => false;
}
}
return expression;
}

If you can possibly avoid it, don't count for pagination. Just return the first page. It's always expensive to count and adds little to the user experience.
And in any case you're building the dynamic search wrong.
You're calling IEnumerable.Count(Func<ParcelOrder,bool>), which will force client-side evaluation where you should be calling IQueryable.Count(Expression<Func<ParcelOrder,bool>>). Here:
Func<ParcelOrder, bool> completeExpression = order => userPrivilegeValidation(order) && criteriaMatchValidation(order);
searchModel.PaginationTotalCount = db.ParcelOrder.Count(completeExpression);
But there's a simpler, better pattern for this in EF: just conditionally add criteria to your IQueryable.
eg put a method on your DbContext like this:
public IQueryable<ParcelOrder> SearchParcels(ParcelOrderSearchModel searchModel)
{
var q = this.ParcelOrders();
if (searchModel.KeyUploadID != null)
{
q = q.Where( po => po.UploadID == searchModel.KeyUploadID );
}
if (searchModel.KeyCustomerID != null)
{
q = q.Where( po.CustomerID == searchModel.KeyCustomerID );
}
//. . .
return q;
}

How to overcome foreach loop for list object dynamically

I'm swapping my values in List Object on some conditions and update List Object value.
Currently, what I'm doing is
- Looping on each object through List
- Check If condition is net
- Swap values
public static void SwapMinMaxIfNull<T>(this IEnumerable<T> rows, string reportfor)
{
if (reportfor.Equals("Comparison"))
{
var row = rows as IEnumerable<RiskBoardDataToExport>;
try
{
if (rows.Any())
{
var Tests = row.Where(min => min.MinGaitSpeed == null && min.MaxGaitSpeed != null).ToList();
if (Tests != null)
{
foreach (RiskBoardDataToExport test in Tests)
{
test.MinGaitSpeed = test.MaxGaitSpeed;
test.MaxGaitSpeed = null;
}
}
// again check for next object
Tests = row.Where(min => min.MinTUGTime == null && min.MaxTUGTime != null).ToList();
if (Tests != null)
{
foreach (RiskBoardDataToExport test in Tests)
{
test.MinTUGTime = test.MaxTUGTime;
test.MaxTUGTime = null;
}
}
// again check for next object
Tests = row.Where(min => min.MinBergScoreSpeed == null && min.MaxBergScoreSpeed != null).ToList();
if (Tests != null)
{
foreach (RiskBoardDataToExport test in Tests)
{
test.MinBergScoreSpeed = test.MaxBergScoreSpeed;
test.MaxBergScoreSpeed = null;
}
}
//.. for brevity
}
}
}
Can I do it in better way? I know about PropertyInfo i.e. Can check property name and get value etc, but, not having any hint to get this done.
Thanks

It's not exactly what you're asking for, but you can combine the clauses in your Where statements and then have a few if statements in the body:
public static void SwapMinMaxIfNull(this IEnumerable<RiskBoardDataToExport> rows,
string reportfor)
{
if (rows = null) return;
if (reportfor.Equals("Comparison", StringComparison.OrdinalIgnoreCase))
{
foreach (var row in rows.Where(r =>
(r.MinGaitSpeed == null && r.MaxGaitSpeed != null) ||
(r.MinBergScoreSpeed == null && r.MaxBergScoreSpeed != null) ||
(r.MinBergScoreSpeed == null && r.MaxBergScoreSpeed != null)))
{
if (row.MinGaitSpeed == null)
{
row.MinGaitSpeed = row.MaxGaitSpeed;
row.MaxGaitSpeed = null;
}
if (row.MinTUGTime == null)
{
row.MinTUGTime = row.MaxTUGTime;
row.MaxTUGTime = null;
}
if (row.MinBergScoreSpeed == null)
{
row.MinBergScoreSpeed = row.MaxBergScoreSpeed;
row.MaxBergScoreSpeed = null;
}
}
}
}

As this is an operation where order of the items in the list does not matter, you can easily speed this up by parallelization (you can read up on that here).
So, what you should do, is handle this foreach loop in a parallel way and combine it with Rufus L's optimized code for the fastest result.
var rows = rows.Where(r =>
(r.MinGaitSpeed == null && r.MaxGaitSpeed != null) ||
(r.MinBergScoreSpeed == null && r.MaxBergScoreSpeed != null) ||
(r.MinBergScoreSpeed == null && r.MaxBergScoreSpeed != null))
Parallel.ForEach(rows, (row) => {
{
if (row.MinGaitSpeed == null)
{
row.MinGaitSpeed = row.MaxGaitSpeed;
row.MaxGaitSpeed = null;
}
if (row.MinTUGTime == null)
{
row.MinTUGTime = row.MaxTUGTime;
row.MaxTUGTime = null;
}
if (row.MinBergScoreSpeed == null)
{
row.MinBergScoreSpeed = row.MaxBergScoreSpeed;
row.MaxBergScoreSpeed = null;
}
}
Note that this requires the System.Threading.Tasks namespace, that's where the Parallel class is.

Is it posible to pass a condition that is not Initialized?

Simply put I was wondering if this is possible?
if (Descriptionsearch.Checked)
{
searchResult = searchResultBuilder(a.Description == textBox.Text))
}
else if (titleSearch.Checked)
{
searchReuslt = searchResultBuilder(a.title == textBox.Text))
}
As you can see I am simply sending a condition of a variable that has not yet been initialized but will be at the time of use.
private List<int> searchResultBuilder(Func<bool> condition)
{
foreach (var element in currentPosition.Where(a => condition()))
{
searchResults.Add(currentPosition.IndexOf(element));
}
return searchResults;
}
I simply wanted to know if there is a way to do this.
since people are asking this is the for loop from my original code
foreach(var element in main.currentPosition.Where(a => (a.key != null && main.msgSigCollection1.msgSig[(int)a.key].Description.IndexOf(searchTextBox.Text, StringComparison.OrdinalIgnoreCase) >= 0) || (a.value != null && main.msgSigCollection2.msgSig[(int)a.value].Description.IndexOf(searchTextBox.Text, StringComparison.OrdinalIgnoreCase) >= 0)))
{
searchResults.Add(main.currentPosition.IndexOf(element));
}
where currentPosition is a List<int?,int?>

You need to specify a lambda syntax to achieve what you want.
Ex: (input parameters) => expression
if(Descriptionsearch.checked == true)
searchReuslt = searchResultBuilder(a => a.Description == textBox.Text))
else if(titleSearch.checked == true)
searchReuslt = searchResultBuilder(a => a.title == textBox.Text))
MSDN

It's hard to tell based on your question, but I think what you want is to be passing lambdas, which is not what you're doing currently.
Something like this seems closer to what you want:
if(Descriptionsearch.checked == true)
searchReuslt = searchResultBuilder(a => a.Description == textBox.Text))
else if(titleSearch.checked == true)
searchReuslt = searchResultBuilder(a => a.title == textBox.Text))
private List<int> searchResultBuilder<T>(Func<T, bool> condition){
var searchResults = new List<int>();
foreach (var element in currentPosition.Where(condition))
{
searchResults.Add(currentPosition.IndexOf(element));
}
return searchResults;
}
In truth though, your question should state what you're trying to accomplish, not just how you're attempting to get there. There probably is a much easier way to accomplish everything you're trying to do here with LINQ.

filtering a collection using Linq - how to reuse the same filter

I am filtering a collection and I perform 2 filters that are the same but for different fields.
There must be a way I can reduce the duplication of code here?
The checks are whether a date has been entered, and whether it is before a cut off date entered by the user.
public override IList<Company> Search()
{
var list = CacheObjects.Subcontractors;
this.ApplicationFormReturnedCheck(ref list);
this.ApplicationFormSentCheck(ref list);
}
private void ApplicationFormReturnedCheck(ref IList<Subcontractor> list)
{
if (this.ApplicationFormNotReturnedFlag == true && this.ApplicationFormReturned != null)
{
list =
list.Where(x => x.ApplicationFormReturned == null || x.ApplicationFormReturned < this.ApplicationFormReturned).ToList();
}
else if (this.ApplicationFormNotReturnedFlag == true)
{
list = list.Where(x => x.ApplicationFormReturned == null).ToList();
}
else if (this.ApplicationFormReturned != null)
{
list = list.Where(x => x.ApplicationFormReturned < this.ApplicationFormReturned).ToList();
}
}
private void ApplicationFormSentCheck(ref IList<Subcontractor> list)
{
if (this.ApplicationFormNotSentFlag == true && this.ApplicationFormSent != null)
{
list =
list.Where(x => x.ApplicationFormSent == null || x.ApplicationFormSent < this.ApplicationFormSent).ToList();
}
else if (this.ApplicationFormNotSentFlag == true)
{
list = list.Where(x => x.ApplicationFormSent == null).ToList();
}
else if (this.ApplicationFormSent != null)
{
list = list.Where(x => x.ApplicationFormSent < this.ApplicationFormSent).ToList();
}
}

I would suggest you can do something as simple as have some instances of Func<Subcontractor,bool> which cover your various scenarios. This is the type of Func that the Where method expects
To demonstrate let me take one of your methods and show you how:
private void ApplicationFormReturnedCheck(ref IList<Subcontractor> list)
{
var isFormReturned = new Func<Subcontractor,bool>(
x => x.ApplicationFormReturned != null);
var isBeforeDate = new Func<Subcontractor,bool>(
x => x.ApplicationFormReturned < this.ApplicationFormReturned);
var isFormReturnedOrBeforeDate= new Func<Subcontractor,bool>(
x => isFormReturned(x) || isFormReturnedBeforeDate(x));
if (this.ApplicationFormNotReturnedFlag == true && this.ApplicationFormReturned != null)
{
list = list.Where(isFormReturnedOrBeforeDate).ToList();
}
else if (this.ApplicationFormNotReturnedFlag == true)
{
list = list.Where(isFormReturned).ToList();
}
else if (this.ApplicationFormReturned != null)
{
list = list.Where(isBeforeDate).ToList();
}
}
The other method you've shown, although having similar logic, uses a different set of variables. (ApplicationFormSent in place of ApplicationFormReturned). The way I see it you have two options
Duplicate the above within the other method, using the differing variable names
Use a more complex method whereby you have these 3 Func's outside of the scope of each method, and able to distinguish which variables (*Sent or *Returned) to use.
The problem with 2. above is that as your perceved "reuse" goes up, the readability of your code goes down.
In all honesty, I see no major problem with your original code! Its clear, its pretty concise and its easy to see what it's doing. By wrapping all this logic up with predicates (which could conceivably be elsewhere in the class), you're making it harder to read and harder to maintain.

A Func(T, TResult) can be used to encapsulate common predicate methods. In your case, the Func would need to be initilialized in the constructor since you are using instance members in the filter. Ex:
private readonly Func<Subcontractor, bool> _pred;
public Subcontractor()
{
_pred = x => x.ApplicationFormReturned == null || x.ApplicationFormReturned < this.ApplicationFormReturned;
}
private void ApplicationFormReturnedCheck( ref IList<Subcontractor> list )
{
if( this.ApplicationFormNotReturnedFlag == true && this.ApplicationFormReturned != null )
{
list = list.Where( _pred ).ToList();
}
else if( this.ApplicationFormNotReturnedFlag == true )
{
list = list.Where( x => x.ApplicationFormReturned == null ).ToList();
}
else if( this.ApplicationFormReturned != null )
{
list = list.Where( x => x.ApplicationFormReturned < this.ApplicationFormReturned ).ToList();
}
}