We Keep Coding

How can I pull a repeated where clause expression from linq into a function?

I have a large project where I have dozens of linq statements where I am looking for a matching record by checking several fields to see if they match or both field and compared field are null.
var testRecord = new { firstField = "bob", secondField = (string)null, thirdField = "ross" };
var matchRecord = dataContext.RecordsTable.FirstOrDefault(vi =>
(vi.first == testRecord.firstField || ((vi.first == null || vi.first == string.Empty) && testRecord.firstField == null))
&& (vi.second == testRecord.secondField || ((vi.second == null || vi.second == string.Empty) && testRecord.secondField == null))
&& (vi.third == testRecord.thirdField || ((vi.third == null || vi.third == string.Empty) && testRecord.thirdField == null)));
//do stuff with matchRecord
Ideally I would replace all that duplicated code (used around 50 times across the system I'm working on) with something like the following
Expression<Func<string, string, bool>> MatchesOrBothNull = (infoItem, matchItem) => (
infoItem == matchItem || ((infoItem == null || infoItem == string.Empty) && matchItem == null));
var matchRecord = dataContext.RecordsTable.FirstOrDefault(vi =>
MatchesOrBothNull(vi.first, testRecord.firstField)
&& MatchesOrBothNull(vi.second, testRecord.secondField)
&& MatchesOrBothNull(vi.third, testRecord.thirdField));
//do stuff with matchRecord
My question is two-fold: First, is there a matched or both null function already available? (I've looked without luck).
Second, the code block above compiles, but throws a "no supported translation to sql" error, is there a way to have a function in the where clause? I know that there is a translation because it works if I don't pull it into the function. How can I get that translated?

First of all you can check whether string is null or empty with single code which is called : String.IsNullOrEmpty(vi.first). You need a method like this one :
public bool MatchesOrBothNull(string first,string second){
if(first==second||String.IsNullOrEmpty(first)||String.IsNullOrEmpty(second))
return true;
else return false;
}
You can use it in where clause
var matchRecord = dataContext.RecordsTable.Where(vi =>
MatchesOrBothNull(vi.first, testRecord.firstField)
&& MatchesOrBothNull(vi.second, testRecord.secondField)
&& MatchesOrBothNull(vi.third, testRecord.thirdField)
).FirstOrDefault();

Entity Framework: Count() very slow on large DbSet and complex WHERE clause

I need to perform a count operation on this Entity Framework (EF6) data set using a relatively complex expression as a WHERE clause and expecting it to return about 100k records.
The count operation is obviously where records are materialized and therefore the slowest of operations to take place. The count operation is taking about 10 seconds in our production environment which is unacceptable.
Note that the operation is performed on the DbSet directly (db being the Context class), so no lazy loading should be taking place.
How can I further optimize this query in order to speed up the process?
The main use case is displaying an index page with multiple filter criteria but the the function is also used for writing generic queries to the ParcelOrderstable as required for other operations in the service classes which might be a bad idea resulting in very complex queries resulting from laziness and might potentially be a future problem.
The count is later used for pagination, and a much smaller number of records (e.g. 500) is actually displayed. This is a database-first project using SQL Server.
ParcelOrderSearchModel is a C#-class that serves to encapsualte query parameters and is used exclusively by service classes in order to call the GetMatchingOrdersfunction.
Note that on the majority of calls, the majority of the parameters of ParcelOrderSearchModel will be null.
public List<ParcelOrderDto> GetMatchingOrders(ParcelOrderSearchModel searchModel)
{
// cryptic id known --> allow public access without login
if (String.IsNullOrEmpty(searchModel.KeyApplicationUserId) && searchModel.ExactKey_CrypticID == null)
throw new UnableToCheckPrivilegesException();
Func<ParcelOrder, bool> userPrivilegeValidation = (x => false);
if (searchModel.ExactKey_CrypticID != null)
{
userPrivilegeValidation = (x => true);
}
else if (searchModel.KeyApplicationUserId != null)
userPrivilegeValidation = privilegeService.UserPrivilegeValdationExpression(searchModel.KeyApplicationUserId);
var criteriaMatchValidation = CriteriaMatchValidationExpression(searchModel);
var parcelOrdersWithNoteHistoryPoints = db.HistoryPoint.Where(hp => hp.Type == (int)HistoryPointType.Note)
.Select(hp => hp.ParcelOrderID)
.Distinct();
Func<ParcelOrder, bool> completeExpression = order => userPrivilegeValidation(order) && criteriaMatchValidation(order);
searchModel.PaginationTotalCount = db.ParcelOrder.Count(completeExpression);
// todo: use this count for pagination
}
public Func<ParcelOrder, bool> CriteriaMatchValidationExpression(ParcelOrderSearchModel searchModel)
{
Func<ParcelOrder, bool> expression =
po => po.ID == 1;
expression =
po =>
(searchModel.KeyUploadID == null || po.UploadID == searchModel.KeyUploadID)
&& (searchModel.KeyCustomerID == null || po.CustomerID == searchModel.KeyCustomerID)
&& (searchModel.KeyContainingVendorProvidedId == null || (po.VendorProvidedID != null && searchModel.KeyContainingVendorProvidedId.Contains(po.VendorProvidedID)))
&& (searchModel.ExactKeyReferenceNumber == null || (po.CustomerID + "-" + po.ReferenceNumber) == searchModel.ExactKeyReferenceNumber)
&& (searchModel.ExactKey_CrypticID == null || po.CrypticID == searchModel.ExactKey_CrypticID)
&& (searchModel.ContainsKey_ReferenceNumber == null || (po.CustomerID + "-" + po.ReferenceNumber).Contains(searchModel.ContainsKey_ReferenceNumber))
&& (searchModel.OrKey_Referencenumber_ConsignmentID == null ||
((po.CustomerID + "-" + po.ReferenceNumber).Contains(searchModel.OrKey_Referencenumber_ConsignmentID)
|| (po.VendorProvidedID != null && po.VendorProvidedID.Contains(searchModel.OrKey_Referencenumber_ConsignmentID))))
&& (searchModel.KeyClientName == null || po.Parcel.Name.ToUpper().Contains(searchModel.KeyClientName.ToUpper()))
&& (searchModel.KeyCountries == null || searchModel.KeyCountries.Contains(po.Parcel.City.Country))
&& (searchModel.KeyOrderStates == null || searchModel.KeyOrderStates.Contains(po.State.Value))
&& (searchModel.KeyFromDateRegisteredToOTS == null || po.DateRegisteredToOTS > searchModel.KeyFromDateRegisteredToOTS)
&& (searchModel.KeyToDateRegisteredToOTS == null || po.DateRegisteredToOTS < searchModel.KeyToDateRegisteredToOTS)
&& (searchModel.KeyFromDateDeliveredToVendor == null || po.DateRegisteredToVendor > searchModel.KeyFromDateDeliveredToVendor)
&& (searchModel.KeyToDateDeliveredToVendor == null || po.DateRegisteredToVendor < searchModel.KeyToDateDeliveredToVendor);
return expression;
}
public Func<ParcelOrder, bool> UserPrivilegeValdationExpression(string userId)
{
var roles = GetRolesForUser(userId);
Func<ParcelOrder, bool> expression =
po => po.ID == 1;
if (roles != null)
{
if (roles.Contains("ParcelAdministrator"))
expression =
po => true;
else if (roles.Contains("RegionalAdministrator"))
{
var user = db.AspNetUsers.First(u => u.Id == userId);
if (user.RegionalAdministrator != null)
{
expression =
po => po.HubID == user.RegionalAdministrator.HubID;
}
}
else if (roles.Contains("Customer"))
{
var customerID = db.AspNetUsers.First(u => u.Id == userId).CustomerID;
expression =
po => po.CustomerID == customerID;
}
else
{
expression =
po => false;
}
}
return expression;
}

If you can possibly avoid it, don't count for pagination. Just return the first page. It's always expensive to count and adds little to the user experience.
And in any case you're building the dynamic search wrong.
You're calling IEnumerable.Count(Func<ParcelOrder,bool>), which will force client-side evaluation where you should be calling IQueryable.Count(Expression<Func<ParcelOrder,bool>>). Here:
Func<ParcelOrder, bool> completeExpression = order => userPrivilegeValidation(order) && criteriaMatchValidation(order);
searchModel.PaginationTotalCount = db.ParcelOrder.Count(completeExpression);
But there's a simpler, better pattern for this in EF: just conditionally add criteria to your IQueryable.
eg put a method on your DbContext like this:
public IQueryable<ParcelOrder> SearchParcels(ParcelOrderSearchModel searchModel)
{
var q = this.ParcelOrders();
if (searchModel.KeyUploadID != null)
{
q = q.Where( po => po.UploadID == searchModel.KeyUploadID );
}
if (searchModel.KeyCustomerID != null)
{
q = q.Where( po.CustomerID == searchModel.KeyCustomerID );
}
//. . .
return q;
}

Generic filtering on query string parameters

I'm looking to convert our current API architecture to a more generic query-able structure that doesn't require us to hard card each possible parameter and build up a custom query. Here is an example of what our current architecture looks like,
[HttpGet]
[Route("Companies/{id:int}/Implementations/Tasks")]
public async Task<IApiResponse<List<CompanyIPTDTO>>> GetCompanyIPTs(int? id = null,
int? taskId = null, int? completedById = null, DateTime? dateCompletedStart = null,
DateTime? dateCompletedEnd = null, int page = 1, int? pageSize = null)
{
// Instantiate generic query
Expression<Func<CompanyIPT, bool>> query = null;
// Check parameters and build the lambda query
// Check if the parameter is given and if the query field is already filled so as to either set or add a query
if (id != null && query != null) query = query.And(t => t.CompanyID == id);
else if (id != null && query == null) query = t => t.CompanyID == id;
if (taskId != null && query != null) query = query.And(t => t.ProcessTaskID == taskId);
else if (taskId != null && query == null) query = t => t.ProcessTaskID == taskId;
if (completedById != null && query != null) query = query.And(t => t.CompletedByID == completedById);
else if (completedById != null && query == null) query = t => t.CompletedByID == completedById;
if ((dateCompletedStart != null && dateCompletedEnd != null) && query != null) query = query.And(a => a.DateCompleted >= dateCompletedStart && a.DateCompleted <= dateCompletedEnd);
else if ((dateCompletedStart != null && dateCompletedEnd != null) && query == null) query = a => a.DateCompleted >= dateCompletedStart && a.DateCompleted <= dateCompletedEnd;
// Execute the GET with the appended query and paging information
var result = _uow.CompanyIPTRepository.List(filter: query, page: page, pageSize: pageSize);
int totalPossibleRecords = _uow.CompanyIPTRepository.GetTotalRecords(filter: query);
return new ApiSuccessResponse<List<CompanyIPTDTO>>(HttpStatusCode.OK, result.Select(x => Mapper.Map<CompanyIPTDTO>(x)).ToList(), result.Count(), Request.RequestUri, totalPossibleRecords, pageSize);
}
And as you can see this gets very cumbersome for a large API on every get request to have to customize the query checks. I assume there must be some way to do this generically based on a query string parameter that comes in. For example in this case I would love to see a request come in that looks like,
https://www.example.com/api/Companies/15/Implementations/Tasks?q=[completedById=5,dateCompletedStart=1/15/18,dateCompletedEnd=1/17/18]
And after coming in like this I assume we could build something that uses Reflection to look at the object and verify these fields exist and build up a generic query to hit our DB?
Anyone know where to point us in the right direction?

linq conditional query

What would be the best practice for setting a status depending on several other "columns" retrieved in a linq query.
var result = (from q in query
select new Item
{
ApprovedDate = q.ApprovedDate,
CreatedDate = q.CreatedDate,
DeclinedDate = q.DeclinedDate,
Status = 0
});
I'd like to set the status to either 0, 1, 2.
(ApprovedDate == null and DeclinedDate == null) --> 0
(ApprovedDate != null and DeclinedDate == null) --> 1
(DeclinedDate != null) --> 3
So perhaps something like:
var result = (from q in query
select new Item
{
ApprovedDate = q.ApprovedDate,
CreatedDate = q.CreatedDate,
DeclinedDate = q.DeclinedDate,
Status = (q.CreatedDate == null && q.DeclinedDate == null) ? 0 : (q.ApprovedDate != null && q.DeclinedDate == null) ? 1 : 2
});
I might add even more status combinations, so should I try and do this in the linq select query, in my repository object.. Or later on in the controller where I would do a .ToList() and then foreach the list to set the correct status code?
Having even more than 3 statuscodes, the linq query gets "hard" to read.

What about moving status calculation to Item class? If status property depends on other properties value, then it's definitely calculated property:
var result = from q in query
select new Item
{
ApprovedDate = q.ApprovedDate,
CreatedDate = q.CreatedDate,
DeclinedDate = q.DeclinedDate
});
And
public class Item
{
// other properties
public int Status
{
get
{
if (ApprovedDate == null and DeclinedDate == null)
return 0;
if (ApprovedDate != null and DeclinedDate == null)
return 1;
if (DeclinedDate != null)
return 3;
// etc
}
}
}
Actually I think it's best option, because in this case status calculation logic will be close to required data. If (for some reason) you can't use this approach, then move setting statuses to local items collection:
var items = result.ToList().ForEach(i => i.Status = CalculateStatus(i));

Maybe wrapped all in a function An do a linq like this
var result = (from q in query sele q).AsEnumerable()
.Select( x => new Item()
{
ApprovedDate = x.ApprovedDate,
CreatedDate = x.CreatedDate,
DeclinedDate = x.DeclinedDate,
Status = MyStatusFunction(x.CreatedDate,q.DeclinedDate)
});
public int MyStatusFunction(DateTime ApprovedDate , Datetime DeclinedDate)
{
if (ApprovedDate == null and DeclinedDate == null) return 0;
else if(ApprovedDate != null and DeclinedDate == null) return 1;
else if (DeclinedDate != null) return 3;
}

omit nullable types in linq-sql query

I have some code like this trying to get some data from a Documents table based on some filters passed in (ContentRef and TypeRef)...
public IQueryable<Document> GetFilteredDocuments(bool archived, int? ContentRef, int? TypeRef)
{
return from document in db.Documents
where document.IsArchived == archived
&& (document.ContentRef == ContentRef)
&& (document.TypeRef == TypeRef )
select document;
}
if either ContentRef or TypeRef are null then i dont want it to do a check for if its null i just want it ignored.
eg if both are null my method should return the equiavalent of
return from document in db.Documents
where document.IsArchived == archived
select document;
how can i do this?

Try this:
public IQueryable<Document> GetFilteredDocuments(bool archived, int? ContentRef, int? TypeRef)
{
return from document in db.Documents
where document.IsArchived == archived
&& (ContentRef == null || document.ContentRef == ContentRef)
&& (TypeRef == null || document.TypeRef == TypeRef )
select document;
}
When ContentRef is null, the document.ContentRef == ContentRef part will not be evaluated.

With deferred execution, you can construct your query like this because the execution only occurs when GetEnumerator is called.
public IQueryable<Document> GetFilteredDocuments(bool archived, int? ContentRef, int? TypeRef)
{
IQueriable<Document> docs = db.Documents.Where(d => d.IsArchived == archived);
if (ContentRef != null)
docs = docs.Where(d => d.ContentRef == ContentRef);
if (TypeRef != null)
docs = docs.Where(d => d.TypeRef == TypeRef);
return docs;
}