We Keep Coding

"Execution Timeout" on converting LINQ query results using ToList()

As the title states, I'm getting a "Wait operation timed out" message (inner exception message: "Timeout expired") on a module I'm maintaining. Everytime the app tries to convert the query results using ToList(), it times out regardless of the number of results.
Reason this needs to be converted to list: Results needed to be exported to Excel for download.
Below is the code:
public Tuple<IEnumerable<ProductPriceSearchResultDto>, int> GetProductPriceSearchResults(ProductPriceFilterDto filter, int? pageNo = null)
{
//// Predicate builder
var predicate = GetProductPriceSearchFilter(filter);
//// This runs for approx. 1 minute before throwing a "Wait operation timed out" message...
var query = this.GetProductPriceSearchQuery()
.Where(predicate)
.Distinct()
.OrderBy(x => x.DosageFormName)
.ToList();
return Tuple.Create<IEnumerable<ProductPriceSearchResultDto>, int>(query, 0);
}
My query:
var query = (from price in this.context.ProductPrice.AsExpandable()
join product in this.context.vwDistributorProducts.AsExpandable()
on price.DosageFormCode equals product.DosageFormCode
join customer in this.context.vwCustomerBranch.AsExpandable()
on price.CustCd equals customer.CustomerCode
where price.CountryId == CurrentUserService.Identity.CountryId && !product.IsInactive
select new { price.PriceKey, price.EffectivityDateFrom, price.ContractPrice, price.ListPrice,
product.DosageFormName, product.MpgCode, product.DosageFormCode,
customer.CustomerName }).GroupBy(x => x.DosageFormCode)
.Select(x => x.OrderByDescending(y => y.EffectivityDateFrom).FirstOrDefault())
.Select(
x =>
new ProductPriceSearchResultDto
{
PriceKey = x.PriceKey,
DosageFormCode = x.DosageFormCode,
DosageFormName = x.DosageFormName,
EffectiveFrom = x.EffectivityDateFrom,
Price = x.ListPrice,
MpgCode = x.MpgCode,
ContractPrice = x.ContractPrice,
CustomerName = x.CustomerName
});
return query;
Notes:
ProductPrice is a table and has a non-clustered index pointing at columns CountryId and DosageFormCode.
vwDistributorProducts and vwCustomerBranch are views copied from the client's database.
I'm already at my wit's end. How do I get rid of this error? Is there something in the code that I need to change?
Edit: As much as possible, I don't want to resort to setting a command timeout because 1.) app's doing okay without it by far...except for this function and 2.) this is already a huge application and I don't want to possibly put the other modules' performances at risk.
Any help would be greatly appreciated. Thank you.

I'd try and log the sql this translates into.
The actual sql may then be used to get the query plan, which may lead you closer to the root cause.

LINQ subquery returns brackets and fieldnames, how to get just the value?

My LINQ query is the following, I query two tables, Settlements and Bills which have a one to many relationship, in particular there can be one settlement with one or more bills.
var TheSettlements = from settlement in context_.Settlements
select new
{
asettlementid = settlement.SettlementId,
SqBills = string.Join(",",
(
from b in context_.Bills
.Where(b => b.SettlementId == settlement.SettlementId)
select new { b.BillMunicipalityId }
))
};
Now the trivial part (for me), is that I would like the bills concatenated, so after many hours of trial and error, I got my results but the Bills(BillMunicipalityId) are presented inside brackets including the fieldname, like this.
The way I export the data, to a txt to be more precise, is this.
foreach (var settlement in TheSettlements)
{
SettlementsText
.Append(settlement.asettlementid).Append(Delimiter)
.Append(settlement.SqBills.ToString()).Append(Delimiter)
.Append(Newline);
}
And the results I get in the txt.
3,{ BillMunicipalityId = f9e47f81-fc97-4008-b93d-d384230c53aa },
6,,
7,{ BillMunicipalityId = 8b66610a-20c1-4f47-9f37-489d1a8ce31a },{ BillMunicipalityId = 003d59d4-7bcb-4603-b42c-dc389dd8fb06 },{ BillMunicipalityId = 0070bb29-e3a1-4317-b5e2-3d1ef08dd20b },
How should I handle this to get only the values?
Just the GUID of every BillMunicipalityId, without the { BillMunicipalityId = } part.

I think rather than selecting as a new object you could just select the value like this:
var TheSettlements = from settlement in context_.Settlements
select new
{
asettlementid = settlement.SettlementId,
SqBills = string.Join(",",
(
from b in context_.Bills
.Where(b => b.SettlementId == settlement.SettlementId)
select b.BillMunicipalityId
))
};

Your Linq statement looks really strange to me. As it shows in the question you're mixing Linq with extensions methods.
If context is a DbContext which is going to the database, concating the results with string.Join won't work as this statement can't be translated to SQL code. If context however contains in memory data this may work. I advise however to not use string.Join within Linq unless you add a clear comment to the code, this Linq should never hit the database.
When this code will hit the database, you'll get an NotSupportedException with the message:
LINQ to Entities does not recognize the method 'System.String Join[Int32]'
The second thing I notice in your query, normally the one-to-many relation is known by the datamodel and you shouldn't need to join the results yourself.
The easiest way to solve this, is to use an intermediate query, which gets the results from the database and after running the query and getting the data into memory, perform the conversion with string.Join()
This would look like:
var TheSettlements =
from settlement in context_.Settlements
select new
{
asettlementid = settlement.SettlementId,
SqBills = (
from b in settlement.Bills
select b.BillMunicipalityId
).ToList(),
};
// Get the results in memory:
var results = TheSettlements.ToArray();
// Format the results:
var printResults = results.Select(s =>
s.asettlementid.ToString() + ", " + string.Join(", ",s.SqBills));

Entity Framework Group By Then Order By

I'm generating a query using Entity Framework which uses a group by clause and then attempts to order each of the groups to get specific data. I attempted to optimize the order by to only happen once using a let statement but the results are incorrect but the query still executes.
Concept:
var results =
(from n in noteEntities.NoteLog
where associatedIDs.Contains(n.AssociatedID)
group n by n.AssociatedID into gn
let ogn = gn.OrderByDescending(t => t.CreatedDateTime)
let successNote = ogn.FirstOrDefault(x => x.Type == "Success")
let lastStatusNote = ogn.FirstOrDefault()
select new { Success = successNote, Status = lastStatusNote, AssociatedID = gn.Key }).ToList();
However, the problem is that using, what should be the ordered let variable, ogn in the subsequent let statements is not using an order by descending list and I'm getting the wrong success and status notes. I've also tried changing things up to create a sub-query and reference the result but that doesn't seem to return an ordered list either, ex:
var subQuery =
(from n in noteEntities.NoteLog
where associatedIDs.Contains(n.AssociatedID)
group n by n.AssociatedID into gn
select gn.OrderByDescending(t => t.CreatedDateTime));
var results =
(from s in subQuery
let successNote = s.FirstOrDefault(x => x.Type == "Success")
let lastStatusNote = s.FirstOrDefault()
select new { Success = successNote, Status = lastStatusNote }).ToList();
I can make this work by using OrderByDescending twice in the select statement or let statements for the success and status notes but this becomes very slow, and redundant, when there are a lot of notes. Is there a way to run the order by only once and get the right results back?

In SQL a subquery with Order By must have a TOP statement (yours does not). And when Linq detects that there is no FirstOrDefault or Takestatements with the ordered subquery it just strips the OrderByDescending.
If you are having a performance problem with the query perhaps you should look into indexing the table.

LINQ to Entities group by and Count()

I have the following LINQ-to-Entities query
from r in ctx.Rs
join p in ctx.Ps on r.RK equals p.RK
group r by r.QK into gr
select new { QK = (int)gr.Key, Num = gr.Count() }
that runs against this schema
Table P Table R Table Q
PK*
RK ----> RK*
Text QK ------> QK*
Text Text
and gives this message if there is any record in Q with no corresponding record in P: "The cast to value type 'Int32' failed because the materialized value is null. Either the result type's generic parameter or the query must use a nullable type."
The problem is the gr.Count() in the last line, but I cannot find a solution. I have tried to test gr for null, but cannot find a way that works.
I have seen a number of solutions to a similar problem using Sum() instead of Count(), but I have not been able to adapt them to my problem.
I tried changing my query to look like the one in Group and Count in Linq issue, but I just got a different message.
I also looked at Group and Count in Entity Framework (and a number of others) but the problem is different.

group Key can't be null
var results = ctx.Rs.Where(r => r.QK != null)
.GroupBy(r => r.QK)
.Select(gr => new { Key = (int)gr.Key, Count = gr.Count() }
.ToList();
PS.
Mostly, You don't need 'JOIN' syntax in Entity Framework. see: Loading Related Entities
Writing descriptive-meaningful variable names would significantly improve Your codes and make it understandable. Readability does matter in real world production.

I'm having trouble reading your format. But can you try:
from r in ctx.Rs
join p in ctx.Ps.DefaultIfEmpty() on r.RK equals p.RK
group r by r.QK into gr
select new { QK = (int)gr.Key, Num = gr.Count(x => x.RK != null) }
With DefaultIfEmpty and x => x.RK != null being the changes.

Using "Match" in a Linq statement

I have a table that has two records (there will be many at runtime). The deviceId of the records are, “DEVICE1” and “DEVICE2”. I want to use a regular expression to extract records.
The code below compiles but fails to return a result. When I hover the cursor on the “devices.ToList()” statement I get the following error:
base {System.SystemException} = {"LINQ to Entities does not recognize the method 'System.Text.RegularExpressions.MatchCollection Matches(System.String)' method, and this method cannot be translated into a store expression."}”
Can anyone show me how I can modify my query so that this would return records based on the expression?
filterText = #"DEVICE.";
Regex searchTerm = new Regex(filterText);
using (var ctx = new MyEntities())
{
var devices = from d in ctx.Devices
let matches = searchTerm.Matches(d.DeviceId)
where matches.Count > 0
select ((Device)d);
return devices.ToList();
}

I don't believe you can use regular expressions with LINQ to Entities. However, it looks like you're just trying to find devices which start with "DEVICE", so the query would be:
return ctx.Devices.Where(d => d.DeviceId.StartsWith("DEVICE"))
.ToList();
EDIT: If you actually need the flexibility of a regular expression, you should probably first fetch the device IDs (and only the device IDs) back to the client, then perform the regular expression on those, and finally fetch the rest of the data which matches those queries:
Regex regex = new Regex(...);
var deviceIds = ctx.Devices.Select(d => DeviceId).AsEnumerable();
var matchingIds = deviceIds.Where(id => regex.IsMatch(id))
.ToList();
var devices = ctx.Devices.Where(d => matchingIds.Contains(d.DeviceId));
That's assuming it would actually be expensive to fetch all the data for all devices to start with. If that's not too bad, it would be a simpler option. To force processing to be performed in process, use AsEnumerable():
var devices = ctx.Devices.AsEnumerable()
.Where(d => regex.IsMatch(d.DeviceId))
.ToList();

You should always remember that your LinqToEntities queries must be translated to SQL queries. Since SQL Server has no support for regular expressions, this can not work.
As suggested in the comment by Paul Ruane, StartsWith will work. This can be translated by LinqToEntities into WHERE DeviceId LIKE 'DEVICE%'.
If StartsWith isn't enough because you may need to look for strings in the middle of database columns, Contains will also work:
var devices = from d in ctx.Devices
where d.DeviceId.Contains("DEVICE")
select d;
This will result in the following: WHERE DeviceId LIKE '%DEVICE%'.

Remember when using Entity Framework or Linq to Sql that your query ends up being translated to SQL. SQL doesn't understand your regular expression object, and can't use its matches on the server side. To use your RegEx easily you could instead retrieve all the devices from the server first, and then use your existing logic. e.g.
using (var ctx = new MyEntities())
{
var devices = from Device d in ctx.Devices select d;
// Retrieve all the devices:
devices = devices.ToList();
devices = from d in devices
let matches = searchTerm.Matches(d.DeviceId)
where matches.Count > 0
select ((Device)d);
return devices.ToList();
}
This does carry the cost of retrieving more data than you need, potentially much more. For complex logic you may want to consider a Stored Procedure, which could potentially use the same RegEx via CLR functions.

LinqToEntities does not support pushing Regex's down to the database. Just do Contains (which gets converted to sql... where DeviceId Like '%DEVICE%').
filterText = #"DEVICE.";
using (var ctx = new MyEntities())
{
var devices = from d in ctx.Devices
d.DeviceId.Contains(filterText)
select d;
return devices.ToList();
}