We Keep Coding

Entity Framework object LINQ query Timeout issues

I've attempted to modify my connection string to include an extended timeout and I've confirmed that on the sql server side the view that feeds my EF Object executes within seconds and returns a total of 3000 or less records.
BUT when I attempt to run it via code I am now running into Timeout issues and I was seeking some advice to fix this issue. I get "Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding."
Most solutions I find on the specific error recommend connection string modifications OR something along this.context.CommandTimeout... which I cannot figure out how to use in this situation.
I've included the Method I use to acquire the desired data. If there is a more efficient way please let me know.
The input arguments are:
int? inputSKU = null
int? inputStoreNum = null
DateTime? inputStartDate = null
The intent is to return the full list.
And it hangs at, because it skips all the conditional bits:
var qUniqueOffers = query.GroupBy(q => q.Plan_Number).ToList();
Thank you.
private List<PromotionItem> QueryPromotion(int? inputSKU, int? inputStoreNum, DateTime? inputStartDate)
{
log.Info("Client requested QueryPromotion");
List<PromotionItem> resultQuery = new List<PromotionItem>();
try
{
using (DWH_Entities db = new DWH_Entities())
{
var query = db.vw_Web_Promotion.AsQueryable();
// filter promotion results that don't match SKU#
if (inputSKU != null)
query = query.Where(q => q.Sku_Number == inputSKU);
// filter promotion results that don't match Store Num
if (inputStoreNum != null)
query = query.Where(q => q.Store_Number == inputStoreNum);
// filter promotion results that don't match Promotion Start Date
if (inputStartDate != null)
query = query.Where(q => q.Start_Date >= inputStartDate);
// Group promotions By Plan Number ('Promotion ID')
var qUniqueOffers = query
.GroupBy(q => q.Plan_Number)
.ToList();
// Select first from each group to get unique details
var qOffers = qUniqueOffers
.Select(g => g.OrderBy(gi => gi.Plan_Number).First())
.ToList();
foreach (var qo in qOffers)
{
resultQuery.Add(new PromotionItem
{
PromotionNumber = qo.Plan_Number.Trim(),
PromotionDescription = qo.Plan_Description.Trim(),
StartDate = qo.Start_Date,
EndDate = qo.End_Date
});
}
}
}
catch (Exception e)
{
log.Error("[" + e.TargetSite + "] | " + e.Message);
throw e;
}
return resultQuery;
}

If you are using latest EF version do the following to increase timeout:
using (DWH_Entities db = new DWH_Entities())
{
db.Database.CommandTimeout = 300;
...
If you want records in the minimum time, try following:
var temp = query.ToList();
var qUniqueOffers = temp.GroupBy(q => q.Plan_Number)
.ToList();

// Group promotions By Plan Number ('Promotion ID')
var qUniqueOffers = query
.GroupBy(q => q.Plan_Number)
.ToList();
// Select first from each group to get unique details
var qOffers = qUniqueOffers
.Select(g => g.OrderBy(gi => gi.Plan_Number).First())
.ToList();
The way you have written the above LINQ means you are pulling a lot of data over the wire (the first ToList) and then getting a subset of the data (using First and the second ToList). Consider changing it to:
// Group promotions By Plan Number ('Promotion ID')
var qUniqueOffers = query
.GroupBy(q => q.Plan_Number)
// Select first from each group to get unique details
var qOffers = qUniqueOffers
.Select(g => g.OrderBy(gi => gi.Plan_Number).First())
.ToList();
This should result in much less data being sent from the database - which will hopefully make it faster.
As https://stackoverflow.com/a/13827077/34092 states:
ToList() always forces everything ahead of it to evaluate immediately,
as opposed to deferred execution.

EF6 cascade object very slow to setup

I have a problem with entity framework, when trying to retreive data.
I split the work into multiple steps:
generate the query
Execute it and retreive dataset from db.
Fill in my ViewModel using dataset.
Actualy, step 1 and 2 are very fast, step 3 can take up to 1 minute (for 200 records). Meaning it's not SQL related (i copied query from debugger to MSSMS, and it's executed in less than a second).
First i was using step 3B, to make it simple, i retreive a Job entity which i transform into a MapMarker object. and i thaught it was the ConvertAll that was slowing down the process.
After some SO reading, I tested using Select but result was the same.
The only this is, if i use the "main object", in this example: Job, everything is fast; as a test, i put the Job.Job_ID into all field, and the execution time is normal (less than a second).
Then i insert again this: ,Latitude = _Job.Maintenance.Equipement.Location.GPS.GPS_Latitude.Value and the slowlyness is back.
I even tried step 3C using a foreach loop (which i knew was not better but ok...) but it is as slow as other solutions.
The main question is:
What am i missing in the EF6 configuration (or somewhere else ?) that make this process so slow ?
I'm going to do it the good old way and execute my own sql query, i started using EF, i guess these entities should be usable, for now using "simple" objects works realy fine but if you cannot cascade them.. what is the added value ?
Below the steps i'm talking about.
Step 1:
IEnumerable<Job> Jobs = db.Job.Include(e => e.Maintenance.MaintenancePlan.MaintenanceType).Include(e => e.Maintenance.MaintenancePlan.MaintenanceType)
.Include(e => e.Maintenance.MaintenancePlan.MaintenanceType.Shape)
.Include(e => e.Maintenance.MaintenanceStatus)
.Include(e => e.Users)
.Include(e => e.Users.Color)
.Include(e => e.Maintenance.Equipement.Location.GPS);
Step 2:
List<Job> listJobs = Jobs.ToList();
Step 3A:
IEnumerable<MapMarker> IEMarkerJobsA = Jobs.AsEnumerable().Select(_Job => new MapMarker
{
ID = string.Format("Job_{0}", _Job.Job_ID)
,Latitude = _Job.Maintenance.Equipement.Location.GPS.GPS_Latitude.Value
,Longitude = _Job.Maintenance.Equipement.Location.GPS.GPS_Longitude.Value
});
List<MapMarker> listMarkerJobsA = IEMarkerJobsA.ToList();
Step 3B:
IEnumerable listMarkerJobs = listJobs.ConvertAll(
new Converter(MapMarker.MapMarkerFactory));
When the Factory is like this:
public static MapMarker MapMarkerFactory(Job _Job)
{
MapMarker A = new MapMarker();
A.ID = String.Format("Job_{0}", _Job.Job_ID);
A.Latitude = _Job.Maintenance.Equipement.Location.GPS.GPS_Latitude.Value;
A.Longitude = _Job.Maintenance.Equipement.Location.GPS.GPS_Longitude.Value;
A.Title = String.Format("{1}", (_Job.Users != null) ? String.Format("[{0}]", _Job.Users.Users_NickName) : "", _Job.Maintenance.Equipement.Equipement_Name);
A.Icon = GetIconePath((_Job.Users != null) ? _Job.Users.Color.Color_Name : "red", _Job.Maintenance.MaintenancePlan.MaintenanceType.Shape.Shape_Name, _Job.Maintenance.MaintenanceStatus.MaintenanceStatus_Description, "13px");
A.IconSize = new Size(13, 13);
A.WindowInfoContent = String.Format("JobID= {0}", _Job.Job_ID);
return A;
}
Step 3C:
List<MapMarker> listMarkerJobs = new List<MapMarker>();
foreach (Job _Job in Jobs)
{
MapMarker A = new MapMarker();
A.ID = String.Format("Job_{0}", _Job.Job_ID);
A.Latitude = _Job.Maintenance.Equipement.Location.GPS.GPS_Latitude.Value:
A.Longitude = _Job.Maintenance.Equipement.Location.GPS.GPS_Longitude.Value;
A.Title = String.Format("{1}", (_Job.Users != null) ? String.Format("[{0}]", _Job.Users.Users_NickName) : "", _Job.Maintenance.Equipement.Equipement_Name);
A.Icon = MapMarker.GetIconePath((_Job.Users != null) ? _Job.Users.Color.Color_Name : "red", _Job.Maintenance.MaintenancePlan.MaintenanceType.Shape.Shape_Name, _Job.Maintenance.MaintenanceStatus.MaintenanceStatus_Description, "13px");
A.IconSize = new System.Drawing.Size(13, 13);
A.WindowInfoContent = String.Format("JobID= {0}", _Job.Job_ID);
listMarkerJobs.Add(A);
}

Try this for part 3
List<MapMarker> listMarkerJobs = Jobs.AsNoTracking().Select(_Job => new MapMarker
{
ID = string.Format("Job_{0}", _Job.Job_ID),
Latitude = _Job.Maintenance.Equipement.Location.GPS.GPS_Latitude.Value,
Longitude = _Job.Maintenance.Equipement.Location.GPS.GPS_Longitude.Value
}).ToList();

The query processor ran out of internal resources and could not produce a query plan in EF

I have a query in EF where there is a List of string value that it checks for existence in another table.
Please consider the below query for more details.
Code
List<string> ItmsStock = item.Select(ds => ds.ItemNum).ToList(); // Currently, This List items count is 80,000 records.
this.Db.Database.CommandTimeout = 180;
var existsStckList = Db.Stocktakes.Where(ds => ItmsStock.Contains(ds.ItemNo)).Select(ds => ds.ItemNo).ToList();
item.RemoveAll(ds => existsStckList.Contains(ds.ItemNum));
var ItmsExists = Db.Items.Where(ds => ItmsStock.Contains(ds.ItemNo)).Select(ds => ds.ItemNo).ToList();
ItmsExists = Db.Stocktakes.Where(ds => !ItmsExists.Contains(ds.ItemNo)).Select(ds => ds.ItemNo).ToList();
I searched on the internet and found the converted sql uses IN to check for existence. so, the limit for the IN makes the problem. My question here is, How can I efficiently perform the above actions without using for loop.
I ll be appreciating you, If anybody can help me out.
Edit
Previously, I had the below code. After facing the performance issue with the below code, I wrote the above one.
foreach (var stockitems in item)
{
if (Db.Stocktakes.Any(a => a.ItemNo == stockitems.ItemNum))
{
StockResult ss = new StockResult();
ss.ItemNumber = stockitems.ItemNum;
ss.FileName = stockitems.FileName;
Stockres.Add(ss);
}
else if (!Db.Stocktakes.Any(a => a.ItemNo == stockitems.ItemNum) && Db.Items.Any(a => a.ItemNo == stockitems.ItemNum))
{
var ItemNo = stockitems.ItemNum;
var AdminId = Convert.ToInt32(Session["AccId"]);
var CreatedOn = System.DateTime.Now;
int dbres = Db.Database.ExecuteSqlCommand("insert into Stocktake values({0},{1},{2})", ItemNo, AdminId, CreatedOn);
Db.SaveChanges();
totalcount = totalcount + 1;
}
else
{
StockResult sss = new StockResult();
sss.ItemNumber = stockitems.ItemNum;
sss.FileName = stockitems.FileName;
Stockitemsdup.Add(sss);
}
}
Thanks.

Issue batches of 1000 item IDs to the database, or use native SQL and submit a table-valued parameter, or a temp table filled with SqlBulkCopy.
I'm surprised you got htis particular message. The parameter limit is about 2000 parameters. Your query should have been rejected.

Nested query in entity framework

I am getting the following exception:
The nested query is not supported. Operation1='Case' Operation2='Collect'
with this query
var Games = context.Games.Select(a => new GameModel
{
Members = (a.Type == 1 ? (a.UsersInGames.Where(b => b.GameID == a.ID && b.StatusID == 1).Select(c => new Member
{
ID = c.UserID,
email = c.UserInfo.EmailAddress,
screenName = c.UserInfo.ScreenName
})) :
(a.Teams.Where(b => b.GameID == a.ID).SelectMany(b => b.UsersInTeams.Where(c => c.StatusID == 1)).Select(d => new Member
{
ID = d.UserID,
email = d.UserInfo.EmailAddress,
screenName = d.UserInfo.ScreenName
)))
})
when I don't include the condition in selecting Members, the query works fine. Is there a way I can do the conditional inside the query?

You're overestimating the power of LINQ translation to SQL. Not everything is translatable and there is no compiler warning for that due to the way LINQ works.
Nested collections are usually either a) not supported or b) end up in horrible SELECT N+1 queries. What you ask EF to do is to return an object tree. SQL does not support tree like results so you run into the object-relational impedance mismatch and it hurts.
I advise you to fetch the nested collection data as a second, completely separate query. That allows you more control and is guaranteed to work.
As a non-essential side-note, you will probably not be able to convince EF to use the ?: operator over sequences. That is very hard to translate. Think how you would write this as SQL - very hard and convoluted.

It looks like Linq to EF doesn't support the following
context.Games.Select(g => new
{
Field = g.IsX? queryable1 : queryable2
});
But, here's a hack you can use to get it to work:
context.Games.Select(g => new
{
Field = queryable1.Where(q => g.IsX)
.Concat(queryable2.Where(q => !g.IsX))
});

I faced the same problem. The solution was to load both results and determine what to use after the query (I know it has performance downside), but at least you can do it temporarily if deadline attacks you:
At the LINQ side
var Games = context.Games.Select(a => new GameModel
{
// carries type1 results
Members = a.UsersInGames.Where(b => b.GameID == a.ID && b.StatusID == 1).Select(c => new Member
{
ID = c.UserID,
email = c.UserInfo.EmailAddress,
screenName = c.UserInfo.ScreenName
})),
//You need to create this temporary carrier to carry type 2 results
MembersOfType2 = a.Teams.Where(b => b.GameID == a.ID).SelectMany(b => b.UsersInTeams.Where(c => c.StatusID == 1)).Select(d => new Member
{
ID = d.UserID,
email = d.UserInfo.EmailAddress,
screenName = d.UserInfo.ScreenName
})))
})
}
After that you may loop Gamesand make the assignment Members = MembersOfType2 if Type == 1 for a certain game.

I had this error too. I had code like this:
var Games = context.Games.Select(a => new GameModel
{
Members = (!filters.GetDatailedDataToo ? null : new List<MemberModel>())
};
This error occurs when null is used in ? : operation.
This is not that case, written up here, but I've wasted lot of time, I think anyone uses this case, who searches this error text..

Is there a neat way of doing a ToList within a LINQ query using query syntax?

Consider the code below:
StockcheckJobs =
(from job in (from stockcheckItem in MDC.StockcheckItems
where distinctJobs.Contains(stockcheckItem.JobId)
group stockcheckItem by new { stockcheckItem.JobId, stockcheckItem.JobData.EngineerId } into jobs
select jobs).ToList()
let date = MJM.GetOrCreateJobData(job.Key.JobId).CompletedJob.Value
orderby date descending
select new StockcheckJobsModel.StockcheckJob()
{
JobId = job.Key.JobId,
Date = date,
Engineer = (EngineerModel)job.Key.EngineerId,
MatchingLines = job.Count(sti => sti.Quantity == sti.ExpectedQuantity),
DifferingLines = job.Count(sti => sti.Quantity != sti.ExpectedQuantity)
}).ToList()
There is a ToList() in the middle because the GetOrCreateJobData method can't be translated into sql.
As a result I've had to surround the first part of my query in brackets to do this, then I've used an outer query to finish up.
I know I could split this into two variables, but I don't want to do that (this is within an object initialiser too).
Is there some other syntax I can use to increase readability, preferably removing the need for an outer an inner query, when I have to do a ToList (or otherwise get to linq-to-objects) in the middle of a linq query?
In an ideal world I'd like something like this (as close as is possible anyway):
StockcheckJobs =
from stockcheckItem in MDC.StockcheckItems
where distinctJobs.Contains(stockcheckItem.JobId)
group stockcheckItem by new { stockcheckItem.JobId, stockcheckItem.JobData.EngineerId } into jobs
MAGIC_DO_BELOW_AS_LINQ-TO-OBJECTS_KEYWORD_OR_SYNTAX
let date = MJM.GetOrCreateJobData(jobs.Key.JobId).CompletedJob.Value
orderby date descending
select new StockcheckJobsModel.StockcheckJob()
{
JobId = jobs.Key.JobId,
Date = date,
Engineer = new ThreeSixtyScheduling.Models.EngineerModel() { Number = jobs.Key.EngineerId },
MatchingLines = jobs.Count(sti => sti.Quantity == sti.ExpectedQuantity),
DifferingLines = jobs.Count(sti => sti.Quantity != sti.ExpectedQuantity)
};

You can fix the issue of GetOrCreateJobData not being translatable to SQL.
By implementing a custom query translator for the specified method call expression, you can gain control over how LINQ-to-SQL interprets the method. There is a good article explaining this procedure and linking to relevant resources available at: http://www.codeproject.com/Articles/32968/QueryMap-Custom-translation-of-LINQ-expressions
Alternatively, you could refactor the GetOrCreateJobData method to an extension method which builds the same logic with expressions, so that LINQ-to-SQL can interpret it naturally. Depending on the complexity of the method, this may be more or less feasible than my first suggestion.

I would raise two points with the question:
I really don't think there's any readability issue with introducing an extra variable here. In fact, I think it makes it more readable as it separates the "locally executing" code from the code executing on the database.
To simply switch to LINQ-To-Objects, AsEnumerable is preferable to ToList.
That said, here's how you can stay in query-land all the way without an intermediate AsEnumerable() / ToList() on the entire query-expression : by tricking the C# compiler into using your custom extension methods rather than the BCL. This is possible since C# uses a "pattern-based" approach (rather than being coupled with the BCL) to turn query-expressions into method-calls and lambdas.
Declare evil classes like these:
public static class To
{
public sealed class ToList { }
public static readonly ToList List;
// C# should target this method when you use "select To.List"
// inside a query expression.
public static List<T> Select<T>
(this IEnumerable<T> source, Func<T, ToList> projector)
{
return source.ToList();
}
}
public static class As
{
public sealed class AsEnumerable { }
public static readonly AsEnumerable Enumerable;
// C# should target this method when you use "select As.Enumerable"
// inside a query expression.
public static IEnumerable<T> Select<T>
(this IEnumerable<T> source, Func<T, AsEnumerable> projector)
{
return source;
}
}
And then you can write queries like this:
List<int> list = from num in new[] { 41 }.AsQueryable()
select num + 1 into result
select To.List;
IEnumerable<int> seq = from num in new[] { 41 }.AsQueryable()
select num + 1 into result
select As.Enumerable into seqItem
select seqItem + 1; // Subsequent processing
In your case, your query would become:
StockcheckJobs =
from stockcheckItem in MDC.StockcheckItems
where distinctJobs.Contains(stockcheckItem.JobId)
group stockcheckItem by new { stockcheckItem.JobId, stockcheckItem.JobData.EngineerId } into jobs
select As.Enumerable into localJobs // MAGIC!
let date = MJM.GetOrCreateJobData(localJobs.Key.JobId).CompletedJob.Value
orderby date descending
select new StockcheckJobsModel.StockcheckJob()
{
JobId = localJobs.Key.JobId,
Date = date,
Engineer = new ThreeSixtyScheduling.Models.EngineerModel() { Number = localJobs.Key.EngineerId },
MatchingLines = localJobs.Count(sti => sti.Quantity == sti.ExpectedQuantity),
DifferingLines = localJobs.Count(sti => sti.Quantity != sti.ExpectedQuantity)
};
I really don't see this as any sort of improvement, though. Rather, it's pretty heavy abuse of a language feature.

I find that using method syntax makes things clearer, but that's just personal preference. It certainly makes the top half of the query better, but using a let, while possible in method syntax, is a bit more work.
var result = stockcheckItem in MDC.StockcheckItems
.Where(item => distinctJobs.Contains(item.JobId))
.GroupBy(item => new { item.JobId, item.JobData.EngineerId })
.AsEnumerable() //switch from Linq-to-sql to Linq-to-objects
.Select(job => new StockcheckJobsModel.StockcheckJob()
{
JobId = job.Key.JobId,
Date = MJM.GetOrCreateJobData(job.Key.JobId).CompletedJob.Value,
Engineer = (EngineerModel)job.Key.EngineerId,
MatchingLines = job.Count(sti => sti.Quantity == sti.ExpectedQuantity),
DifferingLines = job.Count(sti => sti.Quantity != sti.ExpectedQuantity)
})
.Orderby(item => item.Date)
.ToList()

One option is to do all the SQL compatible work up front into an anonymous type,
var jobs =
(from job in (from stockcheckItem in MDC.StockcheckItems
where distinctJobs.Contains(stockcheckItem.JobId)
group stockcheckItem by new
{ stockcheckItem.JobId, stockcheckItem.JobData.EngineerId }
into jobs
select new
{
JobId = job.Key.JobId,
Engineer = (EngineerModel)job.Key.EngineerId,
MatchingLines =
job.Count(sti => sti.Quantity == sti.ExpectedQuantity),
DifferingLines =
job.Count(sti => sti.Quantity != sti.ExpectedQuantity)
}
).AsEnumerable()
StockcheckJobs = jobs.Select(j => new StockcheckJobsModel.StockcheckJob
{
JobId = j.JobId,
Date = MJM.GetOrCreateJobData(j.JobId).CompletedJob.Value,
Engineer = j.EngineerId,
MatchingLines = j.MatchingLines,
DifferingLines = j.DifferingLines
}).OrderBy(j => j.Date).ToList();
Obviously not tested, but you get the idea.