We Keep Coding

LINQ queries take more time to execute

This is my LINQ query and this is taking so long time to execute.
Can any one suggest me how to make it faster.
This query take almost 1.35 min to load.
From the list I am adding in model
var jobList = option.Equals("search")
? _jobService.SearchJobs(_dataModel.SearchFilter, _dataModel.CurrentPage - 1, _dataModel.EntriesPerPage,
_dataModel.KeyWords, _dataModel.Location)
: _jobService.BrowsJobs(_dataModel.BrowseFilterIds, _dataModel.CurrentPage - 1,
_dataModel.EntriesPerPage);
_dataModel.DbJobs = jobList;
_dataModel.Jobs = jobList.Select(job => new JobsViewModel
{
JobId = job.JobId,
JobTitle = job.JobTitle,
SeoFriendlyJobTitle = job.JobTitle.ToSeoFriendly(),
Created = job.Created,
CreatedBy = job.CreatedBy,
LocationName =
job.LocationId != null
? _locationService.GetById((int)job.LocationId).LocationName
: string.Empty,
JobTypeName = _jobTypeService.GetById(job.JobTypeId).JobTypeName,
RecruiterName =
job.RecruiterId != new Guid()
? _recruiterService.GetById(job.RecruiterId).RecruiterName
: string.Empty,
SeoFriendlyRecruiterName =
job.RecruiterId != new Guid()
? _recruiterService.GetById(job.RecruiterId).RecruiterName.ToSeoFriendly()
: string.Empty,
RecruiterId = job.RecruiterId,
StartDate = job.StartDate,
LocationDescription = job.LocationDescription,
SalaryDescription = job.SalaryDescription,
JobSummary = job.JobSummary,
JobSummaryShort = job.JobSummaryShort,
LogoPath = job.Logo.IsNullOrWhiteSpace() ? job.LogoOverride : job.Logo,
PremiumJob = job.Advertising.Count > 0 && job.Advertising.ElementAt(0).PremiumListing,
DaysRemaining = new Helpers().BuildDaysRemainingString(job.EndDate)
}).ToList();

Are those the service calls inside your query ? If so, most probably that is what contributing to slowness.

Some values in LINQ Query Statement aren't saved correctly to class with subsonic 3

I am developing a MVC 3 Application which uses Subsonic 3 for accessing the database.
My Problem is, i don't understand why the Enum "GlobalType" is not being written into the property.
Everytime i check, the value is 0 instead of "One".
The "Name" property contains the "DateCreated" value.
The "DateCreated" property contains a new DateTime instance.
No other fields, as far as i'm aware of, are doing this.
There is no logic inside of the ViewItemModel, it's just a class with properties.
If i add them after this method manually, everything works.
Maybe someone encountered something similar with subsonic (if it even is subsonic itself, maybe i'm making a mistake)?
I have this method in the Backend:
public IEnumerable<ViewItemModel> LoadView(int registratorId)
{
var itemModel = from item in _itemQuery
join header in _headerQuery on item.HeaderID equals header.ID
where header.RegistratorID == registratorId && !(from hidden in _headerHiddenQuery where hidden.ItemID == item.ID && hidden.Type == GlobalType.One && hidden.RegistratorID == registratorId select hidden.ID).Any()
orderby item.ID descending
select new ViewItemModel()
{
Type = GlobalType.One,
ID = item.ID,
Name = header.Name,
DateCreated = header.DateCreated,
TypeOfTransport = header.TypeOfTransport,
TransportType = item.TransportType,
Count = (from subItems in _subItemQuery where subItems.ItemID == item.ID select subItems.ID).Count(),
// For Status
IsArchived = header.IsArchived,
IsCanceled = header.IsCanceled,
Process = header.Process,
End = header.End,
IsPublished = header.IsPublished,
OpenFrom = header.OpenFrom,
OpenTill = header.OpenTill,
IsNextStarted = header.IsNextStarted
};
return itemModel.ToList();
}
Update:
The GlobalType enum looks like this
public enum GlobalType
{
One = 1,
Two = 2,
Individual = 3
}
If i add them manually, i changed the return statement for this:
var result = itemModel.ToList();
foreach (var item in result)
{
var headerId = _itemQuery.Where(it => it.ID == item.ID).Select(it => it.HeaderID).FirstOrDefault();
var created = _itemQuery.Where(it => it.ID == item.ID).Select(it => it.DateCreated).FirstOrDefault();
var name = _headerQuery.Where(it => it.ID == headerId).Select(it => it.Name).FirstOrDefault();
item.AnnouncementType = GlobalType.One;
item.Name = name;
item.DateCreated = created;
}
return result;

try sample code:
public int enum GlobalType
{
One = 1,
Two = 2,
Individual = 3
}
//enum value Convert to int or other data type using casting
item.AnnouncementType = (int) GlobalType.One;
//Suppose if condition using
if((GlobalType)item.AnnouncementType==GlobalType.One)
{
//your code
}

Thanks to DaveParsons comment, i managed to create a workaround.
In this case, the code will have to iterate twice through the list of found elements, but won't load the entire table into memory.
Since there is a bug (throwing exception) with creating an anonymous object containing multiple classes like so:
select new { item, header, subItems }
I managed to get all the data needed, by manually assigning what i need like so:
public IEnumerable<ViewItemModel> LoadView(int registratorId)
{
var itemModel = from item in _itemQuery
join header in _headerQuery on item.AnnouncementHeaderID equals header.ID
where header.RegistratorID == registratorId && !(from hidden in _headerHiddenQuery where hidden.ItemID == item.ID && hidden.Type == GlobalType.One && hidden.RegistratorID == registratorId select hidden.ID).Any()
orderby item.ID descending
select new {
Type = GlobalType.One,
ID = item.ID,
Name = header.Name,
DateCreated = header.DateCreated,
TypeOfTransport = header.TypeOfTransport,
TransportType = item.TransportType,
Count = (from subItems in _subItemQuery where subItems.ItemID == item.ID select subItems.ID).Count(),
// For Status
IsArchived = header.IsArchived,
IsCanceled = header.IsCanceled,
Process = header.Process,
End = header.End,
IsPublished = header.IsPublished,
OpenFrom = header.OpenFrom,
OpenTill = header.OpenTill,
IsNextStarted = header.IsNextStarted
};
return itemModel
.ToList()
.Select(it => new ViewItemModel() {
Type = it.Type,
ID = it.ID,
Name = it.Name,
DateCreated = it.DateCreated,
TypeOfTransport = it.TypeOfTransport,
TransportType = it.TransportType,
Count = it.Count,
// For Status
IsArchived = it.IsArchived,
IsCanceled = it.IsCanceled,
Process = it.Process,
End = it.End,
IsPublished = it.IsPublished,
OpenFrom = it.OpenFrom,
OpenTill = it.OpenTill,
IsNextStarted = it.IsNextStarted
})
.ToList();
}
Notice: The return value of the query is an anonymous object with every single necessary field declared.
After the database returned all fields with the same name as in the database (model), we then have to force execution with ".ToList()" or something similar (deferred execution?).
Since the data is now in memory, we can assign the values from the anonymous object to the original class that was intended for this purpose.
I am sure there is a more reliable way using reflection, but this is what i have come up with.

Linq calling a method in select statement very slow

I am trying to find if there is a way to call a method from within my linq select statement to build a list of objects that doesn't dramatically slow it down. The reason behind this is that I also want to call the same method when trying to get only one of the objects and do not want to have to maintain both versions (i.e. if I have another field added to the object or want to render one of the fields differently I will not have to change it in multiple places).
In the example below, TEST 1 runs over 100 times faster than TEST 2:
// Start timer
var timer = new Stopwatch();
timer.Start();
var test = (from job in dc.Jobs
where !job.archived
select new JobExtended()
{
JobId = job.jobId,
NodeId = job.nodeId,
JobName = job.name != string.Empty ? job.name : "TBC",
Salary = job.salary,
RecruiterId = job.fkRecruiterId,
RecruiterNodeId = job.JobRecruiter != null ? job.JobRecruiter.recruiterNodeId : null,
RecruiterName = job.JobRecruiter != null ? job.JobRecruiter.name : string.Empty,
LocationId = job.fkLocationId,
Location = job.refJobLocation != null ? job.refJobLocation.jobLocation : "",
ContractTypeId = job.fkContractTypeId,
ContractType = job.refJobContractType != null ? job.refJobContractType.contractType : "",
CategoryId = job.fkCategoryId,
Category = job.refJobCategory != null ? job.refJobCategory.category : "",
ClosingDate = job.closingDate,
Featured = job.featured,
JobOfTheWeek = job.jobOfTheWeek,
PublishedDate = job.publishedDate,
Url = "/jobs/" + job.name.Replace(" ", "-").Replace("&", "and").Replace("'", "") + (job.fkLocationId.HasValue ? "-in-" + job.refJobLocation.jobLocation.Replace(" ", "-").Replace("&", "and").Replace("'", "") : "") + "-jn" + job.jobId,
CreatedOn = job.createdOnDate,
PrintWidth = job.printWidth,
PrintHeight = job.printHeight,
UntilFilled = (job.untilFilled != null && job.untilFilled.Value),
AdvertCost = job.advertCost,
DatesToShow = job.relJobDates.Where(x => x.fkJobId == job.jobId).Select(x => x.date).ToList(),
IsParentJob = job.relLinkedJobs != null && job.relLinkedJobs.Any(x => x.fkParentJobId == job.jobId),
IsAlternateWeekJob = job.alternateWeek != null && job.alternateWeek.Value,
Archived = job.archived,
LastModifiedDate = job.lastModifiedDate,
RecruiterContactDetails = job.recruiterContactDetails
}).ToList();
// Stop timer
timer.Stop();
// Output info
litTest.Text = "TEST 1 in " + timer.Elapsed.TotalSeconds + " seconds<br/>";
//Start timer
timer = new Stopwatch();
timer.Start();
var test2 = (from job in dc.Jobs
where !job.archived
select GetJobDetails(job)).ToList();
//Stop timer
timer.Stop();
//Output info
litTest.Text += "TEST 2 in " + timer.Elapsed.TotalSeconds + " seconds<br/>";
This is the method that TEST 2 is calling which should be creating the same object that is being returned in TEST 1:
public static JobExtended GetJobDetails(Data.Job job)
{
return new JobExtended()
{
JobId = job.jobId,
NodeId = job.nodeId,
JobName = job.name != string.Empty ? job.name : "TBC",
Salary = job.salary,
RecruiterId = job.fkRecruiterId,
RecruiterNodeId = job.JobRecruiter != null ? job.JobRecruiter.recruiterNodeId : null,
RecruiterName = job.JobRecruiter != null ? job.JobRecruiter.name : string.Empty,
LocationId = job.fkLocationId,
Location = job.refJobLocation != null ? job.refJobLocation.jobLocation : "",
ContractTypeId = job.fkContractTypeId,
ContractType = job.refJobContractType != null ? job.refJobContractType.contractType : "",
CategoryId = job.fkCategoryId,
Category = job.refJobCategory != null ? job.refJobCategory.category : "",
ClosingDate = job.closingDate,
Featured = job.featured,
JobOfTheWeek = job.jobOfTheWeek,
PublishedDate = job.publishedDate,
Url = "/jobs/" + job.name.Replace(" ", "-").Replace("&", "and").Replace("'", "") + (job.fkLocationId.HasValue ? "-in-" + job.refJobLocation.jobLocation.Replace(" ", "-").Replace("&", "and").Replace("'", "") : "") + "-jn" + job.jobId,
CreatedOn = job.createdOnDate,
PrintWidth = job.printWidth,
PrintHeight = job.printHeight,
UntilFilled = (job.untilFilled != null && job.untilFilled.Value),
AdvertCost = job.advertCost,
DatesToShow = job.relJobDates.Where(x => x.fkJobId == job.jobId).Select(x => x.date).ToList(),
IsParentJob = job.relLinkedJobs != null && job.relLinkedJobs.Any(x => x.fkParentJobId == job.jobId),
IsAlternateWeekJob = job.alternateWeek != null && job.alternateWeek.Value,
Archived = job.archived,
LastModifiedDate = job.lastModifiedDate,
RecruiterContactDetails = job.recruiterContactDetails
};
}
The reason for this is because I want to be able to call "GetJobDetails" for returning a single job for example:
public JobExtended GetJobDetails(int jobId)
{
using (DataContext dc = new DataContext())
{
return dc.Jobs.Where(x => x.jobId == jobId).Select(j => GetJobDetails(j)).FirstOrDefault();
}
}
Doing it like this would allow me to only ever have to update the "GetJobDetails" method if for example I decided to add a new field of change how the "Url" value was generated but doing it this way is a lot slower. Is there a way around this, I have already tried the following which do not seem to help:
var test3 = (from job in dc.Jobs
where !job.archived
select job).AsEnumerable()
.Select(GetJobDetails).ToList();
var test4 = (from job in dc.Jobs
where !job.archived
select GetJobDetails(job));
var test4a = test4.ToList();

The reason TEST 1 is faster because the query is executed once on the server and only returns the selected fields.
var test = (from job in dc.Jobs
where !job.archived
select new JobExtended()
{
JobId = job.jobId,
NodeId = job.nodeId,
...
}).ToList();
When you call GetJobDetails in TEST 2 the parameter j needs to be materialized first before it can be send as parameter to GetJobDetails. And so there are multiple calls of the full objects.
return dc.Jobs.Where(x => x.jobId == jobId).Select(j => GetJobDetails(j)).FirstOrDefault();
In order to achieve something like you want you should use extension methods. This one extends IQueryable.
public static IEnumerable<JobExtended> SelectJobExtended(this IQueryable<Data.Job> query)
{
return query
.Select(o => new JobExtended()
{
JobId = job.jobId,
NodeId = job.nodeId,
...
}
}
Then you can call:
dc.Jobs.Where(x => x.jobId == jobId).SelectJobExtended().FirstOrDefault();

I've seen this kind of issue before. If I recall, what we did was "stacked" the queries.
public IEnumerable<JobExtended> ConvertToJobExtended(IEnumerable<Job> jobs)
{
return
from job in jobs
select new JobExtended()
{
MyString = job.MyInt.ToString(),
...
};
}
Then what you can do is call it the following way:
var query = (from job in dc.Jobs
where !job.archived
select job;
var test2 = ConvertToJobExtended(query).ToList();
There are plenty of alternatives that can go from here... I hope this goes in the right direction of what you're looking to do.

Entity Framework IQueryable Deferred for Big Object with many conditions LinqtoEntity

I'm attempting to refactor some painfully slow code in the repository layer of a large property management system.
There was a pattern followed by the original dev team where they did not understand the nature of deferred execution in regards to the IQueryable.
Here is a code sample:
public PropertyDTO GetPropertyByPropertyAuctionID(Guid propertyAuctionId)
{
PropertyDTO property = null;
var result =
(
from pa in databaseContext.PropertyAuction
join a in databaseContext.Auction on pa.AuctionID equals a.AuctionID into suba
from sa in suba.DefaultIfEmpty()
join an in databaseContext.AuctionNotifications
on sa.AuctionNotificationsID equals an.AuctionNotificationsID into suban
from san in suban.DefaultIfEmpty()
where pa.PropertyAuctionID == propertyAuctionId
where pa.IsDeleted == null || !(bool)pa.IsDeleted
select new
{
PropertyAuction = pa,
AuctionNotifications = san
}
).FirstOrDefault();
if (result != null)
{
var start = DateTime.Now;
property = ConvertPropertyFromDatabase(result.PropertyAuction, result.AuctionNotifications);
//end can be upwards of 10-12 seconds per record
var end = DateTime.Now - start;
}
else
{
throw CustomException.NotFound;
}
return property;
}
internal PropertyDTO ConvertPropertyFromDatabase(PropertyAuction propertyAuction, AuctionNotifications auctionNotifications = null)
{
Property property = null;
PropertyDTO dtoProperty = new PropertyDTO()
{
PropertyAuctionID = propertyAuction.PropertyAuctionID
};
if (propertyAuction.PropertyID.HasValue)
{
dtoProperty.PropertyID = propertyAuction.PropertyID.Value;
property = propertyAuction.Property;
dtoProperty.IsSimulcastAuction = propertyAuction.IsSimulcastAuction;
dtoProperty.IsSold = propertyAuction.IsSold;
dtoProperty.PropertyExecutionInitialValue = propertyAuction.ExecutionInitialValue;
}
if (propertyAuction.User4 != null && !string.IsNullOrEmpty(propertyAuction.User4.Profile1.FirstName) && !string.IsNullOrEmpty(propertyAuction.User4.Profile1.LastName))
{
dtoProperty.OfferAdministratorFirstName = propertyAuction.User4.Profile1.FirstName;
dtoProperty.OfferAdministratorLastName = propertyAuction.User4.Profile1.LastName;
dtoProperty.OfferAdministratorUserID = propertyAuction.OfferAdministratorUserID;
}
if (propertyAuction.AuctionID.HasValue)
{
dtoProperty.AuctionID = propertyAuction.AuctionID;
string auctionStartDateString = String.Empty;
if (propertyAuction.Auction != null)
{
if (propertyAuction.Auction.AuctionStartDate.HasValue)
{
auctionStartDateString = propertyAuction.Auction.AuctionStartDate.Value.Year + "-" + propertyAuction.Auction.AuctionStartDate.Value.Month.ToString().PadLeft(2, '0') + "-" + propertyAuction.Auction.AuctionStartDate.Value.Day.ToString().PadLeft(2, '0') + ": ";
}
dtoProperty.HMOnlineAuctionClosingStartDate = propertyAuction.Auction.AuctionClosingDate;
dtoProperty.AuctionName = auctionStartDateString + propertyAuction.Auction.Title;
dtoProperty.AuctionPublicSiteName = propertyAuction.Auction.PublicSiteName;
dtoProperty.AuctionNumber = propertyAuction.Auction.AuctionNumber;
dtoProperty.AuctionType = propertyAuction.Auction.AuctionType.Value;
dtoProperty.OriginalAuctionID = propertyAuction.Auction.OriginalAuctionID;
dtoProperty.AuctionTermsAndConditions = propertyAuction.Auction.TermsAndConditions;
dtoProperty.BrochureFileName = propertyAuction.Auction.BrochureFileName;
dtoProperty.BrochureName = propertyAuction.Auction.BrochureName;
if (propertyAuction.Auction.LuJohnsAuctionDetails != null)
{
dtoProperty.LuJohnsAuctionLiveWebcastLink = propertyAuction.Auction.LuJohnsAuctionDetails.LiveWebcastLink;
}
}
}
if (propertyAuction.PropertyStatus != null)
{
dtoProperty.PropertyStatusID = propertyAuction.PropertyStatusID.Value;
dtoProperty.PropertyStatusValue = propertyAuction.PropertyStatus.Value;
}
if (propertyAuction.PropertyAuctionAuctionDetails != null)
{
PropertyAuctionAuctionDetails propertyAuctionAuctionDetails = propertyAuction.PropertyAuctionAuctionDetails;
dtoProperty.PropertyAuctionAuctionDetailsId = propertyAuction.PropertyAuctionAuctionDetailsID;
SetShowOnWebsiteProperty(dtoProperty, propertyAuction);
SetWebsiteButtonProperty(dtoProperty, propertyAuctionAuctionDetails, propertyAuctionAuctionDetails.PreSaleMethod,
propertyAuctionAuctionDetails.AuctionMethod, propertyAuctionAuctionDetails.PostSaleMethod);
}
//This goes on for an astounding 900 more lines of code
}
As you can probably guess, this is making significant repeat trips to the database.
What is the fastest way to refactor this code to make the fewest number of trips to the database?

The problem here is that this ConvertPropertyFromDatabase() method triggers lots of queries for lazily loaded properties. I suggest you turn of lazy loading in you DbContext class like this:
Configuration.LazyLoadingEnabled = false;
Then you have to check which properties are accessed in this convert method, and rewrite your query so these are explicitly loaded.

c# linq to sql two threads visit the same db, but one is very slow

I've created two threads in my application and the first one is the main UI thread, which will do the normal job like quering below
var order = db.Orders.Where(x=>x.systemId==id).ToList();
the other thread is checking if there is any warning situation to handle, the code is as below:
TransactionOptions transOptions = new TransactionOptions()
{ IsolationLevel = System.Transactions.IsolationLevel.ReadUncommitted };
using (new TransactionScope(TransactionScopeOption.Required, transOptions))
{
BathDBDataContext dc = new BathDBDataContext(connectionString);
var all_menus = dc.Menu.Where(x => x.addAutomatic);
var menus = all_menus.Select(x => x.name).ToList();
var orders = dc.Orders.Where(x => menus.Contains(x.menu) && !x.paid && x.deleteEmployee == null);
var add_menus = orders.Select(x => x.menu).Distinct();
var ids = orders.Select(x => x.systemId).Distinct();
foreach (var systemId in ids)
{
var seat_orders = orders.Where(x => x.systemId == systemId);
foreach (var add_menu in add_menus)
{
var add_orders = seat_orders.Where(x => x.menu == add_menu && (x.priceType == null || x.priceType != "stop counting"));
if (add_orders.Count() == 0)
continue;
var max_time = add_orders.Max(x => x.inputTime);
var max_order = add_orders.OrderByDescending(x => x.inputTime).FirstOrDefault();
//var max_order = add_orders.FirstOrDefault(x => x.inputTime == max_time);
if (max_order == null || max_order.priceType == "per hour")
continue;
var the_menu = all_menus.FirstOrDefault(x => x.name == add_menu);
string menu_time = the_menu.timeLimitHour.ToString() +
":" + the_menu.timeLimitMiniute.ToString() +
":" + the_menu.timeLimitSecond.ToString();
TimeSpan tsm = TimeSpan.Parse(menu_time);
if (DateTime.Now - max_order.inputTime < tsm)
continue;
if (the_menu.addType == "by unit")
{
Orders new_order = new Orders();
new_order.menu = max_order.menu;
new_order.text = max_order.text;
new_order.systemId = systemId;
new_order.number = 1;
new_order.money = the_menu.price;
new_order.technician = max_order.technician;
new_order.techType = max_order.techType;
new_order.inputTime = DateTime.Now;
new_order.inputEmployee = "computer";
new_order.paid = false;
dc.Orders.InsertOnSubmit(new_order);
}
else if (the_menu.addType == "by time")
{
Orders new_order = new Orders();
new_order.menu = max_order.menu;
new_order.text = max_order.text;
new_order.systemId = systemId;
new_order.number = 1;
new_order.priceType = "per hour";
new_order.money = Convert.ToDouble(the_menu.addMoney);
new_order.technician = max_order.technician;
new_order.techType = max_order.techType;
new_order.inputTime = DateTime.Now;
new_order.inputEmployee = "computer";
new_order.paid = false;
dc.Orders.InsertOnSubmit(new_order);
}
}
dc.SubmitChanges();
}
in this situation, the query in the main UI thread will be very slow or sometimes doesn't even work.
can anyone help me with that?

Correct me if I am wrong but you seem to insert values to Orders in the loop and also trying to use an expression in your main UI.
Besides 5 seconds interval might be short since your LINQ expressions get more complex so that time needed for the calculations increases.
Loading that "BathDBDataContext" might take more time as your database gets new records.
I'd suggest using Cache and increase 5 seconds refresh timer.