We Keep Coding

horrible and big LINQ statement optimisation

I have to to a rather large request to a database to fetch a bunch of data, it's however taking a noticeable time to run. is there some way to increase the performance on this? preemptive apologies for the ugly code (I did have a version that segmented this into multiple smaller functions but that was even slower)
from contact in _database.OBJECTCONTACT
where contact.OBJECTCONTACTOWNER.Any(o => o.OBJECTID == id && o.OBJECTTYPE == type) && contact.ACTIVE >= 1 && CheckViewAccess(contact)
group contact by (contact.OBJECTCONTACTPROJECT.Any() ? contact.OBJECTCONTACTPROJECT.First().OBJECTPROJECT.PROJECTNAME : "General") into projectGroup
select new ProjectViewModel()
{
ProjectName = projectGroup.Key,
ContactGroups = (from g in _database.OBJECTGROUP
where g.GROUPTYPE == "CONTACT" && ContactsModule.CheckUserRole("View", g.OBJECTTYPE, g.GROUPNAME)
select new ContactGroupViewModel()
{
CanEdit = ContactsModule.CheckUserRole("Edit", g.OBJECTTYPE, g.GROUPNAME),
GroupId = g.OBJECTGROUPID,
GroupName = g.GROUPNAME,
Contacts = (from c in projectGroup
join l in _database.OBJECTCONTACTLOCATION on c.OBJECTCONTACTLOCATIONID equals l.OBJECTCONTACTLOCATIONID into lgrp from loc in lgrp.DefaultIfEmpty(null)
orderby c.NAME
select new ContactViewModel()
{
Id = (int)c.OBJECTCONTACTID,
Name = c.NAME,
Description = c.DESCRIPTION,
ContactInformation = CreateContactInfoViewmodels(c),
Owners = c.OBJECTCONTACTOWNER.Where(owner => owner.OBJECTTYPE == "AIRPORT")
.Select(owner => ContactOwnerViewModel.FromOwnerId(owner.OBJECTID, owner.OBJECTTYPE)).ToList(),
Projects = c.OBJECTCONTACTPROJECT.Select(proj => proj.OBJECTPROJECT).ToList(),
Typename = GetTypeName(c),
TypeId = c.OBJECTCONTACTTYPEID ?? 0,
ContactGroupId = c.OBJECTGROUPID,
ContactGroup = g.GROUPNAME,
Editable = CheckAccessBool("EDIT", c),
Location = loc != null ? new LocationViewModel()
{
Address = loc.ADDRESS,
GoogleMapLink = loc.GMAPADDRESS,
LocationId = loc.OBJECTCONTACTLOCATIONID,
LatLon = Tuple.Create(loc.LATITUDE, loc.LONGITUDE)
} : null,
}).ToList()
}).ToList()
}).ToList();
I think I should be able to use joins to move the entire DB fetch code to the top (theoretically improving perfomance) but I am having trouble finding the syntax which would suit my needs

Thanks everyone for coming with suggestions. I am in a situation where I'm not able to do much with the database itself so I'm making the best of what I have. my hands a bit tied in regards to the tools at my disposal (also fairly old codebase I think it's EF 5 or something like that)
this version moves the DB transaction to the top (so that is fewer fetches) and does a lot of data manipulation at the bottom.
// general object is created above
var res = (from contact in _database.OBJECTCONTACT.AsEnumerable() // as enumerable used to allow for defaultifempty in join (minor damage to performance)
join oGroup in _database.OBJECTGROUP on contact.OBJECTGROUPID equals oGroup.OBJECTGROUPID into og from objectGroup in og.DefaultIfEmpty(defaultValue: general)
where contact.OBJECTCONTACTOWNER.Any(o => o.OBJECTTYPE == type && o.OBJECTID == id)
// ReSharper disable once PossibleNullReferenceException (it's taken care of by check using .any() )
group new {contact, objectGroup } by (contact.OBJECTCONTACTPROJECT.Any() ? contact.OBJECTCONTACTPROJECT.FirstOrDefault().OBJECTPROJECT.PROJECTNAME : "General") into pGroup
orderby pGroup.Key == "General" ? pGroup.Key : "􏿽" descending
select new ProjectViewModel()
{
ProjectName = pGroup.Key,
ProjectId = pGroup.FirstOrDefault() != null ? (pGroup.FirstOrDefault().contact.OBJECTCONTACTPROJECT.FirstOrDefault() != null ? pGroup.FirstOrDefault().contact.OBJECTCONTACTPROJECT.FirstOrDefault().OBJECTPROJECTID : -1) : -1,
ContactGroups = (from c in pGroup
group c by c.objectGroup into grp
let canEdit = ContactsModule.CheckUserRole("EDIT", grp.Key.OBJECTTYPE, grp.Key.GROUPNAME)
orderby grp.Key.SORTORDER descending
select new ContactGroupViewModel()
{
GroupName = grp.Key.GROUPNAME,
GroupId = grp.Key.OBJECTGROUPID,
CanEdit = canEdit,
Contacts = grp.Select(item => new ContactViewModel()
{
Id = (int)item.contact.OBJECTCONTACTID,
Name = item.contact.NAME,
Description = item.contact.DESCRIPTION,
Editable = canEdit,
ContactInformation = item.contact.OBJECTCONTACTNUMBER.OrderByDescending(num => num.ISMAININFO).Select(num => new ContactInfoViewmodel()
{
Data = num.NUMBERDATA,
IsMain = num.ISMAININFO > 0,
Type = num.OBJECTCONTACTNUMBERTYPE.NAME
}).ToList()
}).ToList()
}).ToList()
}).ToList();
this seems to (on average) take about a 4th of the time the original query needed (still a noticeable time due to the size of database but within acceptable limits)

Update IQueryable result before using as join in next query

I need to use Linq to Entity Framework to query a LOCATION table to get the record of the location code with the MAX effective date, then use that result as a join in the next query.
I BELIEVE I need to do convert before the IQueryable is used, because I have that last clause in the second query where I want to exclude records where the FLOOR code is in the excludedSchools list. That excludedSchools list will have the newLocationCode in it.
So, I need to update the values in the IQueryable result before I use it. Can I do this? Here is my code:
using (var db = new TheContext())
{
IQueryable<LocationTable> locatinWithMaxEffDate =
(from lc in db.LocationTable
where lc.EFF_STATUS == "A" && lc.EFFDT <= DateTime.Now
group lc by lc.LOCATION into g
select g.OrderByDescending(x => x.EFFDT).FirstOrDefault()
);
foreach (var location in locatinWithMaxEffDate.ToList())
{
string newLocationCode;
if(codeMappingDictionary.TryGetValue(location.FLOOR, out newLocationCode))
{
// how do I update locatinWithMaxEffDate FLOOR value
// with newLocationCode so it works in the query below?
location.FLOOR = newLocationCode;
}
}
var query =
(from fim in db.PS_PPS_FIM_EE_DATA
join mloc in locatinWithMaxEffDate on fim.LOCATION equals mloc.LOCATION
where
fim.EMPL_STATUS == PsPpsFimEeData.EmployeeStatusValues.Active
&& fim.AUTO_UPDATE == PsPpsFimEeData.AutoUpdateValues.Enabled
&& includeJobCodes.Contains(fim.JOBCODE)
&& !excludedSchools.Contains(mloc.FLOOR)
select new PpsAdministratorResult
{
SchoolId = mloc.FLOOR,
Login = fim.OPRID,
EmployeeId = fim.EMPLID,
}
With the code above, the locatinWithMaxEffDate does not have the updated FLOOR values. I can see why this is, but can't seem to fix it.
So far, I have tried introducing another list to ADD() the new location record to, then casting that as an IQueryable, but I get an error about primitive vs concrete types.

I decided to make things easier on myself. Since both sets of data are very small (fewer than 1000 records each) I call take the entire set of data as an annonymous type:
using (var db = new TheContext())
{
IQueryable<LocationTable> locatinWithMaxEffDate =
(from lc in db.LocationTable
where lc.EFF_STATUS == "A" && lc.EFFDT <= DateTime.Now
group lc by lc.LOCATION into g
select g.OrderByDescending(x => x.EFFDT).FirstOrDefault()
);
var query =
(from fim in db.PS_PPS_FIM_EE_DATA
join mloc in locatinWithMaxEffDate on fim.LOCATION equals mloc.LOCATION
where
fim.EMPL_STATUS == PsPpsFimEeData.EmployeeStatusValues.Active
&& fim.AUTO_UPDATE == PsPpsFimEeData.AutoUpdateValues.Enabled
&& includeJobCodes.Contains(fim.JOBCODE)
select new PpsAdministratorResult
{
SchoolId = mloc.FLOOR,
Login = fim.OPRID,
EmployeeId = fim.EMPLID,
}
}
Then, just work with the two objects:
List<PpsAdministratorResult> administratorList = new List<PpsAdministratorResult>();
foreach (var location in query.ToList())
{
string newLocationCode;
if(schoolCodeMappings.TryGetValue(location.SchoolId, out newLocationCode)) // && newLocationCode.Contains(location.LOCATION))
{
location.SchoolId = newLocationCode;
}
if( !excludedSchools.Contains(location.SchoolId) )
{
administratorList.Add(location);
}
}
Now, I have the list I want.

Make a LINQ query dynamic to bring back all rows or only rows with a link to a lookup table?

I have a query that returns a list of currencies and joins to a lookup table. The result is then put into a class object (which works fine):
var queryforobject = from x in db.CurrencyExchangeRates.AsNoTracking()
join c in db.CurrencyTypes.AsNoTracking() on x.CurrencyTypeID equals c.ID
orderby x.ID
select new CurrencyExchangeRateObject
{
ID = x.ID,
CurrencyID = c.ID,
Currency = c.Description,
ExchangeRate = x.ExchangeRate,
LastEditedDate = x.LastEditedDate,
LastEditedBy = x.LastEditedBy,
Active = x.Active
};
I want to make this more dynamic, so if no CurrencyTypeID is supplied then it will return the full list (as it does already) - otherwise if a CurrencyTypeID is supplied it will only show where X.CurrencyTypeID = ID.
Something along the lines of an inline if?

There are a few options for filtering the query based on CurrencyTypeID if a search value (named currencyTypeID in this answer) is supplied, but return all data if no currencyTypeID is supplied.
First option: You could add a where clause to your existing query expression. The WHERE clause below will return every record in the data set if null is passed in for the currencyTypeID variable, otherwise it will filter the results.
from x in db.CurrencyExchangeRates.AsNoTracking()
join c in db.CurrencyTypes.AsNoTracking() on x.CurrencyTypeID equals c.ID
where (currencyTypeID == null || x.CurrencyTypeID == currencyTypeID)
orderby x.ID
select new CurrencyExchangeRateObject {
ID = x.ID,
CurrencyID = c.ID,
Currency = c.Description,
ExchangeRate = x.ExchangeRate,
LastEditedDate = x.LastEditedDate,
LastEditedBy = x.LastEditedBy,
Active = x.Active
};
Alternatively: Since queryforobject is of type IQueryable<T>, you can use LINQ's fluent API to append a WHERE clause to the query inside an if statement. You need to be more careful about timing on this one though as it needs to be done before you force evaluation of the IQueryable with a foreach loop, .ToList(), .Select() or other LINQ methods that force evaluation.
if(currencyTypeID != null)
queryforobject = queryforobject.Where(cerObj => cerObj.CurrencyID == currencyTypeID);

More efficient way of loading children of entity objects in linq to entity query

I have a rather complex linq to entity query that I'm performing, in the end, I have a result set. I loop through that result set, build business objects and return that list of business objects. it's pretty quick, the problem is that 2 of the child properties are complex objects with their own child objects. for every business object in my loop, I then have to make 2 DB calls to fill its child object. Those 2 calls slow down the overall process, is there a better way to do this? noob to EF here. (EF 4,SQL Server 2008,c#)
Get a result set:
var newresult = from r in result // result is another complex query
join subedit in
(from sa in context.Security_Access
join g in context.Security_UserGroup on sa.EntityID equals g.GroupID
where (sa.PrivledgeID == xx) && g.UserID == userId
select new { user = g.UserID, linkid = sa.LinkID }).Distinct() on new { aid = r.AssetId } equals new { aid = subedit.linkid } into theSubEdit
from subEditAccess in theSubEdit.DefaultIfEmpty()
join subdownload in
(from sa in context.Security_Access
join g in context.Security_UserGroup on sa.EntityID equals g.GroupID
where (sa.PrivledgeID == xx|| sa.PrivledgeID == yy) && g.UserID == userId
select new { user = g.UserID, linkid = sa.LinkID }).Distinct() on new { aid = r.AssetId } equals new { aid = subdownload.linkid } into theSubDownload
from subDownloadAccess in theSubDownload.DefaultIfEmpty()
join subView in
(from sa in context.Security_Access
join g in context.Security_UserGroup on sa.EntityID equals g.GroupID
where (sa.PrivledgeID == xx|| sa.PrivledgeID == yy|| sa.PrivledgeID == 101) && g.UserID == userId
select new { user = g.UserID, linkid = sa.LinkID }).Distinct() on new { aid = r.AssetId } equals new { aid = subView.linkid } into theSubView
from subViewAccess in theSubView.DefaultIfEmpty()
select new { r, EditAccess = (int?)subEditAccess.user, DownloadAccess = (int?)subDownloadAccess.user, ViewAccess = (int?)subViewAccess.user };
I then loop through that result set:
foreach (var asset in newresult)
{
// and build a new business object, set its properties
BoAsset boAsset = new BoAsset();
boAsset.HasEditRights = (asset.EditAccess > 0);
boAsset.HasDownloadRights = (asset.DownloadAccess > 0);
boAsset.HasViewRights = (asset.ViewAccess > 0);
boAsset.Description = asset.r.Description;
boAsset.DetailedDescription = asset.r.DetailedDescription;
boAsset.Keywords = asset.r.Keywords;
boAsset.Notes = asset.r.Notes;
boAsset.Photographer = asset.r.Photographer;
boAsset.PhotographerEmail = asset.r.PhotographerEmail;
boAsset.Notes = asset.r.Notes;
boAsset.Author = asset.r.Author;
// these 2 properties i've commented out are
// complex objects/entities, setting them the way I am
// requires me to call 2 separate methods which make 2 DB trips
// per business object.
//boAsset.Domains = GetAssetDomains(asset.r.AssetId);
//boAsset.DomainEntries = GetAssetCustomDomains(asset.r.AssetId);
myListofObjects.Add(boAsset);
}
return myListofObjects;
Is there a better way?

Just add this .Include("Domains").Include("DomainEntries") to your Linq in in context.Security_Access That should get rows from those tables all in one go.
So your "inner" queries would look like:
from sa in context.Security_Access.Include("Domains").Include("DomainEntries")
join g in context.Security_UserGroup on sa.EntityID equals g.GroupID
where (sa.PrivledgeID == xx) && g.UserID == userId
select new { ...
Here is the documentation from MS: http://msdn.microsoft.com/en-us/library/bb738708.aspx

If you want to improve your performance use compile queries !
You can check the example here.
static readonly Func<AdventureWorksEntities, Decimal,
IQueryable<SalesOrderHeader>> s_compiledQuery2 =
CompiledQuery.Compile<AdventureWorksEntities, Decimal, IQueryable<SalesOrderHeader>>((ctx, total) =>
from order in ctx.SalesOrderHeaders.Include("Orders") where order.TotalDue >= total select order);
MSDN
AND
You can Introduce Include suppose to select all the employees along with their departments . If you have a navigational property, you won't need a join at all. You can use Include like this:
List<Employee> employeesWithDepartments = CreateObjectSet<Employee>().
Include(e => e.Department).
ToList();

Linq To Entities

I have a small problem in my where clause in the linq expression below. If I put the number 3 instead of department.Id I get the desired result but when I use department.Id I get nothing in the resultset.
I also want to get a count for the number of filters for that filter name using the query again using distinct.
var dept = Page.RouteData.Values["department"];
var department = (from d in db.Departments
where d.Name.Replace(" ", "-") == dept
select new {d.Id, d.Name}).FirstOrDefault();
var query = from p in db.Products
join f in db.ProductFilters on p.Id equals f.ProductId into filters
from x in filters.Where(x => x.Product.DepartmentId == department.Id
/* if == 3 it works */)
select new { x.Name, x.Id };

Promoted to answer from comments:
Have you checked that the department instance is as you think it should be after the first linq statement - ie has an Id == 3?

Your first query is not finding any valid department and is therefore returning default which most probably means that departmend.Id == 0.