horrible and big LINQ statement optimisation - c#

I have to to a rather large request to a database to fetch a bunch of data, it's however taking a noticeable time to run. is there some way to increase the performance on this? preemptive apologies for the ugly code (I did have a version that segmented this into multiple smaller functions but that was even slower)
from contact in _database.OBJECTCONTACT
where contact.OBJECTCONTACTOWNER.Any(o => o.OBJECTID == id && o.OBJECTTYPE == type) && contact.ACTIVE >= 1 && CheckViewAccess(contact)
group contact by (contact.OBJECTCONTACTPROJECT.Any() ? contact.OBJECTCONTACTPROJECT.First().OBJECTPROJECT.PROJECTNAME : "General") into projectGroup
select new ProjectViewModel()
{
ProjectName = projectGroup.Key,
ContactGroups = (from g in _database.OBJECTGROUP
where g.GROUPTYPE == "CONTACT" && ContactsModule.CheckUserRole("View", g.OBJECTTYPE, g.GROUPNAME)
select new ContactGroupViewModel()
{
CanEdit = ContactsModule.CheckUserRole("Edit", g.OBJECTTYPE, g.GROUPNAME),
GroupId = g.OBJECTGROUPID,
GroupName = g.GROUPNAME,
Contacts = (from c in projectGroup
join l in _database.OBJECTCONTACTLOCATION on c.OBJECTCONTACTLOCATIONID equals l.OBJECTCONTACTLOCATIONID into lgrp from loc in lgrp.DefaultIfEmpty(null)
orderby c.NAME
select new ContactViewModel()
{
Id = (int)c.OBJECTCONTACTID,
Name = c.NAME,
Description = c.DESCRIPTION,
ContactInformation = CreateContactInfoViewmodels(c),
Owners = c.OBJECTCONTACTOWNER.Where(owner => owner.OBJECTTYPE == "AIRPORT")
.Select(owner => ContactOwnerViewModel.FromOwnerId(owner.OBJECTID, owner.OBJECTTYPE)).ToList(),
Projects = c.OBJECTCONTACTPROJECT.Select(proj => proj.OBJECTPROJECT).ToList(),
Typename = GetTypeName(c),
TypeId = c.OBJECTCONTACTTYPEID ?? 0,
ContactGroupId = c.OBJECTGROUPID,
ContactGroup = g.GROUPNAME,
Editable = CheckAccessBool("EDIT", c),
Location = loc != null ? new LocationViewModel()
{
Address = loc.ADDRESS,
GoogleMapLink = loc.GMAPADDRESS,
LocationId = loc.OBJECTCONTACTLOCATIONID,
LatLon = Tuple.Create(loc.LATITUDE, loc.LONGITUDE)
} : null,
}).ToList()
}).ToList()
}).ToList();
I think I should be able to use joins to move the entire DB fetch code to the top (theoretically improving perfomance) but I am having trouble finding the syntax which would suit my needs

Thanks everyone for coming with suggestions. I am in a situation where I'm not able to do much with the database itself so I'm making the best of what I have. my hands a bit tied in regards to the tools at my disposal (also fairly old codebase I think it's EF 5 or something like that)
this version moves the DB transaction to the top (so that is fewer fetches) and does a lot of data manipulation at the bottom.
// general object is created above
var res = (from contact in _database.OBJECTCONTACT.AsEnumerable() // as enumerable used to allow for defaultifempty in join (minor damage to performance)
join oGroup in _database.OBJECTGROUP on contact.OBJECTGROUPID equals oGroup.OBJECTGROUPID into og from objectGroup in og.DefaultIfEmpty(defaultValue: general)
where contact.OBJECTCONTACTOWNER.Any(o => o.OBJECTTYPE == type && o.OBJECTID == id)
// ReSharper disable once PossibleNullReferenceException (it's taken care of by check using .any() )
group new {contact, objectGroup } by (contact.OBJECTCONTACTPROJECT.Any() ? contact.OBJECTCONTACTPROJECT.FirstOrDefault().OBJECTPROJECT.PROJECTNAME : "General") into pGroup
orderby pGroup.Key == "General" ? pGroup.Key : "􏿽" descending
select new ProjectViewModel()
{
ProjectName = pGroup.Key,
ProjectId = pGroup.FirstOrDefault() != null ? (pGroup.FirstOrDefault().contact.OBJECTCONTACTPROJECT.FirstOrDefault() != null ? pGroup.FirstOrDefault().contact.OBJECTCONTACTPROJECT.FirstOrDefault().OBJECTPROJECTID : -1) : -1,
ContactGroups = (from c in pGroup
group c by c.objectGroup into grp
let canEdit = ContactsModule.CheckUserRole("EDIT", grp.Key.OBJECTTYPE, grp.Key.GROUPNAME)
orderby grp.Key.SORTORDER descending
select new ContactGroupViewModel()
{
GroupName = grp.Key.GROUPNAME,
GroupId = grp.Key.OBJECTGROUPID,
CanEdit = canEdit,
Contacts = grp.Select(item => new ContactViewModel()
{
Id = (int)item.contact.OBJECTCONTACTID,
Name = item.contact.NAME,
Description = item.contact.DESCRIPTION,
Editable = canEdit,
ContactInformation = item.contact.OBJECTCONTACTNUMBER.OrderByDescending(num => num.ISMAININFO).Select(num => new ContactInfoViewmodel()
{
Data = num.NUMBERDATA,
IsMain = num.ISMAININFO > 0,
Type = num.OBJECTCONTACTNUMBERTYPE.NAME
}).ToList()
}).ToList()
}).ToList()
}).ToList();
this seems to (on average) take about a 4th of the time the original query needed (still a noticeable time due to the size of database but within acceptable limits)

Related

How to write a recursive Linq query in my case?

I have a recursive Group table - a hierarchical structure in which each of the groups refers to the upper one.
I want to build a query in which I provide the initial value of the group and I want to find all the groups that refer to it.
I wrote something like this, but how to make a universal version (the topmost group is one and it has no reference to the next one (GroupId == null)):
var res = from g in _context.Groups
where g.Status == 3
where g.GroupId == initGroupId || g.GroupNavigation.GroupId == initGroupId ||
g.GroupNavigation.GroupNavigation.GroupId == initGroupId
select new GroupExtendedModel
{
Id = g.Id,
TypeName = g.TypeNavigation.Type,
Name = g.Name,
GroupId = g.GroupId,
CompanyId = g.Company,
TypeId = g.Type,
GroupAccessType = GroupAccessType.Down
};

Build Linq queries and concatenate dynamically based on a table

I'm summarizing data to build an alert system. This query counts how many points an employee has based on some criteria. The one below is working fine. However, the query can be longer depending on the User's location. For example, below I have only have 3 unions but for other Employees' locations more unions may be needed. For example in Location B they may also want to add an alert if a ReasonTypeID == is 3 and any entries in the last 90 days.
So I was thinking on building a table that I can add parameters on a location by location basis. I've read about dynamic linq library and I can use that for the WHERE statements but how would I add another query concatenation on the fly?
Here is what I have now.
public class TMAlert
{
public string EmpID { get; set; }
public string FullName { get; set; }
public decimal? PointSummary { get; set; }
public string WarningLabel { get; set; }
public bool AlertFlag { get; set; }
}
IEnumerable<TMAlert> tmAlert = (
from a in allEntries
where a.Date >= DateTime.Now.AddDays(-30) && a.ReasonTypeID == 1
group a by new
{
a.EmpID,
a.FullName,
a.ReasonTypeID
} into g
select new TMAlert
{
EmpID = g.Key.EmpID,
FullName = g.Key.FullName,
WarningLabel = "Last 30 Days",
PointSummary = g.Sum(a => a.Points),
AlertFlag = (g.Sum(a => a.Points) >= 4) ? true : false
}).Concat(from a in allEntries
where a.Date >= DateTime.Now.AddDays(-90) && a.ReasonTypeID == 1
group a by new
{
a.EmpID,
a.FullName,
a.ReasonTypeID
} into g
select new TMAlert
{
EmpID = g.Key.EmpID,
FullName = g.Key.FullName,
WarningLabel = "Last 90 Days",
PointSummary = g.Sum(a => a.Points),
AlertFlag = (g.Sum(a => a.Points) >= 9) ? true : false
}).Concat(from a in allEntries
where a.Date >= (
from o in allEntries
where o.EmpID == a.EmpID && a.WarningTypeID == 2
select (DateTime?)o.Date).Max()
group a by new
{
a.EmpID,
a.FullName,
a.ReasonTypeID
} into g
select new TMAlert
{
EmpID = g.Key.EmpID,
FullName = g.Key.FullName,
WarningLabel = "60 Since Warning type 2 ",
PointSummary = g.Sum(a => a.Points),
AlertFlag = (g.Sum(a => a.Points) >= 4) ? true : false
});
I was thinking to build a criteria table then build the queries on the fly based on the table.
for example
for each entry in criteriaTable
add dynamic linq entry.Param
.concat(....
next....
is this possible or are there better ways to do this?
There is a way to combine linq queries dynamically. I use it to add conditional filters to my where statements when showing a filtered list of data.
I initialize the lambda expression like this
Expression<Func<CatalogItem, bool>> Filter = CatalogItem => true;
then as I go I add to it.
if(!string.IsNullOrEmpty(searchVal)) {Filter = Filter.Compose(CatalogItem => CatalogItem.Title.ToLower().Contains(searchVal), Expression.And);}
I may have different if conditions that cause me to add more and more to my filter then finally I want to apply that filter against my collection.
IQueryable<CatalogItem> FilteredList = Items.AsQueryable().Where(Filter);
In this case Items is a list of CatalogItems that was pulled from the database earlier. I am using IQueryable instead of just List<> cause I need to act on the results afterwards.
List<CatalogItem> Items;//fill Items list here;
I know this is in-line Linq as opposed to the structure you use but I really can't get into the other form of linq querying as this is so quick and dirty for me. If you have to use the other linq form these rules may still apply.

C# MVC API URL string value remove case sensitive

I have C# MVC API URL localhost/api/APIValues?Name=Nick. All working but only issue is when I typed Name=nick it won't display result. because my database table name field store Nick. Also my Database table name field has some data example Nick, ANN, tristan, Abbott,BUD. How do I remove string(Name) case sensitive MVC API values?
Example, how do I setup both way work localhost/api/APIValues?Name=Nick and localhost/api/APIValues?Name=nick.
This is my C# code.
public IEnumerable<NameDTO> Get(string Name = "")
{
var nameList = (from o in db.People.AsEnumerable()
where o.name == Name
join s in db.Employee on
o.empID equals s.empID
select new
{
s.empID,
o.Id
}).ToList();
}
My finally out put should work both name "Nick or nick"
localhost/api/APIValues?Name=Nick
localhost/api/APIValues?Name=nick
You can use Equals with StringComparison:
public IEnumerable<NameDTO> Get(string Name = "")
{
var nameList = (from o in db.People.AsEnumerable()
where o.name.Equals(Name, StringComparison.OrdinalIgnoreCase)
join s in db.Employee on
o.empID equals s.empID
select new
{
s.empID,
o.Id
}).ToList();
}
Try this, I think it may help you:
// Use if you want same records with name you provide
public List<NameDTO> Get(string Name = "")
{
var nameList = (from o in db.People.AsEnumerable()
where o.name.Trim().ToLower() == Name.Trim().ToLower()
join s in db.Employee on
o.empID equals s.empID
select new NameDTO()
{
EmpId = s.empID,
Id = o.Id
}).ToList();
}
//use this if you want similar records from database
public IEnumerable<NameDTO> Get(string Name = "")
{
var nameList = (from o in db.People.AsEnumerable()
where o.name.Trim().ToLower().Contains(Name.Trim().ToLower())
join s in db.Employee on
o.empID equals s.empID
select new NameDTO()
{
EmpId = s.empID,
Id = o.Id
}).ToList();
}
}
Made it as simple as it can get. Whenever my query's don't work in a single line it's my preference to break it down into several components. Feel happy to write a one liner though.
var nameList= db.People.AsEnumerable();
People people = new People();
foreach (var x in nameList)
{
var result = x.name.ToLower() == Name.ToLower();
if (result)
{
people = x;
}
}
var Employee = db.Employee.FirstOrDefault(e => e.EmpId == people.EmpId);
NameDTO nameDTO = new NameDTO()
{
EmpId = Employee.EmpId,
Id = People.Id
};
SQL is not case sensitive. And as long as yo're using a library that converts your code to SQL (such as EF) this shouldn't be an issue.
var nameList = (from o in db.People
where o.name == Name
join s in db.Employee on
o.empID equals s.empID
select new
{
s.empID,
o.Id
}).ToList();
The problem is that you're using AsEnumerable() which actually executes the query and then compares the objects in memory instead of comparing in DB. Watch it in the SQL Profiler and you will see the difference.

Misbehaving LINQ joins

I have a pretty large linq query driving a section of my system, and i am trying to add in a new feature, but the joins seem to be misbehaving.
The old query used to bring back a list of videos which i used to create a view model and only list each video once, with all of the metadata displayed, VisibleStaff, VisiblePlayers and VisibleTeams are all IEnumerable.
Since adding the par tthat drives VisibleStaff, any video with entries in VideosLinkings where the VideoInType flag is set to staff, displays once for each entry, rather than once, and giving me a list of staff members as metadata for VisibleStaff.
I think i am missing a grouping somewhere, but i have tried multiple groups in multiple places and cannot seem to get it right.
Does anyone have any idea where my joins have gone wrong and how i would return a single Video and multiple staff in each VideoModel?
Full Query
from video in Videos
where
video.ClubID == ClubId &&
(VideoFilter.Category == 0 || video.VideoCategoryID == VideoFilter.Category)
join userStaff in Database.Users on video.AddedByUserID.Value equals userStaff.UserID
join videoInTeams in VideoInTeams on video.VideoID equals videoInTeams.VideoID into teamsForVideo
join playerInVideo in Database.PlayersVideos on video.VideoID equals playerInVideo.VideoId into
playersForVideo
join soapVideoLink in Database.VideosLinkings on new {a = video.VideoID, b = VideoInType.SOAPNote}
equals new {a = soapVideoLink.VideoId, b = soapVideoLink.VideoInType} into soapVideoLinks
join staffVideoLink in Database.VideosLinkings on new {a = video.VideoID, b = VideoInType.Staff}
equals new {a = staffVideoLink.VideoId, b = staffVideoLink.VideoInType} into
staffVideoLinks
from svl in staffVideoLinks.DefaultIfEmpty()
join staff in Staff on svl.VideoInKeyId equals staff.StaffID into visibleStaff
let soapLinks = soapVideoLinks.Any(f => f.VideoInKeyId != -1)
let oldExtension =
video.H264Version == "Uploaded"
? ".mp4"
: (video.FlashVersion == "Uploaded" ? ".flv" : video.FileExtension)
where
VideoFilter.ShowSoapVideos || (VideoFilter.ShowSoapVideos == false && soapLinks == false)
orderby video.DateTimeUploaded descending
select new VideoModel
{
Video = video,
Category = video.VideoCategory,
Staff = userStaff.Staff,
ShowDeleteOption = VideoFilter.ShowDeleteOption,
VisibleTeams = teamsForVideo.Select(f => f.Team),
VisiblePlayers = playersForVideo.Select(f => f.Player),
Thumbnail =
video.ThumbnailURL != "" && video.ThumbnailURL != null
? video.ThumbnailURL
: "/Images/Videos/noimage.png",
IsNew = false,
IsMedicalVideo = soapLinks,
VisibleStaff = visibleStaff,
IsStaffVideo = staffVideoLinks.Any()
}
The New Lines
join staffVideoLink in Database.VideosLinkings on new {a = video.VideoID, b = VideoInType.Staff}
equals new {a = staffVideoLink.VideoId, b = staffVideoLink.VideoInType} into
staffVideoLinks
from svl in staffVideoLinks.DefaultIfEmpty()
join staff in Staff on svl.VideoInKeyId equals staff.StaffID into visibleStaff

More efficient way of loading children of entity objects in linq to entity query

I have a rather complex linq to entity query that I'm performing, in the end, I have a result set. I loop through that result set, build business objects and return that list of business objects. it's pretty quick, the problem is that 2 of the child properties are complex objects with their own child objects. for every business object in my loop, I then have to make 2 DB calls to fill its child object. Those 2 calls slow down the overall process, is there a better way to do this? noob to EF here. (EF 4,SQL Server 2008,c#)
Get a result set:
var newresult = from r in result // result is another complex query
join subedit in
(from sa in context.Security_Access
join g in context.Security_UserGroup on sa.EntityID equals g.GroupID
where (sa.PrivledgeID == xx) && g.UserID == userId
select new { user = g.UserID, linkid = sa.LinkID }).Distinct() on new { aid = r.AssetId } equals new { aid = subedit.linkid } into theSubEdit
from subEditAccess in theSubEdit.DefaultIfEmpty()
join subdownload in
(from sa in context.Security_Access
join g in context.Security_UserGroup on sa.EntityID equals g.GroupID
where (sa.PrivledgeID == xx|| sa.PrivledgeID == yy) && g.UserID == userId
select new { user = g.UserID, linkid = sa.LinkID }).Distinct() on new { aid = r.AssetId } equals new { aid = subdownload.linkid } into theSubDownload
from subDownloadAccess in theSubDownload.DefaultIfEmpty()
join subView in
(from sa in context.Security_Access
join g in context.Security_UserGroup on sa.EntityID equals g.GroupID
where (sa.PrivledgeID == xx|| sa.PrivledgeID == yy|| sa.PrivledgeID == 101) && g.UserID == userId
select new { user = g.UserID, linkid = sa.LinkID }).Distinct() on new { aid = r.AssetId } equals new { aid = subView.linkid } into theSubView
from subViewAccess in theSubView.DefaultIfEmpty()
select new { r, EditAccess = (int?)subEditAccess.user, DownloadAccess = (int?)subDownloadAccess.user, ViewAccess = (int?)subViewAccess.user };
I then loop through that result set:
foreach (var asset in newresult)
{
// and build a new business object, set its properties
BoAsset boAsset = new BoAsset();
boAsset.HasEditRights = (asset.EditAccess > 0);
boAsset.HasDownloadRights = (asset.DownloadAccess > 0);
boAsset.HasViewRights = (asset.ViewAccess > 0);
boAsset.Description = asset.r.Description;
boAsset.DetailedDescription = asset.r.DetailedDescription;
boAsset.Keywords = asset.r.Keywords;
boAsset.Notes = asset.r.Notes;
boAsset.Photographer = asset.r.Photographer;
boAsset.PhotographerEmail = asset.r.PhotographerEmail;
boAsset.Notes = asset.r.Notes;
boAsset.Author = asset.r.Author;
// these 2 properties i've commented out are
// complex objects/entities, setting them the way I am
// requires me to call 2 separate methods which make 2 DB trips
// per business object.
//boAsset.Domains = GetAssetDomains(asset.r.AssetId);
//boAsset.DomainEntries = GetAssetCustomDomains(asset.r.AssetId);
myListofObjects.Add(boAsset);
}
return myListofObjects;
Is there a better way?
Just add this .Include("Domains").Include("DomainEntries") to your Linq in in context.Security_Access That should get rows from those tables all in one go.
So your "inner" queries would look like:
from sa in context.Security_Access.Include("Domains").Include("DomainEntries")
join g in context.Security_UserGroup on sa.EntityID equals g.GroupID
where (sa.PrivledgeID == xx) && g.UserID == userId
select new { ...
Here is the documentation from MS: http://msdn.microsoft.com/en-us/library/bb738708.aspx
If you want to improve your performance use compile queries !
You can check the example here.
static readonly Func<AdventureWorksEntities, Decimal,
IQueryable<SalesOrderHeader>> s_compiledQuery2 =
CompiledQuery.Compile<AdventureWorksEntities, Decimal, IQueryable<SalesOrderHeader>>((ctx, total) =>
from order in ctx.SalesOrderHeaders.Include("Orders") where order.TotalDue >= total select order);
MSDN
AND
You can Introduce Include suppose to select all the employees along with their departments . If you have a navigational property, you won't need a join at all. You can use Include like this:
List<Employee> employeesWithDepartments = CreateObjectSet<Employee>().
Include(e => e.Department).
ToList();

Categories