I have the following LINQ query:
//two different contexts, databases, tables...
NoteSet = lmCtx.LMNotes.AsEnumerable();
EmpSet = tessCtx.Employees.AsEnumerable();
var lmAccountNotes = (from lmnote in NoteSet
join createdby in EmpSet on lmnote.lnt_createdById equals createdby.EmployeeID
join modifiedby in EmpSet on lmnote.lnt_modifiedById equals modifiedby.EmployeeID
where lmnote.lnt_recordId == 5566 && lmnote.lnt_tableId == 1
select new NoteInfo { Note = lmnote, CreatedBy = createdby, ModifiedBy = modifiedby }).ToList();
This works for queries on small tables, but NoteSet is a pretty big table and I'm reaching well over 1.5GB of used memory by the process before the framework just explodes and throw an OutOfMemory exception.
Is there any way to keep the lazy loading feature while executing something like this ?
So as to keep having a query that returns a NoteInfo object, I changed it to this:
//LMNotes is the actual huge database...
var m = lmCtx.LMNotes.Where(x => x.lnt_recordId == 5566).ToList();
var lmAccountNotes = (from lmnote in m
join createdby in EmpSet on lmnote.lnt_createdById equals createdby.EmployeeID
join modifiedby in EmpSet on lmnote.lnt_modifiedById equals modifiedby.EmployeeID
where lmnote.lnt_recordId == 566 && lmnote.lnt_tableId == 1
select new NoteInfo { Note = lmnote, CreatedBy = createdby, ModifiedBy = modifiedby }).ToList();
This is better
As explained in the comments, you cannot really run a single query across two different database, at least not without setting up some help construct (which would then live on either database, and actually who knows if that would improve the performance at all).
However, that does not mean that we cannot improve your query at all. If we can’t rely on the database engine to execute the query, we can do that ourselves. In this case, what you are doing is essentially just a query on the LMNotes entity and then you join employees from the Employees set.
So a naive solution could look like this:
var notes = lmCtx.LMNotes
.Where(lmnote => lmnote.lnt_recordId == 5566 && lmnote.lnt_tableId == 1)
.Select(lmnote =>
{
return new NoteInfo
{
Note = lmnote,
CreatedBy = tessCtx.Employees.FirstOrDefault(e => e.EmployeeId == lmnote.lnt_createdById),
ModifiedBy = tessCtx.Employees.FirstOrDefault(e => e.EmployeeId == lmnote.lnt_modifiedById)
};
})
.ToList();
Of course, while this runs a single query on LMNotes, this still runs two separate queries for each note in the result. So it’s not really better than what EF would have done up there.
What we can do however is add some lookup. I suspect that the set of employees is somewhat limited, so it would make sense to only fetch each employee once. Something like this:
private Dictionary<int, Employee> employees = new Dictionary<int, Employee>();
private Employee GetEmployee(int employeeId)
{
Employee employee;
if (!employees.TryGetValue(employeeId, out employee))
{
employee = tessCtx.Employees.FirstOrDefault(e => e.EmployeeId == employeeId);
employees[employeeId] = employee;
}
return employee;
}
public List<NoteInfo> GetNotes()
{
return lmCtx.LMNotes
.Where(lmnote => lmnote.lnt_recordId == 5566 && lmnote.lnt_tableId == 1)
.Select(lmnote =>
{
return new NoteInfo
{
Note = lmnote,
CreatedBy = GetEmployee(lmnote.lnt_createdById),
ModifiedBy = GetEmployee(lmnote.lnt_modifiedById)
};
})
.ToList();
}
This would only look up each employee once and then cache the employee object.
Alternatively, you could also make a second pass here and fetch all employees at once after reading the notes for the first time. Something like this:
public List<NoteInfo> GetNotes()
{
var employeeIds = new HashSet<int>();
var notes = lmCtx.LMNotes
.Where(lmnote => lmnote.lnt_recordId == 5566 && lmnote.lnt_tableId == 1)
.Select(lmnote =>
{
// remember the ids for later
employeeIds.Add(lmnote.lnt_createdById);
employeeIds.Add(lmnote.lnt_modifiedById);
return new NoteInfo
{
Note = lmnote,
CreatedBy = null,
ModifiedBy = null
};
})
.ToList();
var employees = tessCtx.Employees
.Where(e => employeeIds.Contains(e.EmployeeId))
.ToList()
.ToDictionary(e => e.EmployeeId);
foreach (var noteInfo in notes)
{
noteInfo.CreatedBy = employees[noteInfo.Note.lnt_createdById];
noteInfo.ModifiedBy = employees[noteInfo.Note.lnt_modifiedById];
}
return notes;
}
This would only run a single query against each database.
Related
I have this code to select the city for each person whose Id matches from the list of cities:
public List<PersonelDto> MapPersonelDto(List<Personel> personels,List<City> cities)
{
var result = new List<PersonelDto>();
foreach (var item in personels)
{
var personel = new PersonelDto
{
Id = item.Id,
Maaş = item.Salary,
MedeniHal = item.MartialStatus,
Meslek = item.Job,
Soyisim = item.LastName,
Şehir = (from s in cities where s.Id == item.CityId select s.name).ToString()
};
result.Add(personel);
}
return result;
}
But City's value come out like this:
System.Linq.Enumerable+WhereSelectListIterator`2[Personnel_Registration.City,System.String]
How can I fix this?
The error is because there's nothing in the type system metadata to guarantee you won't have more than one city match, and so the result of the expression is a potential collection of 0, 1, or more cities. What you see is the result of calling .ToString() on that collection.
To fix it, you can do this:
public IEnumerable<PersonelDto> MapPersonelDto(IEnumerable<Personel> personels, IEnumerable<City> cities)
{
return personels.Select( p => {
new PersonelDto() {
Id = item.Id,
Maaş = item.Salary,
MedeniHal = item.MartialStatus,
Meslek = item.Job,
Soyisim = item.LastName,
Şehir = string.Join(",", cities.Where(c => c.Id == p.CityId).Select(c=> c.name));
}
});
}
Or, if you're confident you only want one city, you could do this:
public IEnumerable<PersonelDto> MapPersonelDto(IEnumerable<Personel> personels, IEnumerable<City> cities)
{
return personels.Select( p => {
new PersonelDto() {
Id = item.Id,
Maaş = item.Salary,
MedeniHal = item.MartialStatus,
Meslek = item.Job,
Soyisim = item.LastName,
Şehir = cities.FirstOrDefault(c => c.Id == p.CityId)?.name;
}
});
}
Note the complete lack of Lists. You really ought to get out of the habit of using List<T> everywhere, and instead let things remain as an IEnumerable<T> as much as possible. This can make your code faster and MUCH more efficient with RAM use.
Another benefit is it makes your code more flexible. For example, you can still pass the existing Lists to this new method. And if for some reason you need the result to be a List<> for the next method call (hint: you probably don't really need this at all) you can always add a .ToList() after calling the method. But, again, don't do that unless you really have to!
from s in cities where s.Id == item.CityId select s.name
return an IEnumerable - there might be more than one city
do instead
Şehir = (from s in cities where s.Id == item.CityId select s.name).FirstOrDefault().ToString()
which selects the first element
Note that this assumes that there is always a matching city. If not then you should supply a default value
Şehir = (from s in cities where s.Id == item.CityId select s.name).FirstOrDefault("unknow city").ToString()
This is a .NET Core Web API Task method. I have a flat table that I need to convert into a nested DTOs. The first DTO works but I can't seem to get the second DTO to nest after grouping.
I know I have done the grouping correctly. I am just not sure the second level nesting of the DTO is done correctly, it complains about not being able to translate to some type.
LINQ Query to put data in a nested object
Can someone point me in the right track?
public async Task<List<PointCardViewModel>> GetPointCards() {
var data = (from s in db.Students
join dc in db.DailyCards on s.StudentId equals dc.StudentId
join dcli in db.DailyCardLineItems on dc.CardId equals dcli.CardId
join dcob in db.DailyCardOtherBehaviors on dc.CardId equals dcob.CardId
select new
{
s.StudentName,
s.StudentGrade,
dc.CardId,
dc.CardDate,
dcli.ClassParticipationPoints,
dcli.AssignmentCompletionPoints,
dcli.BonusHomeworkPoints,
dcli.ClassPeriod,
dcob.PersonalAppearancePoints,
dcob.LunchPoints,
dcob.RecessOtherPoints,
dcob.AmHomeroomPoints,
dcob.PmHomeroomPoints
});
var queryPointCards = (data
.GroupBy(x => new
{
x.CardId,
x.StudentGrade,
x.StudentName,
x.CardDate,
x.PersonalAppearancePoints,
x.LunchPoints,
x.RecessOtherPoints,
x.AmHomeroomPoints,
x.PmHomeroomPoints
})
.Select(x => new PointCardViewModel()
{
CardId = x.Key.CardId,
StudentName = x.Key.StudentName,
Grade = x.Key.StudentGrade,
EvaluationDate = x.Key.CardDate,
PersonalAppearancePoints = x.Key.PersonalAppearancePoints,
LunchPoints = x.Key.LunchPoints,
RecessOtherPoints = x.Key.RecessOtherPoints,
AMHomeRoomPoints = x.Key.AmHomeroomPoints,
PMHomeRoomPoints = x.Key.PmHomeroomPoints,
//LineItems = null --> This works!! But not the below
LineItems = x.Select(c => new LineItemViewModel
{
ClassPeriod = c.ClassPeriod,
BonusHomeworkPoints = c.BonusHomeworkPoints,
ClassParticipationPoints = c.ClassParticipationPoints,
AssignmentCompletionPoints = c.AssignmentCompletionPoints
})
}
)
).ToListAsync();
if (db != null)
{
return await queryPointCards;
}
return null;
}
You have hit limitation of Grouping. After groping you cannot access to group items. Only fields from 'Key' and aggregation functions are allowed.
So just put data.AsEnumerable() and do grouping on the client side.
I am trying to use projection in my query and get the following error:
"The result operation MongoDB.Driver.Linq.Expressions.ResultOperators.ListResultOperator is not supported."
Here is the code:
public async Task<IEnumerable<Listing>> LoadAllUserListings(string userId)
{
var result = _context.Listing.Aggregate().Match(l => l.OwnerId == userId || l.Sales.Any(a => a.Owner.Id == userId)).
Project(l => new Listing
{
Id = l.Id,
Reference = l.Reference,
OwnerId = l.OwnerId,
Sales = l.Sales.Where(a => a.Owner.Id == userId || a.Manager.Id == userId).ToList(),
Products = l.Products,
Status = l.Status,
DueDate = l.DueDate
}).ToListAsync();
return await result;
}
It does not appear to like the ToListAsync call. I got this code snippet from the following answer:
https://stackoverflow.com/questions/50904811/mongodb-c-sharp-filter-and-get-all-subdocuments
There reason I am using projection is to omit some fields which the user should not see (depending on the role). Any help on this would be appreciated.
Thanks in advance.
The problem occurs in that line:
Sales = l.Sales.Where(a => a.Owner.Id == userId || a.Manager.Id == userId).ToList()
What happens here ? MongoDB driver takes this expression and tries to translate it to aggregation framework syntax. There's a $filter operator which can be run on nested collection and driver is able to translate .Where() to that operator however there's nothing corresponding for .ToList() at the end of that expression and that's why it fails.
So the fix is fairly easy: you just need to use IEnumerable<T> instead of List<T> for Sales property and then get rid of that .ToList() so your code will look like this:
public async Task<IEnumerable<Listing>> LoadAllUserListings(string userId)
{
var result = _context.Listing.Aggregate().Match(l => l.OwnerId == userId || l.Sales.Any(a => a.Owner.Id == userId)).
Project(l => new Listing
{
Id = l.Id,
Reference = l.Reference,
OwnerId = l.OwnerId,
Sales = l.Sales.Where(a => a.Owner.Id == userId || a.Manager.Id == userId),
Products = l.Products,
Status = l.Status,
DueDate = l.DueDate
}).ToListAsync();
return await result;
}
I have to to a rather large request to a database to fetch a bunch of data, it's however taking a noticeable time to run. is there some way to increase the performance on this? preemptive apologies for the ugly code (I did have a version that segmented this into multiple smaller functions but that was even slower)
from contact in _database.OBJECTCONTACT
where contact.OBJECTCONTACTOWNER.Any(o => o.OBJECTID == id && o.OBJECTTYPE == type) && contact.ACTIVE >= 1 && CheckViewAccess(contact)
group contact by (contact.OBJECTCONTACTPROJECT.Any() ? contact.OBJECTCONTACTPROJECT.First().OBJECTPROJECT.PROJECTNAME : "General") into projectGroup
select new ProjectViewModel()
{
ProjectName = projectGroup.Key,
ContactGroups = (from g in _database.OBJECTGROUP
where g.GROUPTYPE == "CONTACT" && ContactsModule.CheckUserRole("View", g.OBJECTTYPE, g.GROUPNAME)
select new ContactGroupViewModel()
{
CanEdit = ContactsModule.CheckUserRole("Edit", g.OBJECTTYPE, g.GROUPNAME),
GroupId = g.OBJECTGROUPID,
GroupName = g.GROUPNAME,
Contacts = (from c in projectGroup
join l in _database.OBJECTCONTACTLOCATION on c.OBJECTCONTACTLOCATIONID equals l.OBJECTCONTACTLOCATIONID into lgrp from loc in lgrp.DefaultIfEmpty(null)
orderby c.NAME
select new ContactViewModel()
{
Id = (int)c.OBJECTCONTACTID,
Name = c.NAME,
Description = c.DESCRIPTION,
ContactInformation = CreateContactInfoViewmodels(c),
Owners = c.OBJECTCONTACTOWNER.Where(owner => owner.OBJECTTYPE == "AIRPORT")
.Select(owner => ContactOwnerViewModel.FromOwnerId(owner.OBJECTID, owner.OBJECTTYPE)).ToList(),
Projects = c.OBJECTCONTACTPROJECT.Select(proj => proj.OBJECTPROJECT).ToList(),
Typename = GetTypeName(c),
TypeId = c.OBJECTCONTACTTYPEID ?? 0,
ContactGroupId = c.OBJECTGROUPID,
ContactGroup = g.GROUPNAME,
Editable = CheckAccessBool("EDIT", c),
Location = loc != null ? new LocationViewModel()
{
Address = loc.ADDRESS,
GoogleMapLink = loc.GMAPADDRESS,
LocationId = loc.OBJECTCONTACTLOCATIONID,
LatLon = Tuple.Create(loc.LATITUDE, loc.LONGITUDE)
} : null,
}).ToList()
}).ToList()
}).ToList();
I think I should be able to use joins to move the entire DB fetch code to the top (theoretically improving perfomance) but I am having trouble finding the syntax which would suit my needs
Thanks everyone for coming with suggestions. I am in a situation where I'm not able to do much with the database itself so I'm making the best of what I have. my hands a bit tied in regards to the tools at my disposal (also fairly old codebase I think it's EF 5 or something like that)
this version moves the DB transaction to the top (so that is fewer fetches) and does a lot of data manipulation at the bottom.
// general object is created above
var res = (from contact in _database.OBJECTCONTACT.AsEnumerable() // as enumerable used to allow for defaultifempty in join (minor damage to performance)
join oGroup in _database.OBJECTGROUP on contact.OBJECTGROUPID equals oGroup.OBJECTGROUPID into og from objectGroup in og.DefaultIfEmpty(defaultValue: general)
where contact.OBJECTCONTACTOWNER.Any(o => o.OBJECTTYPE == type && o.OBJECTID == id)
// ReSharper disable once PossibleNullReferenceException (it's taken care of by check using .any() )
group new {contact, objectGroup } by (contact.OBJECTCONTACTPROJECT.Any() ? contact.OBJECTCONTACTPROJECT.FirstOrDefault().OBJECTPROJECT.PROJECTNAME : "General") into pGroup
orderby pGroup.Key == "General" ? pGroup.Key : "" descending
select new ProjectViewModel()
{
ProjectName = pGroup.Key,
ProjectId = pGroup.FirstOrDefault() != null ? (pGroup.FirstOrDefault().contact.OBJECTCONTACTPROJECT.FirstOrDefault() != null ? pGroup.FirstOrDefault().contact.OBJECTCONTACTPROJECT.FirstOrDefault().OBJECTPROJECTID : -1) : -1,
ContactGroups = (from c in pGroup
group c by c.objectGroup into grp
let canEdit = ContactsModule.CheckUserRole("EDIT", grp.Key.OBJECTTYPE, grp.Key.GROUPNAME)
orderby grp.Key.SORTORDER descending
select new ContactGroupViewModel()
{
GroupName = grp.Key.GROUPNAME,
GroupId = grp.Key.OBJECTGROUPID,
CanEdit = canEdit,
Contacts = grp.Select(item => new ContactViewModel()
{
Id = (int)item.contact.OBJECTCONTACTID,
Name = item.contact.NAME,
Description = item.contact.DESCRIPTION,
Editable = canEdit,
ContactInformation = item.contact.OBJECTCONTACTNUMBER.OrderByDescending(num => num.ISMAININFO).Select(num => new ContactInfoViewmodel()
{
Data = num.NUMBERDATA,
IsMain = num.ISMAININFO > 0,
Type = num.OBJECTCONTACTNUMBERTYPE.NAME
}).ToList()
}).ToList()
}).ToList()
}).ToList();
this seems to (on average) take about a 4th of the time the original query needed (still a noticeable time due to the size of database but within acceptable limits)
I have a rather complex linq to entity query that I'm performing, in the end, I have a result set. I loop through that result set, build business objects and return that list of business objects. it's pretty quick, the problem is that 2 of the child properties are complex objects with their own child objects. for every business object in my loop, I then have to make 2 DB calls to fill its child object. Those 2 calls slow down the overall process, is there a better way to do this? noob to EF here. (EF 4,SQL Server 2008,c#)
Get a result set:
var newresult = from r in result // result is another complex query
join subedit in
(from sa in context.Security_Access
join g in context.Security_UserGroup on sa.EntityID equals g.GroupID
where (sa.PrivledgeID == xx) && g.UserID == userId
select new { user = g.UserID, linkid = sa.LinkID }).Distinct() on new { aid = r.AssetId } equals new { aid = subedit.linkid } into theSubEdit
from subEditAccess in theSubEdit.DefaultIfEmpty()
join subdownload in
(from sa in context.Security_Access
join g in context.Security_UserGroup on sa.EntityID equals g.GroupID
where (sa.PrivledgeID == xx|| sa.PrivledgeID == yy) && g.UserID == userId
select new { user = g.UserID, linkid = sa.LinkID }).Distinct() on new { aid = r.AssetId } equals new { aid = subdownload.linkid } into theSubDownload
from subDownloadAccess in theSubDownload.DefaultIfEmpty()
join subView in
(from sa in context.Security_Access
join g in context.Security_UserGroup on sa.EntityID equals g.GroupID
where (sa.PrivledgeID == xx|| sa.PrivledgeID == yy|| sa.PrivledgeID == 101) && g.UserID == userId
select new { user = g.UserID, linkid = sa.LinkID }).Distinct() on new { aid = r.AssetId } equals new { aid = subView.linkid } into theSubView
from subViewAccess in theSubView.DefaultIfEmpty()
select new { r, EditAccess = (int?)subEditAccess.user, DownloadAccess = (int?)subDownloadAccess.user, ViewAccess = (int?)subViewAccess.user };
I then loop through that result set:
foreach (var asset in newresult)
{
// and build a new business object, set its properties
BoAsset boAsset = new BoAsset();
boAsset.HasEditRights = (asset.EditAccess > 0);
boAsset.HasDownloadRights = (asset.DownloadAccess > 0);
boAsset.HasViewRights = (asset.ViewAccess > 0);
boAsset.Description = asset.r.Description;
boAsset.DetailedDescription = asset.r.DetailedDescription;
boAsset.Keywords = asset.r.Keywords;
boAsset.Notes = asset.r.Notes;
boAsset.Photographer = asset.r.Photographer;
boAsset.PhotographerEmail = asset.r.PhotographerEmail;
boAsset.Notes = asset.r.Notes;
boAsset.Author = asset.r.Author;
// these 2 properties i've commented out are
// complex objects/entities, setting them the way I am
// requires me to call 2 separate methods which make 2 DB trips
// per business object.
//boAsset.Domains = GetAssetDomains(asset.r.AssetId);
//boAsset.DomainEntries = GetAssetCustomDomains(asset.r.AssetId);
myListofObjects.Add(boAsset);
}
return myListofObjects;
Is there a better way?
Just add this .Include("Domains").Include("DomainEntries") to your Linq in in context.Security_Access That should get rows from those tables all in one go.
So your "inner" queries would look like:
from sa in context.Security_Access.Include("Domains").Include("DomainEntries")
join g in context.Security_UserGroup on sa.EntityID equals g.GroupID
where (sa.PrivledgeID == xx) && g.UserID == userId
select new { ...
Here is the documentation from MS: http://msdn.microsoft.com/en-us/library/bb738708.aspx
If you want to improve your performance use compile queries !
You can check the example here.
static readonly Func<AdventureWorksEntities, Decimal,
IQueryable<SalesOrderHeader>> s_compiledQuery2 =
CompiledQuery.Compile<AdventureWorksEntities, Decimal, IQueryable<SalesOrderHeader>>((ctx, total) =>
from order in ctx.SalesOrderHeaders.Include("Orders") where order.TotalDue >= total select order);
MSDN
AND
You can Introduce Include suppose to select all the employees along with their departments . If you have a navigational property, you won't need a join at all. You can use Include like this:
List<Employee> employeesWithDepartments = CreateObjectSet<Employee>().
Include(e => e.Department).
ToList();