We Keep Coding

linq group by to generate nested POCO

I have four tables joined to produce data something like below:
Name Grade CardID Date Class Listen Read Write
Jane Doe A 1001 2020-10-01 Period 1 - Spanish 500 500 500
John Doe B+ 1002 2010-10-02 Pereiod 2 - English 1000 1000 1000
Jane Doe A 1001 2020-10-01 Period 3 - Englsih 500 1000 1000
How do I convert the above data into a nested form like below using LINQ group by? This is a .NET CORE WEB API project and uses DTO objects projections from the LINQ query data.
[
{
"cardId": 1001,
"studentName": "Jane Doe",
"grade": "A",
"evaluationDate": "2020-10-01T00:00:00",
"Period 1 - Spanish": {
"Listen": 1000,
"Read": 500,
"Write": 500
},
"Period 3 - English": {
"Listen": 1000,
"Read": 500,
"Write": 1000
}
},
{
"cardId": 1002,
"studentName": "John Doe",
"grade": "B+",
"evaluationDate": "2010-10-01T00:00:00",
"Period 2 - English": {
"Listen": 500,
"Read": 500,
"Write": 1000
}
}
]
Below I have two viewModel classes which I am using to generate the nested POCO data stracture to be returned from the query. If I don't use GroupBy, I can generate a simple unnested POCO but I don't want to repeat the response data as separate object. This is for a .NET core web api project .
I feel like I am close, but the group by in LINQ is throwing me off...
public class PointCardViewModel
{
public int CardId { get; set; }
public string StudentName { get; set; }
public string Grade { get; set; }
public DateTime EvaluationDate { get; set; }
public IEnumerable<LineItemViewModel> LineItems { get; set; }
}
public class LineItemViewModel
{
public string ClassPeriod { get; set; }
public int Listen { get; set; }
public int Read { get; set; }
public int Write { get; set; }
}
((from s in db.Students
join dc in db.DailyCards on s.StudentId equals dc.StudentId
join dcli in db.DailyCardLineItems on dc.CardId equals dcli.CardId
join dcob in db.DailyCardOtherBehaviors on dc.CardId equals dcob.CardId
select new
{
s.StudentName,
s.StudentGrade,
dc.CardId,
dc.CardDate,
dcli.ClassParticipationPoints,
dcli.AssignmentCompletionPoints,
dcli.BonusHomeworkPoints,
dcli.ClassPeriod
})
.GroupBy(x => x.CardId)
.Select(g => new PointCardViewModel()
{
CardId = g.Key,
StudentName = g.Select(c => c.StudentName).First(),
Grade = g.Select(c => c.StudentGrade).First(),
EvaluationDate = x.CardDate,
LineItems = g.Select(y => new LineItemViewModel()
{
//Class
//Read
//Listen
//Write
})
}).toList()
Update:
After understanding multiple group By in lINQ, my .NET Core WEB API is still complaining about bad request and doesn't return the nested JSON. I did update the LineItems prop to be IDictionary type with the decorator. Interestingly, if I comment out the DTO portion of LineItems and set it to null, the response comes back fine. Can you help what the issue is here?
public async Task<List<PointCardViewModel>> GetPointCards()
{
var queryPointCards =
((from s in db.Students
join dc in db.DailyCards on s.StudentId equals dc.StudentId
join dcli in db.DailyCardLineItems on dc.CardId equals dcli.CardId
join dcob in db.DailyCardOtherBehaviors on dc.CardId equals dcob.CardId
select new
{
s.StudentName,
s.StudentGrade,
dc.CardId,
dc.CardDate,
dcli.ClassParticipationPoints,
dcli.AssignmentCompletionPoints,
dcli.BonusHomeworkPoints,
dcli.ClassPeriod,
dcob.PersonalAppearancePoints,
dcob.LunchPoints,
dcob.RecessOtherPoints,
dcob.AmHomeroomPoints,
dcob.PmHomeroomPoints
})
.GroupBy(x => new {
x.CardId,
x.StudentGrade,
x.StudentName,
x.CardDate,
x.PersonalAppearancePoints,
x.LunchPoints,
x.RecessOtherPoints,
x.AmHomeroomPoints,
x.PmHomeroomPoints
})
.Select(x => new PointCardViewModel
{
CardId = x.Key.CardId,
StudentName = x.Key.StudentName,
Grade = x.Key.StudentGrade,
EvaluationDate = x.Key.CardDate,
PersonalAppearancePoints = x.Key.PersonalAppearancePoints,
LunchPoints = x.Key.LunchPoints,
RecessOtherPoints = x.Key.RecessOtherPoints,
AMHomeRoomPoints = x.Key.AmHomeroomPoints,
PMHomeRoomPoints = x.Key.PmHomeroomPoints,
LineItems = null
//x.Select(c => new LineItemViewModel
//{
// ClassPeriod = c.ClassPeriod,
// ClassParticipationPoints = c.ClassParticipationPoints,
// AssignmentCompletionPoints = c.AssignmentCompletionPoints,
// BonusHomeworkPoints = c.BonusHomeworkPoints
//}).ToDictionary(key => key.ClassPeriod, value => (object)value)
}
)
).ToListAsync();
if (db != null)
{
return await queryPointCards;
}
return null;
}

You could achieve this with a slight change in your query and resultant Data structure. For example
Changing your Data Structures as
public class PointCardViewModel
{
public int CardId { get; set; }
public string StudentName { get; set; }
public string Grade { get; set; }
public DateTime EvaluationDate { get; set; }
[JsonExtensionData]
public IDictionary<string, object> LineItems { get; set; } //Change Here
}
public class LineItemViewModel
{
public string ClassPeriod { get; set; }
public int Listen { get; set; }
public int Read { get; set; }
public int Write { get; set; }
}
Note that the LineItems has been converted to a Dictionary and decorated with JsonExtensionDataAttribute.
And now you could Change your Group By Query as
.GroupBy(x=> new {x.Name,x.Grade,x.CardID,x.Date})
.Select(x=> new PointCardViewModel
{
CardId=x.Key.CardID,
StudentName = x.Key.Name,
Grade = x.Key.Grade,
EvaluationDate = x.Key.Date,
LineItems = x.Select(c=> new LineItemViewModel
{
ClassPeriod = c.Class,
Listen = c.Listen,
Read = c.Read,
Write = c.Write
}).ToDictionary(key=>key.ClassPeriod,value=>(object)value)
});
Serializing the resultant data would give the required Json
Demo Code

Change the Group by and Select as below:
var result=((from s in db.Students
join dc in db.DailyCards on s.StudentId equals dc.StudentId
join dcli in db.DailyCardLineItems on dc.CardId equals dcli.CardId
join dcob in db.DailyCardOtherBehaviors on dc.CardId equals dcob.CardId
select new
{
s.StudentName,
s.StudentGrade,
dc.CardId,
dc.CardDate,
dcli.ClassParticipationPoints,
dcli.AssignmentCompletionPoints,
dcli.BonusHomeworkPoints,
dcli.ClassPeriod
})
.GroupBy(x => new { x.StudentName, x.CardId, x.StudentGrade, x.CardDate})
.Select(g => new PointCardViewModel()
{
CardId =g.Key.CardId,
StudentName = g.Key.StudentName,
Grade = g.Key.StudentGrade,
EvaluationDate = g.Key.CardDate,
LineItems = g.Select(y => new LineItemViewModel
{
Class=y.Class,
Read=y.ClassParticipationPoints,
Listen=y.AssignmentCompletionPoints,
Write=y.BonusHomeworkPoints
})
}).toList()

Converting SQL query to LINQ or LINQ fluent Syntax

I have SQL query like this
SELECT T.*
FROM
(
SELECT ServiceRecords.DistrictId, Districts.Name as DistrictName, COUNT(Distinct(NsepServiceRecords.ClientRegNo)) AS ClientsServedCount
FROM ServiceRecords
INNER JOIN Districts ON ServiceRecords.DistrictId = Districts.ID
INNER JOIN NsepServiceRecords ON NsepServiceRecords.ServiceRecordId = ServiceRecords.Id
WHERE ServiceRecords.CreatedAtUtc >= #StartDate
AND ServiceRecords.CreatedAtUtc <= #EndDate
AND ServiceRecords.DistrictId = #DistrictId
GROUP BY ServiceRecords.DistrictId, Districts.Name
) AS T
ORDER BY T.DistrictName ASC, T.DistrictId
Query results:
DistrictId DistrictName ClientsServedCount
8d059005-1e6b-44ad-bc2c-0b3264fb4567 Bahawalpur 117
27ab6e24-50a6-4722-8115-dc31cd3127fa Gujrat 492
14b648f3-4912-450e-81f9-bf630a3dfc72 Jhelum 214
8c602b99-3308-45b5-808b-3375d61fdca0 Lodhran 23
059ffbea-7787-43e8-bd97-cab7cb77f6f6 Muzafarghar 22
580ee42b-3516-4546-841c-0bd8cef04df9 Peshawar 211
I'm struggling converting this to LINQ to entities query. I want to get same results (except District Id column) using LINQ.
I have tried like this, but not working as expected. Can somebody tell me what I'm doing wrong?
_dbContext.ServiceRecords
.Include(x => x.District)
.Include(x=>x.NsepServiceRecords)
.GroupBy(x => x.DistrictId)
.Select(x => new DistrictClientsLookUpModel
{
DistrictName = x.Select(record => record.District.Name).FirstOrDefault(),
ClientsServedCount = x.Sum(t=> t.NsepServiceRecords.Count)
});
Model classes are like this
public class BaseEntity
{
public Guid Id { get; set; }
}
public class NsepServiceRecord : BaseEntity
{
public DateTime CreatedAtUtc { get; set; }
public Guid ServiceRecordId { get; set; }
public string ClientRegNo { get; set; }
// other prop .......
public virtual ServiceRecord ServiceRecord { get; set; }
}
public class ServiceRecord : BaseEntity
{
public DateTime CreatedAtUtc { get; set; }
public string DistrictId { get; set; }
public virtual District District { get; set; }
public virtual ICollection<NsepServiceRecord> NsepServiceRecords { get; set; }
}
public class DistrictClientsLookUpModel
{
public string DistrictName { get; set; }
public int ClientsServedCount { get; set; }
}
I'm using Microsoft.EntityFrameworkCore, Version 2.2.4
EDIT
I have also tried like this
var startUniversalTime = DateTime.SpecifyKind(request.StartDate, DateTimeKind.Utc);
var endUniversalTime = DateTime.SpecifyKind(request.EndDate, DateTimeKind.Utc);
return _dbContext.NsepServiceRecords
.Join(_dbContext.ServiceRecords, s => s.ServiceRecordId,
r => r.Id, (s, r) => r)
.Include(i => i.District)
.Where(x => x.DistrictId == request.DistrictId
&& x.CreatedAtUtc.Date >= startUniversalTime
&& x.CreatedAtUtc.Date <= endUniversalTime)
.OrderBy(x => x.DistrictId)
.GroupBy(result => result.DistrictId)
.Select(r => new DistrictClientsLookUpModel
{
DistrictName = r.Select(x=>x.District.Name).FirstOrDefault(),
ClientsServedCount = r.Sum(x=>x.NsepServiceRecords.Count())
});
Another try,
from s in _dbContext.ServiceRecords
join record in _dbContext.NsepServiceRecords on s.Id equals record.ServiceRecordId
join district in _dbContext.Districts on s.DistrictId equals district.Id
group s by new
{
s.DistrictId,
s.District.Name
}
into grp
select new DistrictClientsLookUpModel
{
DistrictName = grp.Key.Name,
ClientsServedCount = grp.Sum(x => x.NsepServiceRecords.Count)
};
It takes too long, I waited for two minutes before I killed the request.
UPDATE
EF core have issues translating GroupBy queries to server side

Assuming the District has a collection navigation property to ServiceRecord as it should, e.g. something like
public virtual ICollection<ServiceRecord> ServiceRecords { get; set; }
you can avoid the GroupBy by simply starting the query from District and use simple projection Select following the navigations:
var query = _dbContext.Districts
.Select(d => new DistrictClientsLookUpModel
{
DistrictName = d.Name,
ClientsServedCount = d.ServiceRecords
.Where(s => s.CreatedAtUtc >= startUniversalTime && s.CreatedAtUtc <= endUniversalTime)
.SelectMany(s => s.NsepServiceRecords)
.Select(r => r.ClientRegNo).Distinct().Count()
});

You don't appear to be doing a join properly.
Have a look at this:
Join/Where with LINQ and Lambda
Here is a start on the linq query, I'm not sure if this will give you quite what you want, but its a good start.
Basically within the .Join method you need to first supply the entity that will be joined. Then you need to decide on what they will be joined on, in this case district=> district.Id, serviceRecord=> serviceRecord.Id.
_dbContext.ServiceRecords
.Join( _dbContext.District,district=> district.Id, serviceRecord=> serviceRecord.Id)
.Join(_dbContext.NsepServiceRecords, Nsep=> Nsep.ServiceRecord.Id,district=>district.Id)
.GroupBy(x => x.DistrictId)
.Select(x => new DistrictClientsLookUpModel
{
DistrictName = x.Select(record => record.District.Name).FirstOrDefault(),
ClientsServedCount = x.Sum(t=> t.NsepServiceRecords.Count)
});

How to use List.Distinct to populate a custom class?

I have the following two classes (in C#)
public class courseList
{
public string MajorName { get; set; }
public string MajorNameID { get; set; }
public string CourseID { get; set; }
public string CourseName { get; set; }
}
public class CourceNames
{
[DataMember]
public string CourseID { get; set; }
[DataMember]
public string CourseName { get; set; }
}
public class Courses
{
[DataMember]
public string MajorNameID { get; set; }
[DataMember]
public string MajorName { get; set; }
[DataMember]
public List<CourceNames> CourseNames { get; set; }
public Courses()
{
Course = new List<CourceNames>();
}
}
I am reading two tables from MYSQL database using SQLreader to
List<courseList> courseList
class.
MY result record is as follows :
MajorNameID MajorName CourseName CourseID
100000 Physics Thermodynamic PHY101
100000 Physics Quantum PHY200
100000 Physics Relativity PHY300
200000 Chemistry Gases CHM300
200000 Chemistry Oreganic CHM500
200000 Chemistry Inroganic CHM120
300000 Mathematics Pure MAT100
300000 Mathematics Applied MAT300
As u could see, I want to populate Courses class. I am not sure how I could do this using Linq.
I recently learnt the following method but it's not working correctly.
List<Courses> courses = courseList .GroupBy(
d => new { d.MajorNameID , d.MajorName },
d => d.MajorName,
(key, g) => new courses
{
MajorNameID = key.MajorNameID,
MajorName = key.MajorName,
CourseNames = g.Distinct().ToList()
}
).ToList();
I get the following error :
> Severity Code Description Project File Line Suppression State
> Error CS0029 Cannot implicitly convert type
> 'System.Collections.Generic.List<string>' to
> 'System.Collections.Generic.List<CourceNames>'...........
I do not fully understand the above. Could someone help me how I could set it up correctly?
I want to load couses per each major in the list.

Try simplifying your group by to just grouping by key and using Select expression on each item in group.
This final trick is that you have to cast the course names to the correct type. The compiler should give you enough hints, but Linq GroupBy is VERY different to MySQL/TSQL GroupBy statements so they can be tricky to master at first.
List<Courses> courses = courseList.GroupBy(
d => new { d.MajorNameID, d.MajorName }
).Select(g => new Courses
{
MajorNameID = g.Key.MajorNameID,
MajorName = g.Key.MajorName,
CourseNames = g.Distinct().Select(c =>
new CourceNames { CourseID = c.CourseID, CourseName = c.CourseName }).ToList()
}
).ToList();
[UPDATE: I don't normally use element projection syntax, so I had to compile this to check]
Using the same GroupBy overload as your question (element projection), this is the same query, note that the element selector expression now selects the elements, the original post has a key expression in there instead:
List<Courses> courses2 = courseList.GroupBy(
d => new { d.MajorNameID, d.MajorName }, // key selector
c => new CourceNames { CourseID = c.CourseID, CourseName = c.CourseName }, // element selector
(key, g) => new Courses
{
MajorNameID = key.MajorNameID,
MajorName = key.MajorName,
CourseNames = g.Distinct().ToList()
}
).ToList();

The lambda variable g is a group, you cannot directly do a .Distinct on it
List<Courses> courses = courseList .GroupBy(
d => new { d.MajorNameID , d.MajorName },
d => d.MajorName,
(key, g) => new Courses
{
MajorNameID = key.MajorNameID,
MajorName = key.MajorName,
CourseNames = g.Distinct().ToList()
}).ToList();
You need to project the course name from the group g and then do a .Distinct on it. Which will be like:
List<Courses> courses = courseList .GroupBy(
d => new { d.MajorNameID , d.MajorName },
d => d.MajorName,
(key, g) => new Courses
{
MajorNameID = key.MajorNameID,
MajorName = key.MajorName,
CourseNames = g.Select(x => x.CourseName).Distinct().ToList()
}).ToList();

Linq two table with sum column

My RowMultiplevaluw table is
public class RowMultipleValues
{
public int ID { get; set; }
public String Year{ get; set; }
public string country { get; set; }
public decial Admin { get; set; }
public decimal Finance { get; set; }
public virtual ICollection<UsedAmount> UsedAmount { get; set; }
}
My used amount table is
public class UsedAmount
{
public int ID { get; set; }
public string Year{ get; set; }
public string country { get; set; }
public decial UsedAmount { get; set; }
public int RowMultipleValues ID { get; set; }
Public virtual RowMultibleValue RowMultibleValue { get; set; }
}
My query is
var query = from mtv in context.multiplerowvaluetable
join usd in dbcontext.usedtsble on mtv.year equal usd.year group g by mtv.country into g
select new { country =g.key,sumadmincolumn =g.sum(Admin),sumfinancecolumn = g.sum(finance) }).tolist();
Result which I want is
ID Year Country Admin. UsedAdmin Finance UsedFinance
1. 2017 USA 100 50 200 300
2. 2017 China 300 300 500 400
Total. 400 350 700 700
Please help me my model design and query for result.Thank.

So you want to join every MultipleValue with the UsedAmount on equal year value. Then group the result into groups of joined items with same country. Finally from every group create one object with the country, the sum of all Admin values and the sum of all finance values.
// first join the two collections on same year.
// we only need properties Country, Admin, Finance:
var result = myDbContext.MultipleRowValueTable.Join(myDbContext.UsedAmountTable,
multipleRow => multipleRow.Year, // from every multipleRow take the year
usedAmount => usedAmount.Year, // from every usedAmount take the year
(multipleRow, usedAmount) => new // when they match make a new object
{
Country = multipleRow.Country,
Admin = multipleRow.Admin,
UsedAdmin = usedAmount.Admin,
Finance = multipleRow.Finance,
UsedFinance = usedAmount.Finance,
})
// group the elements from this join table into groups with same Country
.GroupBy(joinedItem => joinedItem.Country, // all items in the group have this Country
joinedItem => new // the elements of the group
{
Admin = joinedItem.Admin,
UsedAdmin = joinedItem.UsedAdmin,
Finance = joinedItem.Finance,
UsedFinance = joinedItem.UsedFinance,
})
// finally: from every group take the Key (which is the Country)
// and the sum of the Admins and Finances in the group
.Select(group => new
{
Country = group.Key,
SumAdminColumn = group
.Select(groupElement => groupElement.Admin)
.Sum(),
... // others are similar
});
// from every group take the elements and sum the properties
.Select(group => new
{
Id = multipleRowValue.Id,
Year = multipleRowValue.Year,
Country = multipleRowValue.Country,
}

How can I select from an included entity in LINQ and get a flat list similar to a SQL Join would give?

I have two classes:
public class Topic
{
public Topic()
{
this.SubTopics = new HashSet<SubTopic>();
}
public int TopicId { get; set; }
public string Name { get; set; }
public virtual ICollection<SubTopic> SubTopics { get; set; }
}
public class SubTopic
public int SubTopicId { get; set; }
public int Number { get; set; }
public int TopicId { get; set; }
public string Name { get; set; }
public virtual Topic Topic { get; set; }
}
What I would like to do is to get a Data Transfer Object output from LINQ that will show me. I do want to see the TopicId repeated if there is more than one SubTopic inside that topic:
TopicId Name SubTopicId Name
1 Topic1 1 SubTopic1
1 Topic1 2 SubTopic2
1 Topic1 3 SubTopic3
2 Topic2 4 SubTopic4
I tried to code a Linq statement like this:
var r = context.Topics
.Select ( s => new {
id = s.TopicId,
name = s.Name,
sid = s.SubTopics.Select( st => st.SubTopicId),
sidname = s.SubTopics.Select ( st => st.Name)
}).
ToList();
But this does not really work as it returns sid and sidname as lists.
How will it be possible for me to get a flat output showing what I need?

You need SelectMany to expand a nested collection, along these lines
var r = context.Topics.SelectMany(t => t.SubTopics
.Select(st => new
{
TopicID = t.TopicId,
TopicName = t.Name,
SubTopicID = st.SubTopicId,
SubTopicName = st.Name
}));

try this :
var r = context.Topics
.Select ( s => new {
id = s.TopicId,
name = s.Name,
sid = s.SubTopics.Where(st=>st.TopicId==s.TopicId).Select( st => st.SubTopicId ),
sidname = s.SubTopics..Where(st=>st.TopicId==s.TopicId).Select ( st => st.Name)
}).
ToList();
Hope it will help

#Sweko provided an answer that satisfies the exact output that you requested. However, this can be even simpler if you just return the subtopic intact. It may run a bit quicker as well, since you don't need to create a new object for each element in the result.
Lastly, it looks like you wanted your result set ordered. For completeness, I've added those clauses as well.
var r = context.Topics
.SelectMany( topic => topic.SubTopics )
.OrderBy(sub => sub.TopicId)
.ThenBy(sub => sub.SubTopicId);