We Keep Coding

multiple grouping linq nested DTO not translating well

This is a .NET Core Web API Task method. I have a flat table that I need to convert into a nested DTOs. The first DTO works but I can't seem to get the second DTO to nest after grouping.
I know I have done the grouping correctly. I am just not sure the second level nesting of the DTO is done correctly, it complains about not being able to translate to some type.
LINQ Query to put data in a nested object
Can someone point me in the right track?
public async Task<List<PointCardViewModel>> GetPointCards() {
var data = (from s in db.Students
join dc in db.DailyCards on s.StudentId equals dc.StudentId
join dcli in db.DailyCardLineItems on dc.CardId equals dcli.CardId
join dcob in db.DailyCardOtherBehaviors on dc.CardId equals dcob.CardId
select new
{
s.StudentName,
s.StudentGrade,
dc.CardId,
dc.CardDate,
dcli.ClassParticipationPoints,
dcli.AssignmentCompletionPoints,
dcli.BonusHomeworkPoints,
dcli.ClassPeriod,
dcob.PersonalAppearancePoints,
dcob.LunchPoints,
dcob.RecessOtherPoints,
dcob.AmHomeroomPoints,
dcob.PmHomeroomPoints
});
var queryPointCards = (data
.GroupBy(x => new
{
x.CardId,
x.StudentGrade,
x.StudentName,
x.CardDate,
x.PersonalAppearancePoints,
x.LunchPoints,
x.RecessOtherPoints,
x.AmHomeroomPoints,
x.PmHomeroomPoints
})
.Select(x => new PointCardViewModel()
{
CardId = x.Key.CardId,
StudentName = x.Key.StudentName,
Grade = x.Key.StudentGrade,
EvaluationDate = x.Key.CardDate,
PersonalAppearancePoints = x.Key.PersonalAppearancePoints,
LunchPoints = x.Key.LunchPoints,
RecessOtherPoints = x.Key.RecessOtherPoints,
AMHomeRoomPoints = x.Key.AmHomeroomPoints,
PMHomeRoomPoints = x.Key.PmHomeroomPoints,
//LineItems = null --> This works!! But not the below
LineItems = x.Select(c => new LineItemViewModel
{
ClassPeriod = c.ClassPeriod,
BonusHomeworkPoints = c.BonusHomeworkPoints,
ClassParticipationPoints = c.ClassParticipationPoints,
AssignmentCompletionPoints = c.AssignmentCompletionPoints
})
}
)
).ToListAsync();
if (db != null)
{
return await queryPointCards;
}
return null;
}

You have hit limitation of Grouping. After groping you cannot access to group items. Only fields from 'Key' and aggregation functions are allowed.
So just put data.AsEnumerable() and do grouping on the client side.

Operation Intersect with linq

Sorry for strange title of the question, but I don't know how to formulate it more short. If you know how to formulate it better, I will be glad if you edit my question.
So, I have the following table:
I'm tolking about CustomerId and EventType fields. The rest is not important. I think you understand that this table is something like log by customers events. Some customer make event - I have event in the table. Simple.
I need to choice all customers events where each customer had event with type registration and type deposit. In other words, customer had registration before? The same customer had deposit? If yes and yes - I need to select all events of this customer.
How I can do that with the help of LINQ?
So I can write SQL like
select *
From "CustomerEvents"
where "CustomerId" in (
select distinct "CustomerId"
from "CustomerEvents"
where "EventType" = 'deposit'
intersect
select distinct "CustomerId"
from "CustomerEvents"
where "EventType" = 'registration'
)
It works, but how to write it on LINQ?
And second question. SQL above works, but not it is not universal. What if tomorrow I will need to show events of customers who have registration, deposit and - new one event - visit? I have to write new one query. Like:
select *
From "CustomerEvents"
where "CustomerId" in (
select "CustomerId"
from "CustomerEvents"
where "EventType" = 'deposit'
intersect
select distinct "CustomerId"
from "CustomerEvents"
where "EventType" = 'registration'
intersect
select distinct "CustomerId"
from "CustomerEvents"
where "EventType" = 'visit'
)
Uncomfortable :(
As source data, I have List with event types. Is there some way to make it dynamically? I mean, I have new one event in the list - I have new one intersect.
P.S I use Postgres and .NET Core 3.1
Update
I pine here a scheme

I haven't tested to see if this will translate to SQL correctly, but if we assume ctx.CustomerEvents is DbSet<CustomerEvent> you could try this:
var targetCustomerIds = ctx
.CustomerEvents
.GroupBy(event => event.CustomerId)
.Where(grouped =>
grouped.Any(event => event.EventType == "deposit")
&& grouped.Any(event => event.EventType == "registration"))
.Select(x => x.Key)
.ToList();
and then select all events for these customers:
var events = ctx.CustomerEvents.Where(event => targetCustomerIds.Contains(event.CustomerId));
To get targetCustomerIds dynamically with a variable number of event types, you could try this:
// for example
var requiredEventTypes = new [] { "deposit", "registration" };
// First group by customer ID
var groupedByCustomerId = ctx
.CustomerEvents
.GroupBy(event => event.CustomerId);
// Then filter out any grouping which doesn't satisfy your condition
var filtered = GetFilteredGroups(groupedByCustomerId, requiredEventTypes);
// Then select the target customer IDs
var targetCustomerIds = filtered.Select(x => x.Key).ToList();
// Finally, select your target events
var events = ctx.CustomerEvents.Where(event =>
targetCustomerIds.Contains(event.CustomerId));
You can define the GetFilteredGroups method like this:
private static IQueryable<IGrouping<int, CustomerEvent>> GetFilteredGroups(
IQueryable<IGrouping<int, CustomerEvent>> grouping,
IEnumerable<string> requiredEventTypes)
{
var result = grouping.Where(x => true);
foreach (var eventType in requiredEventTypes)
{
result = result.Where(x => x.Any(event => event.EventType == eventType));
}
return result;
}
Alternatively, instead of selecting the target customer IDs, you can try to directly select your target events from the filtered groupings:
// ...
// Filter out any grouping which doesn't satisfy your condition
var filtered = GetFilteredGroups(groupedByCustomerId, requiredEventTypes);
// Select your events here
var results = filtered.SelectMany(x => x).Distinct().ToList();
Regarding the inability to translate the query to SQL
Depending on your database size and particularly on the size of CustomerEvents table, this solution may or may not be ideal, but what you could do is load the optimized collection to memory and perform the grouping there:
// for example
var requiredEventTypes = new [] { "deposit", "registration" };
// First group by customer ID, but load into memory
var groupedByCustomerId = ctx
.CustomerEvents
.Where(event => requiredEventTypes.Contains(event.EventType))
.Select(event => new CustomerEventViewModel
{
Id = event.Id,
CustomerId = event.CustomerId,
EventType = event.EventType
})
.GroupBy(event => event.CustomerId)
.AsEnumerable();
// Then filter out any grouping which doesn't satisfy your condition
var filtered = GetFilteredGroups(groupedByCustomerId, requiredEventTypes);
// Then select the target customer IDs
var targetCustomerIds = filtered.Select(x => x.Key).ToList();
// Finally, select your target events
var events = ctx.CustomerEvents.Where(event =>
targetCustomerIds.Contains(event.CustomerId));
You will need to create a type called CustomerEventViewModel like this (so you don't have to load the entire CustomerEvent entity instances to memory):
public class CustomerEventViewModel
{
public int Id { get; set; }
public int CustomerId { get; set; }
public string EventType { get; set; }
}
And change the GetFilteredGroups like this:
private static IEnumerable<IGrouping<int, CustomerEvent>> GetFilteredGroups(
IEnumerable<IGrouping<int, CustomerEvent>> grouping,
IEnumerable<string> requiredEventTypes)
{
var result = grouping.Where(x => true);
foreach (var eventType in requiredEventTypes)
{
result = result.Where(x => x.Any(event => event.EventType == eventType));
}
return result;
}
It should now work fine.

Thank you for #Dejan Janjušević. He is excpirienced developer. But it seems EF can't translate him solution to SQL (or just my hands grow from wrong place). I publish here my solution for this situation. It's simple stuped. So. I have in the table EventType. It is string. And I have from the client the following filter request:
List<string> eventType
Just list with event types. So, in the action I have the following code of the filter:
if (eventType.Any())
{
List<int> ids = new List<int>();
foreach (var e in eventType)
{
var customerIdsList =
_context.customerEvents.Where(x => x.EventType == e).Select(x => x.CustomerId.Value).Distinct().ToList();
if (!ids.Any())
{
ids = customerIdsList;
}
else
{
ids = ids.Intersect(customerIdsList).ToList();
}
}
customerEvents = customerEvents.Where(x => ids.Contains(x.CustomerId.Value));
}
Not very fast, but works.

Linq query Contains method with where condition with case sensitive less search

I am trying get the data which is contains single word with in the word.Like below query.
List<Models.tbluser> memberslist = new List<Models.tbluser>();
var obct = (from memlist in objcontext.tblusers
where memlist.logname.Contains(member)
select new
{
userid = memlist.userid,
logname = memlist.logname,
decription = memlist.description
}).ToList();
foreach (var item in obct)
{
memberslist.Add(new tbluser
{
userid = item.userid,
logname = item.logname,
description = item.decription
});
}
return Json(memberslist);
But here my problem is i need to search with out case sensitive.
For example
If i search with 'a' i need to get data like Admin,Administrator,User Data.
But i am not getting all these because i am searching with Contains() method.Please let me know how can i get all value either the search value is case sensitive less also.

Change your where condition to be:
memlist.logname.ToUpper().Contains(member.ToUpper())
As a side note, you can shorten your query a bit (you don't need to create an intermediary list):
var memberslist = objcontext.tblusers
.Where(x => x.logname.ToUpper().Contains(member.ToUpper())
.AsEnumerable()
.Select(x => new tbluser
{
userid = x.userid,
logname = x.logname,
decription = x.description
})
.ToList();
return Json(memberslist);

You can change them to Lower or Upper Case when checking the condition using ToLower() or ToUpper():
var obct = (from memlist in objcontext.tblusers
where memlist.logname.ToLower().Contains(member.ToLower())
select new
{
userid = memlist.userid,
logname = memlist.logname,
decription = memlist.description
}).ToList();

query and create objects with a one to many relationship using LINQ

In the DB, I have a two tables with a one-to-many relationship:
orders suborders
----------- -----------
id id
name order_id
name
I'd like to query these tables and end up with a list of order objects, each of which contains a list (or empty list) of suborder objects. I'd also like to do this in a single DB query so it performs well.
In traditional SQL query land, I'd do something like (forgive the pseudocode):
rs = "select o.id, o.name, so.id, so.name from orders o left join suborders so on o.id = so.order_id order by o.id"
orders = new List<Order>
order = null
foreach (row in rs) {
if (order == null || row.get(o.id) != order.id) {
order = new Order(row.get(o.id), row.get(o.name), new List<Suborders>)
orders.add(order)
}
if (row.get(so.id) != null) {
order.suborders.add(new Suborder(row.get(so.id) row.get(so.name))
}
}
Is there a way to get this same resulting object structure using LINQ-to-Entities? Note that I want to get new objects out of the query, not the Entity Framework generated objects.
The following gets me close, but throws an exception: "LINQ to Entities does not recognize the method..."
var orders =
(from o in Context.orders
join so in Context.suborders on o.id equals so.order_id into gj
select new Order
{
id = o.id,
name = o.name,
suborders = (from so in gj select new Suborder
{
so.id,
so.name
}).ToList()
}).ToList();

The solution ends up being pretty simple. The key is to use a group join to get SQL to do the left join to suborders, and add a second ToList() call to force the query to be run so you're not trying to do object creation on the SQL server.
orders = Context.orders
.GroupJoin(
Context.suborders,
o => o.id,
so => so.order_id,
(o, so) => new { order = o, suborders = so })
.ToList()
.Select(r => new Order
{
id = r.order.id,
name = r.order.name,
suborders = r.suborders.Select(so => new Suborder
{
id = so.id,
name = so.name
}.ToList()
}).ToList();
This code only makes a single query to SQL for all objects and their child objects. It also lets you transform the EF objects into whatever you need.

I Always create a virtualized Property for Relations
so just extend (add a property) to your order class :
public class Order{
...
List<Suborder> _suborders;
public List<Suborder> Suborders{
get {
return _suborders ?? (_suborders = MyContext.Suborders.Where(X=>X.order_id==this.id).ToList());
}
...
}
so data will be fetched (pulled) only when you call the getters

How about this code ?
You can get a local cache.
List<Orders> orders = new List<Orders>();
private void UpdateCache(List<int> idList)
{
using (var db = new Test(Settings.Default.testConnectionString))
{
DataLoadOptions opt = new DataLoadOptions();
opt.LoadWith<Orders>(x => x.Suborders);
db.LoadOptions = opt;
orders = db.Orders.Where(x => idList.Contains(x.Id)).ToList();
}
}
private void DumpOrders()
{
foreach (var order in orders)
{
Console.WriteLine("*** order");
Console.WriteLine("id:{0},name:{1}", order.Id, order.Name);
if (order.Suborders.Any())
{
Console.WriteLine("****** sub order");
foreach (var suborder in order.Suborders)
{
Console.WriteLine("\torder id:{0},id{1},name:{2}", suborder.Order_id, suborder.Id, suborder.Name);
}
}
}
}
private void button1_Click(object sender, EventArgs e)
{
UpdateCache(new List<int> { 0, 1, 2 });
DumpOrders();
}
Output example below
*** order
id:0,name:A
****** sub order
order id:0,id0,name:Item001
order id:0,id1,name:Item002
order id:0,id2,name:Item003
*** order
id:1,name:B
****** sub order
order id:1,id0,name:Item003
*** order
id:2,name:C
****** sub order
order id:2,id0,name:Item004
order id:2,id1,name:Item005

Join/Where with LINQ and Lambda

I'm having trouble with a query written in LINQ and Lambda. So far, I'm getting a lot of errors here's my code:
int id = 1;
var query = database.Posts.Join(database.Post_Metas,
post => database.Posts.Where(x => x.ID == id),
meta => database.Post_Metas.Where(x => x.Post_ID == id),
(post, meta) => new { Post = post, Meta = meta });
I'm not sure if this query is correct.

I find that if you're familiar with SQL syntax, using the LINQ query syntax is much clearer, more natural, and makes it easier to spot errors:
var id = 1;
var query =
from post in database.Posts
join meta in database.Post_Metas on post.ID equals meta.Post_ID
where post.ID == id
select new { Post = post, Meta = meta };
If you're really stuck on using lambdas though, your syntax is quite a bit off. Here's the same query, using the LINQ extension methods:
var id = 1;
var query = database.Posts // your starting point - table in the "from" statement
.Join(database.Post_Metas, // the source table of the inner join
post => post.ID, // Select the primary key (the first part of the "on" clause in an sql "join" statement)
meta => meta.Post_ID, // Select the foreign key (the second part of the "on" clause)
(post, meta) => new { Post = post, Meta = meta }) // selection
.Where(postAndMeta => postAndMeta.Post.ID == id); // where statement

You could go two ways with this. Using LINQPad (invaluable if you're new to LINQ) and a dummy database, I built the following queries:
Posts.Join(
Post_metas,
post => post.Post_id,
meta => meta.Post_id,
(post, meta) => new { Post = post, Meta = meta }
)
or
from p in Posts
join pm in Post_metas on p.Post_id equals pm.Post_id
select new { Post = p, Meta = pm }
In this particular case, I think the LINQ syntax is cleaner (I change between the two depending upon which is easiest to read).
The thing I'd like to point out though is that if you have appropriate foreign keys in your database, (between post and post_meta) then you probably don't need an explicit join unless you're trying to load a large number of records. Your example seems to indicate that you are trying to load a single post and its metadata. Assuming that there are many post_meta records for each post, then you could do the following:
var post = Posts.Single(p => p.ID == 1);
var metas = post.Post_metas.ToList();
If you want to avoid the n+1 problem, then you can explicitly tell LINQ to SQL to load all of the related items in one go (although this may be an advanced topic for when you're more familiar with L2S). The example below says "when you load a Post, also load all of its records associated with it via the foreign key represented by the 'Post_metas' property":
var dataLoadOptions = new DataLoadOptions();
dataLoadOptions.LoadWith<Post>(p => p.Post_metas);
var dataContext = new MyDataContext();
dataContext.LoadOptions = dataLoadOptions;
var post = Posts.Single(p => p.ID == 1); // Post_metas loaded automagically
It is possible to make many LoadWith calls on a single set of DataLoadOptions for the same type, or many different types. If you do this lots though, you might just want to consider caching.

Daniel has a good explanation of the syntax relationships, but I put this document together for my team in order to make it a little simpler for them to understand. Hope this helps someone

Your key selectors are incorrect. They should take an object of the type of the table in question and return the key to use in the join. I think you mean this:
var query = database.Posts.Join(database.Post_Metas,
post => post.ID,
meta => meta.Post_ID,
(post, meta) => new { Post = post, Meta = meta });
You can apply the where clause afterwards, not as part of the key selector.

Posting because when I started LINQ + EntityFramework, I stared at these examples for a day.
If you are using EntityFramework, and you have a navigation property named Meta on your Post model object set up, this is dirt easy. If you're using entity and don't have that navigation property, what are you waiting for?
database
.Posts
.Where(post => post.ID == id)
.Select(post => new { post, post.Meta });
If you're doing code first, you'd set up the property thusly:
class Post {
[Key]
public int ID {get; set}
public int MetaID { get; set; }
public virtual Meta Meta {get; set;}
}

I've done something like this;
var certificationClass = _db.INDIVIDUALLICENSEs
.Join(_db.INDLICENSECLAsses,
IL => IL.LICENSE_CLASS,
ILC => ILC.NAME,
(IL, ILC) => new { INDIVIDUALLICENSE = IL, INDLICENSECLAsse = ILC })
.Where(o =>
o.INDIVIDUALLICENSE.GLOBALENTITYID == "ABC" &&
o.INDIVIDUALLICENSE.LICENSE_TYPE == "ABC")
.Select(t => new
{
value = t.PSP_INDLICENSECLAsse.ID,
name = t.PSP_INDIVIDUALLICENSE.LICENSE_CLASS,
})
.OrderBy(x => x.name);

It could be something like
var myvar = from a in context.MyEntity
join b in context.MyEntity2 on a.key equals b.key
select new { prop1 = a.prop1, prop2= b.prop1};

This linq query Should work for you. It will get all the posts that have post meta.
var query = database.Posts.Join(database.Post_Metas,
post => post.postId, // Primary Key
meta => meat.postId, // Foreign Key
(post, meta) => new { Post = post, Meta = meta });
Equivalent SQL Query
Select * FROM Posts P
INNER JOIN Post_Metas pm ON pm.postId=p.postId

Query Syntax for LINQ Join
var productOrderQuery = from product in Product.Setup()//outer sequence
join order in OrderDetails.Setup()//inner sequence
on product.Id equals order.ProductId //key selector
select new//result selector
{
OrderId = order.Id,
ProductId = product.Id,
PurchaseDate = order.PurchaseDate,
ProductName = product.Name,
ProductPrice = product.Price
};
Method Syntax for LINQ Join
var productOrderMethod = Product.Setup().//outer sequence
Join(OrderDetails.Setup(), //inner sequence
product => product.Id//key selector
,order=> order.ProductId //key selector
,(product,order)=> //projection result
new
{
OrderId = order.Id,
ProductId = product.Id,
PurchaseDate = order.PurchaseDate,
ProductName = product.Name,
ProductPrice = product.Price
}
);
Product.cs for reference
class Product
{
public int Id { get; set; }
public string Name { get; set; }
public decimal Price { get; set; }
public static IEnumerable<Product> Setup()
{
return new List<Product>()
{
new Product(){Id=1, Name="Bike", Price=30.33M },
new Product(){Id=2, Name="Car", Price=50.33M },
new Product(){Id=3, Name="Bus", Price=60.33M }
};
}
}
OrderDetails.cs class for reference
class OrderDetails
{
public int Id { get; set; }
public virtual int ProductId { get; set; }
public DateTime PurchaseDate { get; set; }
public static IEnumerable<OrderDetails> Setup()
{
return new List<OrderDetails>()
{
new OrderDetails(){Id=1, ProductId=1, PurchaseDate= DateTime.Now },
new OrderDetails(){Id=2, ProductId=1, PurchaseDate=DateTime.Now.AddDays(-1) },
new OrderDetails(){Id=3, ProductId=2, PurchaseDate=DateTime.Now.AddDays(-2) }
};
}
}

1 equals 1 two different table join
var query = from post in database.Posts
join meta in database.Post_Metas on 1 equals 1
where post.ID == id
select new { Post = post, Meta = meta };