We Keep Coding

How to GroupBy data from many tables in dotnet core Entity Framework

I'm building some marketplace web app, let's say something like e-bay. Typical scenario is:
User makes offer which consists of one or more items and those items are of certain type.After that other users are bidding on that offer.
Here is simplified diagram.
On SQL Fiddle (here) you can see both CREATE TABLE and INSERT INTO statements
Sample data:
There are two offers. On one offer (Id 1) which consists of one item which is type of "watch". There is another offer, (Id 2), which has one item which is of type "headphone".
On both offers there are bids. On watch, there are two bis; one bid with 100 dollars and another with 120. On headphones, there are bids with 50 and 80 dollars.
What I want to achieve is to have average bid per type. In this sample, that means i want to get 110 as average bid for watch and 65 as average bid for headphone. To achieve that using T-SQL, I would write query like this:
SELECT t.name,
avg(amount)
FROM bid b
LEFT JOIN offer o ON b.OfferId = o.id
LEFT JOIN offeritem oi ON o.id = oi.OfferId
LEFT JOIN itemType t ON oi.itemtypeid = t.Id
GROUP BY t.name
So, my question is - how to achieve that in dotnet core 3.0 EntityFramework
Using GroupBy, like this:
_context.Bids
.Include(b => b.Offer)
.ThenInclude(o => o.OfferItems)
.ThenInclude(os => os.ItemType)
.GroupBy(b => b.Offer.OfferItems.First().ItemType.Name);
gives exception:
Client side GroupBy is not supported.
. When I try with projection, like this:
_context.Bids
.Include(b => b.Offer)
.ThenInclude(o => o.OfferItems)
.ThenInclude(os => os.ItemType)
.GroupBy(b => b.Offer.OfferItems.First().ItemType.Name)
.Select(g => new
{
Key = g,
Value = g.Average(b => b.Amount)
});
i get exception again.
Processing of the LINQ .... failed. This may indicate either a bug or
a limitation in EF Core.
EDIT:
This approach
_context.Bids
.Include(b => b.Offer)
.ThenInclude(o => o.OfferItems)
.ThenInclude(os => os.ItemType)
.GroupBy(b => new { b.Offer.OfferItems.First().ItemType.Name}, b => b.Amount)
.Select(g => new
{
Key = g.Key.Code,
Value = g.Average()
});
also threw an exception, but this time:
Cannot use an aggregate or a subquery in an expression used for the
group by list of a GROUP BY clause.
...
So, is there a way to group that data (get simple Average) or should I make another query and iterate throught collection and make calculation myself? That would lower performance for sure (I was hoping I can do server grouping, but as you can see, i got into mentioned issues). Any ideas? Thanks in advance.

In your case it is hard to hide subquery from grouping
You can try it in such way
var joined =
context.Bid
.SelectMany(x =>
x.Offer.OfferItem
.Select(y => new
{
Amount = x.Amount,
Name = y.ItemType.Name
})
.Take(1));
var grouped = from i in joined
group i by i.Name into groups
select new
{
Key = groups.Key,
Amount = groups.Average(x => x.Amount)
};
it gives me a query
SELECT [t].[Name] AS [Key], AVG([t].[Amount]) AS [Amount]
FROM [Bid] AS [b]
INNER JOIN [Offer] AS [o] ON [b].[OfferId] = [o].[Id]
CROSS APPLY (
SELECT TOP(1) [b].[Amount], [i].[Name], [o0].[Id], [i].[Id] AS [Id0], [o0].[OfferId]
FROM [OfferItem] AS [o0]
INNER JOIN [ItemType] AS [i] ON [o0].[ItemTypeId] = [i].[Id]
WHERE [o].[Id] = [o0].[OfferId]
) AS [t]
GROUP BY [t].[Name]

How to convert this SQL statement to Linq for C#

I am trying to convert my simple SQL statement into Linq format for my C# application but I always seem to struggle making the conversion. I downloaded linqpad and have been playing around with it but I'm still having issues with the proper format.
My SQL statement:
SELECT distinct PictureCity, PictureState
FROM Website_Gallery
GROUP BY PictureCity, PictureState, PictureDate
ORDER BY PictureCity, PictureState
The results are ordered by PictureCity and look like this:
Abington MA
Acton MA
Acushnet MA
Agawam MA
Andover MA
Arlington MA
Arlington TX
Ashby MA
Ashland MA
What I have so far in my C# application which I can't seem to get to work. (I suck at linq).
var Results = _context.Website_Gallery
.(g => g.PictureCity, g => g.PictureState).AsEnumerable()
.Select g
.GroupBy(g => g)

Seems like all you need is
var results = _context.Website_Gallery
.OrderBy(x => x.PictureCity)
.ThenBy(x => x.PictureState)
.Select(x => new { x.PictureCity, x.PictureState })
.Distinct();
that would be equivalent to the following SQL
SELECT distinct PictureCity, PictureState
FROM Website_Gallery
ORDER BY PictureCity, PictureState
because what you had did not need the group by
Note you can then either iterate that result in a foreach or tack a ToList to the end to materialize the query into memory.

SQL
SELECT distinct PictureCity, PictureState
FROM Website_Gallery
ORDER BY PictureCity, PictureState
Linq
var Results = _context.Website_Gallery
.Select(g => new { g.PictureCity, g.PictureState })
.Orderby(p => p.PictureCity).ThenBy(p => p.PictureState)
.Distinct().ToList();
Or you can also do this
var Results = _context.Website_Gallery
.GroupBy(x => new { PictureCity = x.PictureCity, PictureState = x.PictureState })
.Select(g => new { g.Key.PictureCity, g.Key.PictureState }).Orderby(p => p.PictureCity).ThenBy(p => p.PictureState)
.ToList();

Get last 2 rows from group on database side

I'm trying to improve performance of linq query for PostgreSQL. There are two tables (Parcles, ParcelStates) with relation 1:n. I need to get last 2 ParcelStates for each Parcel. Looks simple, I have following code:
IQueryable<Parcel> parcels = _dbContext.Parcels
.OrderByDescending(x => x.Id)
.Take(100);
Then getting states:
var states = await parcels
.GroupJoin(_dbContext.ParcelStates, ps => ps.Id, p => p.ParcelId, (ps, p) => new { ps, p })
.SelectMany(x => x.p.DefaultIfEmpty().OrderByDescending(y => y.Id).Take(2), (x,c) => c)
.ToListAsync();
It returns me 180 states, and it is ok. But there is performance issue, because it generates not perform SQL query:
SELECT *
FROM (
SELECT *
FROM parcels AS x
WHERE x.isdeleted = FALSE
ORDER BY c DESC, c0 DESC
LIMIT #__p_1 OFFSET #__p_0
) AS t
LEFT JOIN parcelstates AS p ON t.id = p.parcelid
ORDER BY t.c DESC, t.c0 DESC, t.id
It takes all states from database, when I need only 2.
How to change LINQ to filter result on database side?
In logs I found:
The LINQ expression 'Take(2)' could not be translated and will be evaluated

If you insert the SelectMany expression into the GroupJoin, will it convert to SQL?
var states = await parcels
.GroupJoin(_dbContext.ParcelStates, ps => ps.Id, p => p.ParcelId,
(ps, p) => p.DefaultIfEmpty().OrderByDescending(y => y.Id).Take(2))
.ToListAsync();

We can use a foreach loop which will translate to several very fast SQL lookups (should execute in < 1 second). Not ideal but I would still recommend writing a stored procedure to get this data, instead of relying on LINQ to SQL which doesn't always generate the most optimum query:
// Store a list of parcel states
var parcelStates = new List<ParcelState>();
// Read top 100 parcels from the database
var parcels = dbContext.Parcels
.OrderBy(p => p.Id)
.Take(100);
// For each parcel, use SQL to lookup the 2 most recent parcel states
foreach (var p in parcels)
{
var ps = dbContext.ParcelStates
.Where(ps => ps.ParcelId == p.Id)
.OrderByDescending(ps => ps.Id)
.Take(2);
parcelStates.AddRange(ps);
}
// Now we have all parcel states for those parcels
Console.WriteLine($"Found {parcelStates.Count} parcel states for {parcels.Count} parcels");

translate sql to linq extension method inner join with select group by

Good day,
Thanks for your time.
I want to get the latest messages of a chat table by userId and projectId and this is working fine
select * from chatTable inner join
(select max(SendDate) maxtime,[ProjectId] from [chatTable]
group by [ProjectId])latest
on latest.maxtime=chatTable.SendDate and
latest.[ProjectId]=chatTable.[ProjectId]
order by SendDate
As you can see, I am getting the latest messages from the chatTable , with a join that brings the latest project id and the latest message.
How can I have a linq with extension methods?
var messages= await _dbContext.chatTable.....
Thanks

You can first query the inner part like below:
var latest= _dbContext.chatTable.GroupBy(x=>x.ProjectId).Select(x=>new {maxTime=x.Max(y=>y.SendDate), ProjectId=x.Key });
var res=(from ct in _dbContext.ChatTable
join la in latest on la.maxTime equals ct.SendDate and la.ProjectId equals ct.ProjectId).ToList().OrderBy(x=>x.SendDate);

This query should do what you want
_dbContext.ChatTable
.GroupBy(c => c.ProjectId)
.Select(g => g
.OrderByDescending(c => c.SendDate)
.First())
.OrderBy(c => c.SendDate);

This should give what you need:
_dbContext.chatTable
.GroupBy(c => c.ProjectId)
.Select(g => g.OrderBy(c => c.SendDate).LastOrDefault())

Well guys, it seems that linq entity framework has its advantages
this query is better and gets the latest message sent to the latest project
var ultimos = await _dbContext.chatTable .GroupBy(item => item.ProjectId)
.Select(group => new { pId= group.Key, latestMessage= group.OrderByDescending(r=>r.SendDate)
.FirstOrDefault() })
.ToListAsync();
and all the messages are inside the entity

EF Core 2.0 Group By other properties

I have 2 tables:
USERS
UserId
Name
Scores (collection of table Scores)
SCORES
UserId
CategoryId
Points
I need to show all the users and a SUM of their points, but also I need to show the name of the user. It can be filtered by CategoryId or not.
Context.Scores
.Where(p => p.CategoryId == categoryId) * OPTIONAL
.GroupBy(p => p.UserId)
.Select(p => new
{
UserId = p.Key,
Points = p.Sum(s => s.Points),
Name = p.Select(s => s.User.Name).FirstOrDefault()
}).OrderBy(p => p.Points).ToList();
The problem is that when I add the
Name = p.Select(s => s.User.Name).FirstOrDefault()
It takes so long. I don't know how to access the properties that are not inside the GroupBy or are a SUM. This example is very simple becaouse I don't have only the Name, but also other properties from User table.
How can I solve this?

It takes so long because the query is causing client evaluation. See Client evaluation performance issues and how to use Client evaluation logging to identify related issues.
If you are really on EF Core 2.0, there is nothing you can do than upgrading to v2.1 which contains improved LINQ GroupBy translation. Even with it the solution is not straight forward - the query still uses client evaluation. But it could be rewritten by separating the GroupBy part into subquery and joining it to the Users table to get the additional information needed.
Something like this:
var scores = db.Scores.AsQueryable();
// Optional
// scores = scores.Where(p => p.CategoryId == categoryId);
var points = scores
.GroupBy(s => s.UserId)
.Select(g => new
{
UserId = g.Key,
Points = g.Sum(s => s.Points),
});
var result = db.Users
.Join(points, u => u.UserId, p => p.UserId, (u, p) => new
{
u.UserId,
u.Name,
p.Points
})
.OrderBy(p => p.Points)
.ToList();
This still produces a warning
The LINQ expression 'orderby [p].Points asc' could not be translated and will be evaluated locally.
but at least the query is translated and executes as single SQL:
SELECT [t].[UserId], [t].[Points], [u].[UserId] AS [UserId0], [u].[Name]
FROM [Users] AS [u]
INNER JOIN (
SELECT [s].[UserId], SUM([s].[Points]) AS [Points]
FROM [Scores] AS [s]
GROUP BY [s].[UserId]
) AS [t] ON [u].[UserId] = [t].[UserId]