We Keep Coding

How to write linq query for this sql statement

How would you write a linq query with the following SQL statement. I've tried several methods referenced on stackoverflow but they either don't work with the EF version I'm using (EF core 3.5.1) or the DBMS (SQL Server).
select a.ProductID, a.DateTimeStamp, a.LastPrice
from Products a
where a.DateTimeStamp = (select max(DateTimeStamp) from Products where a.ProductID = ProductID)
For reference, a couple that I've tried (both get run-time errors).
var results = _context.Products
.GroupBy(s => s.ProductID)
.Select(s => s.OrderByDescending(x => x.DateTimeStamp).FirstOrDefault());
var results = _context.Products
.GroupBy(x => new { x.ProductID, x.DateTimeStamp })
.SelectMany(y => y.OrderByDescending(z => z.DateTimeStamp).Take(1))
Thanks!

I understand you would like to have a list of the latest prices of each products?
First of all I prefer to use group by option even over 1st query
select a.ProductID, a.DateTimeStamp, a.LastPrice
from Products a
where a.DateTimeStamp IN (select max(DateTimeStamp) from Products group by ProductID)
Later Linq:
var maxDateTimeStamps = _context.Products
.GroupBy(s => s.ProductID)
.Select(s => s.Max(x => x.DateTimeStamp)).ToArray();
var results = _context.Products.Where(s=>maxDateTimeStamps.Contains(s.DateTimeStamp));
-- all assuming that max datetime stamps are unique

I've managed to do it with the following which replicates the correlated sub query in the original post (other than using TOP and order by instead of the Max aggregate), though I feel like there must be a more elegant way to do this.
var results = from x
in _context.Products
where x.DateTimeStamp == (from y
in _context.Products
where y.ProductID == x.ProductID
orderby y.DateTimeStamp descending
select y.DateTimeStamp
).FirstOrDefault()
select x;

I prefer to break up these queries into IQueryable parts, do you can debug each "step".
Something like this:
IQueryable<ProductOrmEntity> pocoPerParentMaxUpdateDates =
entityDbContext.Products
//.Where(itm => itm.x == 1)/*if you need where */
.GroupBy(i => i.ProductID)
.Select(g => new ProductOrmEntity
{
ProductID = g.Key,
DateTimeStamp = g.Max(row => row.DateTimeStamp)
});
//// next line for debugging..do not leave in for production code
var temppocoPerParentMaxUpdateDates = pocoPerParentMaxUpdateDates.ToListAsync(CancellationToken.None);
IQueryable<ProductOrmEntity> filteredChildren =
from itm
in entityDbContext.Products
join pocoMaxUpdateDatePerParent in pocoPerParentMaxUpdateDates
on new { a = itm.DateTimeStamp, b = itm.ProductID }
equals
new { a = pocoMaxUpdateDatePerParent.DateTimeStamp, b = pocoMaxUpdateDatePerParent.ProductID }
// where
;
IEnumerable<ProductOrmEntity> hereIsWhatIWantItems = filteredChildren.ToListAsync(CancellationToken.None);
That last step, I am putting in an anonymous object. You can put the data in a "new ProductOrmEntity() { ProductID = pocoMaxUpdateDatePerParent.ProductID }...or you can get the FULL ProductOrmEntity object. Your original code, I don't know if getting all columns of the Product object is what you want, or only some of the columns of the object.

How to GroupBy data from many tables in dotnet core Entity Framework

I'm building some marketplace web app, let's say something like e-bay. Typical scenario is:
User makes offer which consists of one or more items and those items are of certain type.After that other users are bidding on that offer.
Here is simplified diagram.
On SQL Fiddle (here) you can see both CREATE TABLE and INSERT INTO statements
Sample data:
There are two offers. On one offer (Id 1) which consists of one item which is type of "watch". There is another offer, (Id 2), which has one item which is of type "headphone".
On both offers there are bids. On watch, there are two bis; one bid with 100 dollars and another with 120. On headphones, there are bids with 50 and 80 dollars.
What I want to achieve is to have average bid per type. In this sample, that means i want to get 110 as average bid for watch and 65 as average bid for headphone. To achieve that using T-SQL, I would write query like this:
SELECT t.name,
avg(amount)
FROM bid b
LEFT JOIN offer o ON b.OfferId = o.id
LEFT JOIN offeritem oi ON o.id = oi.OfferId
LEFT JOIN itemType t ON oi.itemtypeid = t.Id
GROUP BY t.name
So, my question is - how to achieve that in dotnet core 3.0 EntityFramework
Using GroupBy, like this:
_context.Bids
.Include(b => b.Offer)
.ThenInclude(o => o.OfferItems)
.ThenInclude(os => os.ItemType)
.GroupBy(b => b.Offer.OfferItems.First().ItemType.Name);
gives exception:
Client side GroupBy is not supported.
. When I try with projection, like this:
_context.Bids
.Include(b => b.Offer)
.ThenInclude(o => o.OfferItems)
.ThenInclude(os => os.ItemType)
.GroupBy(b => b.Offer.OfferItems.First().ItemType.Name)
.Select(g => new
{
Key = g,
Value = g.Average(b => b.Amount)
});
i get exception again.
Processing of the LINQ .... failed. This may indicate either a bug or
a limitation in EF Core.
EDIT:
This approach
_context.Bids
.Include(b => b.Offer)
.ThenInclude(o => o.OfferItems)
.ThenInclude(os => os.ItemType)
.GroupBy(b => new { b.Offer.OfferItems.First().ItemType.Name}, b => b.Amount)
.Select(g => new
{
Key = g.Key.Code,
Value = g.Average()
});
also threw an exception, but this time:
Cannot use an aggregate or a subquery in an expression used for the
group by list of a GROUP BY clause.
...
So, is there a way to group that data (get simple Average) or should I make another query and iterate throught collection and make calculation myself? That would lower performance for sure (I was hoping I can do server grouping, but as you can see, i got into mentioned issues). Any ideas? Thanks in advance.

In your case it is hard to hide subquery from grouping
You can try it in such way
var joined =
context.Bid
.SelectMany(x =>
x.Offer.OfferItem
.Select(y => new
{
Amount = x.Amount,
Name = y.ItemType.Name
})
.Take(1));
var grouped = from i in joined
group i by i.Name into groups
select new
{
Key = groups.Key,
Amount = groups.Average(x => x.Amount)
};
it gives me a query
SELECT [t].[Name] AS [Key], AVG([t].[Amount]) AS [Amount]
FROM [Bid] AS [b]
INNER JOIN [Offer] AS [o] ON [b].[OfferId] = [o].[Id]
CROSS APPLY (
SELECT TOP(1) [b].[Amount], [i].[Name], [o0].[Id], [i].[Id] AS [Id0], [o0].[OfferId]
FROM [OfferItem] AS [o0]
INNER JOIN [ItemType] AS [i] ON [o0].[ItemTypeId] = [i].[Id]
WHERE [o].[Id] = [o0].[OfferId]
) AS [t]
GROUP BY [t].[Name]

Get last 2 rows from group on database side

I'm trying to improve performance of linq query for PostgreSQL. There are two tables (Parcles, ParcelStates) with relation 1:n. I need to get last 2 ParcelStates for each Parcel. Looks simple, I have following code:
IQueryable<Parcel> parcels = _dbContext.Parcels
.OrderByDescending(x => x.Id)
.Take(100);
Then getting states:
var states = await parcels
.GroupJoin(_dbContext.ParcelStates, ps => ps.Id, p => p.ParcelId, (ps, p) => new { ps, p })
.SelectMany(x => x.p.DefaultIfEmpty().OrderByDescending(y => y.Id).Take(2), (x,c) => c)
.ToListAsync();
It returns me 180 states, and it is ok. But there is performance issue, because it generates not perform SQL query:
SELECT *
FROM (
SELECT *
FROM parcels AS x
WHERE x.isdeleted = FALSE
ORDER BY c DESC, c0 DESC
LIMIT #__p_1 OFFSET #__p_0
) AS t
LEFT JOIN parcelstates AS p ON t.id = p.parcelid
ORDER BY t.c DESC, t.c0 DESC, t.id
It takes all states from database, when I need only 2.
How to change LINQ to filter result on database side?
In logs I found:
The LINQ expression 'Take(2)' could not be translated and will be evaluated

If you insert the SelectMany expression into the GroupJoin, will it convert to SQL?
var states = await parcels
.GroupJoin(_dbContext.ParcelStates, ps => ps.Id, p => p.ParcelId,
(ps, p) => p.DefaultIfEmpty().OrderByDescending(y => y.Id).Take(2))
.ToListAsync();

We can use a foreach loop which will translate to several very fast SQL lookups (should execute in < 1 second). Not ideal but I would still recommend writing a stored procedure to get this data, instead of relying on LINQ to SQL which doesn't always generate the most optimum query:
// Store a list of parcel states
var parcelStates = new List<ParcelState>();
// Read top 100 parcels from the database
var parcels = dbContext.Parcels
.OrderBy(p => p.Id)
.Take(100);
// For each parcel, use SQL to lookup the 2 most recent parcel states
foreach (var p in parcels)
{
var ps = dbContext.ParcelStates
.Where(ps => ps.ParcelId == p.Id)
.OrderByDescending(ps => ps.Id)
.Take(2);
parcelStates.AddRange(ps);
}
// Now we have all parcel states for those parcels
Console.WriteLine($"Found {parcelStates.Count} parcel states for {parcels.Count} parcels");

SQL Azure vs. On-Premises Timeout Issue - EF

I'm working on a report right now that runs great with our on-premises DB (just refreshed from PROD). However, when I deploy the site to Azure, I get a SQL Timeout during its execution. If I point my development instance at the SQL Azure instance, I get a timeout as well.
Goal: To output a list of customers that have had an activity created during the search range, and when that customer is found, get some other information about that customer regarding policies, etc. I've removed some of the properties below for brevity (as best I can)...
UPDATE
After lots of trial and error, I can get the entire query to run fairly consistently within 1000MS so long as this block of code is not executed.
CurrentStatus = a.Activities
.Where(b => b.ActivityType.IsReportable)
.OrderByDescending(b => b.DueDateTime)
.Select(b => b.Status.Name)
.FirstOrDefault(),
With this code in place, things begin to go haywire. I think this Where clause is a big part of it: .Where(b => b.ActivityType.IsReportable). What is the best way to grab the status name?
EXISTING CODE
Any thoughts as to why SQL Azure would timeout whereas on-premises would turn this around in less than 100MS?
return db.Customers
.Where(a => a.Activities.Where(
b => b.CreatedDateTime >= search.BeginDateCreated
&& b.CreatedDateTime <= search.EndDateCreated).Count() > 0)
.Where(a => a.CustomerGroup.Any(d => d.GroupId== search.GroupId))
.Select(a => new CustomCustomerReport
{
CustomerId = a.Id,
Manager = a.Manager.Name,
Customer = a.FirstName + " " + a.LastName,
ContactSource= a.ContactSource!= null ? a.ContactSource.Name : "Unknown",
ContactDate = a.DateCreated,
NewSale = a.Sales
.Where(p => p.Employee.IsActive)
.OrderByDescending(p => p.DateCreated)
.Select(p => new PolicyViewModel
{
//MISC PROPERTIES
}).FirstOrDefault(),
ExistingSale = a.Sales
.Where(p => p.CancellationDate == null || p.CancellationDate <= myDate)
.Where(p => p.SaleDate < myDate)
.OrderByDescending(p => p.DateCreated)
.Select(p => new SalesViewModel
{
//MISC PROPERTIES
}).FirstOrDefault(),
CurrentStatus = a.Activities
.Where(b => b.ActivityType.IsReportable)
.OrderByDescending(b => b.DueDateTime)
.Select(b => b.Disposition.Name)
.FirstOrDefault(),
CustomerGroup = a.CustomerGroup
.Where(cd => cd.GroupId == search.GroupId)
.Select(cd => new GroupViewModel
{
//MISC PROPERTIES
}).FirstOrDefault()
}).ToList();

I cannot give you a definite answer but I would recommend approaching the problem by:
Run SQL profiler locally when this code is executed and see what SQL is generated and run. Look at the query execution plan for each query and look for table scans and other slow operations. Add indexes as needed.
Check your lambdas for things that cannot be easily translated into SQL. You might be pulling the contents of a table into memory and running lambdas on the results, which will be very slow. Change your lambdas or consider writing raw SQL.
Is the Azure database the same as your local database? If not, pull the data locally so your local system is indicative.
Remove sections (i.e. CustomerGroup then CurrentDisposition then ExistingSale then NewSale) and see if there is a significant performance improvement after removing the last section. Focus on the last removed section.
Looking at the line itself:
You use ".Count() > 0" on line 4. Use ".Any()" instead, since the former goes through every row in the database to get you an accurate count when you just want to know if at least one row satisfies the requirements.
Ensure fields referenced in where clauses have indexes, such as IsReportable.

Short answer: use memory.
Long answer:
Because of either bad maintenance plans or limited hardware, running this query in one big lump is what's causing it to fail on Azure. Even if that weren't the case, because of all the navigation properties you're using, this query would generate a staggering number of joins. The answer here is to break it down in smaller pieces that Azure can run. I'm going to try to rewrite your query into multiple smaller, easier to digest queries that use the memory of your .NET application. Please bear with me as I make (more or less) educated guesses about your business logic/db schema and rewrite the query accordingly. Sorry for using the query form of LINQ but I find things such as join and group by are more readable in that form.
var activityFilterCustomerIds = db.Activities
.Where(a =>
a.CreatedDateTime >= search.BeginDateCreated &&
a.CreatedDateTime <= search.EndDateCreated)
.Select(a => a.CustomerId)
.Distinct()
.ToList();
var groupFilterCustomerIds = db.CustomerGroup
.Where(g => g.GroupId = search.GroupId)
.Select(g => g.CustomerId)
.Distinct()
.ToList();
var customers = db.Customers
.AsNoTracking()
.Where(c =>
activityFilterCustomerIds.Contains(c.Id) &&
groupFilterCustomerIds.Contains(c.Id))
.ToList();
var customerIds = customers.Select(x => x.Id).ToList();
var newSales =
(from s in db.Sales
where customerIds.Contains(s.CustomerId)
&& s.Employee.IsActive
group s by s.CustomerId into grouped
select new
{
CustomerId = grouped.Key,
Sale = grouped
.OrderByDescending(x => x.DateCreated)
.Select(new PolicyViewModel
{
// properties
})
.FirstOrDefault()
}).ToList();
var existingSales =
(from s in db.Sales
where customerIds.Contains(s.CustomerId)
&& (s.CancellationDate == null || s.CancellationDate <= myDate)
&& s.SaleDate < myDate
group s by s.CustomerId into grouped
select new
{
CustomerId = grouped.Key,
Sale = grouped
.OrderByDescending(x => x.DateCreated)
.Select(new SalesViewModel
{
// properties
})
.FirstOrDefault()
}).ToList();
var currentStatuses =
(from a in db.Activities.AsNoTracking()
where customerIds.Contains(a.CustomerId)
&& a.ActivityType.IsReportable
group a by a.CustomerId into grouped
select new
{
CustomerId = grouped.Key,
Status = grouped
.OrderByDescending(x => x.DueDateTime)
.Select(x => x.Disposition.Name)
.FirstOrDefault()
}).ToList();
var customerGroups =
(from cg in db.CustomerGroups
where cg.GroupId == search.GroupId
group cg by cg.CustomerId into grouped
select new
{
CustomerId = grouped.Key,
Group = grouped
.Select(x =>
new GroupViewModel
{
// ...
})
.FirstOrDefault()
}).ToList();
return customers
.Select(c =>
new CustomCustomerReport
{
// ... simple props
// ...
// ...
NewSale = newSales
.Where(s => s.CustomerId == c.Id)
.Select(x => x.Sale)
.FirstOrDefault(),
ExistingSale = existingSales
.Where(s => s.CustomerId == c.Id)
.Select(x => x.Sale)
.FirstOrDefault(),
CurrentStatus = currentStatuses
.Where(s => s.CustomerId == c.Id)
.Select(x => x.Status)
.FirstOrDefault(),
CustomerGroup = customerGroups
.Where(s => s.CustomerId == c.Id)
.Select(x => x.Group)
.FirstOrDefault(),
})
.ToList();

Hard to suggest anything without seeing actual table definitions, espectially the indexes and foreign keys on Activities entity.
As far I understand Activity (CustomerId, ActivityTypeId, DueDateTime, DispositionId). If this is standard warehousing table (DateTime, ClientId, Activity), I'd suggest the following:
If number of Activities is reasonably small, then force the use of CONTAINS by
var activities = db.Activities.Where( x => x.IsReportable ).ToList();
...
.Where( b => activities.Contains(b.Activity) )
You can even help the optimiser by specifying that you want ActivityId.
Indexes on Activitiy entity should be up to date. For this particular query I suggest (CustomerId, ActivityId, DueDateTime DESC)
precache Disposition table, my crystal ball tells me that it's dictionary table.
For similar task to avoid constantly hitting Activity table I made another small table (CustomerId, LastActivity, LastVAlue) and updated it as the status changed.

LINQ, Joins, Grouping and Aggregates

Using either a Join or GroupJoin, is there any way to produce aggregates values for fields in both the parent and child tables. Given an Orders table and an OrderDetails table, Using the 2 steps below I can obtain an aggregate (MAX) from the Orders and an aggregate (SUM) from the OrderDetails.
STEP 1:
var query = from o in orders
join d in details on o.OrderId equals d.OrderId
select new
{
order = o.OrderId,
maximum = o.UserId,
quantity = d.Quantity
};
Step 2:
var result = (from q in query
group q by q.order into g
select new
{
OrderId = g.Key,
MaxUnits = g.Max(q => q.maximum),
Available = (g.Max(q => q.maximum) - g.Sum(q => q.quantity))
});
However, when I try to combine these as in:
var finalresult = orders
.GroupJoin( details,
o => o.OrderId,
d => d.OrderDetailId,
(o, grp) => new {
OrderId = o.OrderId,
MaxUnits = grp.Max(o => o.maximum),
Available = (grp.Max(o => o.maximum) - grp.Sum(d => d.Quantity))
});
.. the value 'o' is out of scope inside the grouped set 'grp'. So grp.Max(o => o.maximum) results in an error. It appears that only aggregate values for the child table (OrderDetail) are available.
So does anyone know if it is possible to obtain aggregates from both the Child and Parent tables in a single query?

result is a single query. The beauty of LINQ and deferred execution is that no actual computation has happened in Step 1, only a query has been defined. Step 2 then builds ontop of that query to create another single query. When you execute result that query will be executed as a single block.
I recommend splitting up larger queries into smaller easier to understand pieces like in the first two examples. Using good names for the queries can make them much easier to read. For example, I might name query orderQuantities. from q in query does not convey much meaning, but from oq in orderQuantities lets me know what kind of data the query is over.
If you really think you need them together:
var query = orders.Join(details, o => o.OrderId, d => d.OrderId,
(o, d) => new {
order = o.OrderId,
maximum = o.UserId,
quantity = d.Quantity
}).GroupBy(oq => oq.order)
.Select(g => new {
OrderId = g.Key,
MaxUnits = g.Max(q => q.maximum),
Available = (g.Max(q => q.maximum) - g.Sum(q => q.quantity))
});
Now that is ugly...