I have a query (including LinqKit) of the form:
Expression<Func<Country, DateTime, bool>> countryIndepBeforeExpr =
(ct, dt) => ct.IndependenceDate <= dt;
DateTime someDate = GetSomeDate();
var q = db.Continent.AsExpandable().Select(c =>
new
{
c.ID,
c.Name,
c.Area,
Countries = c.Countries.AsQueryable()
.Where(ct => countryIndepBeforeExpr.Invoke(ct, someDate))
.Select(ct => new {ct.ID, ct.Name, ct.IndependenceDate})
});
Now I want to iterate through q... but since the Countries property of each element is of type IQueryable, it will be lazy loaded, causing n+1 queries to be executed, which isn't very nice.
What is the correct way to write this query so that all necessary data will be fetched in a single query to the db?
EDIT
Hm, well it might have helped if I had actually run a Sql trace before asking this question. I assumed that because the inner property was of type IQueryable that it would be lazy-loaded... but after doing some actual testing, it turns out that Linq to Entities is smart enough to run the whole query at once.
Sorry to waste all your time. I would delete the question, but since it already has answers, I can't. Maybe it can serve as some kind of warning to others to test your hypothesis before assuming it to be true!
Include countries to your model when you call for continents. With something like this:
var continents = db.Continent.Include(c => c.Countries).ToArray();
Then you can make your linq operations without iQueryable object.
I think this should work (moving AsExpandable() to root of IQueryable):
var q = db.Continent
.AsExpandable()
.Select(c => new
{
c.ID,
c.Name,
c.Area,
Countries = c.Countries
.Where(ct => countryIndepBeforeExpr.Invoke(ct, someDate))
.Select(ct => new {ct.ID, ct.Name, ct.IndependenceDate})
});
If not, create two IQueryable and join them together:
var continents = db.Continents;
var countries = db.Countries
.AsExpandable()
.Where(c => countryIndepBeforeExpr.Invoke(c, someDate))
.Select(c => new { c.ID, c.Name, c.IndependenceDate });
var q = continents.GroupJoin(countries,
continent => continent.ID,
country => country.ContinentId,
(continent, countries) => new
{
continent.ID,
continent.Name,
continent.Area,
Countries = countries.Select(c => new
{
c.ID,
c.Name,
c.IndependenceDate
})
});
Related
Let's say I have a table of locations with location ID and location name. And let's say I want to get the revenues for each location (in this simple scenario I might not even need GroupBy - but please assume that I do!)
var revenues = await _context.SaleTransaction.GroupBy(s => s.LocationId)
.Select(x => new LocationDTO {
LocationId = x.Key,
LocationName = ???
Revenues = x.Sum(i => i.Amount)
}).ToListAsync();
I tried to cheat
LocationName = x.Select(i => i.Location.LocationName).First()
since all location names for this ID are the same. But EF can't translate First() unless I use AsEnumerable() and bring the whole sales table into application memory.
Or I can traverse the result the second time:
foreach(var revenue in revenues) {
revenue.LocationName = _context.Location.Find(revenue.LocationId).LocationName;
}
Given that the number of locations is fixed (and relatively small), it may be the best approach. Still, neither going to DB for every location O(n) nor pulling the whole location list into memory doesn't sit well. Maybe there is a way to assign LocationName (and some other attributes) as part of GroupBy statement.
I am using EF Core 5; or if something is coming in EF Core 6 - that would work as well.
From what I can briefly see is that you need a linq join query in order to join the searches. With EF linq query it means those won't be loaded into memory until they are used so it would solve the problem with loading the whole table.
You could write something like:
var revenues = await _context.SaleTransactions.Join(_context.Locations, s => s.LocationId, l => l.Id, (s, l) => new {LocationId = s.LocationId, LocationName = l.LocationName, Revenues = s.Sum(i => i.Amount)});
I will link the whole fiddle with the mock of your possible model
https://dotnetfiddle.net/BGJmjj
You can group by more than one value. eg;
var revenues = await _context.SaleTransaction
.GroupBy(s => new {
s.LocationId,
s.Location.Name
})
.Select(x => new LocationDTO {
LocationId = x.Key.LocationId,
LocationName = x.Key.Name,
Revenues = x.Sum(i => i.Amount)
}).ToListAsync();
Though it seems like you are calculating a total per location, in which case you can build your query around locations instead.
var revenues = await _context.Location
.Select(x => new LocationDTO {
LocationId = x.Id,
LocationName = x.Name,
Revenues = x.SaleTransactions.Sum(i => i.Amount)
}).ToListAsync();
var revenues = await _context.Location
.Select(x => new LocationDTO {
LocationId = x.Id,
LocationName = x.Name,
Revenues = x.SaleTransactions.Sum(i => i.Amount)
}).ToListAsync();
there is example:
.NetFiddle
How would you write a linq query with the following SQL statement. I've tried several methods referenced on stackoverflow but they either don't work with the EF version I'm using (EF core 3.5.1) or the DBMS (SQL Server).
select a.ProductID, a.DateTimeStamp, a.LastPrice
from Products a
where a.DateTimeStamp = (select max(DateTimeStamp) from Products where a.ProductID = ProductID)
For reference, a couple that I've tried (both get run-time errors).
var results = _context.Products
.GroupBy(s => s.ProductID)
.Select(s => s.OrderByDescending(x => x.DateTimeStamp).FirstOrDefault());
var results = _context.Products
.GroupBy(x => new { x.ProductID, x.DateTimeStamp })
.SelectMany(y => y.OrderByDescending(z => z.DateTimeStamp).Take(1))
Thanks!
I understand you would like to have a list of the latest prices of each products?
First of all I prefer to use group by option even over 1st query
select a.ProductID, a.DateTimeStamp, a.LastPrice
from Products a
where a.DateTimeStamp IN (select max(DateTimeStamp) from Products group by ProductID)
Later Linq:
var maxDateTimeStamps = _context.Products
.GroupBy(s => s.ProductID)
.Select(s => s.Max(x => x.DateTimeStamp)).ToArray();
var results = _context.Products.Where(s=>maxDateTimeStamps.Contains(s.DateTimeStamp));
-- all assuming that max datetime stamps are unique
I've managed to do it with the following which replicates the correlated sub query in the original post (other than using TOP and order by instead of the Max aggregate), though I feel like there must be a more elegant way to do this.
var results = from x
in _context.Products
where x.DateTimeStamp == (from y
in _context.Products
where y.ProductID == x.ProductID
orderby y.DateTimeStamp descending
select y.DateTimeStamp
).FirstOrDefault()
select x;
I prefer to break up these queries into IQueryable parts, do you can debug each "step".
Something like this:
IQueryable<ProductOrmEntity> pocoPerParentMaxUpdateDates =
entityDbContext.Products
//.Where(itm => itm.x == 1)/*if you need where */
.GroupBy(i => i.ProductID)
.Select(g => new ProductOrmEntity
{
ProductID = g.Key,
DateTimeStamp = g.Max(row => row.DateTimeStamp)
});
//// next line for debugging..do not leave in for production code
var temppocoPerParentMaxUpdateDates = pocoPerParentMaxUpdateDates.ToListAsync(CancellationToken.None);
IQueryable<ProductOrmEntity> filteredChildren =
from itm
in entityDbContext.Products
join pocoMaxUpdateDatePerParent in pocoPerParentMaxUpdateDates
on new { a = itm.DateTimeStamp, b = itm.ProductID }
equals
new { a = pocoMaxUpdateDatePerParent.DateTimeStamp, b = pocoMaxUpdateDatePerParent.ProductID }
// where
;
IEnumerable<ProductOrmEntity> hereIsWhatIWantItems = filteredChildren.ToListAsync(CancellationToken.None);
That last step, I am putting in an anonymous object. You can put the data in a "new ProductOrmEntity() { ProductID = pocoMaxUpdateDatePerParent.ProductID }...or you can get the FULL ProductOrmEntity object. Your original code, I don't know if getting all columns of the Product object is what you want, or only some of the columns of the object.
I have a situation where OrderBy need to be done for Include object. This is how I have tried so far
Customers query = null;
try
{
query = _context.Customers
.Include(x => x.CustomerStatus)
.ThenInclude(x => x.StatusNavigation)
.Select(x => new Customers()
{
Id = x.Id,
Address = x.Address,
Contact = x.Contact,
Name = x.Name,
CustomerStatus = new List<CustomerStatus>
{
x.CustomerStatus.OrderByDescending(y => y.Date).FirstOrDefault()
}
})
.FirstOrDefault(x => x.Id == 3);
}
catch (Exception ex)
{
throw;
}
The above code successfully ordering the include element but it is not including it's child table.
Eg: Customer include CustomerStatus but CustomerStatus not including StatusNavigation tables.
I even tried with this but neither it can help me
_context.Customers
.Include(x => x.CustomerStatus.OrderByDescending(y => y.Date).FirstOrDefault())
.ThenInclude(x => x.StatusNavigation).FirstOrDefault(x => x.Id == 3);
What am I doing wrong please guide me someone
Even I tried this way
var query = _context.CustomerStatus
.GroupBy(x => x.CustomerId)
.Select(x => x.OrderByDescending(y => y.Date).FirstOrDefault())
.Include(x => x.StatusNavigation)
.Join(_context.Customers, first => first.CustomerId, second => second.Id, (first, second) => new Customers
{
Id = second.Id,
Name = second.Name,
Address = second.Address,
Contact = second.Contact,
CustomerStatus = new List<CustomerStatus> {
new CustomerStatus
{
Id = first.Id,
CustomerId = first.CustomerId,
Date = first.Date,
StatusNavigation = first.StatusNavigation
}
},
}).FirstOrDefault(x => x.Id == 3);
but this is hitting a databases a 3 times and filtering the result in memory.
First select all data from customer status and then from status and then from customer then it filter all the data in memory. Is there any other efficient way to do this??
This is how I have prepared by entity class
As #Chris Pratt mentioned once you are doing new Customer inside the select you are creating a new model. You are discarding the models build by the EntityFramework. My suggestion would be have the query just:
query = _context.Customers
.Include(x => x.CustomerStatus)
.ThenInclude(x => x.StatusNavigation);
Like this you would have an IQueryable object which it would not be executed unless you do a select from it:
var customer3 = query.FirstOrDefault(x=>x.Id==3)
Which returns the customer and the interlinked tables (CustomerStatus and StatusNavigation). Then you can create the object that you want:
var customer = new Customers()
{
Id = customer3.Id,
Address = customer3.Address,
Contact = customer3.Contact,
Name = x.Name,
CustomerStatus = new List<CustomerStatus>
{
customer3.CustomerStatus.OrderByDescending(y => y.Date).FirstOrDefault()
}
})
In this way you can reuse the query for creating different response objects and have a single querying to database, but downside is that more memory is used then the original query (even though it shouldn't be too much of an issue).
If the model that is originally return from database doesn't meet the requirements (i.e. you always need to do: CustomerStatus = new List {...} ) it might indicate that the database schema is not well defined to the needs of the application, so a refactoring might be needed.
What I think is happening is that you are actually overriding the Include and ThenInclude. Include is explicitly to eager-load a navigation property. However, you're doing a couple of things that are likely hindering this.
First, you're selecting into a new Customer. That alone may be enough to break the logic of Include. Second, you're overriding what gets put in the CustomerStatus collection. That should ideally be just loaded in automatically via Include, but by altering it to just have the first entity, you're essentially throwing away the effect of Include. (Selecting a relationship is enough to cause a join to be issued, without explicitly calling Include). Third, the ThenInclude is predicated on the Include, so overriding that is probably throwing out the ThenIncude as well.
All this is conjecture. I haven't done anything exactly like what you're doing here before, but nothing else makes sense.
Try selecting into a new CustomerStatus as well:
CustomerStatus = x.CustomerStatus.OrderByDescending(o => o.Date).Select(s => new CustomerStatus
{
x.Id,
x.Status,
x.Date,
x.CustomerId,
x.Customer,
x.StatusNavigation
})
You can remove the Include/ThenInclude at that point, because the act of selecting these relationships will cause the join.
After Reading from Couple of sources (Source 1) and (Source 2). I think what is happening is that If you use select after Include. It disregards Include even if you are using Include query data in select. So to solve this use .AsEnumerable() before calling select.
query = _context.Customers
.Include(x => x.CustomerStatus)
.ThenInclude(x => x.StatusNavigation)
.AsEnumerable()
.Select(x => new Customers()
{
Id = x.Id,
Address = x.Address,
Contact = x.Contact,
Name = x.Name,
CustomerStatus = new List<CustomerStatus>
{
x.CustomerStatus.OrderByDescending(y => y.Date).FirstOrDefault()
}
})
.FirstOrDefault(x => x.Id == 3);
I have the following situation which, works fine:
IQueryable<Experiment> experiments1 = _db.Experiments.Where(e => e.Projects.Any(p => p.Id == project.Id));
IQueryable<Experiment> experiments2 = _db.Experiments.Where(e => e.Tools.Any(m => m.Project.Id == project.Id));
var experimentsList = experiments1.Union(experiments2).OrderBy(e => e.Date).Select(e => new
{
e.Id,
e.Name,
e.Date
}).ToList();
Checking with SQL Profiles, this translates in 1 SQL query which correctly extract Id, Name and Date from the union.
The problem arises when I perform a join with a many-to-many connected table:
var experimentsList = experiments1.Union(experiments2).OrderBy(e => e.Date).Select(e => new
{
e.Id,
e.Name,
e.Date,
e.Types.Select(t => t.Name)
// or also just e.Types
}).ToList();
The generated SQL query for some reasons not only return the "needed" columns but returns all the columns in the Experiment table + the additional specified columns (i.e. types' names).
This result in a pretty big loss in performances.
Btw, in the code I later need to perform:
experimentsList.Select(e => new
{
e.Id,
e.Name,
e.Date,
Types = string.Join(", ", e.Types)
})
So I need the faster way to have in memory Id, Name, Date + a list of types.
Thanks!
Can you try this? Column names may need to be changed
var result = from exp in _db.Experiments
join t in _db.Types on exp.TypeID = t.Id
select new
{
exp.Id,
exp.Name,
exp.Date,
t.Name
}).ToList();
SELECT
[TimeStampDate]
,[User]
,count(*) as [Usage]
FROM [EFDP_Dev].[Admin].[AuditLog]
WHERE [target] = '995fc819-954a-49af-b056-387e11a8875d'
GROUP BY [Target], [User] ,[TimeStampDate]
ORDER BY [Target]
My database table has the columns User, TimeStampDate, and Target (which is a GUID).
I want to retrieve all items for each date for each user and display count of entries.
The above SQL query works. How can I convert it into LINQ to SQL? Am using EF 6.1 and my entity class in C# has all the above columns.
Create Filter basically returns an IQueryable of the entire AuditLogSet :
using (var filter = auditLogRepository.CreateFilter())
{
var query = filter.All
.Where(it => it.Target == '995fc819-954a-49af-b056-387e11a8875d')
.GroupBy(i => i.Target, i => i.User, i => i.TimeStamp);
audits = query.ToList();
}
Am not being allowed to group by on 3 columns in LINQ and I am also not sure how to select like the above SQL query with count. Fairly new to LINQ.
You need to specify the group by columns in an anonymous type like this:-
var query = filter.All
.Where(it => it.Target == '995fc819-954a-49af-b056-387e11a8875d')
.GroupBy(x => new { x.User, x.TimeStampDate })
.Select(x => new
{
TimeStampDate= x.Key.TimeStampDate,
User = x.Key.User,
Usage = x.Count()
}).ToList();
Many people find query syntax simpler and easier to read (this might not be the case, I don't know), here's the query syntax version anyway.
var res=(from it in filter.All
where it.Target=="995fc819-954a-49af-b056-387e11a8875d"
group it by new {it.Target, it.User, it.TimeStampDate} into g
orderby g.Key.Target
select new
{
TimeStampDate= g.Key.TimeStampDate,
User=g.Key.User,
Usage=g.Count()
});
EDIT: By the way you don't need to group by Target neither OrderBy, since is already filtered, I'm leaving the exact translation of the query though.
To use GroupBy you need to create an anonymous object like this:
filter.All
.Where(it => it.Target == '995fc819-954a-49af-b056-387e11a8875d')
.GroupBy(i => new { i.Target, i.User, i.TimeStamp });
It is unnecessary to group by target in your original SQL.
filter.All.Where( d => d.Target == "995fc819-954a-49af-b056-387e11a8875d")
.GroupBy(d => new {d.User ,d.TimeStampDate} )
.Select(d => new {
User = d.Key.User,
TimeStampDate = d.Key.TimeStampDate,
Usage = d.Count()
} );