I'm working on a report right now that runs great with our on-premises DB (just refreshed from PROD). However, when I deploy the site to Azure, I get a SQL Timeout during its execution. If I point my development instance at the SQL Azure instance, I get a timeout as well.
Goal: To output a list of customers that have had an activity created during the search range, and when that customer is found, get some other information about that customer regarding policies, etc. I've removed some of the properties below for brevity (as best I can)...
UPDATE
After lots of trial and error, I can get the entire query to run fairly consistently within 1000MS so long as this block of code is not executed.
CurrentStatus = a.Activities
.Where(b => b.ActivityType.IsReportable)
.OrderByDescending(b => b.DueDateTime)
.Select(b => b.Status.Name)
.FirstOrDefault(),
With this code in place, things begin to go haywire. I think this Where clause is a big part of it: .Where(b => b.ActivityType.IsReportable). What is the best way to grab the status name?
EXISTING CODE
Any thoughts as to why SQL Azure would timeout whereas on-premises would turn this around in less than 100MS?
return db.Customers
.Where(a => a.Activities.Where(
b => b.CreatedDateTime >= search.BeginDateCreated
&& b.CreatedDateTime <= search.EndDateCreated).Count() > 0)
.Where(a => a.CustomerGroup.Any(d => d.GroupId== search.GroupId))
.Select(a => new CustomCustomerReport
{
CustomerId = a.Id,
Manager = a.Manager.Name,
Customer = a.FirstName + " " + a.LastName,
ContactSource= a.ContactSource!= null ? a.ContactSource.Name : "Unknown",
ContactDate = a.DateCreated,
NewSale = a.Sales
.Where(p => p.Employee.IsActive)
.OrderByDescending(p => p.DateCreated)
.Select(p => new PolicyViewModel
{
//MISC PROPERTIES
}).FirstOrDefault(),
ExistingSale = a.Sales
.Where(p => p.CancellationDate == null || p.CancellationDate <= myDate)
.Where(p => p.SaleDate < myDate)
.OrderByDescending(p => p.DateCreated)
.Select(p => new SalesViewModel
{
//MISC PROPERTIES
}).FirstOrDefault(),
CurrentStatus = a.Activities
.Where(b => b.ActivityType.IsReportable)
.OrderByDescending(b => b.DueDateTime)
.Select(b => b.Disposition.Name)
.FirstOrDefault(),
CustomerGroup = a.CustomerGroup
.Where(cd => cd.GroupId == search.GroupId)
.Select(cd => new GroupViewModel
{
//MISC PROPERTIES
}).FirstOrDefault()
}).ToList();
I cannot give you a definite answer but I would recommend approaching the problem by:
Run SQL profiler locally when this code is executed and see what SQL is generated and run. Look at the query execution plan for each query and look for table scans and other slow operations. Add indexes as needed.
Check your lambdas for things that cannot be easily translated into SQL. You might be pulling the contents of a table into memory and running lambdas on the results, which will be very slow. Change your lambdas or consider writing raw SQL.
Is the Azure database the same as your local database? If not, pull the data locally so your local system is indicative.
Remove sections (i.e. CustomerGroup then CurrentDisposition then ExistingSale then NewSale) and see if there is a significant performance improvement after removing the last section. Focus on the last removed section.
Looking at the line itself:
You use ".Count() > 0" on line 4. Use ".Any()" instead, since the former goes through every row in the database to get you an accurate count when you just want to know if at least one row satisfies the requirements.
Ensure fields referenced in where clauses have indexes, such as IsReportable.
Short answer: use memory.
Long answer:
Because of either bad maintenance plans or limited hardware, running this query in one big lump is what's causing it to fail on Azure. Even if that weren't the case, because of all the navigation properties you're using, this query would generate a staggering number of joins. The answer here is to break it down in smaller pieces that Azure can run. I'm going to try to rewrite your query into multiple smaller, easier to digest queries that use the memory of your .NET application. Please bear with me as I make (more or less) educated guesses about your business logic/db schema and rewrite the query accordingly. Sorry for using the query form of LINQ but I find things such as join and group by are more readable in that form.
var activityFilterCustomerIds = db.Activities
.Where(a =>
a.CreatedDateTime >= search.BeginDateCreated &&
a.CreatedDateTime <= search.EndDateCreated)
.Select(a => a.CustomerId)
.Distinct()
.ToList();
var groupFilterCustomerIds = db.CustomerGroup
.Where(g => g.GroupId = search.GroupId)
.Select(g => g.CustomerId)
.Distinct()
.ToList();
var customers = db.Customers
.AsNoTracking()
.Where(c =>
activityFilterCustomerIds.Contains(c.Id) &&
groupFilterCustomerIds.Contains(c.Id))
.ToList();
var customerIds = customers.Select(x => x.Id).ToList();
var newSales =
(from s in db.Sales
where customerIds.Contains(s.CustomerId)
&& s.Employee.IsActive
group s by s.CustomerId into grouped
select new
{
CustomerId = grouped.Key,
Sale = grouped
.OrderByDescending(x => x.DateCreated)
.Select(new PolicyViewModel
{
// properties
})
.FirstOrDefault()
}).ToList();
var existingSales =
(from s in db.Sales
where customerIds.Contains(s.CustomerId)
&& (s.CancellationDate == null || s.CancellationDate <= myDate)
&& s.SaleDate < myDate
group s by s.CustomerId into grouped
select new
{
CustomerId = grouped.Key,
Sale = grouped
.OrderByDescending(x => x.DateCreated)
.Select(new SalesViewModel
{
// properties
})
.FirstOrDefault()
}).ToList();
var currentStatuses =
(from a in db.Activities.AsNoTracking()
where customerIds.Contains(a.CustomerId)
&& a.ActivityType.IsReportable
group a by a.CustomerId into grouped
select new
{
CustomerId = grouped.Key,
Status = grouped
.OrderByDescending(x => x.DueDateTime)
.Select(x => x.Disposition.Name)
.FirstOrDefault()
}).ToList();
var customerGroups =
(from cg in db.CustomerGroups
where cg.GroupId == search.GroupId
group cg by cg.CustomerId into grouped
select new
{
CustomerId = grouped.Key,
Group = grouped
.Select(x =>
new GroupViewModel
{
// ...
})
.FirstOrDefault()
}).ToList();
return customers
.Select(c =>
new CustomCustomerReport
{
// ... simple props
// ...
// ...
NewSale = newSales
.Where(s => s.CustomerId == c.Id)
.Select(x => x.Sale)
.FirstOrDefault(),
ExistingSale = existingSales
.Where(s => s.CustomerId == c.Id)
.Select(x => x.Sale)
.FirstOrDefault(),
CurrentStatus = currentStatuses
.Where(s => s.CustomerId == c.Id)
.Select(x => x.Status)
.FirstOrDefault(),
CustomerGroup = customerGroups
.Where(s => s.CustomerId == c.Id)
.Select(x => x.Group)
.FirstOrDefault(),
})
.ToList();
Hard to suggest anything without seeing actual table definitions, espectially the indexes and foreign keys on Activities entity.
As far I understand Activity (CustomerId, ActivityTypeId, DueDateTime, DispositionId). If this is standard warehousing table (DateTime, ClientId, Activity), I'd suggest the following:
If number of Activities is reasonably small, then force the use of CONTAINS by
var activities = db.Activities.Where( x => x.IsReportable ).ToList();
...
.Where( b => activities.Contains(b.Activity) )
You can even help the optimiser by specifying that you want ActivityId.
Indexes on Activitiy entity should be up to date. For this particular query I suggest (CustomerId, ActivityId, DueDateTime DESC)
precache Disposition table, my crystal ball tells me that it's dictionary table.
For similar task to avoid constantly hitting Activity table I made another small table (CustomerId, LastActivity, LastVAlue) and updated it as the status changed.
Related
How would you write a linq query with the following SQL statement. I've tried several methods referenced on stackoverflow but they either don't work with the EF version I'm using (EF core 3.5.1) or the DBMS (SQL Server).
select a.ProductID, a.DateTimeStamp, a.LastPrice
from Products a
where a.DateTimeStamp = (select max(DateTimeStamp) from Products where a.ProductID = ProductID)
For reference, a couple that I've tried (both get run-time errors).
var results = _context.Products
.GroupBy(s => s.ProductID)
.Select(s => s.OrderByDescending(x => x.DateTimeStamp).FirstOrDefault());
var results = _context.Products
.GroupBy(x => new { x.ProductID, x.DateTimeStamp })
.SelectMany(y => y.OrderByDescending(z => z.DateTimeStamp).Take(1))
Thanks!
I understand you would like to have a list of the latest prices of each products?
First of all I prefer to use group by option even over 1st query
select a.ProductID, a.DateTimeStamp, a.LastPrice
from Products a
where a.DateTimeStamp IN (select max(DateTimeStamp) from Products group by ProductID)
Later Linq:
var maxDateTimeStamps = _context.Products
.GroupBy(s => s.ProductID)
.Select(s => s.Max(x => x.DateTimeStamp)).ToArray();
var results = _context.Products.Where(s=>maxDateTimeStamps.Contains(s.DateTimeStamp));
-- all assuming that max datetime stamps are unique
I've managed to do it with the following which replicates the correlated sub query in the original post (other than using TOP and order by instead of the Max aggregate), though I feel like there must be a more elegant way to do this.
var results = from x
in _context.Products
where x.DateTimeStamp == (from y
in _context.Products
where y.ProductID == x.ProductID
orderby y.DateTimeStamp descending
select y.DateTimeStamp
).FirstOrDefault()
select x;
I prefer to break up these queries into IQueryable parts, do you can debug each "step".
Something like this:
IQueryable<ProductOrmEntity> pocoPerParentMaxUpdateDates =
entityDbContext.Products
//.Where(itm => itm.x == 1)/*if you need where */
.GroupBy(i => i.ProductID)
.Select(g => new ProductOrmEntity
{
ProductID = g.Key,
DateTimeStamp = g.Max(row => row.DateTimeStamp)
});
//// next line for debugging..do not leave in for production code
var temppocoPerParentMaxUpdateDates = pocoPerParentMaxUpdateDates.ToListAsync(CancellationToken.None);
IQueryable<ProductOrmEntity> filteredChildren =
from itm
in entityDbContext.Products
join pocoMaxUpdateDatePerParent in pocoPerParentMaxUpdateDates
on new { a = itm.DateTimeStamp, b = itm.ProductID }
equals
new { a = pocoMaxUpdateDatePerParent.DateTimeStamp, b = pocoMaxUpdateDatePerParent.ProductID }
// where
;
IEnumerable<ProductOrmEntity> hereIsWhatIWantItems = filteredChildren.ToListAsync(CancellationToken.None);
That last step, I am putting in an anonymous object. You can put the data in a "new ProductOrmEntity() { ProductID = pocoMaxUpdateDatePerParent.ProductID }...or you can get the FULL ProductOrmEntity object. Your original code, I don't know if getting all columns of the Product object is what you want, or only some of the columns of the object.
I have a following working SQL query:
SELECT * FROM truck t
WHERE t.currentlocationdbid IN (SELECT dbid FROM location WHERE name = 'Los Angeles')
OR t.nextdestinationdbid IN (SELECT dbid FROM location WHERE name = 'Chicago' OR name = 'New York');
I'd like to write this in NHibernate. With multiple trips to DB for each entity it works, of course, but I'd like to make it a one trip. Looked into examples with detached queries like this, this or this but none worked for me. Tried to do it also with aliases and criterias.
One of dozens of attempts:
var subQuery1 = QueryOver.Of<LocationEntity>().Where(l => l.Name == LocationNameEnum.LA);
var subQuery2 = QueryOver.Of<LocationEntity>().Where(l => l.Name == LocationNameEnum.CHG || l.Name == LocationNameEnum.NY);
var poc = session.QueryOver<TruckEntity>()
.WithSubquery.WhereProperty(t => t.CurrentLocation).In(subQuery1)
.WithSubquery.WhereProperty(t => t.NextDestination).In(subQuery2)
.List<TruckEntity>();
Thanks in advance for any suggestion.
You got it almost right, you are only missing the .Where(Restrictions.Disjunction()...) for the or in SQL.
Based on your code (assuming that you have a property Id in LocationEntity):
// get IDs to look for in CurrentLocation
var subQuery1 = QueryOver.Of<LocationEntity>()
.Where(l => l.Name == LocationNameEnum.LA)
.Select(x => x.Id);
// get IDs to look for in NextDestination
var subQuery2 = QueryOver.Of<LocationEntity>()
.Where(l => l.Name == LocationNameEnum.CHG || l.Name == LocationNameEnum.NY)
.Select(x => x.Id);
var poc = session.QueryOver<TruckEntity>()
.Where(Restrictions.Disjunction() // this takes care of the OR
.Add(Subqueries.WhereProperty<TruckEntity>(x => x.CurrentLocation.Id).In(subQuery1))
.Add(Subqueries.WhereProperty<TruckEntity>(x => x.NextDestination.Id).In(subQuery2))
)
.List<TruckEntity>();
I have 2 tables:
USERS
UserId
Name
Scores (collection of table Scores)
SCORES
UserId
CategoryId
Points
I need to show all the users and a SUM of their points, but also I need to show the name of the user. It can be filtered by CategoryId or not.
Context.Scores
.Where(p => p.CategoryId == categoryId) * OPTIONAL
.GroupBy(p => p.UserId)
.Select(p => new
{
UserId = p.Key,
Points = p.Sum(s => s.Points),
Name = p.Select(s => s.User.Name).FirstOrDefault()
}).OrderBy(p => p.Points).ToList();
The problem is that when I add the
Name = p.Select(s => s.User.Name).FirstOrDefault()
It takes so long. I don't know how to access the properties that are not inside the GroupBy or are a SUM. This example is very simple becaouse I don't have only the Name, but also other properties from User table.
How can I solve this?
It takes so long because the query is causing client evaluation. See Client evaluation performance issues and how to use Client evaluation logging to identify related issues.
If you are really on EF Core 2.0, there is nothing you can do than upgrading to v2.1 which contains improved LINQ GroupBy translation. Even with it the solution is not straight forward - the query still uses client evaluation. But it could be rewritten by separating the GroupBy part into subquery and joining it to the Users table to get the additional information needed.
Something like this:
var scores = db.Scores.AsQueryable();
// Optional
// scores = scores.Where(p => p.CategoryId == categoryId);
var points = scores
.GroupBy(s => s.UserId)
.Select(g => new
{
UserId = g.Key,
Points = g.Sum(s => s.Points),
});
var result = db.Users
.Join(points, u => u.UserId, p => p.UserId, (u, p) => new
{
u.UserId,
u.Name,
p.Points
})
.OrderBy(p => p.Points)
.ToList();
This still produces a warning
The LINQ expression 'orderby [p].Points asc' could not be translated and will be evaluated locally.
but at least the query is translated and executes as single SQL:
SELECT [t].[UserId], [t].[Points], [u].[UserId] AS [UserId0], [u].[Name]
FROM [Users] AS [u]
INNER JOIN (
SELECT [s].[UserId], SUM([s].[Points]) AS [Points]
FROM [Scores] AS [s]
GROUP BY [s].[UserId]
) AS [t] ON [u].[UserId] = [t].[UserId]
I have Mysql database with ~1 500 000 entities. When I try to execute below statement using EF Core 1.1 and Mysql.Data.EntityFrameworkCore 7.0.7-m61 it takes about 40minutes to finish:
var results = db.Posts
.Include(u => u.User)
.GroupBy(g => g.User)
.Select(g => new { Nick = g.Key.Name, Count = g.Count() })
.OrderByDescending(e => e.Count)
.ToList();
On the other hand using local mysql-cli and below statement, takes around 16 seconds to complete.
SELECT user.Name, count(*) c
FROM post
JOIN user ON post.UserId = user.Id
GROUP BY user.Name
ORDER BY c DESC
Am i doing something wrong, or EF Core performance of MySql is so terrible?
Your queries are doing different things. Some issues in your LINQ-to-Entities query:
You call Include(...) which will eagerly load the User for every item in db.Posts.
You call Count() for each record in each group. This could be rewritten to count the records only once per group.
The biggest issue is that you're only using the Name property of the User object. You could select just this field and find the same result. Selecting, grouping, and returning 1.5 million strings should be a fast operation in EF.
Original:
var results =
db.Posts
.Include(u => u.User)
.GroupBy(g => g.User)
.Select(g => new { Nick = g.Key.Name, Count = g.Count() })
.OrderByDescending(e => e.Count)
.ToList();
Suggestion:
var results =
db.Posts
.Select(x => x.User.Name)
.GroupBy(x => x)
.Select(x => new { Name = x.Key, Count = x.Count() })
.OrderByDescending(x => x.Count)
.ToList();
If EF core still has restrictions on the types of grouping statements it allows, you could call ToList after the first Select(...) statement.
How do fetch data from multiple tables with method syntax without using joins, but only .where() methods?
I'm making a select against EF5 db context which maps to this legacy table structure where I have a persons detail table and another table which refers both to itself to create a hierarchy and to the person details table this way:
PersonSet
.Where(p => p.LastName.ToLower()=="surname")
.Join(SubsetSet, p => p.Id, ss => ss.SubsetLink, (p, ss) => new { PersonDetail = p, person = ss })
.Where(m => m.person.Frame=="a" && m.person.Purpose=="1")
.Join(SubsetSet, ss1 => ss1.person.Owner, person => person.SubsetLink, (ss1, ss2) => new { person = ss1, club = ss2 })
.Where(a => a.club.Frame=="b" && a.club.Purpose=="2")
.Join(SubsetSet, ss => ss.club.Owner, ss2 => ss2.SubsetLink, (ss, ss2) => new { club = ss, association = ss2 })
.Where(a => a.association.Frame=="b" && a.association.Purpose=="3")
.Join(SubsetSet, ss => ss.association.Owner, ss3 => ss3.SubsetLink, (ss, ss3) => new { association = ss, district = ss3})
.Where(d => d.district.Frame=="b" && d.district.Purpose=="4" && d.district.SubsetLink=="12345")
.Select(proj => new { proj.association.club.person, proj.association.club, proj.association, proj.district })
.OrderByDescending(a => a.association.club.person.phyperson.FirstName)
.Take(10).Dump();
The above query works at least in LinqPad but, it seems to me that If I could get rid of those joins the statement might look a bit nicer. Now I know, like in the Albahari example below, that this can be done with query syntax. But I couldn't find an example that would illustrate this situation with method syntax. The way I'm trying to approach this might of course be wrong and that's why I can't find suitable examples.
Here I found something similar, but couldn't make it work in LinQPad:
Is multiple .Where() statements in LINQ a performance issue?
Or this one, where the solution is again in query syntax:
Cross Join with Where clause
Or this example by Albahari: (http://www.linqpad.net/WhyLINQBeatsSQL.aspx)
from p in db.Purchases
where p.Customer.Address.State == "WA" || p.Customer == null
where p.PurchaseItems.Sum (pi => pi.SaleAmount) > 1000
select p
Consider this query:
var q = from order in orders
from orderline in order.Lines
where orderline.Count > 10
select order.Discount * orderline.Price;
this more or less corresponds to
var q = orders
.SelectMany(order => order.Lines, (order, orderline) => new { order, orderline})
.Where(item => item.orderline.Count > 10)
.Select(item => item.order.Discount * item.orderline.Price);
For more information on SelectMany, see the MSDN documentation.
If you don't have associations defined:
var q = from order in orders
from orderline in orderLines
where orderline.OrderId == order.Id
where orderline.Count > 10
select order.Discount * orderline.Price;
this more or less corresponds to
var q = orders
.SelectMany(order => orderLines, (order, orderline) => new { order, orderline})
.Where(item => item.orderline.OrderId == item.order.Id)
.Where(item => item.orderline.Count > 10)
.Select(item => item.order.Discount * item.orderline.Price);