We Keep Coding

Optimize ef-core query

does anyone have any ideas how to improve or optimize this query in terms of performance?
An Include cannot be used due to missing Foreign Keys / Navigation Properties because this is a scaffolded model.
using (var session = new Typo3DBContext())
{
var countryList = session.TxNeustageodataDomainModelCountry
.Where(x => x.Deleted == 0)
.Join(session.TxNeustameinereiseDomainModelTripCountryMm,
country => (uint)country.Uid,
tripMM => tripMM.UidForeign,
(country, tripMM) =>
new
{
country = country,
tripMM = tripMM
})
.Join(session.TxNeustameinereiseDomainModelTrip,
combinedEntry => combinedEntry.tripMM.UidLocal,
trip => trip.Uid,
(combinedEntry, trip) =>
new
{
combinedEntry = combinedEntry,
trip = trip
})
.GroupBy(
temp =>
new
{
Name = temp.combinedEntry.country.Name,
Iso = temp.combinedEntry.country.Iso,
Id = temp.combinedEntry.tripMM.UidForeign,
Status = temp.trip.Status,
Deleted = temp.trip.Deleted
},
temp => temp.combinedEntry.tripMM
)
.Where(x => x.Key.Status == 2 && x.Key.Deleted == 0)
.Select(
group =>
new CountryHelperClass
{
Count = group.Count(),
Iso = group.Key.Iso,
Name = group.Key.Name,
Id = group.Key.Id
})
.ToList();
return countryList;
}

You may analyze the generated SQL first and see if optimal sql is being generated. you may follow the this link to start. Another good tool to work with linq queries is to use LINQPad. Some of the common issue with Linq queries are
The ‘N+1 Select’ problem (If you are using ef core 3 This and other sql related issue re being optimized):
To greedy with row and columns
Change Tracking related issues
Missing indexes
Details of these issue can be found in above link an on internet also
Normally i go for stored procedure approach for complex queries as it saves lot of time of optimization of queries

Group by filtering non-group by field

How can I do a Group by filtering in a non-group by field? I'm using c# connected to a RavenDB database.
I'm new to RavenDB, and I'm trying to run a basic group-by query with no success, neither on RQL nor c#. I'm using Raven.Client lib 4.1.3
var results = await session.Query<Order>()
.Where(s => s.Date <= new DateTime(2019,1,1))
.GroupByArrayValues(x => x.OrderItems.Select(y => y.ProductName))
.Select(x => new
{
Product = x.Key,
Total = x.Sum(s => s.Itens.Sum(i => i.ItemValue))
})
.OrderByDescending(x => x.Total)
.ToListAsync();
When I run this code I got an AggregateException from the Raven.Client lib. Thanks in advance.

I solved the issue... Some strings were bad formatted (with blank spaces in the end) so the aggregate/groupby could not aggregate and sum them properly.
Interesting fact is that SQL Server (the source of my testing data) ignore this and sum all the looked alike results together. So if there is a wrong database here, is SQL Server.

Convert SQL query with multiple GroupBy columns to LINQ

SELECT
[TimeStampDate]
,[User]
,count(*) as [Usage]
FROM [EFDP_Dev].[Admin].[AuditLog]
WHERE [target] = '995fc819-954a-49af-b056-387e11a8875d'
GROUP BY [Target], [User] ,[TimeStampDate]
ORDER BY [Target]
My database table has the columns User, TimeStampDate, and Target (which is a GUID).
I want to retrieve all items for each date for each user and display count of entries.
The above SQL query works. How can I convert it into LINQ to SQL? Am using EF 6.1 and my entity class in C# has all the above columns.
Create Filter basically returns an IQueryable of the entire AuditLogSet :
using (var filter = auditLogRepository.CreateFilter())
{
var query = filter.All
.Where(it => it.Target == '995fc819-954a-49af-b056-387e11a8875d')
.GroupBy(i => i.Target, i => i.User, i => i.TimeStamp);
audits = query.ToList();
}
Am not being allowed to group by on 3 columns in LINQ and I am also not sure how to select like the above SQL query with count. Fairly new to LINQ.

You need to specify the group by columns in an anonymous type like this:-
var query = filter.All
.Where(it => it.Target == '995fc819-954a-49af-b056-387e11a8875d')
.GroupBy(x => new { x.User, x.TimeStampDate })
.Select(x => new
{
TimeStampDate= x.Key.TimeStampDate,
User = x.Key.User,
Usage = x.Count()
}).ToList();

Many people find query syntax simpler and easier to read (this might not be the case, I don't know), here's the query syntax version anyway.
var res=(from it in filter.All
where it.Target=="995fc819-954a-49af-b056-387e11a8875d"
group it by new {it.Target, it.User, it.TimeStampDate} into g
orderby g.Key.Target
select new
{
TimeStampDate= g.Key.TimeStampDate,
User=g.Key.User,
Usage=g.Count()
});
EDIT: By the way you don't need to group by Target neither OrderBy, since is already filtered, I'm leaving the exact translation of the query though.

To use GroupBy you need to create an anonymous object like this:
filter.All
.Where(it => it.Target == '995fc819-954a-49af-b056-387e11a8875d')
.GroupBy(i => new { i.Target, i.User, i.TimeStamp });

It is unnecessary to group by target in your original SQL.
filter.All.Where( d => d.Target == "995fc819-954a-49af-b056-387e11a8875d")
.GroupBy(d => new {d.User ,d.TimeStampDate} )
.Select(d => new {
User = d.Key.User,
TimeStampDate = d.Key.TimeStampDate,
Usage = d.Count()
} );

SQL Azure vs. On-Premises Timeout Issue - EF

I'm working on a report right now that runs great with our on-premises DB (just refreshed from PROD). However, when I deploy the site to Azure, I get a SQL Timeout during its execution. If I point my development instance at the SQL Azure instance, I get a timeout as well.
Goal: To output a list of customers that have had an activity created during the search range, and when that customer is found, get some other information about that customer regarding policies, etc. I've removed some of the properties below for brevity (as best I can)...
UPDATE
After lots of trial and error, I can get the entire query to run fairly consistently within 1000MS so long as this block of code is not executed.
CurrentStatus = a.Activities
.Where(b => b.ActivityType.IsReportable)
.OrderByDescending(b => b.DueDateTime)
.Select(b => b.Status.Name)
.FirstOrDefault(),
With this code in place, things begin to go haywire. I think this Where clause is a big part of it: .Where(b => b.ActivityType.IsReportable). What is the best way to grab the status name?
EXISTING CODE
Any thoughts as to why SQL Azure would timeout whereas on-premises would turn this around in less than 100MS?
return db.Customers
.Where(a => a.Activities.Where(
b => b.CreatedDateTime >= search.BeginDateCreated
&& b.CreatedDateTime <= search.EndDateCreated).Count() > 0)
.Where(a => a.CustomerGroup.Any(d => d.GroupId== search.GroupId))
.Select(a => new CustomCustomerReport
{
CustomerId = a.Id,
Manager = a.Manager.Name,
Customer = a.FirstName + " " + a.LastName,
ContactSource= a.ContactSource!= null ? a.ContactSource.Name : "Unknown",
ContactDate = a.DateCreated,
NewSale = a.Sales
.Where(p => p.Employee.IsActive)
.OrderByDescending(p => p.DateCreated)
.Select(p => new PolicyViewModel
{
//MISC PROPERTIES
}).FirstOrDefault(),
ExistingSale = a.Sales
.Where(p => p.CancellationDate == null || p.CancellationDate <= myDate)
.Where(p => p.SaleDate < myDate)
.OrderByDescending(p => p.DateCreated)
.Select(p => new SalesViewModel
{
//MISC PROPERTIES
}).FirstOrDefault(),
CurrentStatus = a.Activities
.Where(b => b.ActivityType.IsReportable)
.OrderByDescending(b => b.DueDateTime)
.Select(b => b.Disposition.Name)
.FirstOrDefault(),
CustomerGroup = a.CustomerGroup
.Where(cd => cd.GroupId == search.GroupId)
.Select(cd => new GroupViewModel
{
//MISC PROPERTIES
}).FirstOrDefault()
}).ToList();

I cannot give you a definite answer but I would recommend approaching the problem by:
Run SQL profiler locally when this code is executed and see what SQL is generated and run. Look at the query execution plan for each query and look for table scans and other slow operations. Add indexes as needed.
Check your lambdas for things that cannot be easily translated into SQL. You might be pulling the contents of a table into memory and running lambdas on the results, which will be very slow. Change your lambdas or consider writing raw SQL.
Is the Azure database the same as your local database? If not, pull the data locally so your local system is indicative.
Remove sections (i.e. CustomerGroup then CurrentDisposition then ExistingSale then NewSale) and see if there is a significant performance improvement after removing the last section. Focus on the last removed section.
Looking at the line itself:
You use ".Count() > 0" on line 4. Use ".Any()" instead, since the former goes through every row in the database to get you an accurate count when you just want to know if at least one row satisfies the requirements.
Ensure fields referenced in where clauses have indexes, such as IsReportable.

Short answer: use memory.
Long answer:
Because of either bad maintenance plans or limited hardware, running this query in one big lump is what's causing it to fail on Azure. Even if that weren't the case, because of all the navigation properties you're using, this query would generate a staggering number of joins. The answer here is to break it down in smaller pieces that Azure can run. I'm going to try to rewrite your query into multiple smaller, easier to digest queries that use the memory of your .NET application. Please bear with me as I make (more or less) educated guesses about your business logic/db schema and rewrite the query accordingly. Sorry for using the query form of LINQ but I find things such as join and group by are more readable in that form.
var activityFilterCustomerIds = db.Activities
.Where(a =>
a.CreatedDateTime >= search.BeginDateCreated &&
a.CreatedDateTime <= search.EndDateCreated)
.Select(a => a.CustomerId)
.Distinct()
.ToList();
var groupFilterCustomerIds = db.CustomerGroup
.Where(g => g.GroupId = search.GroupId)
.Select(g => g.CustomerId)
.Distinct()
.ToList();
var customers = db.Customers
.AsNoTracking()
.Where(c =>
activityFilterCustomerIds.Contains(c.Id) &&
groupFilterCustomerIds.Contains(c.Id))
.ToList();
var customerIds = customers.Select(x => x.Id).ToList();
var newSales =
(from s in db.Sales
where customerIds.Contains(s.CustomerId)
&& s.Employee.IsActive
group s by s.CustomerId into grouped
select new
{
CustomerId = grouped.Key,
Sale = grouped
.OrderByDescending(x => x.DateCreated)
.Select(new PolicyViewModel
{
// properties
})
.FirstOrDefault()
}).ToList();
var existingSales =
(from s in db.Sales
where customerIds.Contains(s.CustomerId)
&& (s.CancellationDate == null || s.CancellationDate <= myDate)
&& s.SaleDate < myDate
group s by s.CustomerId into grouped
select new
{
CustomerId = grouped.Key,
Sale = grouped
.OrderByDescending(x => x.DateCreated)
.Select(new SalesViewModel
{
// properties
})
.FirstOrDefault()
}).ToList();
var currentStatuses =
(from a in db.Activities.AsNoTracking()
where customerIds.Contains(a.CustomerId)
&& a.ActivityType.IsReportable
group a by a.CustomerId into grouped
select new
{
CustomerId = grouped.Key,
Status = grouped
.OrderByDescending(x => x.DueDateTime)
.Select(x => x.Disposition.Name)
.FirstOrDefault()
}).ToList();
var customerGroups =
(from cg in db.CustomerGroups
where cg.GroupId == search.GroupId
group cg by cg.CustomerId into grouped
select new
{
CustomerId = grouped.Key,
Group = grouped
.Select(x =>
new GroupViewModel
{
// ...
})
.FirstOrDefault()
}).ToList();
return customers
.Select(c =>
new CustomCustomerReport
{
// ... simple props
// ...
// ...
NewSale = newSales
.Where(s => s.CustomerId == c.Id)
.Select(x => x.Sale)
.FirstOrDefault(),
ExistingSale = existingSales
.Where(s => s.CustomerId == c.Id)
.Select(x => x.Sale)
.FirstOrDefault(),
CurrentStatus = currentStatuses
.Where(s => s.CustomerId == c.Id)
.Select(x => x.Status)
.FirstOrDefault(),
CustomerGroup = customerGroups
.Where(s => s.CustomerId == c.Id)
.Select(x => x.Group)
.FirstOrDefault(),
})
.ToList();

Hard to suggest anything without seeing actual table definitions, espectially the indexes and foreign keys on Activities entity.
As far I understand Activity (CustomerId, ActivityTypeId, DueDateTime, DispositionId). If this is standard warehousing table (DateTime, ClientId, Activity), I'd suggest the following:
If number of Activities is reasonably small, then force the use of CONTAINS by
var activities = db.Activities.Where( x => x.IsReportable ).ToList();
...
.Where( b => activities.Contains(b.Activity) )
You can even help the optimiser by specifying that you want ActivityId.
Indexes on Activitiy entity should be up to date. For this particular query I suggest (CustomerId, ActivityId, DueDateTime DESC)
precache Disposition table, my crystal ball tells me that it's dictionary table.
For similar task to avoid constantly hitting Activity table I made another small table (CustomerId, LastActivity, LastVAlue) and updated it as the status changed.

Linq to SQL order by with Distinct

My Environment: ASP.net and C# in VS 2013 Express.
I have been through many similar SO articles trying to work this out. I am amateur with Linq to SQL queries and c# in general.
I'm trying to use Linq to SQL to get the top 5 most recent distinct values from a column, then add them to a list. My application is asp.net using c# and a .dbml file for data abstraction.
I've tried it many different ways. I either get non-distinct yet sorted list, or I get a distinct unsorted list. What I have so far is below
var Top5MFG = (from mfg in db.orders
where mfg.manufacturer.Length > 0 && mfg.customerid == "blahblahblahblahblah"<br />
select new {
manufacturer = mfg.manufacturer,
date = mfg.date_created
})
.Distinct()
.OrderByDescending(s => s.date);
I'm thinking my "Distinct" is looking at the "ID" column, and perhaps I need to tell it I want it to look at the "manufacturer" column, but I haven't worked out how / if it's possible to do that.
I could do this with ease by using a storedproc, but I'm really trying to do it with c# code directly if possible. This is my first post to SO, I hope I have put it together properly. Any help much appreciated.
Thanks

No the Distinct compares manufacturer and date pairs.If you want to get distinct records by manufacturer then I recommend DistinctBy method.It's in the MoreLINQ library.Since its a third library method it's not supported in linq to sql, you still can use it by fetching the records from DB and do the rest in memory
(from mfg in db.orders
where mfg.manufacturer.Length > 0 && mfg.customerid == "blahblahblahblahblah"
select new {
manufacturer = mfg.manufacturer,
date = mfg.date_created
})
.AsEnumerable()
.DistinctBy(x => x.manufacturer)
.OrderByDescending(s => s.date)
.Take(5);

I think you can use the GroupBy to do what you want.
var Top5MFG = db.orders
.Where (x => x.manufacturer.Length > 0 && x.customerid == "blahblahblahblahblah")
.GroupBy(mfg => mfg.manufacturer)
.Select(g => g.First())
.OrderByDescending(d => d.date_created );
.Take(5);

One way you can distinct by a certain field is to replace:
...
.Distinct()
...
with:
...
.GroupBy(x => x.manufacturer )
.Select(g => g.First())
...