Slow LINQ Query on Join - c#

I'm using EF 6 with .NET Core. I have written the following query to select some data into a custom object. At the moment the main query returns over 25000 records. Then the filters are applied. This query is running very slow. I'm just trying to find out if I'm doing any mistakes here. Why it could be so slow and is the filtering happening on the memory rather than on the database side?
public IQueryable<TicketViewModel> GetTickets(int companyId, string locale, TicketFilterViewModel filters, Guid customerRef, int? take, int currentUser)
{
IQueryable<TicketViewModel> tickets = (from to in _context.Ticket
join co in _context.Company on to.CompanyId equals co.CompanyId
join con in _context.Contact on to.ContactId equals con.ContactId
join site in _context.Site on to.SiteId equals site.SiteId
join country in _context.Country on site.CountryId equals country.CountryId
join cus in _context.Customer on con.CustomerId equals cus.CustomerId
join tic_type in _context.TicketType on to.TicketTypeId equals tic_type.TicketTypeId
join ts in _context.TicketStatus on to.TicketStatusId equals ts.TicketStatusId
join sb in _context.ServiceBoard on to.ServiceBoardId equals sb.ServiceBoardId into ob1
from a in ob1.DefaultIfEmpty()
join u in _context.User on to.TechnitianId equals u.Id into ob2
from b in ob2.DefaultIfEmpty()
join pr in _context.Priority on to.PriorityId equals pr.PriorityId into ob3
from c in ob3.DefaultIfEmpty()
where to.CompanyId == companyId && (customerRef == Guid.Empty || cus.RefNo == customerRef)
&& to.MergedIntoTicketId == null
select new TicketViewModel
{
CreatedOn = Helpers.Custom.UtcToStandardTime(locale, to.AddedOnUtc).ToString("dd/MM/yyyy hh:mm tt"),
CustomerName = cus.CustomerName,
TicketNumber = to.TicketNumber,
TicketTitle = to.TicketTitle,
RefNo = to.RefNo,
CompanyName = co.CompanyName,
ContactName = string.Concat(con.FirstName, " ", con.LastName),
SiteAddress = String.Concat(site.AddressLine1, ", ", site.AddressLine2, ", ", site.Suburb, ", ", site.State, ", ", site.PostCode, ", ", country.Name),
ServiceBoardName = a.BoardName,
TechnicianName = string.Concat(b.FirstName, " ", b.LastName),
PriorityName = c.PriorityName,
TicketStatus = ts.StatusName, //d.StatusName,
ServiceBoardId = to.ServiceBoardId,
TicketStatusId = to.TicketStatusId,
CustomerId = cus.CustomerId,
PriorityId = to.PriorityId,
ContractId = site.ContractId,
TechnitianId = to.TechnitianId,
TicketId = to.TicketId,
StatusCategoryId = ts.CategoryId,//,//d.CategoryId,
DueOnUtc = to.DueOnUtc,
DefaultStatusId = ts.DefaultId,//d.DefaultId,
ClosedOnUtc = to.ClosedOnUtc,
ResolvedOnUtc = to.ResolvedOnUtc,
InitialResponseMade = to.InitialResponseMade,
TicketTypeId = to.TicketTypeId,
TicketTypeName = tic_type.TicketTypeName
}).OrderByDescending(o => o.TicketNumber);
bool isFiltersHit = false;
if (filters != null)
{
if (tickets != null && tickets.Count() > 0 && filters.serviceboard_selectedItems != null && filters.serviceboard_selectedItems.Count > 0)
{
isFiltersHit = true;
tickets = tickets.Where(x => x.ServiceBoardId != null && filters.serviceboard_selectedItems.Select(o => o.serviceBoardId).Contains(x.ServiceBoardId.Value));
}
if (tickets != null && tickets.Count() > 0 && filters.status_selectedItems != null && filters.status_selectedItems.Count > 0)
{
isFiltersHit = true;
tickets = tickets.Where(x => x.TicketStatusId != null && filters.status_selectedItems.Select(o => o.ticketStatusId).Contains(x.TicketStatusId.Value));
}
if (tickets != null && tickets.Count() > 0 && filters.type_selectedItems != null && filters.type_selectedItems.Count > 0)
{
isFiltersHit = true;
tickets = tickets.Where(x => filters.type_selectedItems.Select(o => o.ticketTypeId).Contains(x.TicketTypeId));
}
if (tickets != null && tickets.Count() > 0 && filters.technician_selectedItems != null && filters.technician_selectedItems.Count > 0)
{
isFiltersHit = true;
tickets = tickets.Where(x => x.TechnitianId != null && filters.technician_selectedItems.Select(o => o.id).Contains(x.TechnitianId.Value));
}
if (tickets != null && tickets.Count() > 0 && filters.customer_selectedItems != null && filters.customer_selectedItems.Count > 0)
{
isFiltersHit = true;
tickets = tickets.Where(x => x.CustomerId != 0 && filters.customer_selectedItems.Select(o => o.customerId).Contains(x.CustomerId));
}
if (tickets != null && tickets.Count() > 0 && filters.priority_selectedItems != null && filters.priority_selectedItems.Count > 0)
{
isFiltersHit = true;
tickets = tickets.Where(x => x.PriorityId != null && filters.priority_selectedItems.Select(o => o.priorityId).Contains(x.PriorityId.Value));
}
if (tickets != null && tickets.Count() > 0 && filters.contract_selectedItems != null && filters.contract_selectedItems.Count > 0)
{
isFiltersHit = true;
tickets = tickets.Where(x => x.ContractId != null && filters.contract_selectedItems.Select(o => o.contractId).Contains(x.ContractId.Value));
}
if (tickets != null && tickets.Count() > 0 && filters.source_selectedItems != null && filters.source_selectedItems.Count > 0)
{
isFiltersHit = true;
tickets = tickets.Where(x => x.SourceId != 0 && filters.source_selectedItems.Select(o => o.item_id).Contains(x.SourceId));
}
}
if (take.HasValue)
return tickets.Take(take.Value);
return tickets;
}

If you compose the query based on your conditions from the filters, you will see a big improvement. Meaning, create a base query ( base and minimum conditions ) and then add the conditions one by one, depending on what your filters are. Execute the query after all filters are checked.
P.S: you don't really need to call tickets.Count() on every filter. Do it once, in the beginning.

Related

42P18: could not determine data type of parameter $12

I'm trying to run this query, but it tells me this error:
42P18: could not determine data type of parameter $12
bds.DataSource = (from vendas in conexao.Vendas
join nota in conexao.NotaFiscal on vendas.IdNotaFIscal equals nota.Id
join clientes in conexao.Cliente on vendas.IdCliente equals clientes.Id
where (nota.DataEmissao >= periodoInicial || periodoInicial == null) &&
(nota.DataEmissao <= periodoFinal || periodoFinal == null) &&
(clientes.Id == idCliente || idCliente == null) &&
(nota.NumeroNotaFiscal >= notaInicial || notaInicial == null) &&
(nota.NumeroNotaFiscal <= notaFinal || notaFinal == null) &&
(nota.Modelo == mod || mod == null)
orderby nota.Id descending
select new
{
Id = nota.Id,
ClienteId = clientes.NomeRazao,
Chave = nota.Chave,
DataEmissao = nota.DataEmissao,
Status = nota.Status,
NumeroNota = nota.NumeroNotaFiscal,
Modelo = nota.Modelo,
}).ToList();
The error occurs in this line below, if I withdraw, works normal, I have tried to change, however I need this filter.I can't understand what I'm doing wrong.
(nota.Modelo == mod || mod == null)
And here's how I fill out the variable
string mod = String.Empty;
if(modelo != "TODOS") mod = modelo == "NF-E" ? "55" : modelo== "NFC-e" ? "65" : null;
Any idea how I can fix it? Thank you.
Try to separate filtering. Such parameters is bad for LINQ Translation and performance. I hope it will solve your problem.
var query =
from vendas in conexao.Vendas
join nota in conexao.NotaFiscal on vendas.IdNotaFIscal equals nota.Id
join clientes in conexao.Cliente on vendas.IdCliente equals clientes.Id
select new { vendas, nota, clientes };
if (periodoInicial != null)
query = query.Where(x => x.nota.DataEmissao >= periodoInicial);
if (periodoFinal != null
query = query.Where(x => x.nota.DataEmissao <= periodoFinal);
if (idCliente != null)
query = query.Where(x => x.clientes.Id == idCliente);
if (notaInicial != null)
query = query.Where(x => x.nota.NumeroNotaFiscal >= notaInicial);
if (notaFinal != null)
query = query.Where(x => x.nota.NumeroNotaFiscal <= notaFinal);
if (mod != null)
query = query.Where(x => x.nota.Modelo == mod);
var result =
from q in query
orderby q.nota.Id descending
select new
{
Id = q.nota.Id,
ClienteId = q.clientes.NomeRazao,
Chave = q.nota.Chave,
DataEmissao = q.nota.DataEmissao,
Status = q.nota.Status,
NumeroNota = q.nota.NumeroNotaFiscal,
Modelo = q.nota.Modelo,
};
bds.DataSource = result.ToList();

Having problems forming a linq subquery in my Application Context

Here is an example of the SQL I am trying to convert to Linq
SELECT
* FROM assetassignment
WHERE asgn_type = 'trc' AND asgn_id = '54490' AND lgh_number <> 3015097 AND mov_number <>2030782
and asgn_enddate in (select max(asgn_enddate) FROM assetassignment
WHERE asgn_type = 'trc' AND asgn_id = '54490' AND lgh_number <> 3015097 AND mov_number <> 2030782 and asgn_enddate <= '03/19/2017')
I need to convert this to Linq. I am using a repository as follows:
public async task<AssetAssignment> FindPreviousTrip(string tractorNumber, DateTime endDate, int legNumber,string moveNumber)
{
using (var ctx = new AssetAssignmentContext(_nameOrConnectionString))
{
var previousTrip = from a in ctx.Set<AssetAssignment>()
where a.AssignmentType == "TRC" &&
a.AssignmentId == tractorNumber &&
a.LegId == legNumber &&
a.MoveNumber == moveNumber &&
a.EndDate).in()
}
}
As you can see I am making a stab at it, but keep failing on the subquery for the EndDate in statement.
I'll keep plugging at it, but any help is appreciated.
var previousTrip = from a in ctx.Set<AssetAssignment>()
where a.AssignmentType == "TRC" &&
a.AssignmentId == tractorNumber &&
a.LegId != legNumber &&
a.MoveNumber != moveNumber &&
a.EndDate == (from b in ctx.Set<AssetAssignment>()
where b.AssignmentType == "TRC" && b.AssignmentId == tractorNumber && b.LegId != legNumber && b.MoveNumber != moveNumber && b.EndDate <= new DateTime(2017, 19, 3)
select b.EndDate).Max()
select a;
You may want to create a variable for the common query
var baseTrip = from a in ctx.Set<AssetAssignment>()
where a.AssignmentType == "TRC" &&
a.AssignmentId == tractorNumber &&
a.LegId != legNumber &&
a.MoveNumber != moveNumber
select a;
var previousTrip = from a in baseTrip
where a.EndDate == (from b in baseTrip where b.EndDate <= new DateTime(2017, 19, 3) select b.EndDate).Max()
select a;

How to create condtional lambda expression for date ranges?

In my ASP.net MVC project, I've a search form with a total of 7 search variables. None of them are mandatory, however, at least one should be selected to perform search operation, as shown in the image.
What is the best method to perform search using lambda expression in this scenario since any combination of search parameters can be selected by the user? For now, I'm using a long list of 'if-else if' structures and writing individual lambda expressions for each. Is there a better/ more efficient way?
Update 1:
This is just half of the conditions, and now I've been told that even the 'Date To' for each of the three date criteria are optional, which will increase the list of checks even more. Is this the standard way of doing it?
IEnumerable<MyRecord> Request = null;
if (!status.Equals(""))
{
if (deliveryDateFrom == null && caseDateFrom == null && pickupDateFrom == null)
{
//only request status
Request = GetRecord("EquipRequest")
.Where(r => r.Status == status);
}
else if (deliveryDateFrom != null && caseDateFrom == null && pickupDateFrom == null)
{
// request status and delivery date
Request = GetRecord("EquipRequest")
.Where(r => r.Status == status && (r.DeliveryDate >= deliveryDateFrom && r.DeliveryDate <= deliveryDateTo));
}
else if (deliveryDateFrom == null && caseDateFrom != null && pickupDateFrom == null)
{
// request status and case date
Request = GetRecord("EquipRequest")
.Where(r => r.Status == status && (r.CaseDate >= caseDateFrom && r.CaseDate <= caseDateTo));
}
else if (deliveryDateFrom == null && caseDateFrom == null && pickupDateFrom != null)
{
// request status and pickup date
Request = GetRecord("EquipRequest")
.Where(r => r.Status == status && (r.PickupDate >= pickupDateFrom && r.PickupDate <= pickupDateTo));
}
else if (deliveryDateFrom != null && caseDateFrom != null && pickupDateFrom == null)
{
// request status, delivery date and case date
Request = GetRecord("EquipRequest")
.Where(r => r.Status == status && (r.DeliveryDate >= deliveryDateFrom && r.DeliveryDate <= deliveryDateTo)
&& (r.CaseDate >= caseDateFrom && r.CaseDate <= caseDateTo));
}
else if (deliveryDateFrom != null && caseDateFrom == null && pickupDateFrom != null)
{
// request status, delivery date and pickup date
Request = GetRecord("EquipRequest")
.Where(r => r.Status == status && (r.DeliveryDate >= deliveryDateFrom && r.DeliveryDate <= deliveryDateTo)
&& (r.PickupDate >= pickupDateFrom && r.PickupDate <= pickupDateTo));
}
else if (deliveryDateFrom == null && caseDateFrom != null && pickupDateFrom != null)
{
// request status, case date and pickup date
Request = GetRecord("EquipRequest")
.Where(r => r.Status == status && (r.CaseDate >= caseDateFrom && r.CaseDate <= caseDateTo)
&& (r.PickupDate >= pickupDateFrom && r.PickupDate <= pickupDateTo));
}
else if (deliveryDateFrom != null && caseDateFrom != null && pickupDateFrom != null)
{
// request status, delivery date, case date and pickup date
Request = GetRecord("EquipRequest")
.Where(r => r.Status == status && (r.DeliveryDate >= deliveryDateFrom && r.DeliveryDate <= deliveryDateTo)
&& (r.CaseDate >= caseDateFrom && r.CaseDate <= caseDateTo)
&& (r.PickupDate >= pickupDateFrom && r.PickupDate <= pickupDateTo));
}
}
What you could do is use a function like the following:
public static IEnumerable<TSource> WhereIf<TSource>(this IEnumerable<TSource> source, bool condition, Func<TSource, bool> predicate)
{
return condition ? source.Where(predicate) : source;
}
What it will do is if the bool is true, it will narrow it down by your lambda expression, if it is false, it will just continue.
And used it like so:
var whatever = GetRecord("EquipRequest")
.Where(r.Status == status)
.WhereIf(deliveryDateFrom != null, r => r.DeliveryDate >= deliveryDateFrom)
.WhereIf(deliveryDateTo != null, r => r.DeliveryDate <= deliveryDateTo)
.WhereIf(caseDateFrom != null, r => r.CaseDate >= caseDateFrom)
.WhereIf(caseDateTo != null, r => r.CaseDate <= caseDateTo)
.WhereIf(pickupDateFrom != null, r => r.PickupDate >= pickupDateFrom)
.WhereIf(pickupDateTo != null, r => r.PickupDate <= pickupDateTo)
So on and so forth
First I recommend you get LinqPad, learn it, if you want to experiment with things like this.
Then write something like this LinqPad script and run it to get your Linq statement correct:
public class SearchViewModel {
public DateTime DeliveryDateFrom {get; set;}
public DateTime DeliveryDateTo {get; set;}
public DateTime CaseDateFrom {get; set;}
public DateTime CaseDateTo {get; set;}
public DateTime PickupDateFrom {get; set;}
public DateTime PickupDateTo {get; set;}
};
public class Data {
public DateTime DeliveryDate {get; set;}
public DateTime CaseDate {get; set;}
public DateTime PickupDate {get; set;}
};
void Main()
{
var model = new SearchViewModel(); // insert your model stuff here
// you can use LinqPad to hook up directly to your project DBContext dll here
var dataBase = new List<Data>() {
new Data() { DeliveryDate = DateTime.Now, CaseDate = DateTime.Now, PickupDate = DateTime.Now }
};
// experiment with Linq statement here
var result = dataBase.Where(x =>
(model.DeliveryDateFrom >= x.DeliveryDate && model.DeliveryDateTo <= x.DeliveryDate) &&
(model.CaseDateFrom >= x.CaseDate && model.CaseDateTo <= x.CaseDate) &&
(model.PickupDateFrom >= x.PickupDate && model.PickupDateTo <= x.PickupDate)
);
result.Dump();
}
You can chain where clause.
IEnumerable<MyRecord> query = GetRecord("EquipRequest");
if (!status.Equals(""))
{
if (deliveryDateFrom.HasValue)
query = query.Where(x => x.DeliveryDate >= deliveryDateFrom);
if (deliveryDateTo.HasValue)
query = query.Where(x => x.DeliveryDate <= deliveryDateTo);
}
IEnumerable<MyRecord> result = query.ToList();

LINQ to Entity query execution time

I have this LINQ to Entity query which fetches around 5000 Plus records. I have been noticed is that the higher the records returned the more time it takes to execute the query and return the results.
lstUserAssignments = context.UserAssessments.Where(a=>a.UserID == UserID && a.CourseID == ClassID && a.ProductID == a.ProductID && a.AppID == ApplicationID && a.IsDeleted == false && a.Score_Percentage >= 0).Select(a=> new UserAssessmentEntity()
{
ApplicationID = a.AppID,
AttemptNo = a.AttemptNo,
CourseID = a.CourseID,
Score_Percentage = (float)(a.Score_Percentage != null ? a.Score_Percentage : 0),
Status = a.Status,
UserAssessmentID = a.UserAssessmentID,
UserID = a.UserID,
UserScore = a.User_Score,
CreatedDateTime = a.CreatedDateTime,
ModifiedDateTime = a.ModifiedDateTime,
TimeSpent=a.UserAssessmentDetails.Sum(i=>i.TimeSpent??0),
InstructorFeedbackText = a.UserAssessmentDetails.FirstOrDefault()!=null?a.UserAssessmentDetails.FirstOrDefault().InstructorFeedbackText:string.Empty
}).ToList();
To make the query simple. There is UserAssessment and UserAssessmentDetails table. The first one is the parent and the latter the child. There is some performance issue in the query and i feel its in calculating TimeSpent and InstructorFeedback.
Can anyone point my finger on the problem.
Take and Skip:
context.UserAssessments.Where(a=>a.UserID == UserID && a.CourseID == ClassID && a.ProductID == a.ProductID && a.AppID == ApplicationID && a.IsDeleted == false && a.Score_Percentage >= 0).Select(a=> new UserAssessmentEntity()
{
ApplicationID = a.AppID,
AttemptNo = a.AttemptNo,
CourseID = a.CourseID,
Score_Percentage = (float)(a.Score_Percentage != null ? a.Score_Percentage : 0),
Status = a.Status,
UserAssessmentID = a.UserAssessmentID,
UserID = a.UserID,
UserScore = a.User_Score,
CreatedDateTime = a.CreatedDateTime,
ModifiedDateTime = a.ModifiedDateTime,
TimeSpent=a.UserAssessmentDetails.Sum(i=>i.TimeSpent??0),
InstructorFeedbackText = a.UserAssessmentDetails.FirstOrDefault()!=null?a.UserAssessmentDetails.FirstOrDefault().InstructorFeedbackText:string.Empty
}).Take(10).Skip(10).ToList();

Problem with LINQ query

The following works fine:
(from e in db.EnquiryAreas
from w in db.WorkTypes
where
w.HumanId != null &&
w.SeoPriority > 0 &&
e.HumanId != null &&
e.SeoPriority > 0 &&
db.Enquiries.Where(f =>
f.WhereId == e.Id &&
f.WhatId == w.Id &&
f.EnquiryPublished != null &&
f.StatusId != EnquiryMethods.STATUS_INACTIVE &&
f.StatusId != EnquiryMethods.STATUS_REMOVED &&
f.StatusId != EnquiryMethods.STATUS_REJECTED &&
f.StatusId != EnquiryMethods.STATUS_ATTEND
).Any()
select
new
{
EnquiryArea = e,
WorkType = w
});
But:
(from e in db.EnquiryAreas
from w in db.WorkTypes
where
w.HumanId != null &&
w.SeoPriority > 0 &&
e.HumanId != null &&
e.SeoPriority > 0 &&
EnquiryMethods.BlockOnSite(db.Enquiries.Where(f => f.WhereId == e.Id && f.WhatId == w.Id)).Any()
select
new
{
EnquiryArea = e,
WorkType = w
});
+
public static IQueryable<Enquiry> BlockOnSite(IQueryable<Enquiry> linq)
{
return linq.Where(e =>
e.EnquiryPublished != null &&
e.StatusId != STATUS_INACTIVE &&
e.StatusId != STATUS_REMOVED &&
e.StatusId != STATUS_REJECTED &&
e.StatusId != STATUS_ATTEND
);
}
I get the following error:
base {System.SystemException}: {"Method 'System.Linq.IQueryable1[X.Enquiry]
BlockOnSite(System.Linq.IQueryable1[X.Enquiry])' has no supported translation to SQL."}
Linq to Sql only translates certain method calls to SQL, and yours (BlockOnSite) is not one of them. Hence the error. The fact that your method takes an IQueryable<T> and returns an IQueryable<T> doesn't make it special.
Ok I solved it using:
IQueryable<Enquiry> visibleOnSite = EnquiryMethods.VisibleOnSite(db.Enquiries);
var combinations = (from e in db.EnquiryAreas
from w in db.WorkTypes
where
w.HumanId != null &&
w.SeoPriority > 0 &&
e.HumanId != null &&
e.SeoPriority > 0 &&
visibleOnSite.Where(f => f.WhereId == e.Id && f.WhatId == w.Id).Any()
select
new
{
EnquiryArea = e,
WorkType = w
});

Categories