Slow Performance on searching through List in LINQ - c#

I had a table which has more than 200,000 records for any particular month.
Getting records from a table is not a problem it is working as expected but searching through records shows very slow performance
var listEmpShiftDetails =ctx.tblTO_ShiftSchedule
.Where(m => m.CompanyId == companyId &&
m.ShiftDate >= fromdate &&
m.ShiftDate <= todate)
.Select(m => m).ToList();
Records fetched from database around: 200 000
var data = (from a in ctx.tblEmployee
join b in ctx.tblTO_Entry on a.Id equals b.EmployeeId
where a.CompanyId == companyId && b.CompanyId == companyId &&
(b.Entry_Date >= fromDate && b.Entry_Date <= toDate)
select new { a, b }).ToList();
*ote: No database called are made in below code.all the data is fetched above
Linq Query to fetch one by one record
foreach (var item in data) // Data consist of employee details 3k Records
{
if (listEmpShiftDetails
.Any(m => m.EmployeeId == item.a.Id &&
m.ShiftDate == item.b.Entry_Date))
{
var shiftDetails = listEmpShiftDetails
.Where(m => m.EmployeeId == item.a.Id &&
m.ShiftDate ==item.b.Entry_Date)
.Select(m => m)
.FirstOrDefault();
//Other Calculations
}
}
Above 2 Lines takes too much time to execute, below is output from Visual Studio. How to improve the performance?
Profiler Output

var listEmpShiftDetails = Records Fetched from Database around :- 2 Lakh Recordd;
foreach (var item in data) // Data consist of employee details 3k Records
{
var selectedItem = listEmpShiftDetails.FirstOrDefault(m => m.EmployeeId == item.Id &&
m.ShiftDate == item.Entry_Date);
if (selectedItem != null)
{
// Other Calculations
}
}
No need to iterate same query several times. you just need first item if it matches otherwise null. Hope above query gives you better performance.

What Comes on my head after looking your query:
Extra query load, no need to check If(){}.
If your "listEmpShiftDetails" is IQueryable then listEmpShiftDetails.Any() is OK. But if it is List then it hampers the persormance.
Worth spending 5 mins Count Vs Any performance.
Keep your query simple.
var shiftDetails = listEmpShiftDetails.FirstOrDefault(m => m.EmployeeId == item.Id && m.ShiftDate == item.Entry_Date);

To start with, do it this way to avoid executing the query twice:
foreach (var item in data) // Data consist of employee details 3k Records
{
var shiftDetails = listEmpShiftDetails.Where(m => m.EmployeeId == item.Id && m.ShiftDate == item.Entry_Date).FirstOrDefault()
if (shiftDetails != null)
{
//Other Calculations
}
}
Next, it appears that you're doing some sort of join it would be ideal to see what makes up data so that we could further suggest a way to improve the time significantly.
It's possible that this might give you some improvement:
var query =
(
from a in ctx.tblEmployee.Where(x => x.CompanyId == companyId)
join b in ctx.tblTO_Entry.Where(x => x.CompanyId == companyId) on a.Id equals b.EmployeeId
where b.Entry_Date >= fromDate
where b.Entry_Date <= toDate
join m in ctx.tblTO_ShiftSchedule.Where(x => x.CompanyId == companyId) on new
{
a.Id,
b.Entry_Date
} equals new
{
Id = m.EmployeeId,
Entry_Date = m.ShiftDate
} into g
from m2 in g.Where(x => x.ShiftDate >= fromDate).Where(x => x.ShiftDate <= toDate).Take(1)
select m2
).ToList();
foreach (var shiftDetails in query)
{
//Other Calculations
}

Related

Check if parameter value is null or not inside query

In my query I am getting records based on RoleId and LocationId, some times the user may not pass location in that case I want to remove that filter and get information from all locations.
Currently I am doing this way
if(loc > 0)
{
var myResult = (from x in CSDB.Allocations
join s in CSDB.Managers
on x.ManagerId equals s.Id
Where x.RoleId == 2 && s.LocationId == loc
select new
{
x.name,
x.Date
}).ToList();
}
else
{
var myResult = (from x in CSDB.Allocations
join s in CSDB.Managers
on x.ManagerId equals s.Id
Where x.RoleId == 2
select new
{
x.name,
x.Date
}).ToList();
}
I am seeing if I can check if loc is null or not inside the query instead of using if else.
You can do something like this:
Where x.RoleId == 2 && (loc == null || s.LocationId == loc)
Also, you can do smth like this.
Where x.RoleId == 2 && (loc?.Equals(s.LocationId) ?? true)
If loc just int I would prefer to use a little bit changed #Salah Akbari answer:
Where x.RoleId == 2 && (loc == 0 || s.LocationId == loc)
Simply extract your managers and filter them if needed. That way you can as well easily apply more filters and code readability isn't hurt.
var managers = CSDB.Managers.AsQueryable();
if(loc > 0)
managers = managers.Where(man => man.LocationId == loc);
var myResult = from allocation in CSDB.Allocations
join manager in managers on allocation.ManagerId equals manager.Id
where allocation.RoleId == 2
select new
{
allocation.name,
allocation.Date
};

Should I apply counting in DB or in Code?

I have the following code where I'm trying to get the amount of rows in the same dataset with various matches.
My question is if should I get the count in C# code with a IEnumerable or by querying a IQueryable from database?
Which one is more efficient, multiple database transactions or IEnumerable filtering and count?
public List<Tuple<string, int>> CalismaVeIzinleriHesapla(long personelId, DateTime baslangic, DateTime bitis)
{
var hesaplamalar = new List<Tuple<string, int>>();
var puantajList = puantajlar.Where(p => p.PersonelId == personelId && (p.Tarih >= baslangic && p.Tarih <= bitis));
var haftaTatili = puantajList.Where(p => p.Secenek.Deger == "Ht").Count();
var resmiTatil = puantajList.Where(p => p.Secenek.Deger == "Rt").Count();
var yillikIzin = puantajList.Where(p => p.Secenek.Deger == "Yi").Count();
var odenecekRapor = puantajList.Where(p => p.Secenek.Deger == "R+").Count();
var dogumIzni = puantajList.Where(p => p.Secenek.Deger == "Di").Count();
var olumIzni = puantajList.Where(p => p.Secenek.Deger == "Öi").Count();
var evlilikIzni = puantajList.Where(p => p.Secenek.Deger == "Ei").Count();
var odenmeyecekRapor = puantajList.Where(p => p.Secenek.Deger == "R-").Count();
var ucretsizIzin = puantajList.Where(p => p.Secenek.Deger == "Üi").Count();
var devamsizlik = puantajList.Where(p => p.Secenek.Deger == "D").Count();
return hesaplamalar;
}
As for your case, querying and counting in the db is more efficient.
something like this would be efficient.
puantajlar
.Where(p => p.PersonelId == personelId && (p.Tarih >= baslangic && p.Tarih <= bitis))
.GroupBy(x => x.Secenek.Deger)
.Select(group => new { group.Key, Count = group.Count() })
My question is if should I get the count in C# code with a IEnumarable or by querying a IQueryable in DB
If you need only the count of the rows then count must be done in database, not in memory. If you do count in memory by pulling the data list from database into memory then it will waste your server memory unnecessarily and cost performance.
Complexity and Performance, both of them depends on your situation, if there are no huge data the Performance is no matter, but sometime you have to make a decision based on your situation.
By running your code it should connect to DB and run the count query in each line of code
it is 100 per cent clear that counting same rows in DB is more efficient in one shoot so you can do something like :
select p.Secenek.Deger,
....
sum(case when p.Secenek.Deger = 'Ht' then 1 else 0 end) haftaTatili,
sum(case when p.Secenek.Deger = 'Rt' then 1 else 0 end) resmiTatil
.....
from puantajlar p
group by p.Secenek.Deger
or you can do it in more efficient way by grouping them in one shoot also like #amd mentioned:
puantajlar
.Where(p => p.PersonelId == personelId && (p.Tarih >= baslangic && p.Tarih <= bitis))
.GroupBy(x => x.Secenek.Deger)
.Select(group => new { group.Key, Count = group.Count() })

Dynamically Building LINQ-To-Entities Where Clause

How can I build the where clause dynamically, Some time the OwnerID will be zero only itemID and LocationIDwill be provided as the search criteria, in that case the LINQ should be
(from s in repository.ItemOwners.Include("OwnerDetails")
where s.ItemId == searchCriteria.ItemID &&
s.OwnerDetails.LocationId == searchCriteria.LocationID
select new { s.OwnerDetails.OwnerName, s.OwnerDetails.MobNumber }).ToList();
Some time the OwnerID and ItemId will be zero then only the LocationID will be provided as the search criteria, in that case the LINQ should be
(from s in repository.ItemOwners.Include("OwnerDetails")
where s.OwnerDetails.LocationId == searchCriteria.LocationID
select new { s.OwnerDetails.OwnerName, s.OwnerDetails.MobNumber }).ToList();
Some time the whole OwnerID, ItemID and LocationID will be provided as the search criteria, then the LINQ will be like this
(from s in repository.ItemOwners.Include("OwnerDetails")
where s.OwnerId == searchCriteria.OwnerID &&
s.ItemId == searchCriteria.ItemID &&
s.OwnerDetails.LocationId == searchCriteria.LocationID
select new { s.OwnerDetails.OwnerName, s.OwnerDetails.MobNumber }).ToList();
Here only the where clause is changing, Please help me to solve. How I can I build the where clause dynamically (Please note, here I am having a navigation property which is OwnerDetails.LocationId).
You can easily do it by using method-based query. You can add the conditions one at a time and call Select and ToList at the end:
// Where(x => true) might not be necessary, you can try skipping it.
var query = repository.ItemOwners.Include("OwnerDetails").Where(x => true);
if (searchCriteria.OwnerID != null)
query = query.Where(s => s.OwnerID == searchCriteria.OwnerID);
if (searchCriteria.ItemID != null)
query = query.Where(s => s.ItemID == searchCriteria.ItemID);
if (searchCriteria.OwnerID != null)
query = query.Where(s => s..OwnerDetails.LocationId == searchCriteria.LocationID);
var results = query.Select(s => new { s.OwnerDetails.OwnerName, s.OwnerDetails.MobNumber }).ToList();
Simplest is just check the zero condition in the Where clause:
(from s in repository.ItemOwners.Include("OwnerDetails")
where (searchCriteria.OwnerID == 0 || s.OwnerId == searchCriteria.OwnerID) &&
(searchCriteria.ItemID == 0 || s.ItemId == searchCriteria.ItemID) &&
s.OwnerDetails.LocationId == searchCriteria.LocationID
select new { s.OwnerDetails.OwnerName, s.OwnerDetails.MobNumber }).ToList();

Select two columns from the table via linq

I use the query below to get all columns(20 more) in Entity Framework Linq. Because of out of memory exception, I only want to get two of them. One is "FileName", the other one is "FilePath". How to modify my code?
var query = DBContext.Table1
.Where(c => c.FacilityID == facilityID && c.FilePath != null && c.TimeStationOffHook < oldDate)
.OrderBy(c => c.FilePath)
.Skip(1000)
.Take(1000)
.ToList();
foreach(var t in query)
{
Console.WriteLine(t.FilePath +"\\"+t.FileName);
}
var query = DBContext.Table1.Where(c => c.FacilityID == facilityID && c.FilePath != null && c.TimeStationOffHook < oldDate)
.OrderBy(c => c.FilePath)
.Skip(1000)
.Take(1000)
.Select(c => new { c.FilePath, c.FileName })
.ToList();
foreach(var t in query)
{
Console.WriteLine(t.FilePath +"\\"+t.FileName);
}
You need to use Select.
Just select out two of the columns:
DBContext.Table1.Select(c => new { c.FileName, c.FilePath });
How about something like
using (var entity = new MyModel(ConnectionString))
{
var query = (from myTable in entity.theTable
where myTable.FacilityID == facilityID &&
myTable.FilePath != null &&
myTable.TimeStationOffHook < oldDate
orderby myTable.FilePath
select new
{
myTable,FileName,
myTable.FilePath
}).Skip(1000).Take(1000).ToList();
//do what you want with the query result here
}

How to apply a filter in a LINQ to SQL expression only if results exist when the filter is applied?

I have a function I'd like to transform into a LINQ to SQL expression, but I can't figure out how. This function is called from within a LINQ query, once for each row in the result set. The productAreaId being passed in may or may not refer to valid data, so I have to check, and then only filter by productAreaId if any rows exist after applying the filter:
//Forgive the contrived example...
public static IQueryable<Order> GetOrders(int orderNumber, int? productAreaId,
OSDataContext db)
{
var orders = db.Orders.Where(o => o.OrderNumber == orderNumber &&
o.Group.GroupTypeId != (int)GroupTypeId.INTERNAL &&
!o.Deleted);
if (productAreaId != null)
{
var orders2 = orders.Where(o => o.ProductAreaId == productAreaId);
if (orders2.Any()) return orders2;
}
return orders;
}
I don't want to do it like this. I need a function that returns an expression without arbitrary code, so it will be composable. The above function is only displayed because it's the only way I know how to encapsulate this in a function.
I'd like to do something like this, with the contrived "ApplyFilterIfAnyResultExists" replaced with something that actually works:
public static Expression<Func<Order,bool>>
GetOrdersExpr(int orderNumber, int? productAreaId)
{
return o => o.OrderNumber == orderNumber &&
o.Group.GroupTypeId != (int)GroupTypeId.INTERNAL &&
!o.Deleted && (productAreaId == null ||
//Making up a fictional function. Is there a real way to do this?
o.ApplyFilterIfAnyResultExists(row =>
row.ProductAreaId == productAreaId)
);
}
Is there any way to apply that kind of filter within a LINQ to SQL expression? If not, any suggestions?
Thank you!Roy
EDIT:
This is the main query as I'd like it to look:
var customerData =
from c in db.Customers
select new
{
id = c.Id,
name = c.Name,
lastOrder =
db.Orders
.Where(GetOrdersExpr(c.LastOrderNumber,
c.PreferredProductAreaId))
.FirstOrDefault(),
allOrders = c.OrderForms
.Select(form =>
db.Orders
.Where(GetOrdersExpr(form.OrderNumber,
c.PreferredProductAreaId))
.FirstOrDefault()
)
.Where(o => o != null)
//How lastOrder used to be queried
//lastOrder =
// GetOrders(c.LastOrderNumber, c.PreferredProductAreaId, db)
// .FirstOrDefault()
};
It's also worth noting that Orders and Customers are in two different databases on the database server, but they're both referenced from the same DataContext here.
Maybe something like this:
var customerData =
from c in db.Customers
let orders = db.Orders.Where(o => o.OrderNumber == c.orderNumber &&
o.Group.GroupTypeId != (int)GroupTypeId.INTERNAL &&
!o.Deleted)
let orders2 = orders.Where(o => o.ProductAreaId == c.productAreaId)
select new
{
id = c.Id,
name = c.Name,
lastOrder = c.productAreaId != null && orders2.Any() ?
orders2.FirstOrDefault() :
orders.FirstOrDefault()
};
For your original method, this might work better:
public static IQueryable<Order> GetOrders(int orderNumber, int? productAreaId,
OSDataContext db)
{
var orders = db.Orders.Where(o => o.OrderNumber == orderNumber &&
o.Group.GroupTypeId != (int)GroupTypeId.INTERNAL &&
!o.Deleted);
if(productAreaId != null)
{
orders = orders.Where(
o => !orders.Any(o2 => o2.ProductAreaId == productAreaId) ||
o.ProductAreaId == productAreaId);
}
return orders;
}
This makes it so you're only doing a single database roundtrip. If the product area ID is provided, you will return orders where either:
none of the orders in the original query have that area ID, or
this order does have that area ID
It does make the query more complex, so I'd test it a bit to see if it really gives you any performance gains.
This won't translate very well to the function that you're suggesting, but if you share more information about how this code is getting called, I could probably give you some advice on how to avoid calling this function 20+ times.
Edit
Something like this should work:
var customerData =
from c in db.Customers
let productAreaId = c.PreferredProductAreaId
let orders =
db.Orders
.Where(o => o.OrderNumber == c.LastOrderNumber &&
o.Group.GroupTypeId != (int)GroupTypeId.INTERNAL &&
!o.Deleted)
.OrderBy(o => o.Date)
let lastOrderInProductArea = productAreaId != null
? orders.FirstOrDefault(o => o.ProductAreaId == productAreaId)
: null
select new
{
id = c.Id,
name = c.Name,
lastOrder = lastOrderInProductArea != null
? lastOrderInProductArea
: orders.FirstOrDefault()
};
public static IQueryable<Order> GetOrders(int orderNumber, int? productAreaId, OSDataContext db)
{
var orders = db.Orders.Where(o => o.OrderNumber == orderNumber &&
o.Group.GroupTypeId != (int)GroupTypeId.INTERNAL &&
!o.Deleted);
if (productAreaId != null)
{
var orders2 = orders.Where(o => o.ProductAreaId == productAreaId);
return orders2.Select(x => new {x, Type = 2 }).Concat(orders.Select(x => new {x, Type = 1 })).OrderBy(x => x.Type);
}
return orders;
}
This will return both results concatenated. First the results from orders2 then from orders2. This might help you.
If you really only want orders from one group you can do
GetOrders().Where(x => x.Type == GetOrders().Max(x => x.Type))
in order to restrict the query to the highest-priority orders. This will have suboptimal performance.

Categories