I have a foreach that goes through response, response is a list that can take up to 230,000 records, foreach response it will filter and sum a second list called _MaterialIssued, this list contains max 30,000 records, in the following if it will only enter if Issued is bigger than 0 which will occur only about 15% of the time, next it will try to get the itemOnhand from the _Onhand list that contains max 17,000 records, in the following if it will go in about 85% of the time, when I wrote the code inside this block is where the performance dropped dramatically, in this if I will go back and filter response for all the child items and loop through them changing the _onhand list and the response list!
To go through this foreach depends on the machine I use but takes from 45 to 75 minutes and I cannot find what line is my bottleneck or how I can improve performance for this code block.
foreach (var b in response)
{
var Issued = _MaterialIssued.Where(x => x.ItemId == b.ItemId && x.Job == b.Job).Sum(c => c.Qty);
if (Issued > 0)
{
var prctIssued = Issued / b.QtyDemand;
var childItems = response.Where(x => x.Job == b.Job && x.BomPath.StartsWith(b.BomPath));
foreach (var child in childItems)
{
child.QtyIssued = child.QtyDemand * prctIssued;
}
}
var itemOnhand = _OnHand.Where(x => x.ItemId == b.ItemId).FirstOrDefault();
if (itemOnhand.Onhand > 0)
{
decimal prctOnhand = 1;
var childItems = response.Where(x => x.Job == b.Job && x.BomPath.StartsWith(b.BomPath) && x.ItemId != b.ItemId && x.SiteRef == b.SiteRef && x.QtyIssued < x.QtyDemand);
var DemandWithIssued = b.QtyDemand - b.QtyIssued;
if (itemOnhand.Onhand < DemandWithIssued)
{
prctOnhand = itemOnhand.Onhand / DemandWithIssued;
}
itemOnhand.Onhand -= DemandWithIssued * prctOnhand;
foreach (var child in childItems)
{
child.QtyParentAvailable = (child.QtyDemand - child.QtyIssued) * prctOnhand;
}
}
}
The model for _OnHand is
private class ItemOnhand
{
public string ItemId { get; set; }
public string SiteRef { get; set; }
public decimal Onhand { get; set; }
}
The model for _MaterialIssued is
public class FiniteDemandBase
{
public string ItemId { get; set; }
public string Item { get; set; }
public string JobId { get; set; }
public string Job { get; set; }
public DateTime StartDate { get; set; }
public string SiteRef { get; set; }
public decimal Qty { get; set; }
}
The model for response is
public class FiniteDemand
{
public int Id { get; set; }
public DateTime StartDate { get; set; }
public string BomPath { get; set; }
public string SiteRef { get; set; }
public string Job { get; set; }
public string ItemId { get; set; }
public string JobId { get; set; }
public string ParentItemId { get; set; }
public decimal QtyPerUnit { get; set; }
public decimal QtyDemand { get; set; }
public decimal QtyOnhand { get; set; }
public decimal QtyWip { get; set; }
public decimal QtyOnhandRunning { get; set; }
public decimal QtyInSchedule { get; set; }
public decimal QtyIssued { get; set; }
public decimal QtyParentAvailable { get; set; }
public decimal QtyDemandNeeded { get; set; }
public decimal QtyDemandNeededRunning { get; set; }
}
I tried changing the _OnHand list to a HashSet but the performance was the same.
Let's break down your code into parts which contain an explicit of implicit iteration:
Iteration over response — n-times:
foreach (var b in response)
{
Nested iteration over _MaterialIssued — m-times:
var Issued = _MaterialIssued.Where(x => x.ItemId == b.ItemId && x.Job == b.Job).Sum(c => c.Qty);
Nested iteration over response (again) — n-times:
var childItems = response.Where(x => x.Job == b.Job && x.BomPath.StartsWith(b.BomPath));
Nested iteration over _OnHand — up to h-times:
var itemOnhand = _OnHand.Where(x => x.ItemId == b.ItemId).FirstOrDefault();
Nested iteration over response (again) — n-times:
var childItems = response.Where(x => x.Job == b.Job && x.BomPath.StartsWith(b.BomPath) && x.ItemId != b.ItemId && x.SiteRef == b.SiteRef && x.QtyIssued < x.QtyDemand);
Hence, the complexity of your algorithm is: O(n * (m + n + h + n)) = O(2*n^2 + n*(m+h))!
If the number of responses is 230,000, we are speaking about 52.9 billion iterations! No wonder it runs for ages :)
This sounds an awful lot like you're trying to manage a database. I would suggest you go all the way and do exactly that, and look into using something like MySQL instead.
E.g.:
Create a database and add tables in Visual Studio
Create a SQL Server Database programmatically by using ADO.NET and Visual C# .NET
Whilst, technically, a List can contain up to two billion elements, once you're above a few thousand you'll start to run into trouble. They aren't really meant for the sort of heavy data processing you're attempting. That is exactly what database engines are for.
Related
*I have written a recursive query to get unlimited menu layer. The query works fine providing the exact results but it takes too much time to load. It takes probably 10 to 15 seconds. Please help me if I need to do anything to improve the performance. I have provided all the code to find out the problem. for mapping from entity to view model I have used automapper. *
Entity:
public class Menus
{
public int Id { get; set; }
public string Icon { get; set; }
public string Label { get; set; }
public string To { get; set; }
[ForeignKey("Parents")]
public int? ParentsId { get; set; }
public string Key { get; set; }
public bool? Permitted { get; set; }
public Menus Parents { get; set; }
public ICollection<Menus> Subs { get; set; }
public ICollection<MenusRole> MenusRoles { get; set; }
}
Query:
public async Task<IEnumerable<Menus>> GetAllMenusAsync()
{
List<Menus> temp = await ApplicationDbContext
.Menus
.Include(x => x.Subs)
.Where(x => x.Parents == null)
.Select(f => new Menus
{
Id = f.Id,
Key = f.Key,
Label = f.Label,
To = f.To,
Icon = f.Icon,
ParentsId = f.ParentsId,
Subs = f.Subs
}).ToListAsync();
return Get_all_menus(temp);
}
public List<Menus> Get_all_menus(List<Menus> menus)
{
int z = 0;
List<Menus> menuList = new List<Menus>();
if (menus.Count > 0)
{
menuList.AddRange(menus);
}
foreach (Menus item in menus)
{
Menus menu = ApplicationDbContext
.Menus
.Include(y => y.Subs)
.Where(y => y.Id == item.Id)
.Select(y => new Menus
{
Id = y.Id,
Key = y.Key,
Label = y.Label,
To = y.To,
Icon = y.Icon,
ParentsId = y.ParentsId,
Subs = y.Subs,
Permitted = true
}).First();
if (menu.Subs == null)
{
z++;
continue;
}
List<Menus> subMenu = menu.Subs.ToList();
menu.Subs = Get_all_menus(subMenu);
menuList[z] = menu;
z++;
}
return menuList;
}
In Controller
[HttpGet("get-all-menus")]
public async Task<ActionResult> GetAllMenus()
{
var menus = await _menusService.GetAllMenus();
var menusResources = _mapper.Map<IEnumerable<Menus>, IEnumerable<MenusResourceForSidebar>>(menus);
return Ok(menusResources);
}
View Model
public string Id { get; set; }
public string Icon { get; set; }
public string Label { get; set; }
public string To { get; set; }
public bool? Permitted { get; set; }
public ICollection<MenusResourceForSidebar> Subs { get; set; }
Instead of loading the root menus, then loading the children in separate queries, just load the whole collection in one query, and then populate the navigation links afterwards.
public async Task<IEnumerable<Menus>> GetAllMenusAsync()
{
List<Menus> temp = await ApplicationDbContext.Menus.ToList();
List<Menus> topLevel = new List<Menu>();
foreach (var menu in temp)
{
if (menu.ParentsId == null)
{
topLevel.Add(menu);
continue;
}
var parent = temp.Find(x => x.Id == temp.ParentsId.Value);
if (parent.Subs == null)
parent.Subs = new List<Menus>();
parent.Subs.Add(menu);
}
return topLevel;
}
You should just be able to do:
context.Menus.Include(m => m.Subs).ToList();
The relationship fixup in EFCore will link all the menus together in a tree for you. In later EFs you don't even need the Include..
context.Menus.ToList();
Here is a table in SSMS:
Here is the data:
Here it is chopped up in a paint program and rearranged into a tree:
Here's the scaffolded entity:
// <auto-generated> This file has been auto generated by EF Core Power Tools. </auto-generated>
#nullable disable
using System.Collections.Generic;
namespace ConsoleApp7net5.Models
{
public partial class Menu
{
public Menu()
{
InverseParent = new HashSet<Menu>();
}
public int Id { get; set; }
public int? ParentId { get; set; }
public string Label { get; set; }
public virtual Menu Parent { get; set; }
public virtual ICollection<Menu> InverseParent { get; set; }
}
}
Here's what we see after asking EFC (5, in my case) to download it all with just a ToList:
Of course it might make sense to start with a root (or multiple roots but my data only has one)
Don't give classes plural names (Menus), btw, and don't give properties plural names if they aren't collections/enumerables (Parents) - it makes for very confusing code
I have those two entities:
public class Ticket
{
public int Id { get; set; }
public int ScheduleId { get; set; }
public int SeatId { get; set; }
[DataType(DataType.Date)]
[Column(TypeName = "Date")]
[DisplayFormat(DataFormatString = "{0:yyyy-MM-dd}", ApplyFormatInEditMode = true)]
public DateTime ForDate { get; set; }
public Seat Seat { get; set; }
public Schedule Schedule { get; set; }
public ICollection<TicketStop> TicketStops { get; set; }
}
public class TicketStop
{
public int Id { get; set; }
public int TicketId { get; set; }
public int LineStopStationId { get; set; }
public Ticket Ticket { get; set; }
public LineStopStation LineStopStation { get; set; }
}
public class LineStopStation
{
public int Id { get; set; }
public int LineId { get; set; }
public int StopId { get; set; }
public int Order { get; set; }
public bool IsLastStop { get; set; }
public Line Line { get; set; }
public Stop Stop { get; set; }
}
The business case is that I am implementing a bus ticket reservation system (for learning purposes mostly) and I want to find overlapping tickets.
The combination of LineId + ScheduleId + ForDate identifies uniquely that that a ticket is for certain bus at a certain date and departure time.
The problem I have is to identify, given starting and end location whether or not two tickets overlap for one or more stops.
The LineStopStation entity holds information about the StopId and the Order in which it is visited during the bus trip. So basically, overlapping tickets will have same Order number at some point (unless it is the last stop, which means that the passenger is leaving the bus).
So what I have is LineId, ScheduleId, StartId and EndId where starId and endId correspond to LineStopStation.StopId so eventually I am able to get the Order out of them like this.
int startStationOrder = _context.LineStopStations
.First(l => l.LineId == lineId && l.StopId == startId).Order;
int endStationOrder = _context.LineStopStations
.First(l => l.LineId == lineId && l.StopId == endId).Order;
So I am pretty convinced that having all this information I should be able to find if in TicketStop table I have ticket which overlaps with the data in question. TicketStop works this way - if someone bought a ticket for 3 stops I will have three records there with the same TicketId and three different LineStopStationId.
I feel that this question got a bigger than needed. So basically I have this:
public List<Seat> GetAvailableSeats(int lineId, int scheduleId, int startId, int endId, DateTime forDate)
{
int startStationOrder = _context.LineStopStations
.First(l => l.LineId == lineId && l.StopId == startId).Order;
int endStationOrder = _context.LineStopStations
.First(l => l.LineId == lineId && l.StopId == endId).Order;
var reservedSeats = _context.TicketStops
.Where(t => t.Ticket.ScheduleId == scheduleId)
.Where(t => t.Ticket.ForDate == forDate)
//basically I don't know how to proceed here.
//In pseudo code it should be something like:
.Where(t => t.Min(Order) >= endStationOrder || t.Max(Order) <= startStationOrder
}
But obs. this is not how LINQ works. So how can I find all tickets where this range overlaps?
Without an in-depth analysis of your model, perhaps something like this could give you an idea?
var reservedSeats = _context.TicketStops
.GroupBy(t => new { t.Ticket.ScheduleId, t.Ticket.ForDate })
.Where(tg => tg.Key == new { ScheduleId = scheduleId, ForDate = forDate })
.Where(tg => tg.Min(t => t.LineStopStation.Order) >= endStationOrder || tg.Max(t => t.LineStopStation.Order) <= startStationOrder);
You could also filter first and do an empty GroupBy:
var reservedSeats = _context.TicketStops
.Where(t => t.Ticket.ScheduleId == scheduleId && t.Ticket.ForDate == forDate)
.GroupBy(t => 1)
.Where(tg => tg.Min(t => t.LineStopStation.Order) >= endStationOrder || tg.Max(t => t.LineStopStation.Order) <= startStationOrder);
To return all the SeatIds, you just need to select them from the group.
var reservedSeats = _context.TicketStops
.Where(t => t.Ticket.ScheduleId == scheduleId && t.Ticket.ForDate == forDate)
.GroupBy(t => 1)
.Where(tg => tg.Min(t => t.LineStopStation.Order) >= endStationOrder || tg.Max(t => t.LineStopStation.Order) <= startStationOrder);
.SelectMany(tg => tg.Select(t => t.Ticket.SeatId));
Multiple answers have led me to the following 2 solutions, but both of them do not seem to be working correctly.
What I have are 2 objects
public class DatabaseAssignment : AuditableEntity
{
public Guid Id { get; set; }
public string User_Id { get; set; }
public Guid Database_Id { get; set; }
}
public class Database : AuditableEntity
{
public Guid Id { get; set; }
public string Name { get; set; }
public string Server { get; set; }
public bool IsActive { get; set; }
public Guid DatabaseClientId { get; set; }
}
Now, the front-end will return all selected Database objects (as IEnumerable) for a given user. I am grabbing all current DatabaseAssignments from the database for the given user and comparing them to the databases by the Database.ID property. My goal is to find the DatabaseAssignments that I can remove from the database. However, my solutions keep returning all DatabaseAssignments to be removed.
if (databases != null)
{
var unitOfWork = new UnitOfWork(_context);
var userDatabaseAssignments = unitOfWork.DatabaseAssignments.GetAll().Where(d => d.User_Id == user.Id);
//var assignmentsToRemove = userDatabaseAssignments.Where(ud => databases.Any(d => d.Id != ud.Database_Id));
var assignmentsToRemove = userDatabaseAssignments.Select(ud => userDatabaseAssignments.FirstOrDefault()).Where(d1 => databases.All(d2 => d2.Id != d1.Database_Id));
var assignmentsToAdd = databases.Select(d => new DatabaseAssignment { User_Id = user.Id, Database_Id = d.Id }).Where(ar => assignmentsToRemove.All(a => a.Database_Id != ar.Database_Id));
if (assignmentsToRemove.Any())
{
unitOfWork.DatabaseAssignments.RemoveRange(assignmentsToRemove);
}
if (assignmentsToAdd.Any())
{
unitOfWork.DatabaseAssignments.AddRange(assignmentsToAdd);
}
unitOfWork.SaveChanges();
}
I think u are looking for an Except extension, have a look at this link
LINQ: Select where object does not contain items from list
Or other way is with contains see below Fiddler link :
https://dotnetfiddle.net/lKyI2F
Im having a problem here with performance.
I got to List contains (50k items)
and List contains (120k items)
WholeSaleEntry is
public class WholeSaleEntry
{
public string SKU { get; set; }
public string Stock { get; set; }
public string Price { get; set; }
public string EAN { get; set; }
}
and ProductList
public class ProductList
{
public string SKU { get; set; }
public string Price { get; set; }
public string FinalPrice { get; set; }
public string AlternateID { get; set; }
}
I need to filter WholeSaleEntry By its EAN and Its SKU in case that their EAN or SKU is in the ProductList.AlternateID
I wrote this code which works but the performance is really slow
List<WholeSaleEntry> filterWholeSale(List<WholeSaleEntry> wholeSaleEntry, List<ProductList> productList)
{
List<WholeSaleEntry> list = new List<WholeSaleEntry>();
foreach (WholeSaleEntry item in wholeSaleEntry)
{
try
{
string productSku = item.SKU;
string productEan = item.EAN;
var filteredCollection = productList.Where(itemx => (itemx.AlternateID == productEan) || (itemx.AlternateID == productSku)).ToList();
if (filteredCollection.Count > 0)
{
list.Add(item);
}
}
catch (Exception)
{
}
}
return list;
}
Is there any better filtering system or something that can filter it in bulk?
The use of .Where(...).ToList() will find every matching item, and in the end you only need to know if there is a matching item. That can be fixed using Any(...) which stops as soon as a match is found, like this:
var hasAny = productList.Any(itemx => itemx.AlternateID == productEan || itemx.AlternateID == productSku);
if (hasAny)
{
list.Add(item);
}
Update: the algorithm can be simplified to this. First get the unique alternate IDs using a HashSet, which stores duplicate items just once, and which is crazy fast for lookup. Then get all WholeSale items that match with that.
There will not be many faster strategies, and the code is small and I think easy to understand.
var uniqueAlternateIDs = new HashSet<string>(productList.Select(w => w.AlternateID));
var list = wholeSaleEntry
.Where(w => uniqueAlternateIDs.Contains(w.SKU)
|| uniqueAlternateIDs.Contains(w.EAN))
.ToList();
return list;
A quick test revealed that for 50k + 50k items, using HashSet<string> took 28ms to get the answer. Using a List<string> filled using .Distinct().ToList() took 48 seconds, with all other code the same.
If you do not want a specific method and avoid the list just do this.
var filteredWholeSale = wholeSaleEntry.Where(x => productList.Any(itemx => itemx.AlternateID == x.EAN || itemx.AlternateID == x.SKU));
The following code is throwing an InvalidOperationException: Sequence contains more than one element. The errors occurs whether I have the Include statement or not.
long[] customerIDs; //Method parameter. Has valid values
var results = from x in DB.CycleCounts
//.Include(y => y.CustomerInventoryItem)
select x;
if (customerIDs != null && customerIDs.Length > 0)
{
results = from x in results
where customerIDs.Contains(x.CustomerInventoryItem.CustomerID)
select x;
}
var cycleCounts = await results.ToListAsync(); //throws InvalidOperationException
I am using ASP5 RC1 (Core), and Entity Framework 7
I not sure how look your models, but I think that they looks like something below (relationship between CycleCounts and CustomerInventoryItems as many-to-one as I expect):
Models:
[Table("CustomerInventorys")]
public class CustomerInventory
{
public long ID { get; set; }
public long CustomerID { get; set; }
public virtual ICollection<CycleCount> CycleCounts { get; set; }
}
[Table("CycleCounts")]
public class CycleCount
{
public long ID { get; set; }
public long CustomerInventoryItemID { get; set; }
public virtual CustomerInventory CustomerInventoryItem { get; set; }
}
So if my suggestions are correct I recommend you to rewrite you code like this:
IQueryable<CycleCount> results = null;
if (customerIDs != null && customerIDs.Length > 0)
{
results = from invent in DB.CustomerInventorys
where customerIDs.Contains(invent.CustomerID)
join cycle in DB.CycleCounts on invent.ID equals cycle.CustomerInventoryItemID
select cycle;
}
else
results = DB.CycleCounts;
var cycleCounts = await results.ToListAsync();