We Keep Coding

EF Core 5 check if all ids from filter exists in related entities

I have two models:
public class Employee
{
public int Id { get; set; }
public IList<Skill> { get; set; }
}
public class Skill
{
public int Id { get; set; }
}
And I have filter with list of skill ids, that employee should contain:
public class Filter
{
public IList<int> SkillIds { get; set; }
}
I want to write query to get all employees, that have all skills from filter.
I tried:
query.Where(e => filter.SkillIds.All(id => e.Skills.Any(skill => skill.Id == id)));
And:
query = query.Where(e => e.Skills
.Select(x => x.Id)
.Intersect(filter.SkillIds)
.Count() == filter.SkillIds.Count);
But as a result I get exception says that query could not be translated.

It is going to be a difficult, if not impossible task, to run a query like this on the sql server side.
This is because to make this work on the SQL side, you would be grouping each set of employee skills into a single row which would need to have a new column for every skill listed in the skills table.
SQL server wasn't really made to handle grouping with an unknown set of columns passed into a query. Although this kind of query is technically possible, it's probably not very easy to do through a model binding framework like ef core.
It would be easier to do this on the .net side using something like:
var employees = _context.Employees.Include(x=>x.Skill).ToList();
var filter = someFilter;
var result = employees.Where(emp => filter.All(skillID=> emp.skills.Any(skill=>skill.ID == skillID))).ToList()

This solution works:
foreach (int skillId in filter.SkillIds)
{
query = query.Where(e => e.Skills.Any(skill => skill.Id == skillId));
}
I am not sure about it's perfomance, but works pretty fast with small amount of data.

I've also encountered this issue several times now, this is the query I've come up with that I found works best and does not result in an exception.
query.Where(e => e.Skills.Where(s => filter.SkillIds.Contains(s.Id)).Count() == filter.SkillIds.Count);

How to query IQueryable with Include - ThenInclude?

In .NET Core 2.2 I'm stuck with filtering IQueryable built as:
_context.Ports.Include(p => p.VesselsPorts)
.ThenInclude(p => p.Arrival)
.Include(p => p.VesselsPorts)
.ThenInclude(p => p.Departure)
.OrderBy(p => p.PortLocode);
in many-to-many relation. And the entity models are such as:
public class PortModel
{
[Key]
public string PortLocode { get; set; }
public double? MaxKnownLOA { get; set; }
public double? MaxKnownBreadth { get; set; }
public double? MaxKnownDraught { get; set; }
public virtual ICollection<VesselPort> VesselsPorts { get; set; }
}
public class VesselPort
{
public int IMO { get; set; }
public string PortLocode { get; set; }
public DateTime? Departure { get; set; }
public DateTime? Arrival { get; set; }
public VesselModel VesselModel { get; set; }
public PortModel PortModel { get; set; }
}
Based on this this SO answer I managed to create LINQ like that:
_context.Ports.Include(p => p.VesselsPorts).ThenInclude(p => p.Arrival).OrderBy(p => p.PortLocode)
.Select(
p => new PortModel
{
PortLocode = p.PortLocode,
MaxKnownBreadth = p.MaxKnownBreadth,
MaxKnownDraught = p.MaxKnownDraught,
MaxKnownLOA = p.MaxKnownLOA,
VesselsPorts = p.VesselsPorts.Select(vp => vp.Arrival > DateTime.UtcNow.AddDays(-1)) as ICollection<VesselPort>
}).AsQueryable();
BUT what I need is to find all port records, where:
VesselsPorts.Arrival > DateTime.UtcNow.AddDays(-1) quantity is greater than int x = 5 value (for the example). And I have no clue how to do it :/

Thanks to #GertArnold comment, I ended up with query:
ports = ports.Where(p => p.VesselsPorts.Where(vp => vp.Arrival > DateTime.UtcNow.AddDays(-1)).Count() > x);

When using entity framework people tend to use Include instead of Select to save them some typing. It is seldom wise to do so.
The DbContext holds a ChangeTracker. Every complete row from any table that you fetch during the lifetime of the DbContext is stored in the ChangeTracker, as well as a clone. You get a reference to the copy. (or maybe a reference to the original). If you change properties of the data you got, they are changed in the copy that is in the ChangeTracker. During SaveChanges, the original is compared to the copy, to see if the data must be saved.
So if you are fetching quite a lot of data, and use include, then every fetched items is cloned. This might slow down your queries considerably.
Apart from this cloning, you will probably fetch more properties than you actually plan to use. Database management systems are extremely optimized in combining tables, and searching rows within tables. One of the slower parts is the transfer of the selected data to your local process.
For example, if you have a database with Schools and Students, with the obvious one to many-relation, then every Student will have a foreign key to the School he attends.
So if you ask for School [10] with his 2000 Students, then every Student will have a foreign key value of [10]. If you use Include, then you will be transferring this same value 10 over 2000 times. What a waste of processing power!
In entity framework, when querying data, always use Select to select the properties, and Select only the properties that you actually plan to use. Only use Include if you plan to change the fetched items.
Certainly don't use Include to save you some typing!
Requirement: Give me the Ports with their Vessels
var portsWithTheirVessels = dbContext.Ports
.Where(port => ...) // if you don't want all Ports
.Select(port => new
{
// only select the properties that you want:
PortLocode = port.PortLoCode,
MaxKnownLOA = port.MaxKnownLOA,
MaxKnownBreadth = prot.MaxKnownBreadth,
MaxKnownDraught = ports.MaxKnownDraught,
// The Vessels in this port:
Vessels = port.VesselsPort.Select(vessel => new
{
// again: only the properties that you plan to use
IMO = vessel.IMO,
...
// do not select the foreign key, you already know the value!
// PortLocode = vessle.PortLocode,
})
.ToList(),
});
Entity framework knows your one-to-many relation, and knows that if you use the virtual ICollection that it should do a (Group-)Join.
Some people prefer to do the Group-Join themselves, or they use a version of entity framework that does not support using the ICollection.
var portsWithTheirVessels = dbContext.Ports.GroupJoin(dbContext.VesselPorts,
port => port.PortLocode, // from every Port take the primary key
vessel => vessel.PortLocode, // from every Vessel take the foreign key to Port
// parameter resultSelector: take every Port with its zero or more Vessels to make one new
(port, vesselsInThisPort) => new
{
PortLocode = port.PortLoCode,
...
Vessels = vesselsInThisPort.Select(vessel => new
{
...
})
.ToList(),
});
Alternative:
var portsWithTheirVessels = dbContext.Ports.Select(port => new
{
PortLocode = port.PortLoCode,
...
Vessels = dbContext.VesselPorts.Where(vessel => vessel.PortLocode == port.PortLocode)
.Select(vessel => new
{
...
}
.ToList(),
});
Entity framework will translate this also to a GroupJoin.

Only primitive types do I need to cast?

I am trying to use anonymous types in Entity Framework, but I am getting an error about
Unable to create a constant value
MinQty and MaxQty are int so I don't know if I need to add to Convert.ToInt32?
Unable to create a constant value of type 'Anonymous type'. Only primitive types or enumeration types are supported in this context.
This builds a list object
var listOfLicense = (from l in db.License
select new
{
l.ProductId,
l.MinLicense,
l.MaxLicense
}).tolist();
This is the larger EF object where I am getting the error am I missing a casting?
var ShoppingCart = (from sc in db.ShoppingCarts
Select new model.Shoppingchart{
ShoppingCartId= sc.Id,
MinQty = (int)listOfLicense
.Where(mt => (int)mt.ProductId == sc.ProductId)
.Select(mt => (int)mt.MinLicense)
.Min(mt => mt.Value),
MaxQty = (int)listOfLicense
.Where(mt => (int)mt.ProductId == p.ProductId)
.Select(mt =>(int) mt.MaxQty)
.Max(mt => mt.Value)}.tolist();

This builds a list object
var listOfLicense = (from l in db.License
select new
{
l.ProductId,
l.MinLicense,
l.MaxLicense
})
The above example does not build a list of objects. It builds a query to return objects of that anonymous type.
This builds an in-memory list of objects of that type:
var listOfLicense = (from l in db.License
select new
{
l.ProductId,
l.MinLicense,
l.MaxLicense
}).ToList();
Using .ToList() here will execute the query and return a materialized list of the anonymous types. From there, your code may work as expected without the exception. However, this is effectively loading the 3 columns from all rows in your database table, which may be a problem as the system matures and rows are added.
The error you are getting isn't a casting issue, it is a translation issue. Because your initial query is still just an EF Query, (IQueryable) any further querying against it will need to conform to EF limitations. EF has to be able to translate what your expressions are trying to select back into SQL. In your case, what your real code is trying to do is breaking those rules.
Generally it is better to let EF work with the IQueryable rather than materializing an entire list to memory. Though to accomplish that we'd need to either see the real code, or a minimum reproducible example.
This code:
MinQty = (int)listOfLicense
.Where(mt => (int)mt.ParentProductId == p.ProductId)
.Select(mt => (int)mt.MinLicense)
.Min(mt => mt.Value),
... does not fit with the above anonymous type as there is no correlation between what mt.ParentProductId is in relation to the anonymous type. (p seems to be associated with that type, not mt so there looks to be a lot of Query code missing from your example.)
Edit: based on your updated example:
var ShoppingCart = (from sc in db.ShoppingCarts
Select new model.Shoppingchart{
ShoppingCartId= sc.Id,
MinQty = (int)listOfLicense
.Where(mt => (int)mt.ProductId == sc.ProductId)
.Select(mt => (int)mt.MinLicense)
.Min(mt => mt.Value),
MaxQty = (int)listOfLicense
.Where(mt => (int)mt.ProductId == p.ProductId)
.Select(mt =>(int) mt.MaxQty)
.Max(mt => mt.Value)}.ToList();
It may be possible to build something like this into a single query expression depending on the relationships between ShoppingCart, Product, and Licence. It almost looks like "Licence" really refers to a "Product" which contains a min and max quantity that you're interested in.
Assuming a structure like:
public class Product
{
[Key]
public int ProductId { get; set; }
public int MinQuantity { get; set; }
public int MaxQuantity { get; set; }
// ...
}
// Here lies a question on how your shopping cart to product relationship is mapped. I've laid out a many-to-many relationship using ShoppingCartItems
public class ShoppingCart
{
[Key]
public int ShoppingCartId { get; set; }
// ...
public virtual ICollection<ShoppingCartItem> ShoppingCartItems { get; set; } = new List<ShoppingCartItem>();
}
public class ShoppingCartItem
{
[Key, Column(0), ForeignKey("ShoppingCart")]
public int ShoppingCartId { get; set; }
public virtual ShoppingCart ShoppingCart{ get; set; }
[Key, Column(1), ForeignKey("Product")]
public int ProductId { get; set; }
public virtual Product Product { get; set; }
}
With something like this, to get shopping carts with their product min and max quantities:
var shoppingCarts = db.ShoppingCarts
.Select(sc => new model.ShoppingCart
{
ShoppingCartId = sc.Id,
Products = sc.ShoppingCartItems
.Select(sci => new model.Product
{
ProductId = sci.ProductId,
MinQuantity = sci.MinQuantity,
MaxQuantity = sci.MaxQuantity
}).ToList()
}).ToList();
This would provide a list of Shopping Carts with each containing a list of products with their respective min/max quantities.
If you also wanted a Lowest min quantity and highest max quantity across all products in a cart:
var shoppingCarts = db.ShoppingCarts
.Select(sc => new model.ShoppingCart
{
ShoppingCartId = sc.Id,
Products = sc.ShoppingCartItems
.Select(sci => new model.Product
{
ProductId = sci.ProductId,
MinQuantity = sci.MinQuantity,
MaxQuantity = sci.MaxQuantity
}).ToList(),
OverallMinQuantity = sc.ShoppingCartItems
.Min(sci => sci.MinQuantity),
OverallMaxQuantity = sc.ShoppingCartItems
.Max(sci => sci.MaxQuantity),
}).ToList();
Though I'm not sure how practical a figure like that might be in relation to a shopping cart structure. In any case, with navigation properties set up for the relationship between your entities, EF should be perfectly capable of building an IQueryable query for the data you want to retrieve without resorting to pre-fetching lists. One issue with pre-fetching and re-introducing those lists into further queries is that there will be a maximum # of rows that EF can handle. Like with SQL IN clauses, there is a maximum # of items that can be parsed from a set.
In any case it sounds like it's provided you with some ideas to try and get to the figures you want.

How can I make a more efficient Entity Framework LINQ call than a nested loop

Is there a more efficent way to write this LINQ statement?
Feature uses the same model as Product (as they are almost identical).
var products = db.Products.Where(q => q.IsFeature == false).ToList();
foreach (Product pr in products)
{
var features = db.Products.Where(q => q.ParentID == pr.ID).ToList();
foreach(Product feature in features)
{
pr.Features.Add(feature);
}
}
And the model:
public class Product
{
//START
public Product()
{
Features = new HashSet<Product>();
}
public int ID { get; set; }
public string Code { get; set; }
public string Name { get; set; }
public bool IsFeature { get; set; }
public int ParentID { get; set; }// If IsFeature is true we need to know its parent
//Nav Props
public ICollection<Product> Features { get; set; }// Features can be represented with the same model as product as they are almost identical
}
This does what I need ie a List of Product (Products), each with its own List of Feature (Features) but are there more elegant/efficient alternatives?

What you need is a join:
var productFeatures =
db.Products.Where(q => !q.IsFeature)
.Join(db.Products, p => p.ID, f => f.ParentId, (p, f) => new { product: p, features: f.ToList() })
.ToList();
For each object in db.Products.Where(q => !q.IsFeature) this will find each of the db.Products that have a ParentId equal to its ID field.
The results will be projected into a List() of anonymous types with the fields product (Product) and features (List<Product>)

If this operation has a complexity which rises to the point where the generated Linq's SQL becomes a hindrance or this business logic is done in multiple places; consider placing the logic in a stored procedure.
Once in the database simply map the results in EF which later can easily be consumed by the code. Hence the complexity is moved off of the client and directly on the database.
Steps (Database First)
Create Stored procedure.
Update the model from the database and choose the stored procedure.
Verify the resultant mappings in EF's Model Browser.
Consume the named stored procedure off of the data context.
I discuss the ins-and-outs of mapping stored procedures on my blog article Entity Framework Stored Procedure Instructions.

You're almost there you can use Addrange method of List collection:
var products = db.Products.Where(q => !q.IsFeature).ToList();
foreach(Product pr in db.Products)
if(!pr.IsFeature) pr.Features.AddRange(db.Products.Where(q => q.ParentID == pr.ID));
P.S. other idea is to extract features sort them, convert to array and use for loop :
var features = db.Products.Where(q => q.IsFeature).OrderBy(f => f.ParentID).ToArray();
p = db.Products.First(p => p.ID == features[0].ParentID);
for(int i = 0; i < features.Length; i++ )
{
if(i > 0 && features[i - 1].ParentID != features[i].ParentID)
p = db.Products.First(p => p.ID == features[i].ParentID);
p.Features.Add(features[i]);
}

You need to use GroupJoin for this. This is exactly what it's designed for.
var products = db.Products.Where(p => !p.IsFeature)
.GroupJoin(db.Products, p => p.ID, q => q.ParentID, (p, eq) => {
p.Features = eq.ToList();
return p;
})
.ToList();
This code is untested, but should work.

Nhibernate QueryOver - Performing a Has All for a Many-to-Many relationship

To explain my problem more easily I will create the following fictional example, illustrating a very basic many-to-many relationship. A Car can have many Parts, and a Part can belong to many Cars.
DB SCHEMA:
CAR_TABLE
---------
CarId
ModelName
CAR_PARTS_TABLE
---------------
CarId
PartId
PARTS_TABLE
-----------
PartId
PartName
CLASSES:
public class Car
{
public int CarId {get;set;}
public string Name {get;set;}
public IEnumerable<Part> Parts {get;set;}
}
public class Part
{
public int PartId {get;set;}
public string Name {get;set}
}
Using this very simple model I would like to get any cars that have all the parts assigned to them from a list of parts I am searching on.
So say I have an array of PartIds:
var partIds = new [] { 1, 3, 10};
I want to mimic the following c# code in terms of a database call:
var allCars = /* code to retrieve all cars */
var results = new List<Car>();
foreach (var car in allCars)
{
var containsAllParts = true;
foreach (var carPart in car.Parts)
{
if (false == partIds.Contains(carPart.PartId))
{
containsAllParts = false;
break;
}
}
if (containsAllParts)
{
results.Add(car);
}
}
return results;
To be clear: I want to get the Cars that have ALL of the Parts specified from the partIds array.
I have the following query, which is REALLY inefficient as it creates a subquery for each id within the partIds array and then does an IsIn query on each of their results. I am desperate to find a much more efficient manner to execute this query.
Car carAlias = null;
Part partAlias = null;
var searchCriteria = session.QueryOver<Car>(() => carAlias);
foreach (var partId in partIds)
{
var carsWithPartCriteria = QueryOver.Of<Car>(() => carAlias)
.JoinAlias(() => carAlias.Parts, () => partAlias)
.Where(() => partAlias.PartId == partId)
.Select(Projections.Distinct(Projections.Id()));
searchCriteria = searchCriteria
.And(Subqueries.WhereProperty(() => carAlias.Id).In(carsWithPartCriteria));
}
var results = searchCriteria.List<Car>();
Is there a decent way to execute this sort of query using NHibernate?

This should be exactly what you want...
Part part = null;
Car car = null;
var qoParts = QueryOver.Of<Part>(() => part)
.WhereRestrictionOn(x => x.PartId).IsIn(partIds)
.JoinQueryOver(x => x.Cars, () => car)
.Where(Restrictions.Eq(Projections.Count(() => car.CarId), partIds.Length))
.Select(Projections.Group(() => car.CarId));

Part partAlias=null;
Session.QueryOver<Car>().JoinQueryOver(x=>x.Parts,()=>partAlias)
.WhereRestrictionOn(()=>partAlias.Id).IsIn(partIds) //partIds should be implement an ICollection
.List<Car>();
Hope that helps.

modified version of this answer
var cars = session.CreateQuery("SELECT c.id FROM Car c JOIN c.Parts p WHERE p.PartId IN(:parts) GROUP BY c HAVING COUNT(DISTINCT p) = :partsCount")
.SetParameterList("parts", partIds)
.SetInt32("partsCount", partIds.Count)
.List();
AFAIK the having clause is only available in HQL. You could modify the select/group by list if you want more columns. Another possible way would be too generate an HQL query that inner joins to each specific part id. I don't think that is possible with ICritiria either as it only lets you join to a property once.