We Keep Coding

How to query IQueryable with Include - ThenInclude?

In .NET Core 2.2 I'm stuck with filtering IQueryable built as:
_context.Ports.Include(p => p.VesselsPorts)
.ThenInclude(p => p.Arrival)
.Include(p => p.VesselsPorts)
.ThenInclude(p => p.Departure)
.OrderBy(p => p.PortLocode);
in many-to-many relation. And the entity models are such as:
public class PortModel
{
[Key]
public string PortLocode { get; set; }
public double? MaxKnownLOA { get; set; }
public double? MaxKnownBreadth { get; set; }
public double? MaxKnownDraught { get; set; }
public virtual ICollection<VesselPort> VesselsPorts { get; set; }
}
public class VesselPort
{
public int IMO { get; set; }
public string PortLocode { get; set; }
public DateTime? Departure { get; set; }
public DateTime? Arrival { get; set; }
public VesselModel VesselModel { get; set; }
public PortModel PortModel { get; set; }
}
Based on this this SO answer I managed to create LINQ like that:
_context.Ports.Include(p => p.VesselsPorts).ThenInclude(p => p.Arrival).OrderBy(p => p.PortLocode)
.Select(
p => new PortModel
{
PortLocode = p.PortLocode,
MaxKnownBreadth = p.MaxKnownBreadth,
MaxKnownDraught = p.MaxKnownDraught,
MaxKnownLOA = p.MaxKnownLOA,
VesselsPorts = p.VesselsPorts.Select(vp => vp.Arrival > DateTime.UtcNow.AddDays(-1)) as ICollection<VesselPort>
}).AsQueryable();
BUT what I need is to find all port records, where:
VesselsPorts.Arrival > DateTime.UtcNow.AddDays(-1) quantity is greater than int x = 5 value (for the example). And I have no clue how to do it :/

Thanks to #GertArnold comment, I ended up with query:
ports = ports.Where(p => p.VesselsPorts.Where(vp => vp.Arrival > DateTime.UtcNow.AddDays(-1)).Count() > x);

When using entity framework people tend to use Include instead of Select to save them some typing. It is seldom wise to do so.
The DbContext holds a ChangeTracker. Every complete row from any table that you fetch during the lifetime of the DbContext is stored in the ChangeTracker, as well as a clone. You get a reference to the copy. (or maybe a reference to the original). If you change properties of the data you got, they are changed in the copy that is in the ChangeTracker. During SaveChanges, the original is compared to the copy, to see if the data must be saved.
So if you are fetching quite a lot of data, and use include, then every fetched items is cloned. This might slow down your queries considerably.
Apart from this cloning, you will probably fetch more properties than you actually plan to use. Database management systems are extremely optimized in combining tables, and searching rows within tables. One of the slower parts is the transfer of the selected data to your local process.
For example, if you have a database with Schools and Students, with the obvious one to many-relation, then every Student will have a foreign key to the School he attends.
So if you ask for School [10] with his 2000 Students, then every Student will have a foreign key value of [10]. If you use Include, then you will be transferring this same value 10 over 2000 times. What a waste of processing power!
In entity framework, when querying data, always use Select to select the properties, and Select only the properties that you actually plan to use. Only use Include if you plan to change the fetched items.
Certainly don't use Include to save you some typing!
Requirement: Give me the Ports with their Vessels
var portsWithTheirVessels = dbContext.Ports
.Where(port => ...) // if you don't want all Ports
.Select(port => new
{
// only select the properties that you want:
PortLocode = port.PortLoCode,
MaxKnownLOA = port.MaxKnownLOA,
MaxKnownBreadth = prot.MaxKnownBreadth,
MaxKnownDraught = ports.MaxKnownDraught,
// The Vessels in this port:
Vessels = port.VesselsPort.Select(vessel => new
{
// again: only the properties that you plan to use
IMO = vessel.IMO,
...
// do not select the foreign key, you already know the value!
// PortLocode = vessle.PortLocode,
})
.ToList(),
});
Entity framework knows your one-to-many relation, and knows that if you use the virtual ICollection that it should do a (Group-)Join.
Some people prefer to do the Group-Join themselves, or they use a version of entity framework that does not support using the ICollection.
var portsWithTheirVessels = dbContext.Ports.GroupJoin(dbContext.VesselPorts,
port => port.PortLocode, // from every Port take the primary key
vessel => vessel.PortLocode, // from every Vessel take the foreign key to Port
// parameter resultSelector: take every Port with its zero or more Vessels to make one new
(port, vesselsInThisPort) => new
{
PortLocode = port.PortLoCode,
...
Vessels = vesselsInThisPort.Select(vessel => new
{
...
})
.ToList(),
});
Alternative:
var portsWithTheirVessels = dbContext.Ports.Select(port => new
{
PortLocode = port.PortLoCode,
...
Vessels = dbContext.VesselPorts.Where(vessel => vessel.PortLocode == port.PortLocode)
.Select(vessel => new
{
...
}
.ToList(),
});
Entity framework will translate this also to a GroupJoin.

Only primitive types do I need to cast?

I am trying to use anonymous types in Entity Framework, but I am getting an error about
Unable to create a constant value
MinQty and MaxQty are int so I don't know if I need to add to Convert.ToInt32?
Unable to create a constant value of type 'Anonymous type'. Only primitive types or enumeration types are supported in this context.
This builds a list object
var listOfLicense = (from l in db.License
select new
{
l.ProductId,
l.MinLicense,
l.MaxLicense
}).tolist();
This is the larger EF object where I am getting the error am I missing a casting?
var ShoppingCart = (from sc in db.ShoppingCarts
Select new model.Shoppingchart{
ShoppingCartId= sc.Id,
MinQty = (int)listOfLicense
.Where(mt => (int)mt.ProductId == sc.ProductId)
.Select(mt => (int)mt.MinLicense)
.Min(mt => mt.Value),
MaxQty = (int)listOfLicense
.Where(mt => (int)mt.ProductId == p.ProductId)
.Select(mt =>(int) mt.MaxQty)
.Max(mt => mt.Value)}.tolist();

This builds a list object
var listOfLicense = (from l in db.License
select new
{
l.ProductId,
l.MinLicense,
l.MaxLicense
})
The above example does not build a list of objects. It builds a query to return objects of that anonymous type.
This builds an in-memory list of objects of that type:
var listOfLicense = (from l in db.License
select new
{
l.ProductId,
l.MinLicense,
l.MaxLicense
}).ToList();
Using .ToList() here will execute the query and return a materialized list of the anonymous types. From there, your code may work as expected without the exception. However, this is effectively loading the 3 columns from all rows in your database table, which may be a problem as the system matures and rows are added.
The error you are getting isn't a casting issue, it is a translation issue. Because your initial query is still just an EF Query, (IQueryable) any further querying against it will need to conform to EF limitations. EF has to be able to translate what your expressions are trying to select back into SQL. In your case, what your real code is trying to do is breaking those rules.
Generally it is better to let EF work with the IQueryable rather than materializing an entire list to memory. Though to accomplish that we'd need to either see the real code, or a minimum reproducible example.
This code:
MinQty = (int)listOfLicense
.Where(mt => (int)mt.ParentProductId == p.ProductId)
.Select(mt => (int)mt.MinLicense)
.Min(mt => mt.Value),
... does not fit with the above anonymous type as there is no correlation between what mt.ParentProductId is in relation to the anonymous type. (p seems to be associated with that type, not mt so there looks to be a lot of Query code missing from your example.)
Edit: based on your updated example:
var ShoppingCart = (from sc in db.ShoppingCarts
Select new model.Shoppingchart{
ShoppingCartId= sc.Id,
MinQty = (int)listOfLicense
.Where(mt => (int)mt.ProductId == sc.ProductId)
.Select(mt => (int)mt.MinLicense)
.Min(mt => mt.Value),
MaxQty = (int)listOfLicense
.Where(mt => (int)mt.ProductId == p.ProductId)
.Select(mt =>(int) mt.MaxQty)
.Max(mt => mt.Value)}.ToList();
It may be possible to build something like this into a single query expression depending on the relationships between ShoppingCart, Product, and Licence. It almost looks like "Licence" really refers to a "Product" which contains a min and max quantity that you're interested in.
Assuming a structure like:
public class Product
{
[Key]
public int ProductId { get; set; }
public int MinQuantity { get; set; }
public int MaxQuantity { get; set; }
// ...
}
// Here lies a question on how your shopping cart to product relationship is mapped. I've laid out a many-to-many relationship using ShoppingCartItems
public class ShoppingCart
{
[Key]
public int ShoppingCartId { get; set; }
// ...
public virtual ICollection<ShoppingCartItem> ShoppingCartItems { get; set; } = new List<ShoppingCartItem>();
}
public class ShoppingCartItem
{
[Key, Column(0), ForeignKey("ShoppingCart")]
public int ShoppingCartId { get; set; }
public virtual ShoppingCart ShoppingCart{ get; set; }
[Key, Column(1), ForeignKey("Product")]
public int ProductId { get; set; }
public virtual Product Product { get; set; }
}
With something like this, to get shopping carts with their product min and max quantities:
var shoppingCarts = db.ShoppingCarts
.Select(sc => new model.ShoppingCart
{
ShoppingCartId = sc.Id,
Products = sc.ShoppingCartItems
.Select(sci => new model.Product
{
ProductId = sci.ProductId,
MinQuantity = sci.MinQuantity,
MaxQuantity = sci.MaxQuantity
}).ToList()
}).ToList();
This would provide a list of Shopping Carts with each containing a list of products with their respective min/max quantities.
If you also wanted a Lowest min quantity and highest max quantity across all products in a cart:
var shoppingCarts = db.ShoppingCarts
.Select(sc => new model.ShoppingCart
{
ShoppingCartId = sc.Id,
Products = sc.ShoppingCartItems
.Select(sci => new model.Product
{
ProductId = sci.ProductId,
MinQuantity = sci.MinQuantity,
MaxQuantity = sci.MaxQuantity
}).ToList(),
OverallMinQuantity = sc.ShoppingCartItems
.Min(sci => sci.MinQuantity),
OverallMaxQuantity = sc.ShoppingCartItems
.Max(sci => sci.MaxQuantity),
}).ToList();
Though I'm not sure how practical a figure like that might be in relation to a shopping cart structure. In any case, with navigation properties set up for the relationship between your entities, EF should be perfectly capable of building an IQueryable query for the data you want to retrieve without resorting to pre-fetching lists. One issue with pre-fetching and re-introducing those lists into further queries is that there will be a maximum # of rows that EF can handle. Like with SQL IN clauses, there is a maximum # of items that can be parsed from a set.
In any case it sounds like it's provided you with some ideas to try and get to the figures you want.

How do I group on one of two possible fields using LINQ?

I am trying to get the latest contact with a given user, grouped by user:
public class ChatMessage
{
public string SentTo { get; set; }
public string SentFrom { get; set; }
public string MessageBody { get; set; }
public string SendDate { get; set; }
}
The user's contact info could either be in SentTo or SentFrom.
List<ChatMessage> ecml = new List<ChatMessage>();
var q = ecml.OrderByDescending(m => m.SendDate).First();
would give me the latest message, but I need the last message per user.
The closest solution I could find was LINQ Max Date with group by, but I cant seem to figure out the correct syntax. I would rather not create multiple List objects if I don't have to.
If the user's info is in SentTo, my info will be in SentFrom, and vice-versa, so I do have some way of checking where the user's data is.
Did I mention I was very new to LINQ? Any help would be greatly appreciated.

Since you need to interpret each record twice - i.e. as a SentTo and a SentFrom, the query becomes a bit tricky:
var res = ecml
.SelectMany(m => new[] {
new { User = m.SentFrom, m.SendDate }
, new { User = m.SentTo, m.SendDate }
})
.GroupBy(p => p.User)
.Select(g => new {
User = g.Key
, Last = g.OrderByDescending(m => m.SendDate).First()
});
The key trick is in SelectMany, which makes each ChatMessage item into two anonymous items - one that pairs up the SentFrom user with SendDate, and one that pairs up the SentTo user with the same date.
Once you have both records in an enumerable, the rest is straightforward: you group by the user, and then apply the query from your post to each group.

It should be pretty easy, look at this code:
string username = "John";
var q = ecml.Where(i=>i.SentFrom == username || i.SentTo == username).OrderByDescending(m => m.SendDate).First();
It simply filter your collection be choosing items which either SentFrom or SentTo is equal to username.

Orchard CMS ContentQuery with N-to-N relation

In my project I've implemented N-to-N relation between records using this tutorial on OrchardProject web-site. I have 2 parts: MaterialPart & CategoryPart and association record.
Material part
public class MaterialPartRecord : ContentPartRecord {
public MaterialPartRecord() {
Categories = new List<ContentMaterialCategoryRecord>();
}
}
public class MaterialPart : ContentPart<MaterialPartRecord> {
public IEnumerable<CategoryPartRecord> Categories {
get { return Record.Categories.Select(cmcr => cmcr.CategoryPartRecord); }
}
}
CategoryPartRecord
public class CategoryPartRecord : ContentPartRecord {
...
}
public class CategoryPart : ContentPart<CategoryPartRecord> {
...
}
association record:
public class ContentMaterialCategoryRecord {
public virtual int Id { get; set; }
public virtual MaterialPartRecord MaterialPartRecord { get; set; }
public virtual CategoryPartRecord CategoryPartRecord { get; set; }
}
Now I need to select MaterialItems which are linked to certain category. So far I have this method to extract them. It works but I'm not sure that it is correct way to do this.
public IEnumerable<MaterialPart> GetMaterialsByCategory(int catId) {
var cs = new CategoriesService(_oServices);
CategoryPartRecord cat = cs.GetItem(catId).Record;
return _oServices.ContentManager
.Query(VersionOptions.Latest, _contentType)
.Join<CommonPartRecord>()
.OrderByDescending(cpr => cpr.PublishedUtc);
.List()
.Where(ci => ci.IsPublished())
.Select(ci => ci.As<MaterialPart>())
.Where(mp => mp.Categories.Contains(cat)); // < ---- ?
}
So my question is: what is correct way to select materials for required category, which produces optimal SQL query, as we simply need to inner join associated record table with required CategoryPartRecord_Id field value.
thaks!

In case, of M : N with pairing object, we can use QueryOver and subquery. The biggest benefit would be, that we recieve the plain set of material Items, which we can use for paging (Take(), Skip())
var session = ... // get curretn session
CategoryPartRecord category = null;
ContentMaterialCategoryRecord pair = null;
MaterialPartRecord material = null;
var subquery = QueryOver.Of<ContentMaterialCategoryRecord>(() => pair)
// now we will join Categories to be able to filter whatever property
.JoinQueryOver(() => pair.CategoryPartRecord, () => category)
// here is the filter
// there could be IN, >= <= ...
.Where(() => category.ID == 1)
// or
.WhereRestrictionOn(c => c.category.ID).IsIn(new[] {1, 2, 3})
...
// now we will return IDs of the Material we are interested in
.Select(x => pair.MaterialPartRecord.Id);
// finally the clean query over the Materials...
var listOfUsers = session.QueryOver<MaterialPartRecord>(() => material )
.WithSubquery
.WhereProperty(() => material.Id)
.In(subquery)
// paging
.Take(10)
.Skip(10)
.List<MaterialPartRecord>();
So, this will produce the most effective SQL Script, with one subselect, and clean select from material table
NOTE: similar stuff could be done even with LINQ. But QueryOver is NHibernate most native way I'd say. Anyhow, the principe - subquery to filter by category, and main query to load materials will remain the same. Only ONE SQL Select call

RavenDB index for nested query

I'm pretty new to RavenDB and am struggling to find a solution to the following:
I have a collection called ServiceCalls that look like this:
public class ServiceCall
{
public int ID { get; set; }
public string IncidentNumber { get; set; }
public string Category { get; set; }
public string SubCategory { get; set; }
public DateTime ReportedDateTime { get; set; }
public string Block { get; set; }
public decimal Latitude { get; set; }
public decimal Longitude { get; set; }
}
I have an index named ServiceCalls/CallsByCategory that looks like this:
Map = docs => from doc in docs
select new
{
Category = doc.Category,
CategoryCount = 1,
ServiceCalls = doc,
};
Reduce = results => from result in results
group result by result.Category into g
select new
{
Category = g.Key,
CategoryCount = g.Count(),
ServiceCalls = g.Select(i => i.ServiceCalls)
};
So the output is:
public class ServiceCallsByCategory
{
public string Category { get; set; }
public int CategoryCount { get; set; }
public IEnumerable<ServiceCall> ServiceCalls { get; set; }
}
using this query everything works as it should
var q = from i in session.Query<ServiceCallsByCategory>("ServiceCalls/CallsByCategory") select i
Where I am absolutely lost is writing an index that would allow me to query by ReportedDateTime. Something that would allow me to do this:
var q = from i in session.Query<ServiceCallsByCategory>("ServiceCalls/CallsByCategory")
where i.ServiceCalls.Any(x=>x.ReportedDateTime >= new DateTime(2012,10,1))
select i
Any guidance would be MUCH appreciated.

A few things,
You can't have a .Count() method in your reduce clause. If you look closely, you will find your counts are wrong. As of build 2151, this will actually throw an exception. Instead, you want CategoryCount = g.Sum(x => x.CategoryCount)
You always want the structure of the map to match the structure of the reduce. If you're going to build a list of things, then you should map a single element array of each thing, and use .SelectMany() in the reduce step. The way you have it now only works due to a quirk that will probably be fixed at some point.
By building the result as a list of ServiceCalls, you are copying the entire document into the index storage. Not only is that inefficient, but it's unnecessary. You would do better keeping a list of just the ids. Raven has an .Include() method that you can use if you need to retrieve the full document. The main advantage here is that you are guaranteed to have the most current data for each item you get back, even if your index results are still stale.
Putting all three together, the correct index would be:
public class ServiceCallsByCategory
{
public string Category { get; set; }
public int CategoryCount { get; set; }
public int[] ServiceCallIds { get; set; }
}
public class ServiceCalls_CallsByCategory : AbstractIndexCreationTask<ServiceCall, ServiceCallsByCategory>
{
public ServiceCalls_CallsByCategory()
{
Map = docs => from doc in docs
select new {
Category = doc.Category,
CategoryCount = 1,
ServiceCallIds = new[] { doc.ID },
};
Reduce = results => from result in results
group result by result.Category
into g
select new {
Category = g.Key,
CategoryCount = g.Sum(x => x.CategoryCount),
ServiceCallIds = g.SelectMany(i => i.ServiceCallIds)
};
}
}
Querying it with includes, would look like this:
var q = session.Query<ServiceCallsByCategory, ServiceCalls_CallsByCategory>()
.Include<ServiceCallsByCategory, ServiceCall>(x => x.ServiceCallIds);
When you need a document, you still load it with session.Load<ServiceCall>(id) but Raven will not have to make a round trip back to the server to get it.
NOW - that doesn't address your question about how to filter the results by date. For that, you really need to think about what you are trying to accomplish. All of the above would assume that you really want every service call shown for each category at once. Most of the time, that's not going to be practical because you want to paginate results. You probably DON'T want to even use what I've described above. I am making some grand assumptions here, but most of the time one would filter by category, not group by it.
Let's say you had an index that just counts the categories (the above index without the list of service calls). You might use that to display an overview screen. But you wouldn't be interested in the documents that were in each category until you clicked one and drilled into a details screen. At that point, you know which category you're in, and you can filter by it and reduce to a date range without a static index:
var q = session.Query<ServiceCall>().Where(x=> x.Category == category && x.ReportedDateTime >= datetime)
If I am wrong and you really DO need to show all documents from all categories, grouped by category, and filtered by date, then you are going to have to adopt an advanced technique like the one I described in this other StackOverflow answer. If this is really what you need, let me know in comments and I'll see if i can write it for you. You will need Raven 2.0 to make it work.
Also - be very careful about what you are storing for ReportedDateTime. If you are going to be doing any comparisons at all, you need to understand the difference between calendar time and instantaneous time. Calendar time has quirks like daylight savings transitions, time zone differences, and more. Instantaneous time tracks the moment something happened, regardless of who's asking. You probably want instantaneous time for your usage, which means either using a UTC DateTime, or switching to DateTimeOffset which will let you represent instantaneous time without losing the local contextual value.
Update
I experimented with trying to build an index that would use that technique I described to let you have all results in your category groups but still filter by date. Unfortunately, it's just not possible. You would have to have all ServiceCalls grouped together in the original document and express it in the Map. It doesn't work the same way at all if you have to Reduce first. So you really should just consider simple query for ServiceCalls once you are in a specific Category.

Could you add ReportedDateTime to the Map and aggregate it in the Reduce? If you only care about the max per category, something like this should be sufficient.
Map = docs => from doc in docs
select new
{
Category = doc.Category,
CategoryCount = 1,
ServiceCalls = doc,
ReportedDateTime
};
Reduce = results => from result in results
group result by result.Category into g
select new
{
Category = g.Key,
CategoryCount = g.Sum(x => x.CategoryCount),
ServiceCalls = g.Select(i => i.ServiceCalls)
ReportedDateTime = g.Max(rdt => rdt.ReportedDateTime)
};
You could then query it just based on the aggregated ReportedDateTime:
var q = from i in session.Query<ServiceCallsByCategory>("ServiceCalls/CallsByCategory")
where i.ReportedDateTime >= new DateTime(2012,10,1)
select i