We Keep Coding

Entity Framework Include directive not getting all expected related rows

While debugging some performance issues I discovered that Entity framework was loading a lot of records via lazy loading (900 extra query calls ain't fast!) but I was sure I had the correct include. I've managed to get this down to quite a small test case to demonstrate the confusion I'm having, the actual use case is more complex so I don't have a lot of scope to re-work the signature of what I'm doing but hopefully this is a clear example of the issue I'm having.
Documents have Many MetaInfo rows related. I want to get all documents grouped by MetaInfo rows with a specific value, but I want all the MetaInfo rows included so I don't have to fire off a new request for all the Documents MetaInfo.
So I've got the following Query.
ctx.Configuration.LazyLoadingEnabled = false;
var DocsByCreator = ctx.Documents
.Include(d => d.MetaInfo) // Load all the metaInfo for each object
.SelectMany(d => d.MetaInfo.Where(m => m.Name == "Author") // For each Author
.Select(m => new { Doc = d, Creator = m })) // Create an object with the Author and the Document they authored.
.ToList(); // Actualize the collection
I expected this to have all the Document / Author pairs, and have all the Document MetatInfo property filled.
That's not what happens, I get the Document objects, and the Authors just fine, but the Documents MetaInfo property ONLY has MetaInfo objects with Name == "Author"
If I move the where clause out of the select many it does the same, unless I move it to after the actualisation (which while here might not be a big deal, it is in the real application as it means we're getting a huge amount more data than we want to deal with.)
After playing with a bunch of different ways to do this I think it really looks like the issue is when you do a select(...new...) as well as the where and the include. Doing the select, or the Where clause after actualisation makes the data appear the way I expected it to.
I figured it was an issue with the MetaInfo property of Document being filtered, so I rewrote it as follows to test the theory and was surprised for find that this also gives the same (I think wrong) result.
ctx.Configuration.LazyLoadingEnabled = false;
var DocsByCreator = ctx.Meta
.Where(m => m.Name == "Author")
.Include(m => m.Document.MetaInfo) // Load all the metaInfo for Document
.Select(m => new { Doc = m.Document, Creator = m })
.ToList(); // Actualize the collection
Since we're not putting the where on the Document.MetaInfo property I expected this to bypass the problem, but strangely it doesn't the documents still only appear to have "Author" MetaInfo object.
I've created a simple test project and uploaded it to github with a bunch of test cases in, as far as I can tell they should all pass, bug only the ones with premature actualisation pass.
https://github.com/Robert-Laverick/EFIncludeIssue
Anyone got any theories? Am I abusing EF / SQL in some way I'm missing? Is there anything I can do differently to get the same organisation of results? Is this a bug in EF that's just been hidden from view by the LazyLoad being on by default, and it being a bit of an odd group type operation?

This is a limitation in EF in that Includes will be ignored if the scope of the entities returned is changed from where the include was introduced.
I couldn't find the reference to this for EF6, but it is documented for EF Core. (https://learn.microsoft.com/en-us/ef/core/querying/related-data) (see "ignore includes") I suspect it is a limit in place to stop EF's SQL generation from going completely AWOL in certain scenarios.
So while var docs = context.Documents.Include(d => d.Metas) would return the metas eager loaded against the document; As soon as you .SelectMany() you are changing what EF is supposed to return, so the Include statement is ignored.
If you want to return all documents, and include a property that is their author:
var DocsByCreator = ctx.Documents
.Include(d => d.MetaInfo)
.ToList() // Materialize the documents and their Metas.
.SelectMany(d => d.MetaInfo.Where(m => m.Name == "Author") // For each Author
.Select(m => new { Doc = d, Creator = m })) // Create an object with the Author and the Document they authored.
.ToList(); // grab your collection of Doc and Author.
If you only want documents that have authors:
var DocsByCreator = ctx.Documents
.Include(d => d.MetaInfo)
.Where(d => d.MetaInfo.Any(m => m.Name == "Author")
.ToList() // Materialize the documents and their Metas.
.SelectMany(d => d.MetaInfo.Where(m => m.Name == "Author") // For each Author
.Select(m => new { Doc = d, Creator = m })) // Create an object with the Author and the Document they authored.
.ToList(); // grab your collection of Doc and Author.
This means you will want to be sure that all of your filtering logic is done above that first 'ToList() call. Alternatively you can consider resolving the Author meta after the query such as when view models are populated, or an unmapped "Author" property on Document that resolves it. Though I generally avoid unmapped properties because if their use slips into an EF query, you get a nasty error at runtime.
Edit: Based on the requirement to skip & take I would recommend utilizing view models to return data rather than returning entities. Using a view model you can instruct EF to return just the raw data you need, compose the view models with either simple filler code or utilizing Automapper which plays nicely with IQueryable and EF and can handle most deferred cases like this.
For example:
public class DocumentViewModel
{
public int DocumentId { get; set; }
public string Name { get; set; }
public ICollection<MetaViewModel> Metas { get; set; } = new List<MetaViewModel>();
[NotMapped]
public string Author // This could be update to be a Meta, or specialized view model.
{
get { return Metas.SingleOrDefault(x => x.Name == "Author")?.Value; }
}
}
public class MetaViewModel
{
public int MetaId { get; set; }
public string Name { get; set; }
public string Value { get; set; }
}
Then the query:
var viewModels = context.Documents
.Select(x => new DocumentViewModel
{
DocumentId = x.DocumentId,
Name = x.Name,
Metas = x.Metas.Select(m => new MetaViewModel
{
MetaId = m.MetaId,
Name = m.Name,
Value = m.Value
}).ToList()
}).Skip(pageNumber*pageSize)
.Take(PageSize)
.ToList();
The relationship of an "author" to a document is implied, not enforced, at the data level. This solution keeps the entity models "pure" to the data representation and lets the code handle transforming that implied relationship into exposing a document's author.
The .Select() population can be handled by Automapper using .ProjectTo<TViewModel>().
By returning view models rather than entities you can avoid issues like this where .Include() operations get invalidated, plus avoid issues due to the temptation of detaching and reattaching entities between different contexts, plus improve performance and resource usage by only selecting and transmitting the data needed, and avoiding lazy load serialization issues if you forget to disable lazy-load or unexpected #null data with it.

Initializing var type in LINQ

Question: How do I initialize var in the following code? [Then, of course, I'll remove var declaration from the if statement]
The last line of the following code returns the well know error: The name lstCurrent does not exists in current context. Clearly the error is because the var is defined inside if statement and used outside if statement
Note: I'm selecting only a few columns from the table and hence dealing with anonymous type. Some examples I saw online did not work - probably since my code is selecting anonymous type. But this is just a guess.
var lstCurrent =????;
if(Type==1)
var lstCurrent =_context.Customers().Where(t =>t.type=="current").Select(c => new { c.LastName, c.City});
else
lstCurrent = _context.Customers().Where(...).Select(...)
return View(lstCurrent.ToList());

var is not a type - it means "I don't care to (or can't) specify what the type is - let the compiler do it for me".
In your case, you're assigning it the result of one of two queries, one of which returns an anonymous type, so you can't specify the type since you don't know the name of the anonymous type (hence the term "anonymous").
In order to use var, the compiler needs some expression at initialization to know what the actual type is.
I'd suggest something like:
var lstCurrent = Type==1
? _context.Customers().Where(t =>t.type=="current").Select(c => new { c.LastName, c.City})
: _context.Customers().Where(...).Select(...)
But note that your "selects" must return the same type (or anonymous types with the exact same fields) or you won't be able to use var.
In the end I would try to bake the condition into your Where clause for less repetetive code:
bool isTypeOne = Type==1;
var lstCurrent = _context.Customers()
.Where(t => isTypeOne ? t.type=="current" : ...)
.Select(c => new { c.LastName, c.City})

Try
IEnumerable lstCurrent;
if(Type == 1)
lstCurrent = foo;
else
lstCurrent = bar;

How do I initialize var in the following code? [Then, of course, I'll
remove var declaration from the if statement]
This is not possible since the type of the object you assign at the left should be known. For instance
var a = "text";
The type of a is known at compile time since the right hand expression is a string. This cannot be done with a sequence of anonymous types, like the one you define.
I can see two options. One is that D Stanley already mentioned. The other is to define a class with two properties like below:
public class PersonCity
{
public string LastName { get; set; }
public string City { get; set; }
}
and then project each element of your query to a PersonCity object.
lstCurrent context.Customers()
.Where(t =>t.type=="current")
.Select(c => new PersonCity
{
LastName = c.LastName,
City = c.City
});
Doing so, you can define now you lstCurrent as below:
var lstCurrent = Enumerable.Empty<PersonCity>();
Important Note
In case of your queries return different types, the above are meaningless. Both queries (one in if and the other at else) should return the same type.

This is a common trap when expecting to use an implicit type declaration or when refactoring code that already has an implicit type (var). The current accepted answer is very much valid but reducing all expression variations into a single 1-liner linq expression can easily impact on readability of the code.
The issue in this case is complicated by the projection to an anonymous type at the end of the query, which can be solved by using an explicit type definition for the projection, but it is simpler to break up the query construction into multiple steps:
var customerQuery = _context.Customers().AsQueryable();
if (Type == 1)
customerQuery = customerQuery.Where(t => t.type == "current");
else
customerQuery = customerQuery.Where(...);
... // any other filtering or sorting expressions?
var lstCurrent = customQuery.Select(c => new { c.LastName, c.City});
return View(lstCurrent.ToList());
or of course that last segment could have been a one liner or if there are no further references to lstCurrent the compiler may optimise that into the following:
return View(customQuery.Select(c => new { c.LastName, c.City}).ToList());
In this example I have deliberately cast to IQueryable<T> to ensure this solution is compatible with both IQueryable<T>/DbSet<T> contexts and repository style IEnumerable<T> contexts.
This variation is usually the first that comes to mind, but we are still declaring the source of the query in two places, which increases the ambiguity of this code and the risk of divergence in later refactoring (by accidentally editing only one branch and not maintaining the code in the other branch):
IQueryable<Customer> customerQuery = null;
if (Type == 1)
customerQuery = _context.Customers().Where(t => t.type == "current");
else
customerQuery = _context.Customers().Where(...);
... // any other filtering or sorting expressions?
var lstCurrent = customQuery.Select(c => new { c.LastName, c.City});
return View(lstCurrent.ToList());
A different solution is to explicitly define the output as its own concrete class:
public class CustomerSummary
{
public string LastName { get;set; }
public string City { get;set; }
}
...
List<Customers> customers = null;
if (Type == 1)
customers = _context.Customers().Where(c => c.type == "current")
.Select(c => new CustomerSummary
{
LastName = c.LastName,
City = c.City
}).ToList();
else
customers = _context.Customers().Where(c => ...)
.Select(c => new CustomerSummary
{
LastName = c.LastName,
City = c.City
}).ToList();
... // any other filtering or sorting expressions?
return View(customers);
It's a lot of code for a once-off, but if you make it abstract enough it could be re-used for other scenarios, I would still combine this with the first code example, that keeps the source, filter and projection logic separated, over the lifetime of an application these 3 elements tend to evolve differently, so separating out the code makes refactoring or future maintenance easier to complete and review.

LINQ using Automapper for a third level property

I have three entity framework objects that are nested - see image below. Also, I have generated similar objects as DTO to map them.
So my question is, as the companyDocument object has a list of ListCompanyDocumentsType, when I execute a query to pick all companies, it returns a list of Companies and its Documents plus a list of documents types with a list of all documents that each type has within the database (like a looping).
Is there any way to return the companyDocument with only one document type? Or I have designed it incorrectly? To solve this problem, I've used a mapped object and a for each that returns the type name within a non-mapped DTO property. So it is working, but I'm not sure if it is the correct way.
Also, I've tried to use the EF .include() to go up to the ListCompanyDocumentTypes, but it is still returning all documents that each type has.
var config = new MapperConfiguration(cfg => { cfg.CreateMap<Company, CompanyDto>(); cfg.CreateMap<CompanyDocument,CompanyDocumentDto>();});
var mapper = config.CreateMapper();
var newDtoTest = mapper.Map<List<CompanyDto>>(companiesReturn);
var db = new entities();
foreach (var companyDto in newDtoTest)
{
foreach (var companyDtoCompanyDocument in companyDto.CompanyDocuments)
{
companyDtoCompanyDocument.dtoTypeName = (await db.ListCompanyDocumentsTypes.FirstOrDefaultAsync(p=> p.id.Equals(companyDtoCompanyDocument.typeId))).typeName;
}
}
Thank you,

You should use ForMember(),using this method you will be able to set the appropriate value for the specific field:
cfg.CreateMap<Company, CompanyDto>()
.ForMember(dest => dest.dtoTypeName, opt => opt.MapFrom(src => src.CompanyDocuments.ListCompanyDocumentsTypes.FirstOrDefault().typeName));
And companiesReturn should be a IQueryable<Company> type, so your sample method GetAll Company need include CompanyDocuments and ListCompanyDocumentsTypes and should look like:
public IQueryable<AttributeElement> GetAll()
{
return CompanyRepository.GetQueryable()
.Include(d => d.CompanyDocuments)
.ThenInclude(d => d.ListCompanyDocumentsTypes);
}
Automapper has easy mapping, you do not need to use additional loops, everything can be set up in configuration class.

Multiple Include and Where Clauses Linq

I have a database where I'm wanting to return a list of Clients.
These clients have a list of FamilyNames.
I started with this
var query = DbContext.Clients.Include(c => c.FamilyNames).ToList() //returns all clients, including their FamilyNames...Great.
But I want somebody to be able to search for a FamilyName, ifany results are returned, then show the clients to the user.
so I did this...
var query = DbContext.Clients.Include(c => c.FamilyNames.Where(fn => fn.familyName == textEnteredByUser)).ToList();
I tried...
var query = DbContext.Clients.Include(c => c.FamilyNames.Any(fn => fn.familyName == textEnteredByUser)).ToList();
and...
var query = DbContext.FamilyNames.Include(c => c.Clients).where(fn => fn.familyname == textEnteredByUser.Select(c => c.Clients)).ToList();
What I would like to know (obviously!) is how I could get this to work, but I would like it if at all possible to be done in one query to the database. Even if somebody can point me in the correct direction.
Kind regards

In Linq to Entities you can navigate on properties and they will be transformed to join statements.
This will return a list of clients.
var query = DbContext.Clients.Where(c => c.FamilyNames.Any(fn => fn == textEnteredByUser)).ToList();
If you want to include all their family names with eager loading, this should work:
var query = DbContext.Clients.Where(c => c.FamilyNames.Any(fn => fn == textEnteredByUser)).Include(c => c.FamilyNames).ToList();
Here is some reference about loading related entities if something doesn't work as expected.

You can use 'Projection', basically you select just the fields you want from any level into a new object, possibly anonymous.
var query = DbContext.Clients
.Where(c => c.FamilyNames.Any(fn => fn == textEnteredByUser))
// only calls that can be converted to SQL safely here
.Select(c => new {
ClientName = c.Name,
FamilyNames = c.FamilyNames
})
// force the query to be materialized so we can safely do other transforms
.ToList()
// convert the anon class to what we need
.Select(anon => new ClientViewModel() {
ClientName = anon.ClientName,
// convert IEnumerable<string> to List<string>
FamilyNames = anon.FamilyNames.ToList()
});
That creates an anonymous class with just those two properties, then forces the query to run, then performs a 2nd projection into a ViewModel class.
Usually I would be selecting into a ViewModel for passing to the UI, limiting it to just the bare minimum number of fields that the UI needs. Your needs may vary.

LINQ Query - Only get Order and MAX Date from Child Collection

I'm trying to get a list that displays 2 values in a label from a parent and child (1-*) entity collection model.
I have 3 entities:
[Customer]: CustomerId, Name, Address, ...
[Order]: OrderId, OrderDate, EmployeeId, Total, ...
[OrderStatus]: OrderStatusId, StatusLevel, StatusDate, ...
A Customer can have MANY Order, which in turn an Order can have MANY OrderStatus, i.e.
[Customer] 1--* [Order] 1--* [OrderStatus]
Given a CustomerId, I want to get all of the Orders (just OrderId) and the LATEST (MAX?) OrderStatus.StatusDate for that Order.
I've tried a couple of attempts, but can seem to get the results I want.
private IQueryable<Customer> GetOrderData(string customerId)
{
var ordersWithLatestStatusDate = Context.Customers
// Note: I am not sure if I should add the .Expand() extension methods here for the other two entity collections since I want these queries to be as performant as possible and since I am projecting below (only need to display 2 fields for each record in the IQueryable<T>, but thinking I should now after some contemplation.
.Where(x => x.CustomerId == SelectedCustomer.CustomerId)
.Select(x => new Custom
{
CustomerId = x.CustomerId,
...
// I would like to project my Child and GrandChild Collections, i.e. Orders and OrderStatuses here but don't know how to do that. I learned that by projecting, one does not need to "Include/Expand" these extension methods.
});
return ordersWithLatestStatusDate ;
}
---- UPDATE 1 ----
After the great solution from User: lazyberezovsky, I tried the following:
var query = Context.Customers
.Where(c => c.CustomerId == SelectedCustomer.CustomerId)
.Select(o => new Customer
{
Name = c.Name,
LatestOrderDate = o.OrderStatus.Max(s => s.StatusDate)
});
In my hastiness from my initial posting, I didn't paste everything in correctly since it was mostly from memory and didn't have the exact code for reference at the time. My method is a strongly-typed IQueryabled where I need it to return a collection of items of type T due to a constraint within a rigid API that I have to go through that has an IQueryable query as one of its parameters. I am aware I can add other entities/attributes by either using the extension methods .Expand() and/or .Select(). One will notice that my latest UPDATED query above has an added "new Customer" within the .Select() where it was once anonymous. I'm positive that is why the query failed b/c it couldn't be turn into a valid Uri due to LatestOrderDate not being a property of Customer at the Server level. FYI, upon seeing the first answer below, I had added that property to my client-side Customer class with simple { get; set; }. So given this, can I somehow still have a Customer collection with the only bringing back those 2 fields from 2 different entities? The solution below looked so promising and ingenious!
---- END UPDATE 1 ----
FYI, the technologies I'm using are OData (WCF), Silverlight, C#.
Any tips/links will be appreciated.

This will give you list of { OrderId, LatestDate } objects
var query = Context.Customers
.Where(c => c.CustomerId == SelectedCustomer.CustomerId)
.SelectMany(c => c.Orders)
.Select(o => new {
OrderId = o.OrderId,
LatestDate = o.Statuses.Max(s => s.StatusDate) });
.
UPDATE construct objects in-memory
var query = Context.Customers
.Where(c => c.CustomerId == SelectedCustomer.CustomerId)
.SelectMany(c => c.Orders)
.AsEnumerable() // goes in-memory
.Select(o => new {
OrderId = o.OrderId,
LatestDate = o.Statuses.Max(s => s.StatusDate) });
Also grouping could help here.

If I read this correctly you want a Customer entity and then a single value computed from its Orders property. Currently this is not supported in OData. OData doesn't support computed values in the queries. So no expressions in the projections, no aggregates and so on.
Unfortunately even with two queries this is currently not possible since OData doesn't support any way of expressing the MAX functionality.
If you have control over the service, you could write a server side function/service operation to execute this kind of query.