We Keep Coding

EF Core: Correct way to query data multiple levels deep in related one-to-many entities

I'm trying learn to write efficient Entity Framework queries when data has to be fetched based on multiple joins, including a many-to-many via a junction table. In the following example, I'd like to fetch all States that contain a particular Book.
Let's use a model with the following tables/entities, all linked by navigation properties:
State, City, Library, Book, LibraryBook (junction table for many-to-many relationship between library and book.)
Each State has 1 or more Cities
Each City has 1 or more Libraries
Each Library has many Books & Each Book may exist at more than 1 library.
How can I best return all of the States that contain a particular Book? I'm inclined to think separate queries may work better than 1 large one, but I'm not certain what the best implementation is. I think that getting the LibraryId from the many-to-many relation first in a separate query is probably a good way to start.
So for that:
var bookId = 12;
var libraryIds = _context.LibraryBook.Where(l => l.BookId == bookId).Select(s => s.LibraryId);
If that comes first, I'm uncertain how to best query the next data in order to get the cities which contain each of those LibraryIds. I could use a foreach:
var cities = new List<City>;
foreach(var libraryId in libraryIds)
{
var city = _context.City.Where(c => c.Library = libraryId)
cities.Add(city);
}
But then I'd have to do yet another foreach for the states that contain the city, and this all adds up to a lot of separate SQL queries!
Is this really the only way to go about this? If not, what is a better alternative?
Thanks in advance!

Database management systems are extremely optimized in combining tables and selecting columns from the result. The transport of the selected data is the slower part.
Hence it is usually better to limit the data that needs to be transported: let the DBMS do all the joining and selecting.
For this, you don't need to put everything in one big LINQ statement that is hard to understand (and thus hard to test, reuse, maintain). As long as your LINQ statements remain IQuerayble<...>, the query is not executed. Concatenating several of these LINQ statements is not costly.
Back to your question
If you followed the entity framework conventions, your one-to-many relations and your many-to-many will have resulted in classes similar to the following:
class State
{
public int Id {get; set;}
public string Name {get; set;}
...
// every State has zero or more Cities (one-to-many)
public virtual ICollection<City> Cities {get; set;}
}
class City
{
public int Id {get; set;}
public string Name {get; set;}
...
// Every City is a City in exactly one State, using foreign key:
public int StateId {get; set;}
public virtual State State {get; set;}
// every City has zero or more Libraries (one-to-many)
public virtual ICollection<Library> Libraries {get; set;}
}
Library and Books: many-to-many:
class Library
{
public int Id {get; set;}
public string Name {get; set;}
...
// Every Library is a Library in exactly one City, using foreign key:
public int CityId {get; set;}
public virtual City City {get; set;}
// every Library has zero or more Books (many-to-many)
public virtual ICollection<Book> Books {get; set;}
}
class Book
{
public int Id {get; set;}
public string Title {get; set;}
...
// Every Book is a Book in zero or more Libraries (many-to-many)
public virtual ICollection<Book> Books {get; set;}
}
This is all that entity framework needs to know to recognize your tables, the columns in the tables and the relations between the tables.
You will only need attributes or fluent API if you want to deviate from the conventions: different identifiers for columns or tables, non-default types for decimals, non default behaviour for cascade on delete, etc.
In entity framework, the columns in the tables are represented by the non-virtual properties; the virtual properties represent the relations between the tables.
The foreign key is an actual column in the table, hence it is non-virtual. The one-to-many has virtual ICollection<Type> on the "one" side and virtual Type on the "many" side. The many-to-many has virtual ICollection<...> on both sides.
There is no need to specify the junction table. Entity framework recognizes the many-to-many and creates the junction table for you. If you use database first, you might need to use fluent API to specify the junction table.
But how am I supposed to do the joins without a junction table?
Answer: don't do the (group-)joins yourself, use the virtual ICollections!
How can I best return all of the States that contain a particular Book?
int bookId = ...
var statesWithThis = dbContext.States
.Where(state => state.Cities.SelectMany(city => city.Libraries)
.SelectMany(library => library.Books)
.Select(book => book.Id)
.Contains(bookId);
In words: you have a lot of States. From every State, get all Books that are in all Libraries that are in all Cities in this State. Use SelectMany to make this one big sequence of Books. From every Book Select the Id. The result is one big sequence of Ids (of Books that are in Libraries that are in Cities that are in the State). Keep only those States that have at least one Book.
Room for Optimization
If you regularly need to do similar questions, like: "Give me all States that have a Book from a certain Author", or "Give me all Libraries that have a Book with a certain title", consider to create extension methods for this. This way you can concatenate them as any LINQ method. The extension method creates the query, it will not execute them, so this won't be a performance penalty.
Advantages of the extension method: simpler to understand, reusable, easier to test and easier to change.
If you are not familiar with extension methods, read Extension Methods Demystified
// you need to convert them to IQueryable with the AsQueryable() method, if not
// you get an error since the receiver asks for an IQueryable
// and a ICollection was given
public static IQueryable<Book> GetBooks(this IQueryable<Library> libraries)
{
return libraries.SelectMany(library => library.AsQueryable().Books);
}
public static IQueryable<Book> GetBooks(this IQueryable<City> cities)
{
return cities.SelectMany(city => city.Libraries.AsQueryable().GetBooks());
}
Usage:
Get all states that have a book by Karl Marx:
string author = "Karl Marx";
var statesWithCommunistBooks = dbContext.States.
.Where(state => state.GetBooks()
.Select(book => book.Author)
.Contains(author));
Get all Cities without a bible:
string title = "Bible";
var citiesWithoutBibles = dbContext.Cities
.Where(city => !city.GetBooks()
.Select(book => book.Title)
.Contains(title));
Because you extended your classes with method GetBooks(), it is as if States and Cities have Books. You've seen the reusability above. Changes can be easy, if for instance you extend your database such, that Cities have BookStores. GetBooks can check the libraries and the BookStores. Your change will be in one place. Users of GetBooks(), won't have to change.

Number of results in DbSet<TEntity>

I was going through some sample code about DBEntities and DbContext. Is there any limit on number of rows the DbSet pulls from the database? In the below code sample, lets say there is a DbSet<History> history or DbSet<Logs> logs, when a dbcontext is created, will dbcontext.logs or dbcontext.history has all the logs present in the database? If its so, what if the tables have millions of rows. Doesnt it hit the performance when during linq or any udpates and saving the context?
public virtual DbSet<Course> Courses { get; set; }
public virtual DbSet<Standard> Standards { get; set; }
public virtual DbSet<Student> Students { get; set; }
public virtual DbSet<StudentAddress> StudentAddresses { get; set; }
using (var context = await _contextFactory.CreateContext())
{
context.History.Add(history);
context.SaveChanges();
}

Entity framework doesn’t need to pull any rows to do an insert which is what the Add() method and SaveChanged() does. It should do what you would do in SQL to add a row to the table in question.

As in your example, it doesn't "explode"
The following line basically only adds an item to an empty change tracker:
context.History.Add(history);
If you would execute
context.History.ToList()
Then the query is executed as a "select * from History" and you will definitely hit a performance issue if it contains millions of rows.
Key point is that EF is "smart enough" to not load everything in memory as a whole set. You could attach a profiler (or enable EF logging) to see the actual queries being executed. Fiddle around a bit with it to gain some experience.
If you expand the set, for example with a debugger, then basically you don't apply any filter and will retrieve the whole set. With the misuse of navigation properties you would even be able to load your whole database in memory.
The subtitle difference is within the difference between the IQueryable and other IEnumerable-like interfaces.
While the object is still only IQueryable the actual query is still to be executed and can be expand with filters. As I said; once you are starting to enumerate, the actual query is executed and hence, an unfiltered dbset will return all rows in a table.
Also note the mentioned linq methods
.Skip
And
.Take
There are several more, like group, join, where, etc.

You have to realise that a DbSet<Student> does not represent your collection of Students, it represents the Students table in your database. This means that you can query for sequences of properties of Students.
If desired, you can query the complete sequence, but that will lead to performance problems, if not memory problems.
Therefore, if you ask for Student data, you have to keep in mind what you will be using of the fetched data: don't Select properties that you already know the value of, don't Select items that you don't plan to use.
An example: a database with Schools and Students, with a one-to-many relation, every School has zero or more Students, every Student attends exactly one School:
class School
{
public int Id {get; set;}
public string Name {get; set;}
...
// every School has zero or more Students (one-to-many)
public virtual ICollection<Student> Students {get; set;}
}
class Student
{
public int Id {get; set;}
public string Name {get; set;}
...
// Every Student attends exactly one School, using foreign key:
public int SchoolId {get; set;}
public virtual School School {get; set;}
}
In entity framework the columns of the tables are represented by non-virtual properties. The virtual properties represent the relations between the tables (one-to-many, many-to-many, ...)
Don't do the following!
public IEnumerable<School> GetSchoolByLocation(string city)
{
return mySchoolWithItsStudents = dbContext.Schools
.Where(school => school.City == city)
.Include(school => school.Students)
.ToList();
}
Why not? This looks like perfect code, doesn't it?
Well maybe you are fetching way more data than your caller will use:
var mySchoolId = GetSchoolByLocation("Oxford")
.Where(school => schoolStreet == "Main Street")
.Select(school => school.Id)
.FirstOrDefault();
What a waste, to first fetch all Oxford schools, and then keep only this one!
Furthermore: you get the school with all its Students, and all you use if the Id of the school?
Try to return IQueryable<...> as long as possible, and let your caller decide what to do with the returned data.
Maybe he wants to do ToList, or Count, or FirstOrDefault. Maybe he only wants the Id and the Name. As long as you don't know that, don't make the decision for him, it only makes your code less reusable.
Always use Select to select the properties, and select only the data you actually plan to use. Only use Include if you plan to update the included data.
var schools = dbContext.Schools.Where(school => ...)
// Keep only the Schools that you actually plan to use:
.Select(school => new
{
// only select the properties that you plan to use
Id = school.Id,
Name = school.Name,
...
// Only the Students you plan to use:
Students = school.Students.Where(student => ...)
.Select(student => new
{
// Again, only the properties you plan to use
Id = student.Id,
Name = student.Name,
// no need for the foreign key: you already know the value
// SchoolId = student.SchoolId,
}),
});
Finally, if you want access to all Students to show them, but you don't want to fetch all million Students at once, consider fetching them by Page. Remember the primary key of the last item of the last fetched page, and use `.Where(item => item.Id > lastFetchedPrimaryKey).Take(pageSize) to get the next page, until there are no more pages.
This way, you might ask for 50 students, while you'll only display 25 of them, but at least you don't have all million students in memory. Fetching the next page is fairly fast, because there is already an index on primary key, and fetched items are already ordered by primary key.

EF Core Select one Entity filtered by related entities

I have a question and I'm struggling to find the answer.
I have 3 Entity classes with a relation like that (simplified):
class Article {
public int id
public string Name
public int ArticleStandard Id
public int CompanyId
public ArticleStandard ArticleStandardNavigation
public Company CompanyNavigation
}
class ArticleStandard {
public int id
public ICollection<Article> Articles
}
class Company {
public in id
public ICollection<Article> Articles
}
So the relation is 1 company can have many articles, 1 standardarticle can have many articles. They are all setup as DBSets in the DBContext.
Given is the Id of the ArticleStandard. I now want to get all companies who have this standard article through the article entity.
The TSQL I would like to "produce" is:
select c.* from Company c
inner join Article a on c.ID = a.CompanyId
inner join ArticleStandard on a.ArticleStandard = arts.id
Where arts.Id = 1
Which gives the result I want to have.
I tried a lot around and I don't want to post all my trials to work around the issue, that could make it work with just Include and ThenInclude. But I dont want to get all Articles and ArticleStandard. If I only select the company the Includes are ignored:
https://learn.microsoft.com/en-us/ef/core/querying/related-data#ignored-includes
var vtp = context.Company.Include(a => a.Article).ThenInclude(ars => ars.ArticleStandardNavigation).ToList();
Here I also struggle to even use a where clause on the included entities.
I read about that issue here:
How to add where clause to ThenInclude
But I couldnt make it really work, especially only retrieving the company and not the other entities.
I know I could load all and just take the companies out. That would work. But I want to reduce the amount of data sent and also keep the query as one.
Any hints? I'm still pretty inexperienced with EF Core and LinQ is also still sometimes confusing for me.
If you need more information or can point me to a similar problem (I found some kind of similar cases, but couldn't use them correctly) I would be very thankful.
Thank you for your time.

Well, forget about SQL and joins. In EF (Core) targeting LINQ queries you use the navigation properties to access the related data. In the context of the query, the entities represent the database table records and navigations - joins and related table records.
For reference navigation properties you use simple criteria like:
Where(entity => entity.Reference.SomeProperty == someValue)
and for collection navigation properties, usually Any with the criteria needed, e.g.:
Where(entity => entity.Collection.Any(related => related.SomeProperty == someValue))
in other words, I want records having at least one related record with this value.
Applying the above rules to your model, the equivalent query would be like this:
var query = db.Companies
.Where(c => c.Articles.Any(a => a.ArticleStandardNavigation.id == 1));
The generated SQL most likely will not be the same as the one you would write by hand (in general we cannot control the SQL generation of the ORM), but the result should be.
Btw, there is no need to append Navigation to the navigation property names. The default (and more intuitive) convention is to use the class name, e.g.
public class Article
{
// ...
public ArticleStandard ArticleStandard { get; set; }
public Company Company { get; set; }
}

Entity framework map entity to multiple tables in cascade

I'm facing a problem using EF.
I have the following situation:
From this database schema i'd like to generate the following entity by merge tables data:
// Purchases
public class Purchase
{
//Fields related to Purchases
public int IdPurchase { get; set; }
public string CodPurchase { get; set; }
public int IdCustomer { get; set; }
public decimal Total { get; set; }
//Fields related to Customers table
public string CodCustomer { get; protected set; }
public string CompanyTitle { get; protected set; }
public string CodType { get; protected set; }
//Fields related to CustomersType table
public string DescrType { get; protected set; }
}
As you can see, in my context i don't want 3 separated entities for each table. I want a single one with the fields related to all tables. All fields of Customers and CustomersType tables must be readonly (so i've set the relative setters protected) and the others must be editables so that EF can track changes. In particular, i'd like to have the ability to change the "IdCustomer" field and let EF to automatically update "CodCustomer", "CompanyTitle", "DescrType"....and so on by doing cross table select.
To do that, i wrote this configuration class:
internal class PurchaseConfiguration : EntityTypeConfiguration<Purchase>
{
public PurchaseConfiguration(string schema = "dbo")
{
ToTable(schema + ".Purchases");
HasKey(x => x.IdPurchase);
Property(x => x.IdPurchase).HasColumnName("IdPurchase").IsRequired().HasDatabaseGeneratedOption(DatabaseGeneratedOption.Identity);
Property(x => x.IdCustomer).HasColumnName("IdCustomer").IsRequired();
Property(x => x.Total).HasColumnName("Total").IsRequired().HasPrecision(19, 4);
Map(mc =>
{
mc.Properties(n => new
{
n.CodCustomer,
n.CompanyTitle,
n.CodType
});
mc.ToTable("Customers");
});
Map(mc =>
{
mc.Properties(n => new
{
n.DescrType,
});
mc.ToTable("CustomersType");
});
}
}
I've tested it but it doesn't work as expected. I always get this message:
Properties for type 'Purchase' can only be mapped once. The non-key
property 'CodCustomer' is mapped more than once. Ensure the
Properties method specifies each non-key property only once.
Maybe there's something wrong or i forget something (for example the join fields of Map<> that i don't know where to specify them).
How can i accomplish in the correct way this task?
I don't want to have "Customers" and "CustomersType" DBSets in my context.
Is there a way to avoid it?
I even thought to add into the "IdCustomer" setter a custom query to update manually "Customers" and "CustomersType" related fields, but i don't want to do that for 2 reasons:
I don't have any DbConnection avaiable into the "Purchases" class, so i can't create a DbCommand to read data from DB.
I want entity class to be persistent-ignorant
EF seems to be a powerfull tool that can do these sort of things and i don't want to reinvent the wheel by writing custom procedures.
I've uploaded the example C# source and the tables CREATE scripts (MS SQLServer) here.
All entities are autogenerated by the "EF reverse POCO generator" T4 template (the T4 template is disabled, to activate it set CustomTool = TextTemplatingFileGenerator).
Do not forget to update the ConnectionString in the app.config.
Thanks in advance.

Not the right mapping
I'm afraid the bad news is that this mapping is not possible with this table structure. What you're trying to achieve here is known as entity splitting. However, entity splitting requires 1:1 associations, because sets of records in the involved tables represent one entity. With this mapping, you can't have a Customer belonging to more than one Purchase. That would mean that you could modify multiple Purchase entities by modifying a Customer property of only one of them.
Maybe the news isn't that bad, because I think you actually want to have 1-n associations. But then you can't have these "flattened" properties in Purchase.
As an alternative you could create delegated properties like so:
public string CodCustomer
{
get { return this.Customer.CodCustomer; }
set { this.Customer.CodCustomer = value; }
}
You'd have to Include() Customers and CustomersTypes when you fetch Purchases.
Another alternative is to use a tool like AutoMapper to map Purchase to a DTO type having the flattened properties.
But what does the exception tell me?
You map the Purchase entity to the Purchases table. But you don't specify which properties you want to map to this table. So EF assumes that all properties should be mapped to it. So that's the first (implicit) mapping of CodCustomer. The second one is the one in the mc.ToTable statement. (EF only reports the first problem.)
To fix this, you should add a mapping statement for the left-over Purchase properties:
Map(mc =>
{
mc.Properties(n => new
{
n.IdPurchase,
n.CodPurchase,
n.IdCustomer,
n.Total,
});
mc.ToTable("Purchases");
});
By the way, you should also remove the mapping configuration classes of Customer and CustomersType, they're redundant.
But, as said, the database schema doesn't match the required structure. If you try to save a Purchase you will get a foreign key constraint exception. This is because EF expects the following table structure:
Where the columns IdPurchase in Customer and CustomersType are both primary key and foreign key to Purchase. I don't think this is what you had in mind when designing the database.

Loading virtual properties

I am retrieving data from the database (with Entity Framework) using DTOs:
IQueryable<User> users = repository.ListFiltered<User>(n => n.Active);
var usersDTO = from user in users
select new UserAccountDTO{
UserId = user.Id,
UserName = user.Name,
InstitutionName = user.Institution.Name,
InstitutionCountry = user.Institution.Country.Name
};
I'm doing this because the user entity, and the institution entity, have a lot of data i don't need right now, so i don't want to retrieve it from the database.
But the problem is that i don't like the code, i would like to split the code and concatenate the selects, is there anyway to do this?
I'd like to arrive to something like:
users.LoadUserData().LoadInstitutionData();
What do you say? is it possible?

The problem is what LoadUserData() and LoadInstitutionData() should look like. Let's say that the end product of this method chain should be an IEnumerable<UserAccountDTO>. The last method in fluent syntax always determines the output. So LoadInstitutionData() should return the required IEnumerable. Clear.
But what about the input? I see two alternatives:
The input is IEnumerable<User> (or IQueryable<User>). OK, that would mean that LoadInstitutionData() can't do anything else than the code you already have. And that's not a solution, because you don't like the code. Moreover, the method requires that user.Institution and the Country must be loaded or lazy loadable, which is hard to express in any form of code contract.
The input is IEnumerable<UserAccountDTO> in which LoadUserData has set the direct user properties and that LoadInstitutionData should replenish with Institution data. Now the question is: how should LoadInstitutionData do that? Anyhow, what was done in one query now will take at least two queries, maybe even 1 + n.
More complex alternatives could consist of passing and composing expressions, but surely you'll be jumping from the frying pan into the fire, reinventing LINQ's extension methods, or AutoMapper, or...
Wait, AutoMapper.
Did you know that AutoMapper could do this for you in a way that may appeal to you?
All you need is a class
class UserAccountDTO
{
public int Id { get; set; }
public string Name { get; set; }
public string InstitutionName { get; set; }
public string InstitutionCountryName { get; set; } // Notice the Name part!
};
A mapping
AutoMapper.Mapper.CreateMap<User, UserAccountDTO>();
A using:
using AutoMapper.QueryableExtensions;
And a concise syntax:
var usersDTO = users.Project().To<UserAccountDTO>();
(And AutoMapper from NuGet, of course).
Automapper will resolve a property like InstitutionCountryName as user.Institution.Country.Name and the result of the projection is one expression that is translated into one SQL statement with joins and all. That's code I do like.