We Keep Coding

How can I know if related data is included when using eager loading in Entity Framework?

I am working on a project that is using Repository pattern. We are using Entity Framework to retrieve data from the database. To load related data we use eager loading which means that we have to explicitly include related data.
Say that we want to have all blogs and their related posts we would have to write:
_dbContext.Blogs.Include(b => b.Posts)
The disadvantages with this is that you can only see which related data is loaded in the repository class.
I have tried to shown an example below. Here the problem occurs in the class BlogViewModel.cs because you try to access blogs.Posts, but posts is not included in the original query. So if you are trying to get related data, one would have to check the origin of query and see if it is included.
Repository.cs:
public IEnumerable<BlogDbModel> GetBlogs()
{
return _dbContext.Blogs.ToList();
}
Service.cs:
public IEnumerable<BlogViewModel> GetBlogs()
{
return _repository.Select(x => new BlogViewModel(x));
}
BlogViewModel.cs:
public class BlogViewModel
{
public BlogViewModel(BlogDbModel blogDbModel)
{
Name = blogDbModel.Name;
Posts = blogDbModel.Posts;
}
public string Name { get; set; }
public IEnumerable<Posts> Posts { get; set; }
}
Does anyone have a smart solution to this? Is it possible give an compilation error if you are trying to access data that is not included?
Two options that I have thought about are:
Use lazy-loading instead
Always return an IQueryable from the repository
Thanks

No you dont have any option to give compilation error on this context. Data either way are loaded in the run-time.
LazyLoading
Assuming your displaying Blogs, Posts, Comments in view, So you need to load everything when the page load.
When you enable lazyload and map properties that not been included in the data layer query it will load related entities
public BlogViewModel(BlogDbModel blogDbModel)
{
Name = blogDbModel.Name;
Posts = blogDbModel.Posts;
}
If you enable lazyload and do this post will be fetched using lazyload feature. What if post contains a reference to User or any other object it will load those too.
When you need everything at first place its not recommended to use lazyload because it will additional round-trip to database.
IQueryable from repository
I would say IQueryable to a viewmodel is leaky abstraction. Ideally your data layer should be performing these. Providing IQuerable<BlogDbModel> to Viewmodel is like developer able to fetch 1Blogs=>Post=>User=>Accounts1 anything related from this.
IMHO I would write data layer methods which are needed for the view.
LazyloadingEnabled = false;
public IEnumerable<BlogDbModel> GetBlogs()
{
return _dbContext.Blogs.ToList();
}
public IEnumerable<BlogDbModel> GetBlogsAndPosts()
{
return _dbContext.Blogs.Include("Posts").ToList();
}
//This will fetch only what is needed (You can customize to get columns what is needed)
public IEnumerable<NewCustomDTO> GetBlogsAndPostCount()
{
return _dbContext.Blogs.Select(x=> NewCustomDTO
{
BlogName = x.BlogName,
Count = x.Posts.Count(),
});
}

How can I implement "Soft Deletes" with "Entity Framework Core" (aka EF7)?

I'm trying to implement a "Soft Delete" using EF7. My Item table has a field named IsDeleted of type bit. All of the examples that I see around SO and elsewhere are using something like this:
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<Item>().Map(m => m.Requires("IsDeleted").HasValue(false));
}
but Map() is no longer a method of ModelBuilder.
EDIT: Let me clarify. I'm mostly only interested in reading the data right now. I want EF to automatically filter out all records in my Item table where IsDeleted == 1 (or true). I do not want to require an && x.IsDeleted == false at the end of every query.

It's 2021, and it occurred to me to add a more modern, standard, built-in solution that pertains to current versions of EF Core.
With global query filters you can ensure that certain filters are always applied to certain entities. And you can define your soft deletion properties via an interface, which facilitates programmatically adding the filter to all relevant entities. See:
...
public interface ISoftDeletable
{
public string DeletedBy { get; }
public DateTime? DeletedAt { get; }
}
...
// Call it from DbContext.OnModelCreating()
private static void ConfigureSoftDeleteFilter(ModelBuilder builder)
{
foreach (var softDeletableTypeBuilder in builder.Model.GetEntityTypes()
.Where(x => typeof(ISoftDeletable).IsAssignableFrom(x.ClrType)))
{
var parameter = Expression.Parameter(softDeletableTypeBuilder.ClrType, "p");
softDeletableTypeBuilder.SetQueryFilter(
Expression.Lambda(
Expression.Equal(
Expression.Property(parameter, nameof(ISoftDeletable.DeletedAt)),
Expression.Constant(null)),
parameter)
);
}
}
Then, to make sure this flag is used during deletion instead of hard deletion (alternative to e.g. repositories setting the flag instead of deleting the entity):
public override Task<int> SaveChangesAsync(bool acceptAllChangesOnSuccess, CancellationToken cancellationToken = default)
{
foreach (var entry in ChangeTracker.Entries<ISoftDeletable>())
{
switch (entry.State)
{
case EntityState.Deleted:
// Override removal. Unchanged is better than Modified, because the latter flags ALL properties for update.
// With Unchanged, the change tracker will pick up on the freshly changed properties and save them.
entry.State = EntityState.Unchanged;
entry.Property(nameof(ISoftDeletable.DeletedBy)).CurrentValue = _currentUser.UserId;
entry.Property(nameof(ISoftDeletable.DeletedAt)).CurrentValue = _dateTime.Now;
break;
}
}
return base.SaveChangesAsync(acceptAllChangesOnSuccess, cancellationToken);
}
Caveat 1: Cascade Delete Timing
One crucial aspect is to take into account the cascade deletion of related entities, and either disable cascade delete, or understand and control the cascade delete timing behavior of EF Core. The default value of the CascadeDeleteTiming setting is CascadeTiming.Immediate, which causes EF Core to immediately flag all navigation properties of the 'deleted' entity as EntityState.Deleted, and reverting the EntityState.Deleted state only on the root entity won't revert it on the navigation properties. So if you have navigation properties which don't use soft deletion, and you want to avoid them being deleted, you must handle their change tracker state too (instead of just handling it for e.g. ISoftDeletable entities), or change the CascadeDeleteTiming setting as shown below.
The same is true for owned types used on the soft-deleted entities. With the default deletion cascade timing EF Core also flags these owned types as 'deleted', and in case they are set as Required/non-nullable, you will encounter SQL update failures when trying to save the soft-deleted entities.
public ApplicationDbContext(DbContextOptions<ApplicationDbContext> options) : base(options)
{
ChangeTracker.CascadeDeleteTiming = CascadeTiming.OnSaveChanges;
}
Caveat 2: Effect on other root entities
If you define a global query filter this way, EF Core will diligently hide all other entities that reference a soft-deleted entity.
For example if you've soft-deleted a Partner entity, and you have Order entities where each of them references a partner through a (required) navigation property, then, when you retrieve the list of orders and you include the partner, all orders that reference a soft-deleted Partner will be missing from the list.
This behavior is discussed at the bottom of the documentation page.
Sadly, the global query filters as of EF Core 5 don't provide an option to limit them to root entities, or to disable only one of the filters. The only available option is to use the IgnoreQueryFilters() method, which disables ALL filters. And since the IgnoreQueryFilters() method takes an IQueryable and also returns an IQueryable, you cannot use this method to transparently disable the filter inside your DbContext class for an exposed DbSet.
Though, one important detail is that this occurs only if you Include() the given navigation property while querying. And there is an interesting solution for getting a result set that has query filters applied to certain entities but doesn't have them applied to other entities, relying on a lesser known feature of EF, relational fixup. Basically, you load a list of EntityA that has navigation property EntityB (without including EntityB). And then you separately load the list of EntityB, using IgnoreQueryFilters(). What happens is that EF automatically sets the EntityB navigation property on EntityA to the loaded EntityB instances. This way the query filter was applied to EntityA itself, but wasn't applied to the EntityB navigational property, so you can see EntityAs even with soft-deleted EntityBs. See this answer on another question. (Of course this has performance implications, and you still can't encapsulate it in DbContext.)

Disclaimer: I'm the owner of the project Entity Framework Plus
As you will see in #Adem link, our library supports query filtering.
You can easily enable/disable a global/instance filter
QueryFilterManager.Filter<Item>(q => q.Where(x => !x.IsDeleted));
Wiki: EF Query Filter
Edit: Answer sub question
Care to explain how this works behind the scene?
Firstly, you can either initialize filter globally or by instance
// Filter by global configuration
QueryFilterManager.Filter<Customer>(q => q.Where(x => x.IsActive));
var ctx = new EntitiesContext();
// TIP: You can also add this line in EntitiesContext constructor instead
QueryFilterManager.InitilizeGlobalFilter(ctx);
// Filter by instance configuration
var ctx = new EntitiesContext();
ctx.Filter<Post>(MyEnum.EnumValue, q => q.Where(x => !x.IsSoftDeleted)).Disable();
Under the hood, the library will loop on every DbSet of the context and checks if a filter can be applied to the generic type.
In this case, the library will filter the original/filtered query from the DbSet using the filter then modify the current internal query for the new filtered query.
In summary, we changed some DbSet internal value to use the filtered query.
The code is FREE & Open Source if you want to learn about how it works.
Edit: Answer sub question
#jonathan will this filter included navigation collections too?
For EF Core, it's not supported yet since Interceptor is not available yet. But starting from EF Core 2.x, the EF Team has implemented Global query filters which should allow this.

If you can migrate to EF Core 2.0 you can use
Model-level query filters
https://learn.microsoft.com/en-us/ef/core/what-is-new/index
If you use EF Core 1.0
You can make some trick with available EF Core features:
Inheritance
https://learn.microsoft.com/en-us/aspnet/core/data/ef-mvc/inheritance
Shadow properties
https://learn.microsoft.com/en-us/ef/core/modeling/shadow-properties
public class Attachment : AttachmentBase
{}
public abstract class AttachmentBase
{
public const string StatePropertyName = "state";
public Guid Id { get; set; }
}
public enum AttachmentState
{
Available,
Deleted
}
public class AttachmentsDbContext : DbContext
{
public AttachmentsDbContext(DbContextOptions options)
: base(options)
{
}
public DbSet<Attachment> Attachments { get; set; }
public override Task<int> SaveChangesAsync(CancellationToken cancellationToken = default(CancellationToken))
{
IEnumerable<EntityEntry<Attachment>> softDeletedAttachments = ChangeTracker.Entries<Attachment>().Where(entry => entry.State == EntityState.Deleted);
foreach (EntityEntry<Attachment> softDeletedAttachment in softDeletedAttachments)
{
softDeletedAttachment.State = EntityState.Modified;
softDeletedAttachment.Property<int>(AttachmentBase.StatePropertyName).CurrentValue = (int)AttachmentState.Deleted;
}
return base.SaveChangesAsync(cancellationToken);
}
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<AttachmentBase>()
.HasDiscriminator<int>(AttachmentBase.StatePropertyName)
.HasValue<Attachment>((int)AttachmentState.Available);
modelBuilder.Entity<AttachmentBase>().Property<int>(AttachmentBase.StatePropertyName).Metadata.IsReadOnlyAfterSave = false;
modelBuilder.Entity<Attachment>()
.ToTable("available_attachment");
modelBuilder.Entity<AttachmentBase>()
.ToTable("attachment");
base.OnModelCreating(modelBuilder);
}
}

Limit collection to retrieve only recent entries for readonly entity

The User entity can have thousands of UserOperations. Sometimes I don't want to retrieve (for readonly entity) all of them but only "the recent 10 OR not completed".
public class SimpleForm
{
public class User : EntityBase
{
// ...
private ISet<UserOperation> _recentOperations = new HashedSet<UserOperation>();
public virtual ISet<UserOperation> RecentOperations { get { return _recentOperations; } set { _recentOperations = value; } }
}
}
So how can I specify it? I think I could use mapping overrides?
I understand I could make this with a seperate query but can it be done by entity mapping?
Also I wonder if there is a possibility to do the some for non-readonly entity where I can modify the collection of operations?
UPDATE
I tried to use
DateTime dateTime = (DateTime.UtcNow - TimeSpan.FromDays(15));
mapping.HasMany(x => x.RecentOperations)
.Where(x => x.EndedAt == null || x.EndedAt < dateTime);
but it says "Unable to convert expression to SQL".
I replaced it with
mapping.HasMany(x => x.RecentOperations)
.Where(x => x.EndedAt == null);
and now it throws null reference exception inside
в FluentNHibernate.Utils.ExpressionToSql.Convert(Object value)
в FluentNHibernate.Utils.ExpressionToSql.Convert(ConstantExpression expression)
в FluentNHibernate.Utils.ExpressionToSql.Convert[T](Expression`1 expression, UnaryExpression body)

There are 2 general ways how to filter mapped collections.
The first is a bit rigid, fixed, in a mapping defined where="" clause:
6.2. Mapping a Collection (...in fluent .Where(bool expr) or .Where(Sql statement string)
The second and maybe really suitable in this scenario, is dynamic version called filter:
18.1. NHibernate filters
NHibernate adds the ability to pre-define filter criteria and attach those filters at both a class and a collection level. A filter criteria is the ability to define a restriction clause very similiar to the existing "where" attribute available on the class and various collection elements. Except these filter conditions can be parameterized. The application can then make the decision at runtime whether given filters should be enabled and what their parameter values should be. Filters can be used like database views, but parameterized inside the application....
The implementation in fluent would look like this:
public class RecentFilter : FilterDefinition
{
public RecentFilter()
{
WithName("RecentFilter")
.WithCondition("( :EndedAtDate IS NULL OR EndedAt < :EndedAtDate )")
.AddParameter("EndedAtDate",NHibernate.NHibernateUtil.DateTime);
}
}
this is the filter, and here is its usage in a fluent mapping:
mapping
.HasMany(x => x.RecentOperations)
...
.ApplyFilter<RecentFilter>();
In runtime, we can turn filter on/off on the ISession level:
session.EnableFilter("RecentFilter")
.SetParameter("EndedAtDate",DateTime.Now.AddDays(-15));
See also:
property filter with fluent nHibernate automapping
Syntax to define a NHibernate Filter with Fluent Nhibernate?
Is it possible to use NHibernate Filters to filter through references?

Loading virtual properties

I am retrieving data from the database (with Entity Framework) using DTOs:
IQueryable<User> users = repository.ListFiltered<User>(n => n.Active);
var usersDTO = from user in users
select new UserAccountDTO{
UserId = user.Id,
UserName = user.Name,
InstitutionName = user.Institution.Name,
InstitutionCountry = user.Institution.Country.Name
};
I'm doing this because the user entity, and the institution entity, have a lot of data i don't need right now, so i don't want to retrieve it from the database.
But the problem is that i don't like the code, i would like to split the code and concatenate the selects, is there anyway to do this?
I'd like to arrive to something like:
users.LoadUserData().LoadInstitutionData();
What do you say? is it possible?

The problem is what LoadUserData() and LoadInstitutionData() should look like. Let's say that the end product of this method chain should be an IEnumerable<UserAccountDTO>. The last method in fluent syntax always determines the output. So LoadInstitutionData() should return the required IEnumerable. Clear.
But what about the input? I see two alternatives:
The input is IEnumerable<User> (or IQueryable<User>). OK, that would mean that LoadInstitutionData() can't do anything else than the code you already have. And that's not a solution, because you don't like the code. Moreover, the method requires that user.Institution and the Country must be loaded or lazy loadable, which is hard to express in any form of code contract.
The input is IEnumerable<UserAccountDTO> in which LoadUserData has set the direct user properties and that LoadInstitutionData should replenish with Institution data. Now the question is: how should LoadInstitutionData do that? Anyhow, what was done in one query now will take at least two queries, maybe even 1 + n.
More complex alternatives could consist of passing and composing expressions, but surely you'll be jumping from the frying pan into the fire, reinventing LINQ's extension methods, or AutoMapper, or...
Wait, AutoMapper.
Did you know that AutoMapper could do this for you in a way that may appeal to you?
All you need is a class
class UserAccountDTO
{
public int Id { get; set; }
public string Name { get; set; }
public string InstitutionName { get; set; }
public string InstitutionCountryName { get; set; } // Notice the Name part!
};
A mapping
AutoMapper.Mapper.CreateMap<User, UserAccountDTO>();
A using:
using AutoMapper.QueryableExtensions;
And a concise syntax:
var usersDTO = users.Project().To<UserAccountDTO>();
(And AutoMapper from NuGet, of course).
Automapper will resolve a property like InstitutionCountryName as user.Institution.Country.Name and the result of the projection is one expression that is translated into one SQL statement with joins and all. That's code I do like.

Virtual Navigation Properties and Multi-Tenancy

I have a standard DbContext with code like the following:
public DbSet<Interest> Interests { get; set; }
public DbSet<User> Users { get; set; }
I've recently implemented multi-tenancy by creating a TenantContext that contains the following:
private readonly DbContext _dbContext;
private readonly Tenant _tenant;
public TenantContext(Tenant tenant)
: base("name=DefaultConnection") {
this._tenant = tenant;
this._dbContext = new DbContext();
}
public IQueryable<User> Users { get { return FilterTenant(_dbContext.Users); } }
public IQueryable<Interest> Interests { get { return FilterTenant(_dbContext.Interests); } }
private IQueryable<T> FilterTenant<T>(IQueryable<T> values) where T : class, ITenantData
{
return values.Where(x => x.TenantId == _tenant.TenantId);
}
So far, this has been working great. Whenever any of my services creates a new TenantContext, all queries directly off of that context are filtered through this FilterTenant method that guarantees I'm only returning tenant-relevant entities.
The problem that I'm encountering is my usage of navigation properties that do not take this into account:
using (var db = CreateContext()) // new TenantContext
{
return db.Users.
Include(u => u.Interests).FirstOrDefault(s => s.UserId == userId);
}
This query pulls up the tenant-specific Users, but then the Include() statement pulls in Interests for that user only - but across all tenants. So if a user has Interests across multiple Tenants, I get all of the user's Interests with the above query.
My User model has the following:
public int UserId { get; set; }
public int TenantId { get; set; }
public virtual ICollection<Interest> Interests { get; set; }
Is there any way that I can somehow modify these navigation properties to perform tenant-specific queries? Or should I go and tear out all navigation properties in favor of handwritten code?
The second option scares me because a lot of queries have nested Includes. Any input here would be fantastic.

As far as I know, there's no other way than to either use reflection or query the properties by hand.
So in your IQueryable<T> FilterTenant<T>(IQueryable<T> values) method, you'll have to inspect your type T for properties that implement your ITenantData interface.
Then you're still not there, as the properties of your root entity (User in this case) may be entities themselves, or lists of entities (think Invoice.InvoiceLines[].Item.Categories[]).
For each of the properties you found by doing this, you'll have to write a Where() clause that filters those properties.
Or you can hand-code it per property.
These checks should at least happen when creating and editing entities. You'll want to check that navigation properties referenced by an ID property (e.g. ContactModel.AddressID) that get posted to your repository (for example from an MVC site) are accessible for the currently logged on tenant. This is your mass assignment protection, which ensures a malicious user can't craft a request that would otherwise link an entity to which he has permissions (a Contact he is creating or editing) to one Address of another tenant, simply by posting a random or known AddressID.
If you trust this system, you only have to check the TenantID of the root entity when reading, because given the checks when creating and updating, all child entities are accessible for the tenant if the root entity is accessible.
Because of your description you do need to filter child entities. An example for hand-coding your example, using the technique explained found here:
public class UserRepository
{
// ctor injects _dbContext and _tenantId
public IQueryable<User> GetUsers()
{
var user = _dbContext.Users.Where(u => u.TenantId == _tenantId)
.Select(u => new User
{
Interests = u.Interests.Where(u =>
u.TenantId == _tenantId),
Other = u.Other,
};
}
}
}
But as you see, you'll have to map every property of User like that.

Just wanted to offer an alternative approach to implementing multi-tenancy, which is working really well in a current project, using EF5 and SQL 2012. Basic design is (bear with me here...):
Every table in the database has a column (ClientSid binary, default constraint = SUSER_SID()) and is never queried directly, only ever via a dedicated view
Each view is a direct select over the table with WHERE (ClientSid = SUSER_SID()) but doesn't select the ClientSid (effectively exposing the interface of the table)
EF5 model is mapped to the VIEW, not the TABLE
The connection string is varied based on the context of the tenant (user / client whatever multi-tenant partition requirement may be)
That's pretty much it - though it might be useful to share. I know it's not a direct answer to your question, but this has resulted in basically zero custom code in the C# area.