We Keep Coding

Problem with EF Core updating nested entities when using automapper

I am maintaining an application which uses EF Core to persist data to a SQL database.
I am trying to implement a new feature which requires me to retrieve an object from the database (Lets pretend its an order) manipulate it and some of the order lines which are attached to it and save it back into the database. Which wouldn't be a problem but I have inherited some of this code so need to try to stick to the existing way of doing things.
The basic process for data access is :
UI -> API -> Service -> Repository -> DataContext
The methods in the repo follow this pattern (Though I have simplified it for the purposes of this question)
public Order GetOrder(int id)
{
return _context.Orders.Include(o=>o.OrderLines).FirstOrDefault(x=>x.Id == id);
}
The service is where business logic and mapping to DTOs are applied, this is what the GetOrder method would look like :
public OrderDTO GetOrder(int id)
{
var ord = _repo.GetOrder(id);
return _mapper.Map<OrderDto>(ord);
}
So to retrieve and manipulate an order my code would look something like this
public void ManipulateAnOrder()
{
// Get the order DTO from the service
var order = _service.GetOrder(3);
// Manipulate the order
order.UpdatedBy = "Daneel Olivaw";
order.OrderLines.ForEach(ol=>ol.UpdatedBy = "Daneel Olivaw");
_service.SaveOrder(order);
}
And the method in the service which allows this to be saved back to the DB would look something like this:
public void SaveOrder(OrderDTO order)
{
// Get the original item from the database
var original = _repo.GetOrder(order.Id);
// Merge the original and the new DTO together
_mapper.Map(order, original);
_repo.Save(original);
}
Finally the repositories save method looks like this
public void Save(Order order){
_context.Update(order)
_context.SaveChanges();
}
The problem that I am encountering is using this method of mapping the Entities from the context into DTOs and back again causes the nested objects (in this instance the OrderLines) to be changed (or recreated) by AutoMapper in such a way that EF no longer recognises them as being the entities that it has just given to us.
This results in errors when updating along the lines of
InvalidOperationException the instance of ProductLine cannot be tracked because another instance with the same key value for {'Id'} is already being tracked.
Now to me, its not that there is ANOTHER instance of the object being tracked, its the same one, but I understand that the mapping process has broken that link and EF can no longer determine that they are the same object.
So, I have been looking for ways to rectify this, There are two ways that have jumped out at me as being promising,
the answer mentioned here EF & Automapper. Update nested collections
Automapper.Collection
Automapper.collection seems to be the better route, but I cant find a good working example of it in use, and the implementation that I have done doesn't seem to work.
So, I'm looking for advice from anyone who has either used automapper collections before successfully or anyone that has any suggestions as to how best to approach this.
Edit, I have knocked up a quick console app as an example, Note that when I say quick I mean... Horrible there is no DI or anything like that, I have done away with the repositories and services to keep it simple.
I have also left in a commented out mapper profile which does work, but isn't ideal.. You will see what I mean when you look at it.
Repo is here https://github.com/DavidDBD/AutomapperExample

Ok, after examining every scenario and counting on the fact that i did what you're trying to do in my previous project and it worked out of the box.
Updating your EntityFramework Core nuget packages to the latest stable version (3.1.8) solved the issue without modifying your code.

AutoMapper in fact "has broken that link" and the mapped entities you are trying to save are a set of new objects, not previously tracked by your DbContext. If the mapped entities were the same objects, you wouldn't have get this error.
In fact, it has nothing to do with AutoMapper and the mapping process, but how the DbContext is being used and how the entity states are being managed.
In your ManipulateAnOrder method after getting the mapped entities -
var order = _service.GetOrder(3);
your DbContext instance is still alive and at the repository layer it is tracking the entities you just retrieved, while you are modifying the mapped entities -
order.UpdatedBy = "Daneel Olivaw";
order.OrderLines.ForEach(ol=>ol.UpdatedBy = "Daneel Olivaw");
Then, when you are trying to save the modified entities -
_service.SaveOrder(order);
this mapped entities reach the repository layer and DbContext tries to add them to its tracking list, but finds that it already has entities of same type with same Ids in the list (the previously fetched ones). EF can track only one instance of a specific type with a specific key. Hence, the complaining message.
One way to solve this, is when fetching the Order, tell EF not to track it, like at your repository layer -
public Order GetOrder(int id, bool tracking = true) // optional parameter
{
if(!tracking)
{
return _context.Orders.Include(o=>o.OrderLines).AsNoTracking().FirstOrDefault(x=>x.Id == id);
}
return _context.Orders.Include(o=>o.OrderLines).FirstOrDefault(x=>x.Id == id);
}
(or you can add a separate method for handling NoTracking calls) and then at your Service layer -
var order = _repo.GetOrder(id, false); // for this operation tracking is false

Net Core: Generic Repository Primary Id Key Performance in Entity Framework

We are reviewing two different methods in generic repository patterns.
Currently, want to map primary keys to Ids. The purpose of this is to map to the Generic Repository Interface which utilizes Id. Two solutions are provided below.
What are performance implications of .FindPrimaryKey().Properties. Does it cause a schema lock on database table in trying to find the primary key? Does it cause any application slowness?
How does it compare in performance vs Partial Class Method Solution 2?
What option is better performance-wise?
Note: Architects demand the use of repository pattern at the workplace, so implementing it. Know there is debate surrounding this issue, but not my call.
Scaffolded Model Example:
namespace Datatest
{
public partial class Property
{
public int Property { get; set; }
public int DocumentId { get; set; }
public string Address { get; set; }
}
}
Sample Generic Base Repository for all tables:
public T Get(int id)
{
return Table.Find(id);
}
public async Task<T> GetAsync(int id)
{
return await Table.FindAsync(id);
}
public T Single(Expression<Func<T, bool>> predicate)
{
return All.Single(predicate);
}
public async Task<T> SingleAsync(Expression<Func<T, bool>> predicate)
{
return await All.SingleAsync(predicate);
}
public T FirstOrDefault(int id)
{
return All.FirstOrDefault(CreateEqualityExpressionForId(id));
}
Solution 1: FindPrimaryKey()
Generic Repository in C# Using Entity Framework
use EF FindPrimaryKey()
var idName = _context.Model.FindEntityType(typeof(TEntity))
.FindPrimaryKey().Properties.Single().Name;
Solution 2: Partial classes Mapping
Net Core: Create Generic Repository Interface Id Mapping for All Tables Auto Code Generation
public partial class Property: IEntity
{
[NotMapped]
public int Id { get => PropertyId; set => PropertyId = value; }
}

Regarding the first approach (using EF Core metadata services):
First, EF Core is ORM (Object Relational Mapper), with most important here is Mapper.
Second, it uses the so called code based model, which means all the mappings are provided by code and not the actual database (even though the model is created by reverse engineering of an existing database).
In simple words, EF Core creates at runtime a memory data structure containing the information (metadata) about classes and properties, and their mappings to database tables, columns and relationships. All that information is based on pure code model - the entity classes, conventions, data annotations and fluent configuration.
All EF Core runtime behaviors are based on that metadata model. EF Core uses it internally when building queries, mapping the query results to objects, linking navigation properties, generating create/update/delete commands and their order of execution, updating temporary FK property values after getting the real autogenerated principal key values etc.
Hence the metadata model and discovering services (methods) use optimized data structures and are (has to be) quite efficient. And again, no database operations are involved.
So the first approach is quite efficient. The performance impact of obtaining the PK property name via metadata service is negligible compared to actual query building, execution and materialization.
Also the performance of the first approach is similar to EF Core Find method which you are using in another method. Note that when calling Find method you just pass the PK value(s) and not the properties. So the method implementation should somehow know how to build the Where expression, right? And what it does internally is very similar to the suggested snippet.
Regarding the second approach:
It's simply not comparable because it doesn't work. It's possible to use base class/interface, but only if the actual property name is mapped - like all classes have Id property, and it's mapped to different column name in the database tables using [Column] data annotation or HasColumnName fluent API.
In your example, the Id property is [NotMapped] (ignored). Which means EF Core cannot map to the table column. The fact that your are mapping it to another property via code (property getter/setter) doesn't matter. EF Core is not a (de)compiler, it can't see your code, hence cannot translate a LINQ query using such properties to SQL.
Which in EF Core 2.x leads to either client evaluation (very inefficient, reading to whole table and applying the filter in memory), or exception if client evaluation is configured to do so. And in EF Core 3.0+ it will always be an exception.
So in case you don't remove properties like PropertyId and map the property Id (which would be hard with "database first" models), the second "approach" should be avoided. And even if you can map the actual Id property, all you'll save would be a few milliseconds. And again, when using Find you don't bother about performance, why bother with methods that uses the same (or similar) approach.

Using DbContext instance inside the entity class

Recently I've started learning Entity Framework Core and I'm curious if it's fine to use DbContext instance inside the entity class.
Sample code:
class User {
public int Id { get; set; }
public string Name { get; set; }
public ICollection<Order> Orders { get; set; }
public void LoadOrders() {
using (var context = new StoreContext()) {
Orders = context.Orders
.Where(x => x.UserId == Id)
.ToList();
}
}
}
User entity has a relation with Order class, both of them have appropriate tables in the DataBase created using migrations from Entity Framework.
The purpose of LoadOrders() method is simply to load related entities for current user when it's necessary.
Now I wanted to know if that's a valid approach?
Or maybe I should always load related entities at the same time when I'm loading the parent object? (E.g. .Include().ThenInclude())
Or maybe the code of LoadOrders() method should be located in some additional class like UserHelper that would be used along with the User entity.

You should avoid using an approach like this because the User will be loaded by one DbContext, while it's orders would be associated to another, disposed context. When you go to update a User, you would be facing errors or duplicate orders, or a messy business of reassociating the order (and other child entities) to contexts before saving. Down the road there will undoubtedly be confusion if orders are mapped to users and someone goes and writes code to .Include(x => x.Orders) If you completely detach related entities from EF and rely on load on demand, you lose out on a lot of the capability that EF gives you.
Issues like this typically stem from mixing up the scope/lifespan of entities vs. the scope of the context they are loaded from. For example loading entities in one method with a DbContext, returning them, then later deciding you want to access related entities but the DbContext was disposed. The simplest method I can recommend using is adopting POCO view models and ensuring that entities never exit the scope of their DbContext, only view models do. That way you can sculpt a view model structure to represent the data you need, then use entities and their references to populate those view models using .Select() without worrying about lazy loading or eager loading.
For instance:
using (var context = new StoreContext())
{
var userViewModel = context.Users.Where(x => x.UserId == userId)
.Select(x => new UserViewModel
{
UserId = x.UserId,
UserName = x.UserName,
Orders = x.Orders
.Where(o => o.IsActive)
.Select( o => new OrderViewModel
{
OrderId = o.OrderId,
OrderNumber = o.OrderNumber
Price = o.OrderItems.Sum(i => i.Price)
}).ToList()
}).SingleOrDefault();
return userViewModel;
}
Automapper can assist with mapping entities to view models. It's not a one-to-one tree structure map, but rather aligning the view model to represent the data the view needs, then filling that with the entity structure. You just need to be a bit careful to only pull data and supported aggregate methods from the entities because these will be passed to SQL, so no .Net or custom functions in the .Select. Let the view models accept raw values and provide alternate properties to perform formatting, or use .Select() to fetch anonymous types, get EF to materialize those into POCO instances with .ToList()/.Single()/etc. and then populate your view models from those using Linq2Object.
Working with entities on demand and view models / DTOs for the to-and-fro of data avoids a lot of hassle with entities. Done right, EF can pull this data extremely fast and it avoids performance pitfalls such as tripping lazy loads during serialization. It means that when you're done with the View Model you will need to re-load the entity to apply changes. It may seem to make more sense to simply use entities then have EF magically re-attach them and persist changes, but your view model will have all the info needed to quickly fetch that entity by ID if needed, and you will need to consider cases where the data may have changed between the time you first retrieved the entity, and the time you are prepared to alter it.

Using AutoMapper to load entities from the database?

Most of what I've read (e.g. from the author) indicates that AutoMapper should be used to map an an entity to a DTO. It should not load anything from the database.
But what if I have this:
public class Customer {
public int Id { get; set; }
public string Name { get; set; }
public virtual ICollection<Order> Orders { get; set; }
}
public class CustomerDto {
public int Id { get; set; }
public string Name { get; set; }
public IEnumerable<int> OrderIds { get; set; } // here is the problem
}
I need to map from DTO to entity (i.e. from CustomerDto to Customer), but first I must use that list of foreign keys to load corresponding entities from the database. AutoMapper can do that with a custom converter.
I agree that it doesn't feel right... but what are the alternatives? Sticking that logic into a controller, service, a repository, some manager class? All that seems to be pushing the logic somewhere else, in the same tier. And if I do that, I must also perform the mapping manually!
From a DDD perspective, the DTO should not be part of the domain. So AutoMapper is also not part of the domain, because it knows about that DTO. So AutoMapper is in the same tier as the controllers, services, etc.
So does it make sense to put the DTO-to-entity logic (which includes accessing the database, and possibly throwing exceptions) into an AutoMapper mapping?
EDIT
#ChrisSimon's great answer below explains from a DDD perspective why I shouldn't do this. From a non-DDD perspective, is there a compelling reason not to use AutoMapper to load from the db?

To start with, I'm going to summarise my understanding of Entities in DDD:
Entities can be created - often using a factory. This is the start of their life-cycle.
Entities can be mutated - have their state modified - by calling methods on the entity. This is how they progress through their lifecycle. By ensuring that the entity owns its own state, and can only have its state modified by calling its methods, the logic that controls the entity's state is all within the entity class, leading to cleaner separation of business logic and more maintainable systems.
Using Automapper to convert from a Dto to the entity means the entity is giving up ownership of its state. If the dto is in an invalid state and you map that directly onto the entity, the entity may end up in an invalid state - you have lost the value of making entities contain data + logic, which is the foundation of the DDD entity.
To make a suggestion as to how you should approach this, I'd ask - what is the operation you are trying to achieve? DDD encourages us not to think about CRUD operations, but to think about real business processes, and to model them on our entities. In this case it looks like you are linking Orders to the Customer entity.
In an Application Service I would have a method like:
void LinkOrdersToCustomer(CustomerDto dto)
{
using (var dbTxn = _txnFactory.NewTransaction())
{
var customer = _customerRepository.Get(dto.Id);
foreach (var orderId in dto.OrderIds)
{
var order = _orderRepository.Get(orderId);
customer.LinkToOrder(order);
}
dbTxn.Save();
}
}
Within the LinkToOrder method, I would have explicit logic that did things like:
Check that order is not null
Check that the customer's state permits adding the order (are they currently active? is their account closed? etc.)
Check that the order actually does belong to the customer (what would happen if the order referenced by orderId belonged to another customer?)
Ask the order (via a method on the order entity) if it is in a valid state to be added to a customer.
Only then would I add it to the Customers Order's collection.
This way, the application 'flow' and infrastructure management is contained within the application/services layer, but the true business logic is contained within the domain layer - within your entities.
If the above requirements are not relevant in your application, you may have other requirements. If not, then perhaps it is not necessary to go the route of DDD - while DDD has a lot to add, its overheads are generally only worth it in systems with lots of complex business logic.
This isn't related to the question you asked, but I'd also suggest you take a look at the modelling of Customer and Order. Are they both independent Aggregates? If so, modelling Customer as containing a collection of Order may lead to problems down the road - what happens when a customer has a million orders? Even if the collection is lazy loaded, you know at some point something will attempt to load it, and there goes your performance. There's some great reading about aggregate design here: http://dddcommunity.org/library/vernon_2011/ which recommends modelling references by Id rather than reference. In your case, you could have a collection of OrderIds, or possibly even a completely new entity to represent the link - CustomerOrderLink which would have two properties - CustomerId, and OrderId. Then none of your entities would have embedded collections.

Advice on my ASP.NET MVC, EF layer, and Automapper setup for a project

I'm trying to figure out the best approach to architecting this project. Basically, it's a "band" profile site. I'm using ASP.NET 4, EF, and Automapper (structuremap too, but that's not important). I'm running into performance issues and need advice on whether my approach is right or not (my guess is not). I'll focus on specific sections and provide stripped down examples.
I have a EntityFramework repository class that interacts directly onto the EF objects using LINQ:
[Pluggable("Repository")]
public class EntityDataRepository : IRepository
{
static EntityDataRepository()
{
// other mappings removed
// Data. objects are EF objects, mapping to my DTO classes
Mapper.CreateMap<Data.Event, Models.EventModel>();
Mapper.CreateMap<Data.Genre, Models.GenreModel>();
Mapper.CreateMap<Data.Band, Models.BandModel>();
}
public IEnumerable<BandModel> GetBandsByUser(Guid userId)
{
using (var ctx = new DbContext())
{
var user = GetCurrentUserModel(ctx, userId);
var efBands = from r in user.BandRelations
orderby r.Date
select r.Band;
return Mapper.Map<IEnumerable<Data.Band>, IEnumerable<Models.BandModel>>(efBands);
}
}
}
Bands have genres and events. Note that it maps the EF objects to my DTO object, and returns a list of them. It acts as a proxy to enable my controllers to invoke methods on to obtain the data that it needs (actual logic altered to show what I need):
namespace OpenGrooves.Web.Areas.Edit.Controllers
{
[Authorize]
public class MyBandsController : BaseController
{
public ActionResult ShowBands()
{
IEnumerable<BandModel> bands = repository.GetBandsByUser(loggedUserGuid).First();
return View(bands);
}
}
}
Finally, here's the BandModel class, which is mirroring the Band class entity in EF:
public class BandModel
{
// fluff and scalar properties removed
public IEnumerable<EventModel> Events { get; set; }
public IEnumerable<GenreModel> Genres { get; set; }
}
Basically, am I doing this right? In my EF to DTO classes, the Band EF entity has navigational properties, such as Genres and Events. The problem is, during the mapping that takes place in automapper, these list properties are being populated, especially if one of my proxy methods returns a list of BandModels. It seems to be invoking the Genres and Event EF queries for each record, which is a major performance killer obviously (at least 2 queries for Events and Genres are ran for each BandModel object returned).
Is it OK practice to use EF objects directly in my controllers, possibly even used as models for views?
Is there something I need to change in my mappings to enable lazy loading for these navigational properties (events, genres off a BandModel object)?
Thanks!!

Is it OK practice to use EF objects directly in my controllers, possibly even used as models for views?
Yes, Kinda.
This answer is subjective and depends on how you view your separation of concerns. Most MVC developers, including me, swear on view models. It decouples your data or domain classes from the presentation layer. This is awesome.
Some people don't like being awesome, including other languages and frameworks like every PHP MVC framework, Rails, and Django. Nobody can say these languages "do it wrong" but us .NET devs subscribe to a different paradigm.
Your second question is strange you say "is there something to enable lazy loading" right after you say lazy loading is happening. Care to explain?
Lazy loading is on by default in EF4.