We Keep Coding

How to compute object property in database with C# code?

As an example let's say my database has a table with thousands of ships with every ship potentially having thousands of passengers as a navigation property:
public DbSet<Ship> Ship { get; set; }
public DbSet<Passenger> Passenger { get; set; }
public class Ship
{
public List<Passenger> passengers { get; set; }
//properties omitted for example
}
public class Passenger
{
//properties omitted for example
}
The example use case is that someone is fetching all ships per API and would like to know for each ship whether it is empty (0 passengers), so the returned JSON will contain a list of ships each with a bool whether it is empty.
My current code seems very inefficient (including all passengers just to determine if a ship is empty):
List<Ship> ships = dbContext.Ship
.Include(x => x.passengers)
.ToList();
and later when the ships are serialized to JSON:
jsonShip.isEmpty = !ship.passengers.Any();
I would like a more performant (and not bloated) alternative to including all passengers. What options do I have?
I have looked at computed columns but they only seem to support sql as string. If possible I would like to stay in the C# code world, so for example having a property which is set correctly by being automatically woven in the SQL query would be optimal.

Create a Data Transfer Object for Ship that reflects the shape of your JSON result, like -
public class ShipDto
{
public int Id { get; set; }
public string Name { get; set; }
public bool IsEmpty { get; set; }
}
Then use projection in your query -
var ships = dbCtx.Ships
.Select(p => new ShipDto
{
Id = p.Id,
Name = p.Name,
IsEmpty = !p.Passengers.Any()
})
.ToList();
Usually, APIs need to produce responses of various shapes and DTOs give you well defined models to represent the shape of your API response. Domain entities are not always suitable for this.
If your domain entity (Ship) has a lot of properties, then copying all those properties in the .Select() method might be cumbersome. You can use AutoMapper to map them for you. AutoMapper has a ProjectTo<T>() method that can generate the SQL and return the projected result. For example, you can achieve the above result with a mapping configuration -
CreateMap<Ship, ShipDto>()
.ForMember(d => d.IsEmpty, opt => opt.MapFrom(s => !s.Passengers.Any()));
and a query -
var ships = Mapper.ProjectTo<ShipDto>(dbCtx.Ships).ToList();
assuming all other properties in ShipDto are named similar as in Ship entity.
EDIT :
If you don't want a DTO model -
you can add a NotMapped property in Ship model -
public class Ship
{
public int Id { get; set; }
public string Name { get; set; }
[NotMapped]
public bool IsEmpty { get; set; }
public List<Passenger> passengers { get; set; }
}
and then do the query like -
var ships = dbCtx.Ships
.Select(p => new Ship
{
Id = p.Id,
Name = p.Name,
IsEmpty = !p.Passengers.Any()
})
.ToList();
You can return an anonymous type -
var ships = dbCtx.Ships
.Select(p => new
{
Id = p.Id,
Name = p.Name,
IsEmpty = !p.Passengers.Any()
})
.ToList();

If I understand your intention correctly...
One way is to store the number of passengers inside each Ship entity. This can work well if you use Domain Driven Design, treat the Ship as an aggregate root, and only add or remove passengers through methods exposed on the given Ship entity, e.g. RegisterPassenger() / RemovePassenger(). Inside these methods, increment or decrement the passenger number along with adding or removing the passenger.
Then, obviously, you can query the Ships dbset with a PassengerCount < 0 projection to the bool you need. And, again obviously, it won't even touch the Passengers table.
In traditional anemic domain ASP.NET systems this sort of data redundancy might be a bit more risky, because properties are usually publicly mutable, and you have multiple services that 'massage' the entities, which is a potential source of data integrity loss.

EntityFramework(Repository Pattern, Data Validation, Dto's)

I've been having some rough time sorting out how to create an Restful API with EntityFramework. The problem is mainly because this API should be used in a long time forward and i want to be it as maintainable and clean, with good performance. Enough of that, let's get into the problem.
Disclamer:
Because of companypolicy and can't post too much here, but i will try to address the problem the best way possible. There will also just be snippets of code, and may not be valid. I'm also fairly new to C# and as a JuniorD i have never touched an API before.. And excuse my english, this is my second language.
Every model derives from the BaseModel class
public class BaseModel
{
[Required]
public Guid CompanyId { get; set; }
public DateTime CreatedDateTime { get; set; }
[StringLength(100)]
public string CreatedBy { get; set; }
public DateTime ChangedDateTime { get; set; }
[StringLength(100)]
public string ChangedBy { get; set; }
public bool IsActive { get; set; } = true;
public bool IsDeleted { get; set; }
}
public class Carrier : BaseModel
{
[DatabaseGenerated(DatabaseGeneratedOption.Identity)]
[Key]
public Guid CarrierId { get; set; }
public int CarrierNumber { get; set; }
[StringLength(100)]
public string CarrierName { get; set; }
[StringLength(100)]
public string AddressLine { get; set; }
public Guid? PostOfficeId { get; set; }
public PostOffice PostOffice { get; set; }
public Guid? CountryId { get; set; }
public Country Country { get; set; }
public List<CustomerCarrierLink> CustomerCarrierLinks { get; set; }
}
Every repository derives from Repository and has it's own interface.
public class CarrierRepository : Repository<Carrier>, ICarrierRepository
{
public CarrierRepository(CompanyMasterDataContext context, UnitOfWork unitOfWork) : base(context, unitOfWork) { }
#region Helpers
public override ObjectRequestResult<Carrier> Validate(Carrier carrier, List<string> errorMessages)
{
var errorMessages = new List<string>();
if(carrier != null)
{
var carrierIdentifier = (carrier.CarrierName ?? carrier.CarrierNumber.ToString()) ?? carrier.CarrierGLN;
if (string.IsNullOrWhiteSpace(carrier.CarrierName))
{
errorMessages.Add($"Carrier({carrierIdentifier}): Carrier name is null/empty");
}
}
else
{
errorMessages.Add("Carrier: Cannot validate null value.");
}
return CreateObjectResultFromList(errorMessages, carrier); // nonsense
}
}
UnitOfWork derives from class UnitOfWorkDiscoverySet, this class initializes the repository properties using reflection and also contains a method (OnBeforeChildEntityProcessed) for calling every OnBeforeChildEntityProcessed.
public class UnitOfWork : UnitOfWorkDiscoverySet
{
public UnitOfWork(CompanyMasterDataContext context)
: base(context){}
public CarrierRepository Carriers { get; internal set; }
public PostOfficeRepository PostOffices { get; internal set; }
public CustomerCarrierLinkRepository CustomerCarrierLinks { get; internal set; }
}
public IRepository<Entity> where Entity : BaseModel
{
ObjectRequestResult<Entity> Add(Entity entity);
ObjectRequestResult<Entity> Update(Entity entity);
ObjectRequestResult<Entity> Delete(Entity entity);
ObjectRequestResult<Entity> Validate(Entity entity);
Entity GetById(Guid id);
Guid GetEntityId(Entity entity);
}
public abstract class Repository<Entity> : IRepository<Entity> where Entity : BaseModel
{
protected CompanyMasterDataContext _context;
protected UnitOfWork _unitOfWork;
public Repository(CompanyMasterDataContext context, UnitOfWork unitOfWork)
{
_context = context;
_unitOfWork = unitOfWork;
}
public ObjectRequestResult<Entity> Add(Entity entity)
{
if (!EntityExist(GetEntityId(entity)))
{
try
{
var validationResult = Validate(entity);
if (validationResult.IsSucceeded)
{
_context.Add(entity);
_context.UpdateEntitiesByBaseModel(entity);
_context.SaveChanges();
return new ObjectRequestResult<Entity>()
{
ResultCode = ResultCode.Succceeded,
ResultObject = entity,
Message = OBJECT_ADDED
};
}
return validationResult;
}
catch (Exception exception)
{
return new ObjectRequestResult<Entity>()
{
ResultCode = ResultCode.Failed,
ResultObject = entity,
Message = OBJECT_NOT_ADDED,
ErrorMessages = new List<string>()
{
exception?.Message,
exception?.InnerException?.Message
}
};
}
}
return Update(entity);
}
public virtual ObjectRequestResult Validate(Entity entity)
{
if(entity != null)
{
if(!CompanyExist(entity.CompanyId))
{
return EntitySentNoCompanyIdNotValid(entity); // nonsense
}
}
return EntitySentWasNullBadValidation(entity); // nonsense
}
}
DbContext class:
public class CompanyMasterDataContext : DbContext {
public DbSet<PostOffice> PostOffices { get; set; }
public DbSet<Carrier> Carriers { get; set; }
public DbSet<Company> Companies { get; set; }
public DbSet<CustomerCarrierLink> CustomerCarrierLinks { get; set; }
public UnitOfWork Unit { get; internal set; }
public CompanyMasterDataContext(DbContextOptions<CompanyMasterDataContext> options)
: base(options)
{
Unit = new UnitOfWork(this);
}
public void UpdateEntitiesByBaseModel(BaseModel baseModel)
{
foreach (var entry in ChangeTracker.Entries())
{
switch (entry.State)
{
case EntityState.Added:
entry.CurrentValues["CompanyId"] = baseModel.CompanyId;
entry.CurrentValues["CreatedDateTime"] = DateTime.Now;
entry.CurrentValues["CreatedBy"] = baseModel.CreatedBy;
entry.CurrentValues["IsDeleted"] = false;
entry.CurrentValues["IsActive"] = true;
Unit.OnBeforeChildEntityProcessed(entry.Entity, enumEntityProcessState.Add);
break;
case EntityState.Deleted:
entry.State = EntityState.Modified;
entry.CurrentValues["ChangedDateTime"] = DateTime.Now;
entry.CurrentValues["ChangedBy"] = baseModel.ChangedBy;
entry.CurrentValues["IsDeleted"] = true;
Unit.OnBeforeChildEntityProcessed(entry.Entity, enumEntityProcessState.Delete);
break;
case EntityState.Modified:
if (entry.Entity != null && entry.Entity.GetType() != typeof(Company))
entry.CurrentValues["CompanyId"] = baseModel.CompanyId;
entry.CurrentValues["ChangedDateTime"] = DateTime.Now;
entry.CurrentValues["ChangedBy"] = baseModel.ChangedBy;
Unit.OnBeforeChildEntityProcessed(entry.Entity, enumEntityProcessState.Update);
break;
}
}
}
}
DiscoveryClass:
public abstract class UnitOfWorkDiscoverySet
{
private Dictionary<Type, object> Repositories { get; set; }
private CompanyMasterDataContext _context;
public UnitOfWorkDiscoverySet(CompanyMasterDataContext context)
{
_context = context;
InitializeSets();
}
private void InitializeSets()
{
var discoverySetType = GetType();
var discoverySetProperties = discoverySetType.GetProperties();
Repositories = new Dictionary<Type, object>();
foreach (var child in discoverySetProperties)
{
var childType = child.PropertyType;
var repositoryType = childType.GetInterfaces()
.Where( i => i.IsGenericType && i.GetGenericTypeDefinition() == typeof(IRepository<>))
.FirstOrDefault();
if (repositoryType != null)
{
var repositoryModel = repositoryType.GenericTypeArguments.FirstOrDefault();
if (repositoryModel != null)
{
if (repositoryModel.IsSubclassOf(typeof(BaseModel)))
{
var repository = InitializeProperty(child); //var repository = child.GetValue(this);
if (repository != null)
{
Repositories.Add(repositoryModel, repository);
}
}
}
}
}
}
private object InitializeProperty(PropertyInfo property)
{
if(property != null)
{
var instance = Activator.CreateInstance(property.PropertyType, new object[] {
_context, this
});
if(instance != null)
{
property.SetValue(this, instance);
return instance;
}
}
return null;
}
public void OnBeforeChildEntityProcessed(object childObject, enumEntityProcessState processState)
{
if(childObject != null)
{
var repository = GetRepositoryByObject(childObject);
var parameters = new object[] { childObject, processState };
InvokeRepositoryMethod(repository, "OnBeforeEntityProcessed", parameters);
}
}
public void ValidateChildren<Entity>(Entity entity, List<string> errorMessages) where Entity : BaseModel
{
var children = BaseModelUpdater.GetChildModels(entity);
if(children != null)
{
foreach(var child in children)
{
if(child != null)
{
if (child.GetType() == typeof(IEnumerable<>))
{
var list = (IEnumerable<object>) child;
if(list != null)
{
foreach (var childInList in list)
{
ValidateChild(childInList, errorMessages);
}
}
}
ValidateChild(child, errorMessages);
}
}
}
}
public void ValidateChild(object childObject, List<string> errorMessages)
{
if(childObject != null)
{
var repository = GetRepositoryByObject(childObject);
var parameters = new object[] { childObject, errorMessages };
InvokeRepositoryMethod(repository, "Validate", parameters);
}
}
public void InvokeRepositoryMethod(object repository, string methodName, object[] parameters)
{
if (repository != null)
{
var methodToInvoke = repository.GetType().GetMethod(methodName);
var methods = repository.GetType().GetMethods().Where(x => x.Name == methodName);
if (methodToInvoke != null)
{
methodToInvoke.Invoke(repository, parameters);
}
}
}
public object GetRepositoryByObject(object objectForRepository)
{
return Repositories?[objectForRepository.GetType()];
}
public object GetObject<Entity>(Type type, Entity entity) where Entity : BaseModel
{
var childObjects = BaseModelUpdater.GetChildModels(entity);
foreach (var childObject in childObjects)
{
if (childObject.GetType().FullName == type.FullName)
{
return childObject;
}
}
return null;
}
}
}
The problem:
I want to validate data in every model and the childmodels properties/list, know you might say this can be done using attributes, but the validation can be fairly complex and i prefer to separate this in it's own space.
The way i have attacked this problem is by using reflection from the UnitDiscoverySet class, here i can find every child of the Entity i'm trying to process and search for an appropriate Repository containg the UnitOfWork. This works by all means, just need some more work and cleanup, but for some reason i feel like this is the cheating/wrong way to attack the problem, and i'm also not getting the compile-time errors + reflection comes at a cost.
I could validate children of the entity in the entity repository, but then i would be repeating myself all over the place, and this solution don't seem right either.
I don't want this solution to depend to heavy on entityframework, since it's not given that we will use this forever.
This solution also heavly depends on the UpdateEntitiesByBaseModel method inside The DbContext. So it only changes the fields that should be changed.
Not sure that i addressed this problem as good as thought i would, but i appreciate every contribution that will lead me down the right path. Thank you!
Solution(Edit):
I ended up only using the navigation properties for GET operations, and excluded it for insert operations. Made everything more flexible and faster, this way i didn't need to use the EF Tracker, which made an insert operation of 5000 entities from a 13 minute operation, down to 14.3 seconds.

This question would probably be best asked in CodeReview rather than SO which is geared towards specific code-related questions. You can ask 10 different developers and get 10 different answers. :)
Reflection definitely does have a cost, and it's not something I like to use very much at all.
I don't want this solution to depend to heavy on entityframework,
since it's not given that we will use this forever.
This is a fairly common theme I see in applications and frameworks that development teams I work with try to cope with when working with ORMs. To me, abstracting EF from the solution is like trying to abstract away parts of .Net. There is literally no point because you forfeit access to much of the flexibility and capability that Entity Framework offers. It leads to more, complex code to deal with things that EF can do natively leaving room for bugs as you re-invent the wheel, or leaving gaps that later have to be worked around. You either trust it, or you shouldn't use it.
I could validate children of the entity in the entity repository, but
then i would be repeating myself all over the place, and this solution
don't seem right either.
This is actually the pattern I've come to advocate for with projects. Many people are against the Repository pattern, but it is a great pattern to serve as a boundary for the domain for testing purposes. (No need to set up in-memory databases or attempt to mock a DbContext/DbSets) However, IMO the Generic Repository pattern is an anti-pattern. It separates entities concerns from one another, however in many cases we are dealing with entity "graphs" not individual entity types. Rather than defining repositories per entity, I opt for something that is effectively a repository per controller. (With repositories for truly common entities such as lookups for example.) There are two reasons behind this:
Fewer dependency references to pass around / mock
Better serves SRP
Avoid pigeon-holing DB operations
The biggest problem I have with generic or per-entity repositories is that while they appear to be conforming to SRP (responsible for operations for a single Entity) I feel they violate it because SRP is about having only one reason to change. If an I have an Order entity and an Order repository I may have several areas of the application that load and interact with orders. Methods to interact with Order entities now are called in several different places, and that forms many potential reasons for methods to be adjusted. You end up either with complex, conditional code, or several very similar methods to serve specific scenarios. (Orders for listing orders, orders by customer, orders by store, etc.) When it comes to validating entities, this is commonly done in context to the entire graph so it makes sense to centralize this in code related to the graph rather than individual entities. This goes for generic base operations like Add/Update/Delete. 80% of the time this works and saves effort, but that remaining 20% either has to get shoe-horned into the pattern, leading to inefficient and/or error-prone code, or work-arounds. K.I.S.S. should always trump D.N.R.Y when it comes to software design. Consolidation into base classes and the like is an optimization that should be done as the code evolves when "identical" functionality is identified. When it's done up-front as an architecture decision, I consider this premature optimization which creates obstacles for development, performance issues, and bugs when "similar" but not "identical" behaviour is grouped together leading to conditional code creeping in for edge cases.
So instead of an OrderRepository to serve orders, if I have something like a ManageOrderController, I will have a ManageOrderRepository to serve it.
For instance I like to use DDD styled methods for managing entities where my repositories play a part in the construction since they are privy to the data domain and can validate/retrieve related entities. So a typical repository would have:
IQueryable<TEntity> GetTEntities()
IQueryable<TEntity> GetTEntityById(id)
IQueryable<TRelatedEntity> GetTRelatedEntities()
TEntity CreateTEntity({all required properties/references})
void DeleteTEntity(entity)
TChildEntity CreateTChildEntity(TEntity, {all required properties/references})
Retrieval methods, including a "By ID" as it's a common scenario, return IQueryable so that callers can control how the data is consumed. This negates the need to try and abstract the Linq capabilities that EF can leverage so callers can apply filters, perform paging, sorting, then consume the data how they need. (Select, Any, etc.) The repository enforces core rules such as IsActive, and tenancy/authorization checks. This serves as a boundary for tests since mocks just have to return List<TEntity>.AsQueryable() or wrapped with a async-friendly collection type. (Unit-testing .ToListAsync() using an in-memory) The repository also serves as a go-to place for retrieving any related entities via whatever criteria is applicable. This can be viewed as potential duplication but changes to this repository will only be needed when the controller/view/area of the application needs to change. Common stuff like lookups would be pulled via their own repository. This cuts down on the need for lots of individual repository dependencies. Each area takes care of itself so changes/optimizations here don't need to consider or impact other areas of the application.
"Create" methods manage the rules around creating and associating entities to the Context to ensure that entities are always created in a minimally complete and valid state. This is where validation comes into play. Any value that is not null-able gets passed in, along with FKs (keys or references) needed to ensure that if SaveChanges() was the next call after Create, the entity would be valid.
"Delete" methods similarly come in here to manage validating data state/authorization, and apply consistent behaviour. (hard vs. soft delete, auditing, etc.)
I don't use "Update" methods. Updates are handled by DDD methods on the entity itself. Controllers define the unit of work, use the repository to retrieve the entity(ies), call the entity methods, then commit the unit of work. Validation can be done at the entity level, or via a Validator class.
In any case, that is just a summary of one approach out of 10+ you may get out there, and hopefully highlights some things to consider with whatever approach you take. My emphasis when it comes to working with EF would be:
Keep it simple. (K.I.S.S. > D.N.R.Y)
Leverage what EF has to offer rather than attempting to hide it.
Complex, clever code ultimately leads to more code, and more code leads to bugs, performance issues, and makes it difficult to adjust for requirements you haven't thought of up-front. (Leading to more complexity, more conditional paths, and more headaches) Frameworks like EF have been tested, optimized, and vetted so leverage them.

Entity Framework core select causes too many queries

I have the following method which is meant to build me up a single object instance, where its properties are built via recursively calling the same method:
public ChannelObjectModel GetChannelObject(Guid id, Guid crmId)
{
var result = (from channelObject in _channelObjectRepository.Get(x => x.Id == id)
select new ChannelObjectModel
{
Id = channelObject.Id,
Name = channelObject.Name,
ChannelId = channelObject.ChannelId,
ParentObjectId = channelObject.ParentObjectId,
TypeId = channelObject.TypeId,
ChannelObjectType = channelObject.ChannelObjectTypeId.HasValue ? GetChannelObject(channelObject.ChannelObjectTypeId.Value, crmId) : null,
ChannelObjectSearchType = channelObject.ChannelObjectSearchTypeId.HasValue ? GetChannelObject(channelObject.ChannelObjectSearchTypeId.Value, crmId) : null,
ChannelObjectSupportingObject = channelObject.ChannelObjectSupportingObjectId.HasValue ? GetChannelObject(channelObject.ChannelObjectSupportingObjectId.Value, crmId) : null,
Mapping = _channelObjectMappingRepository.Get().Where(mapping => mapping.ChannelObjectId == channelObject.Id && mapping.CrmId == crmId).Select(mapping => new ChannelObjectMappingModel
{
CrmObjectId = mapping.CrmObjectId
}).ToList(),
Fields = _channelObjectRepository.Get().Where(x => x.ParentObjectId == id).Select(field => GetChannelObject(field.Id, crmId)).ToList()
}
);
return result.First();
}
public class ChannelObjectModel
{
public ChannelObjectModel()
{
Mapping = new List<ChannelObjectMappingModel>();
Fields = new List<ChannelObjectModel>();
}
public Guid Id { get; set; }
public Guid ChannelId { get; set; }
public string Name { get; set; }
public List<ChannelObjectMappingModel> Mapping { get; set; }
public int TypeId { get; set; }
public Guid? ParentObjectId { get; set; }
public ChannelObjectModel ParentObject { get; set; }
public List<ChannelObjectModel> Fields { get; set; }
public Guid? ChannelObjectTypeId { get; set; }
public ChannelObjectModel ChannelObjectType { get; set; }
public Guid? ChannelObjectSearchTypeId { get; set; }
public ChannelObjectModel ChannelObjectSearchType { get; set; }
public Guid? ChannelObjectSupportingObjectId { get; set; }
public ChannelObjectModel ChannelObjectSupportingObject { get; set; }
}
this is connecting to a SQL database using Entity Framework Core 2.1.1
Whilst it technically works, it causes loads of database queries to be made - I realise its because of the ToList() and First() etc. calls.
However because of the nature of the object, I can make one huge IQueryable<anonymous> object with a from.... select new {...} and call First on it, but the code was over 300 lines long going just 5 tiers deep in the hierarchy, so I am trying to replace it with something like the code above, which is much cleaner, albeit much slower..
ChannelObjectType, ChannelObjectSearchType, ChannelObjectSupportingObject
Are all ChannelObjectModel instances and Fields is a list of ChannelObjectModel instances.
The query takes about 30 seconds to execute currently, which is far too slow and it is on a small localhost database too, so it will only get worse with a larger number of db records, and generates a lot of database calls when I run it.
The 300+ lines code generates a lot less queries and is reasonably quick, but is obviously horrible, horrible code (which I didn't write!)
Can anyone suggest a way I can recursively build up an object in a similar way to the above method, but drastically cut the number of database calls so it's quicker?

I work with EF6, not Core, but as far as I know, same things apply here.
First of all, move this function to your repository, so that all calls share the DbContext instance.
Secondly, use Include on your DbSet on properties to eager load them:
ctx.DbSet<ChannelObjectModel>()
.Include(x => x.Fields)
.Include(x => x.Mapping)
.Include(x => x.ParentObject)
...
Good practice is to make this a function of context (or extension method) called for example BuildChannelObject() and it should return the IQueryable - just the includes.
Then you can start the recursive part:
public ChannelObjectModel GetChannelObjectModel(Guid id)
{
var set = ctx.BuildChannelObject(); // ctx is this
var channelModel = set.FirstOrDefault(x => x.Id == id); // this loads the first level
LoadRecursive(channelModel, set);
return channelModel;
}
private void LoadRecursive(ChannelObjectModel c, IQueryable<ChannelObjectModel> set)
{
if(c == null)
return; // recursion end condition
c.ParentObject = set.FirstOrDefault(x => x.Id == c?.ParentObject.Id);
// all other properties
LoadRecursive(c.ParentObject, set);
// all other properties
}
If all this code uses the same instance of DbContext, it should be quite fast. If not, you can use another trick:
ctx.DbSet<ChannelObjectModel>().BuildChannelObjectModel().Load();
This loads all objects to memory cache of your DbContext. Unfortunately, it dies with context instance, but it makes those recursive calls much faster, since no database trip is made.
If this is still to slow, you can add AsNoTracking() as last instruction of BuildChannelObjectModel().
If this is still to slow, just implement application wide memory cache of those objects and use that instead of querying database everytime - this works great if your app is a service that can have long startup, but then work fast.
Whole another approach is to enable lazy loading by marking navigation properties as virtual - but remember that returned type will be derived type anonymous proxy, not your original ChannelObjectModel! Also, properties will load only as long you don't dispose the context - after that you get an exception. To load all properties with the context and then return complete object is also a little bit tricky - easiest (but not the best!) way to do it to serialize the object to JSON (remember about circural references) before returning it.
If that does not satisfy you, switch to nHibernate which I hear has application wide cache by default.

Entity Framework 6 Update Graph

What is the correct way to save a graph of objects whose state you don't know? By state I mean whether they are new or existing database entries that are being updated.
For instance, if I have:
public class Person
{
public int Id { get; set; }
public int Name { get; set; }
public virtual ICollection<Automobile> Automobiles { get; set; }
}
public class Automobile
{
public int Id { get; set; }
public int Name { get; set; }
public short Seats { get; set; }
public virtual ICollection<MaintenanceRecord> MaintenanceRecords { get; set ;}
public virtual Person Person { get; set; }
}
public class MaintenanceRecord
{
public int Id { get; set; }
public int AutomobileId { get; set; }
public DateTime DatePerformed { get; set; }
public virtual Automobile Automobile{ get; set; }
}
I'm editing models, similar to these objects above, and then passing those models into the data layer to save, where for this instance I happen to be using entity framework. So I'm translating these models into POCO entities internal to the DAL.
It appears that unless my models have a state indicating whether they are new or updated, I have quite a bit of work to do to "Save" the changes. I have to first select the Person entity, update it, then match any existing Automobiles and update those and add any new, then for each automobile check for any new or updated maintenance records.
Is there a faster/easier way of doing this? It's possible I can keep track of the Model state, which I guess would be helpful with this, but it would mean changes to code outside of the data layer which i would prefer to avoid. I'm just hoping there is a pattern of usage out there that I can follow for updates like this.

I ran into this issue a while back and have been following this thread on the EF Codeplex site. https://entityframework.codeplex.com/workitem/864
Seems like it is being considered for the next release, I'm assuming EF 7, which apparently is a pretty large internal overhaul of EF. This may be worth checking out... http://www.nuget.org/packages/RefactorThis.GraphDiff/
Back when I was working on this I found another EF post on SO, and someone had an example of how to do this manually. At the time I decided to do it manually, not sure why, GraphDiff looks pretty cool. Here is an example of what I did.
public async Task<IHttpActionResult> PutAsync([FromBody] WellEntityModel model)
{
try
{
if (!ModelState.IsValid)
{
return BadRequest(ModelState);
}
var kne = TheContext.Companies.First();
var entity = TheModelFactory.Create(model);
entity.DateUpdated = DateTime.Now;
var currentWell = TheContext.Wells.Find(model.Id);
// Update scalar/complex properties of parent
TheContext.Entry(currentWell).CurrentValues.SetValues(entity);
//We don't pass back the company so need to attached the associated company... this is done after mapping the values to ensure its not null.
currentWell.Company = kne;
// Updated geometry - ARGHHH NOOOOOO check on this once in a while for a fix from EF-Team https://entityframework.codeplex.com/workitem/864
var geometryItemsInDb = currentWell.Geometries.ToList();
foreach (var geometryInDb in geometryItemsInDb)
{
// Is the geometry item still there?
var geometry = entity.Geometries.SingleOrDefault(i => i.Id == geometryInDb.Id);
if (geometry != null)
// Yes: Update scalar/complex properties of child
TheContext.Entry(geometryInDb).CurrentValues.SetValues(geometry);
else
// No: Delete it
TheContext.WellGeometryItems.Remove(geometryInDb);
}
foreach (var geometry in entity.Geometries)
{
// Is the child NOT in DB?
if (geometryItemsInDb.All(i => i.Id != geometry.Id))
// Yes: Add it as a new child
currentWell.Geometries.Add(geometry);
}
// Update Surveys
var surveyPointsInDb = currentWell.SurveyPoints.ToList();
foreach (var surveyInDb in surveyPointsInDb)
{
// Is the geometry item still there?
var survey = entity.SurveyPoints.SingleOrDefault(i => i.Id == surveyInDb.Id);
if (survey != null)
// Yes: Update scalar/complex properties of child
TheContext.Entry(surveyInDb).CurrentValues.SetValues(survey);
else
// No: Delete it
TheContext.WellSurveyPoints.Remove(surveyInDb);
}
foreach (var survey in entity.SurveyPoints)
{
// Is the child NOT in DB?
if (surveyPointsInDb.All(i => i.Id != survey.Id))
// Yes: Add it as a new child
currentWell.SurveyPoints.Add(survey);
}
// Update Temperatures - THIS IS A HUGE PAIN = HOPE EF is updated to handle updating disconnected graphs.
var temperaturesInDb = currentWell.Temperatures.ToList();
foreach (var tempInDb in temperaturesInDb)
{
// Is the geometry item still there?
var temperature = entity.Temperatures.SingleOrDefault(i => i.Id == tempInDb.Id);
if (temperature != null)
// Yes: Update scalar/complex properties of child
TheContext.Entry(tempInDb).CurrentValues.SetValues(temperature);
else
// No: Delete it
TheContext.WellTemperaturePoints.Remove(tempInDb);
}
foreach (var temps in entity.Temperatures)
{
// Is the child NOT in DB?
if (surveyPointsInDb.All(i => i.Id != temps.Id))
// Yes: Add it as a new child
currentWell.Temperatures.Add(temps);
}
await TheContext.SaveChangesAsync();
return Ok(model);
}
catch (Exception ex)
{
Trace.WriteLine(ex.Message);
}
return InternalServerError();
}

This is a huge pain to me too. I extracted the answer from #GetFuzzy to a more reusable method:
public void UpdateCollection<TCollection, TKey>(
DbContext context, IList<TCollection> databaseCollection,
IList<TCollection> detachedCollection,
Func<TCollection, TKey> keySelector) where TCollection: class where TKey: IEquatable<TKey>
{
var databaseCollectionClone = databaseCollection.ToArray();
foreach (var databaseItem in databaseCollectionClone)
{
var detachedItem = detachedCollection.SingleOrDefault(item => keySelector(item).Equals(keySelector(databaseItem)));
if (detachedItem != null)
{
context.Entry(databaseItem).CurrentValues.SetValues(detachedItem);
}
else
{
context.Set<TCollection>().Remove(databaseItem);
}
}
foreach (var detachedItem in detachedCollection)
{
if (databaseCollectionClone.All(item => keySelector(item).Equals(keySelector(detachedItem)) == false))
{
databaseCollection.Add(detachedItem);
}
}
}
With this method in place I can use it like this:
public void UpdateProduct(Product product)
{
...
var databaseProduct = productRepository.GetById(product.Id);
UpdateCollection(context, databaseProduct.Accessories, product.Accessories, productAccessory => productAcccessory.ProductAccessoryId);
UpdateCollection(context, databaseProduct.Categories, product.Categories, productCategory => productCategory.ProductCategoryId);
...
context.SubmitChanges();
}
However when the graph gets deeper, I have a feeling this will not be sufficient.

What your looking for is the Unit of Work pattern:
http://msdn.microsoft.com/en-us/magazine/dd882510.aspx
You can either track UoW on the client and pass it in with the DTO or have the server figure it out. Both the veritable DataSet and EF Entities have their own internal implementation of UoW. For something stand alone there is this framework, but I have never used it so have no feedback:
http://genericunitofworkandrepositories.codeplex.com/
Alternatively another option is to do real time updates with undo functionality, kind of like when you go into Gmail contacts and it saves the changes as you make them with the option to undo.

It depends HOW you are accomplishing adding/changing the entities.
I think you may be trying to do too much with an entity at any given time. Allowing editing and adding at the same time can get you into a situation where your not sure what is being done with the entity, especially in a disconnected scenario. You should only perform a single action on a single entity at a time, unless you are deleting entities. Does this seem monotonous, sure, but 99% of your users want a clean and easily understandable interface. Many time we end up making screens of our applications "god" screens where everything and anything can be done. Which 9/10 times isn't needed (YAGNI).
This way, when you edit a user, you know you are doing an update operation. If you are adding a new maintenance record, you know you are creating a new record that is attached to an automobile.
To summarize, you should limit how many operations you are making available for a single screen and make sure you provide some type of unique information for the entity so you can try to look up the entity to see if it exists.

I had the similar problem, and couldnt find my own solution. I think that problem is complex. Complete solution for updating graphs in disconected scenario with EF6 I find in extension method RefactoringThis.GraphDiff produced by Brent McKendric.
Exemple brings by author is:
using (var context = new TestDbContext())
{
// Update the company and state that the company 'owns' the collection Contacts.
context.UpdateGraph(company, map => map
.OwnedCollection(p => p.Contacts, with => with
.AssociatedCollection(p => p.AdvertisementOptions))
.OwnedCollection(p => p.Addresses)
);
context.SaveChanges();
}
See more at:
http://blog.brentmckendrick.com/introducing-graphdiff-for-entity-framework-code-first-allowing-automated-updates-of-a-graph-of-detached-entities/

How to Return Extra Data with IQueryable method?

I am using Entity Framework and Breeze. For an Entity, there is a bit of associated data I would like to provide with the entity. Getting this data is most efficiently done by querying the Entity table and joining to other tables; this query includes a group by sub-query.
I am attempting to tack this extra data on by adding it as a [NotMapped] field to the entity:
[NotMapped]
public string NotMappedField { get; set; }
So then I basically want to replace this webapi controller method
[HttpGet]
public IQueryable<MyObject> MyObjects()
{
return _contextProvider.Context.MyObjects;
}
With something like this:
public IQueryable<MyObject> MyObjectsWithExtraData()
{
var query = from o in _contextProvider.Context.MyObjects
// big complex query
select new MyObject
{
FieldA = o.FieldA,
FieldB = o.FieldB,
// all fields
NotMappedField = x.ResultFromComplexJoin
}
return query;
}
This gives me an error:
The entity or complex type 'MyObject' cannot be constructed in a LINQ to Entities query.
I've tried this a few ways and it seems to fight me both from the EF side and the Breeze side. I need to keep this as returning something like IQueryable so I can filter from the client through webapi because doing something like a ToList() here causes memory issues due to the dataset size.
So my question is - is there a best practices kind of way to accomplish what I am attempting or can anyone provide a solution?
Update:
I have found you can return extra data alongside of your entity and still have access to the entity as a queryable from Breeze:
public object MyObjectsWithExtraData()
{
var query = from o in _contextProvider.Context.MyObjects
// big complex query....
select new
{
theObject = MyObject,
NotMappedField = x.ResultFromComplexJoin
};
return query;
}
and then from the client breeze side you can do something like this:
var query = breeze.EntityQuery
.from("MyObjectsWithExtraData")
.where("theObject.FieldA", "Equals", 1)
.expand("theObject.SomeNavigationalProperty")
.orderBy("theObject.FieldB");
Still not exactly what I was looking for but it is actually pretty slick.

Take a look at the EntityQuery.withParameters method.
// client side
var q = EntityQuery.from("CustomersStartingWith")
.withParameters({ companyName: "C" });
// server side
[HttpGet]
public IQueryable<Customer> CustomersStartingWith(string companyName) {
var custs = ContextProvider.Context.Customers.Where(c => c.CompanyName.StartsWith(companyName));
return custs;
}
You can also mix and match a combination of regular query predicates with these custom parameters.

LINQ to entity can only construct pur "Data Transfert Object" : class containing only public properties with trivial getter and setter and without constructor.
See my answer to a similar question here : https://stackoverflow.com/a/21174654/3187237
I specify my answer
An Entity class can't be instanciated in a LINQ to Entities query.
If you want to construct similar (or almost similar) in the query you have to define an other class.
In your case you want to return object almost similar to your MyObject. So you have to define a class:
public class MyObjectExtended
{
public string FieldA { get; set; }
public string FieldB { get; set; }
// ... all other MyObjetc fields
public string ExtraFieldA { get; set; }
public string ExtraFieldB { get; set; }
}
Now, your service can return a IQueryable<MyObjectExtended>:
public IQueryable<MyObjectExtended> MyObjectsWithExtraData() {
var myQuery = from o in _contextProvider.Context.MyObjects
// big complex query....
select new MyObjectExtended {
FieldA = o.FieldA,
FieldB = o.FieldB,
//... all fields ...
ExtraFieldA = x.ResultFromComplexJoinA,
ExtraFieldB = x.ResultFromComplexJoinB
};
return myQuery;
}
I hope this is what you are looking for.