We Keep Coding

audit-proof (immutable) database entries with Entity Framework Core (INSERT instead of UPDATE)

I am using Entity Framework Core 6.0 and I am currently looking for a way on how to persist data audit-proof (immutably).
Concretely, any intended modification of an existing database entry (SQL UPDATE statement) shall be translated into a new version/ revision of the existing entity (SQL INSERT statement) and, thus, result in a new database entry which supersedes the existing one (correlation among related entities is established by CorrelationId). Only the metadata of an entry are allowed to be updated, such that predecessor and successor relate to each other by ID (see PredecessorId and SucessorId, respectively).
Of course, this should not only work for entity relations (one-to-one, one-to-many, many-to-many) on first level but all along a complete object graph (arbitrary level).
I've added sample code for entity modeling for a contrived example to illustrate the idea:
Once I have placed an Order Order any subsequent changes of an ordered product Product must not have any influence on the order placed (not the names, not the prices). Instead a change in a product shall result in a new database entry with the same CorrelationId and metadata info on current and previous entry to relate to each other.
The example is contrived and the data modeling in the example should not be considered from a domain perspective.
I already looked into Entity Framework Plus and its audit functionality but from what I understood, it does only provide a separate table for the changes made to entities. However, an update on an entity results in a SQL UPDATE and in the example above a change in product price would affect the order already placed (Total would be different). The old state of the entity does not exist in its form any longer but rather one can reconstruct and comprehend who, when and how the entity changed with the audit information.
So, is there any clever solution on how to handle this in Entity Framework Core? I already tried myself in overriding the SaveChanges(Async) methods in the DbContext but I was not successful in preserving the entity relations:
case 1: principal changed, dependent(s) unchanged
-> new principal entity with reference to unchanged dependent(s)
case 2: principal unchanged, dependent(s) changed
-> no new principal entity with reference to new dependent(s)
case 3: principal changed, dependent(s) changed
-> new principal entity with reference to new dependent(s)
case 4: principal unchanged, dependent(s) unchanged
-> no action required
using System;
using System.Linq;
using System.Collections.Generic;
public abstract record BaseEntity
{
public long Id { get; init; }
public long? PredecessorId { get; private init; }
public long? SuccessorId { get; private init; }
public DateTimeOffset CreatedTimestamp { get; private init; }
public bool IsDeleted { get; private init; }
public DateTimeOffset DeletedTimestamp { get; private init; }
public Guid CorrelationId { get; init; }
}
public record Product : BaseEntity
{
public string Name { get; init; }
public double Price { get; init; }
}
public record Order : BaseEntity
{
public IReadOnlyCollection<Product> Products { get; init; }
public double Total => Products.Sum(o => o.Price);
}

Adding Complex Objects in Entity Framework

I'm setting up .NET Core API. My application is divided into 3 layers:
Data Access
Business Logic/Services
API
I'm having troubles added related objects with the API method. Let's say I have following classes.
public class Part
{
[Key]public int Id {get; set;}
public string Name {get; set;}
public string ICollection<PartAttribute> PartAttributes{get; set;}
}
public class PartAttribute
{
[Key]public int Id {get; set;}
public string Name {get; set;}
public string Value {get; set;}
}
And following methods for interaction with DB - context is EF DbContext:
public virtual void Add(T entity)
{
Context.Set<T>()
.Add(entity);
}
I'm having trouble adding Part that have already existing PartAttributes, if I send following JSON via API (assuming that following record in PartAttributes already exist)
{
"Name":"Example1",
"PartAttributes":[
{
"attributeId":1,
"name":"Color",
"value":"black"
}]
}
I'm getting a following exception: "Cannot insert explicit value for identity column in table 'PartAttributes' when IDENTITY_INSERT is set to OFF." - This leads me to conclusion that EF is not recognizing the existing record and tries to insert it as new one. This results in fail because of identity insert setting in SQL Server itself.
What I'd like to achieve is that the existing objects would be recognized and EF would not try to insert the existing attributes as new records in the database. What are the best practices to achieve that behaviour?

You need to take your primary key (which here looks to be attributeId) and load the record using the dbContext. Once loaded update the values and call SaveChanges() or SaveChangesAsync() on the context.
You can even have a method that does insert or update.
for example (SUDO CODE!):
public void InsertOrUpdate(PartAttribute model)
{
var existingRecord = dbContext.Set<PartAttribute>().Find(model.attributeId);
if(existingRecord == null)
{ // Insert (Should not include value for attributeId)
dbContext.Add(model);
}
else
{ // Update
existingRecord = model;
dbContext.SaveChanges();
}
}
Please let me know if you still have issues

Entity Framework - Non Key Relationships

Problem
I have a situation whereby I need to use Entity Framework 6, Code First, with a legacy database structure which cannot be changed. The database has a very generic table which stores text based data alongside some non key data which can be used to relate the record back to another table.
To illustrate:
Assume the Notes table has a model as follows:
[Table("Notes")]
public class Notes
{
[Key]
public int RecordId { get; set; }
[Required]
public string RelatedTableName { get; set; }
[Required]
public int RelatedTableRecordId { get; set; }
[Required]
public string NotesText { get; set; }
}
I then have another model which could look like so:
[Table("Drivers")]
public class Drivers
{
[Key]
public int RecordId { get; set; }
[Required]
public string DriverName { get; set; }
public ICollection<Notes> DriverNotes { get; private set; }
}
There is no foreign key which links the tables. The Drivers table is linked to the Notes table by way of the RelatedTableName and RelatedTableRecordId fields.
I do not have a problem reading data from the database and hydrating the models using entity framework.
The problem I have is that I want to be able to save a new Driver and its newly created Notes in one transaction and have the RelatedTableRecordId field set to the primary key of the Driver.
If a foreign key existed entity framework would know to back fill the property but in this case it doesn't know about the relationship.
Key Points
Database Structure must not change.
Must use Entity Framework 6 Code First
Must be able to use an Execution Strategy.
Require a relationship between non key fields.
Need to be able to persist all data in a single transaction.
What I've Tried
I had a similar issue with Audit type data and solved it by doing something similar to the following (note that this is very pseudo here):
public override int SaveChanges()
{
int changes = 0;
//Disable the current execution strategy as the default ones do not support user instantiated transactions.
this.ContextConfiguration.SuspendExecutionStrategy();
try
{
//Wrap a whole transaction inside an execution strategy so that auditing can be combined with regular saving of changes.
this.ExecutionStrategy.Execute(
() =>
{
using (var transaction = this.Database.BeginTransaction())
{
//Reset the change count so that it doesn't increase each time the transaction fails.
changes = 0;
//Remove any audit records created by previous failed transactions.
this.AuditTableChanges.Local.Clear();
//Evaluate the change tracker to identify entities which will potentially require an audit trail.
var insertedEntities = this.ChangeTracker.Entries().Where(entryEntity => entryEntity.State == EntityState.Added).ToList();
//Save all changes to get identities.
changes = base.SaveChanges();
//Create the audit trail for inserted entities. This step must occur after the initial call to SaveChanges() so that the identities are set.
foreach (DbEntityEntry entryEntity in insertedEntities)
{
//For each inserted record, get the audit record entries and add them
foreach (AuditTableChange auditTableChange in GetAuditRecords(entryEntity, AuditTableChangeType.Insert).Result)
this.AuditTableChanges.Add(auditTableChange);
}
//Save the audit trail for inserted entities.
changes += base.SaveChanges();
//Commit all changes to the database
transaction.Commit();
}
});
}
finally
{
//Re-enable the execution strategy so that other calls can benefit from the retry policy.
this.ContextConfiguration.UnSuspendExecutionStrategy();
}
return changes;
}
This worked fine for the Audit data as the implementation was hidden away in the framework. I do not want my development team to have to do all of the above each time they persist records.
In its simplistic form this is as much as I'd want people to be doing:
public void CreateDriver()
{
using (MyContext context = new MyContext())
{
Drivers driver = new Drivers();
driver.DriverName = "Joe Bloggs";
Notes driverNote = new Notes();
driverNote.RelatedTableName = "Drivers";
driverNote.NotesText = "Some very long text";
driver.DriverNotes.Add(driverNote);
context.Drivers.Add(driver);
context.SaveChanges();
}
}
In a way I want a foreign key which exists in code but not in the database so that entity framework knows to fill in the RelatedTableRecordId field. I've read some articles on hacking the EDMX but this project is purely Code First only.
There are older questions on stack overflow which are similar but relate to older versions of entity framework and don't help much or have as much detail as the above.
I'm hoping that someone may have experienced a similar problem and has an answer which may involve perhaps some custom mapping/metadata or some overrides to entity framework logic.
Any help would be appreciated.
Thanks,
Greg

New Breeze 1.4.9 - Duplicate entity - possible bug?

This problem started appearing after I upgraded to Breeze 1.4.9.
I have the following entity:
public class ProjectMember
{
public int ProjectId { get; set; }
[ForeignKey("ProjectId")]
[InverseProperty("ProjectMembers")]
public Project Project { get; set; }
public int TeamMemberId { get; set; }
[ForeignKey("TeamMemberId")]
[InverseProperty("ProjectMembers")]
public TeamMember TeamMember { get; set; }
}
And its configuration:
public class ProjectMemberConfiguration : EntityTypeConfiguration<ProjectMember>
{
public ProjectMemberConfiguration()
{
HasKey(a => new { a.ProjectId, a.TeamMemberId });
// ProjectMember has 1 project, projects have many projectmember records
HasRequired(a => a.Project)
.WithMany(s => s.ProjectMembers)
.HasForeignKey(a => a.ProjectId)
.WillCascadeOnDelete(true);
}
}
The metadata looks:
I create this entity on the client side with the following:
manager.createEntity('ProjectMember', { projectId: projectId, teamMemberId: teamMemberId });
All good so far, however when this entity is saved back to the server it gets duplicated on the client side as shown belo (the screenshot below shows what is in the cache after saveChanges succeeded callback is reached.
QUESTION
Why is Breeze duplicating this entity although it should not be allowed?
EDIT
I reverted back to Breeze 1.4.8 and the problem disappeared. Here is what the manager contains after a save operation:

Updated March 7 2014
This was a bug and is now fixed and available on GitHub. It will be released with a complete zip within the next few days. Any version above 1.4.9 should contain the fix.
Original post
Sorry, I can't tell from the screenshots that anything is duplicated. Are you seeing two entities in the EntityManager cache with the same key, And if so how?. Are you also seeing some form of duplication on the database as well?
Or is the problem that a 'new' entity is being created on the client after the save?
Is it possible that one part of these keys is an Identity column on the database? If so, then it's worth checking the metadata to insure that the autoGeneratedKeyType property for this EntityType is set to Identity. This would cause the database to generate a new key on insert and this entity would then be sent back to the client. Merging this entity with its previous incarnation will only occur if the AutoGeneratedKeyType is set to Identity. Otherwise, you will end up with both the original entity with the old key and a cloned version its new key.
Otherwise, I think we need more information.

Cannot update many-to-many relationships in Entity Framework

I am using Entity Framework 4.3 Code First, and I have problem with updating many-to-many relationships.
I defined the following classes:
public abstract class Entity
{
[Column(Order = 0)]
[Key]
public int Id { get; set; }
[Timestamp]
[Column(Order = 1)]
public byte[] Version { get; set; }
}
public class Video : Entity
{
public string Title { get; set; }
public string Description { get; set; }
public TimeSpan Length { get; set; }
public virtual ICollection<Coworker> Coworkers { get; set; }
}
public class Coworker : Entity
{
public string FirstName { get; set; }
public string LastName { get; set; }
public virtual ICollection<Video> Videos { get; set; }
}
When the database is created, the schema look right:
There is a Videos, Coworkers and VideoCoworkers table too, without
I use repository pattern in an N-Tier application to access database, my Insert and Update method looks like this:
public T Insert(T entity)
{
//Creates database context. When it disposes, it calls context.SaveChanges()
using (var session = new DatabaseSession())
{
session.Context.Set<T>().Add(entity);
}
}
public T Update(T entity)
{
//Creates database context. When it disposes, it calls context.SaveChanges()
using (var session = new DatabaseSession())
{
entity = session.Context.Set<T>().Attach(entity);
session.Context.Entry(entity).State = EntityState.Modified;
}
return entity;
}
When I update an entity, I create the entity object from a DTO, that's why use DbSet.Attach instead of selecting it and updating the properties one-by-one.
When I initialize the database, I add some test data:
Create 3 Coworkers, where I set first and last name. (A, B, C)
Create 3 Videos, where I set title, description and length, and also set some coworkers. First video has A,B, second has B,C and third has A,C.
When I list the Videos from code, I can see that Video.Coworkers collection is filled with good values, and when I query the link table (VideoCoworkers) in SQL Server Management Studio, it also looks good.
My problem is
when I update for example the title of the Video, it works. But when I try to delete from Video2 the existing coworkers (B and C), and try to add coworker A, then the relationship is not updated. It also does not work when I only try to add new coworker, or only try to delete one. I create the entity which is used as the parameter of the Update() method by creating a new Video entity with a new collection of Coworkers (which are selected from the database with Find() method by Id).
What is the correct way to update many-to-many relationships?

But when I try to delete from Video2 the existing coworkers (B and C),
and try to add coworker A, then the relationship is not updated.
Without using a generic repository the correct procedure would be:
using (var session = new DatabaseSession())
{
video2 = session.Context.Set<Video>().Include(v => v.Coworkers)
.Single(v => v.Id == video2Id);
coworkerA = new Coworker { Id = coworkerAId };
session.Context.Set<Coworker>().Attach(coworkerA);
video2.Coworkers.Clear();
video2.Coworkers.Add(coworkerA)
session.Context.SaveChanges();
}
The essential part is that you must load or attach the entity in its original state, change the entity, i.e. remove and add children, and then save the changes. EF's change detection will create the necessary INSERT and DELETE statements for the link table entries. The simple procedure to set the state to Modified you are trying in your generic Update method is suited only for updating scalar properties - like changing the video title - but won't work for updating relationships between entities.

For solve this problem:
attach the entity to context
load the collection(the collection is not loaded, because )
change the state of entity to modified
save changes
So your code for update should be like this:
public Video Update(Video entity)
{
//Creates database context. When it disposes, it calls context.SaveChanges()
using (var session = new DatabaseSession())
{
entity = session.Context.Set<Video>().Attach(entity);
session.Context.Entry(entity).Collection(p => p.Coworkers).Load();
session.Context.Entry(entity).State = EntityState.Modified;
}
return entity;
}

Please refer here to see how to save master detail in asp.net mvc with database first. Hopefully it will give you the idea about the code first. You may also have a look at knokout.js example