I am designing a database with Entity Framework Core which should contain two entity types:
An entity named "Channel" with a unique ChannelId
An entity named "Message" with a foreign key ChannelId and a MessageId
The MessageId must be unique for each channel, and it should be counted starting at 1.
My first try to implement this was to use a composite key for the Message entity with ChannelId and MessageId, but it does not have to stay this way. However I don't how to auto-generate the MessageId with EF Core.
So I tried to get the last MessageId for the current Channel incremented it and tried to insert:
public class DatabaseContext : DbContext
{
public void AddMessage(Message message)
{
long id = Messages
.Where(m => m.ChannelId == message.ChannelId)
.Select(m => m.MessageId)
.OrderByDescending(i => i)
.FirstOrDefault()
+ 1;
while (true)
{
try
{
message.MessageId = id;
Messages.Add(insert);
SaveChanges();
return;
}
catch
{
id++;
}
}
}
}
This code does not work. After an exception occurred EF Core does not insert the item with the incremented ID. In addition to that it seems to be very inefficient in situation with concurrent inserts.
Is there a more elegant solution to solve this problem when I use an additional ID in the messages table as primary key and maybe some additional tables?
Concept
After long research I found a solution for the problem:
I added a MessageIdCounter row to my Channels table.
Unlike classical code, SQL allows an atomic conditional write. This can be used for optimistic concurrency handling. First we read the counter value and increment it. Then we try to apply the changes:
UPDATE Channels SET MessageIdCounter = $incrementedValue
WHERE ChannelId = $channelId AND MessageIdCounter = $originalValue;
The database server will return the number of changes. If no changes have been made, the MessageIdCounter must have changed in the meantime. Then we have to run the operation again.
Implementation
Entities:
public class Channel
{
public long ChannelId { get; set; }
public long MessageIdCounter { get; set; }
public IEnumerable<Message> Messages { get; set; }
}
public class Message
{
public long MessageId { get; set; }
public byte[] Content { get; set; }
public long ChannelId { get; set; }
public Channel Channel { get; set; }
}
Database context:
public class DatabaseContext : DbContext
{
public DbSet<Channel> Channels { get; set; }
public DbSet<Message> Messages { get; set; }
protected override void OnModelCreating(ModelBuilder builder)
{
var channel = builder.Entity<Channel>();
channel.HasKey(c => c.ChannelId);
channel.Property(c => c.MessageIdCounter).IsConcurrencyToken();
var message = builder.Entity<Message>();
message.HasKey(m => new { m.ChannelId, m.MessageId });
message.HasOne(m => m.Channel).WithMany(c => c.Messages).HasForeignKey(m => m.ChannelId);
}
}
Utility method:
/// <summary>
/// Call this method to retrieve a MessageId for inserting a Message.
/// </summary>
public long GetNextMessageId(long channelId)
{
using (DatabaseContext ctx = new DatabaseContext())
{
bool saved = false;
Channel channel = ctx.Channels.Single(c => c.ChannelId == channelId);
long messageId = ++channel.MessageIdCounter;
do
{
try
{
ctx.SaveChanges();
saved = true;
}
catch (DbUpdateConcurrencyException ex)
{
var entry = ex.Entries.Single();
var proposedValues = entry.CurrentValues;
var databaseValues = entry.GetDatabaseValues();
const string name = nameof(Channel.MessageIdCounter);
proposedValues[name] = messageId = (long)databaseValues[name] + 1;
entry.OriginalValues.SetValues(databaseValues);
}
} while (!saved);
return messageId;
}
}
For successfully using EF Core's concurrency tokens I had to set MySQL's transaction isolation at least to READ COMMITTED.
Summary
It is possible to implement an incremental id per foreign key with EF Core.
This solution is not perfect because it needs two transactions for one insert and is therefore slower than an auto-increment row. Furthermore it's possible that MessageIds are skipped when the application crashes while inserting a Message.
Related
The minimal project sources to reproduce the issue is here :
https://wetransfer.com/downloads/8d9325ce7117bb362bf0d61fc7c8571a20220708100401/326add
===================
This error is a classic; In layman's terms it is usually caused by a "bad" insertion when a navigation is not properly taken in account, causing a faulty Ef state somewhere.
Many solutions have been posted along the years but I fail to see how my specific scenario could cause the issue!
My schema is a many-to-many between Groups and Users. The middle entity is named GroupUser.
There's a twist : Each GroupUser has an owned entity containing extra data, DataPayload. This choice was made for versatility -- we wanted that payload to be stored in its own table.
Schema:
public class User {
public Guid Id { get; private set; }
public IList<GroupUser> GroupUsers { get; private set; } = new List<GroupUser>();
public User(Guid id) { Id = id; }
}
public class Group {
public Guid Id { get; private set; }
public Group(Guid id) { Id = id; }
}
public class GroupUser {
public Guid Id { get; private set; }
public Guid GroupId { get; private set; }
public Guid UserId { get; private set; }
// Navigation link
public Group? Group { get; private set; }
public DataPayload? Data { get; private set; }
public GroupUser(Guid groupId, Guid userId, DataPayload data) {
Id = Guid.NewGuid(); //Auto generated
UserId = userId;
GroupId = groupId;
Data = data;
}
// This extra constructor is only there to make EF happy! We do not use it.
public GroupUser(Guid id, Guid groupId, Guid userId) {
Id = id;
UserId = userId;
GroupId = groupId;
}
}
public class DataPayload {
//Note how we did not defined an explicit Id; we let EF do it as part of the "Owned entity" mechanism.
///// <summary>foreign Key to the Owner</summary>
public Guid GroupUserId { get; private set; }
public int DataValue { get; private set; }
public DataPayload(int dataValue) {
DataValue = dataValue;
}
public void SetDataValue(int dataValue) {
DataValue = dataValue;
}
}
To make it all work, we configure the navigations like this :
// --------- Users ---------
builder
.ToTable("Users")
.HasKey(u => u.Id);
// --------- Groups ---------
builder
.ToTable("Groups")
.HasKey(g => g.Id);
// --------- GroupUsers ---------
builder
.ToTable("GroupUsers")
.HasKey(gu => gu.Id);
builder
.HasOne<User>() //No navigation needed
.WithMany(u => u.GroupUsers)
.HasForeignKey(gu => gu.UserId);
builder
.HasOne<Group>(gu => gu.Group) //Here, we did define a navigation
.WithMany()
.HasForeignKey(gu => gu.GroupId);
builder
.OwnsOne(gu => gu.Data,
navBuilder => {
navBuilder.ToTable("PayloadDatas");
navBuilder.Property<Guid>("Id"); //Note: Without this EF would try to use 'int'
navBuilder.HasKey("Id");
//Configure an explicit foreign key to the owner. It will make our life easier in our Unit Tests
navBuilder.WithOwner().HasForeignKey(d => d.GroupUserId);
}
);
//.OnDelete(DeleteBehavior.Cascade) // Not needed (default behaviour for an owned entity)
Now, you know how everything is defined.
Basic setup : works!
var group = new Group(groupId);
await dbContext.Groups.AddAsync(group);
await dbContext.SaveChangesAsync();
var user = new User(userId);
await dbContext.Users.AddAsync(user);
await dbContext.SaveChangesAsync();
Follow-up scenario : fails!
var groupUser = new GroupUser(groupId, userId, new DataPayload(dataValue: 777777));
user.GroupUsers.Add(groupUser);
await dbContext.SaveChangesAsync(); // Crash happens here!!!
Error:
Microsoft.EntityFrameworkCore.DbUpdateConcurrencyException : The database operation was expected to affect 1 row(s), but actually affected 0 row(s); data may have been modified or deleted since entities were loaded.
I suspect that EF gets confused by the addition of two entities at once, where it has to compute some Ids itself : the new GroupUser and the DataPayload it contains. I'm not sure how it's supposed to know that it needs to give an Id to the GroupUser first and then use that Id as the foreign key in PayloadData. But that's just me; it might or might not be related to the issue.
But what do I need to change?
The mistake was in GroupUser's id-less constructor:
Id = Guid.NewGuid();
The code needs to let EF manage the keys when it comes to owned entities such as DataPayload which rely on a foreign key (GroupUserId) that's still in the making at the time of saving.
If you set a key value (Guid.NewGuid()) yourself, then EF gets confused between:
linking the new DataPayload entity to the GroupUser entity where you've shoehorned an Id value,
OR
just expecting an empty value (foreign key) and setting all the keys (both the GroupUser's Id and DataPayload's GroupUserId) itself.
All in all, EF feels like you announced that you were about to let it create 1 entity, but you've pulled the rug under its feet and done it yourself, so it ends up with 0 entity to work with. Hence the error message.
It should have been :
Id = Guid.Empty;
With Guid.Empty, EF clearly identifies that this entity is new and that has to be the same one as the one you told it to create and link to the new PayloadData -- that is, the instance that you've set in GroupUser.Data.
So I'm using Entity Framework Core to build a database of Guilds (Another name for Discord Servers) and Users, with the Discord.NET Library. Each Guild has many users, and each user can be in many guilds. First time using EF and I'm having some teething issues. The two classes are:
public class Guild
{
[Key]
[DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public int Id { get; set; }
public ulong Snowflake { get; set; }
public DateTimeOffset CreatedAt { get; set; }
public string Name { get; set; }
public ICollection<User> Users { get; set; }
}
public class User
{
[Key]
[DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public int Id { get; set; }
public ulong Snowflake { get; set; }
public string Username { get; set; }
public ushort DiscriminatorValue { get; set; }
public string AvatarId { get; set; }
public ICollection<Guild> Guilds { get; set; }
public DateTimeOffset CreatedAt { get; set; }
}
With the goal of having 3 tables: Guild, Users, and GuildUsers. This is my current function for getting the guilds:
using var context = new AutomataContext();
var discordGuilds = this.client.Guilds.ToList();
var dbGuilds = context.Guilds;
List<Guild> internalGuilds = discordGuilds.Select(g => new Guild
{
Snowflake = g.Id,
Name = g.Name,
CreatedAt = g.CreatedAt,
Users = g.Users.Select(gu => new User
{
Id = context.Users.AsNoTracking().FirstOrDefault(u => u.Snowflake == gu.Id)?.Id ?? default(int),
}).ToList(),
}).ToList();
// Upsert Guilds to db set.
foreach (var guild in internalGuilds)
{
var existingDbGuild = dbGuilds.AsNoTracking().FirstOrDefault(g => g.Snowflake == guild.Snowflake);
if (existingDbGuild != null)
{
guild.Id = existingDbGuild.Id;
dbGuilds.Update(guild); // Hits the second Update here and crashes
}
else
{
dbGuilds.Add(guild);
}
}
await context.SaveChangesAsync();
I should note, a 'snowflake' is a unique ID that discord uses, but I wanted to keep my own unique ID for each table.
High level overview, guilds are collected into Discord.NET models. These are then transformed into internalGuilds (my guild class, which includes the list of users). Each of these is looped through and upserted to the database.
The issue arises in the second guild loop, where an error is thrown in the "Update" that a User ID is already being tracked (Inside the guild). So the nested ID is already being tracked? Not sure what's going on here, any help would be appreciated. Thanks.
This exception is most likely occurring because you are loading Users without tracking then looping through and potentially trying to update or insert guilds /w the same user reference, especially using the Update method.
I would suggest removing the use of AsNoTracking. Working with detached entity references via AsNoTracking is more of a performance tweak for when reading large amounts of data. You can pre-fetch all of the User references by their snowflake:
using (var context = new AutomataContext())
{
var discordGuilds = this.client.Guilds.ToList();
// Get the user snowflakes from the guilds, and pre-fetch them.
var userSnowflakes = discordGuilds.SelectMany(g => g.Users.Select(u => u.Id)).ToList();
var users = await context.Users
.Where(x => userSnowflakes.Contains(x.Snowflake))
.ToListAsync();
// We need to add references for any New user snowflakes.
var existingSnowflakes = users.Select(x => x.Snowflake).ToList();
// If more detail is needed for new user records, it will need to be fetched from the passed in Guild.User.
var newUsers = userSnowflakes.Except(existingSnowFlakes)
.Select(x => new User { SnowflakeId = x }).ToList();
if(newUsers.Any())
users.AddRange(newUsers);
List<Guild> internalGuilds = discordGuilds.Select(g => new Guild
{
Snowflake = g.Id,
Name = g.Name,
CreatedAt = g.CreatedAt,
Users = g.Users
.Select(gu => users.Single(u => u.Snowflake == gu.Id))
.ToList(),
}).ToList(),
// Upsert Guilds to db set.
foreach (var guild in internalGuilds)
{
var existingGuildId = context.Guilds
.Where(x => x.Snowflake == guild.Snowflake)
.Select(x => x.Id)
.SingleOrDefault();
if (existingGuildId != 0)
{
guild.Id = existingGuildId;
dbGuilds.Update(guild);
}
else
{
dbGuilds.Add(guild);
}
}
await context.SaveChangesAsync();
This should help ensure that the User references for existing users are pointing at the same instances, whether existing users or new user references that will be associated to the DbContext when first referenced.
Ultimately I don't recommend using Update for "Upsert" scenarios, instead since the Db Record needs to be fetched anyways, updating values on the fetched instance or inserting a new one. Update will want to send all fields from an entity to the database each time, rather than just sending what has changed. It means enforcing a bit more control over what can possibly be changed vs. what should not be.
I'm using EF Core 2.1, database first approach. I'm trying to Include() a foreign key entity when fetching my target entity collection, but something strange is happening.
The entity structure is Job -> JobStatus. I'm fetching some Job entities, and want to include the Job's JobStatus foreign key property. The issue is that the JobStatus entity has a ICollection[Job] property that is populating every single Job from the database. This is causing the payload to be gigabytes in size.
When I include the JobStatus on the Job, I'd like to satisfy one of the following solutions. I'm also open to other solutions or workarounds I haven't thought of.
*how can I prevent the JobStatus' ICollection property from populating?
*Or can I prevent Entity Framework from generating that property in the first place?
I've already explored Ignoring the ReferenceLoopHandling
services.AddMvc().AddJsonOptions(options => {
options.SerializerSettings.ReferenceLoopHandling = Newtonsoft.Json.ReferenceLoopHandling.Ignore;
});
Here are the entities, automatically generated by Entity Framework.
public partial class Job
{
public long Id { get; set; }
public long StatusId { get; set; }
public JobStatus Status { get; set; }
}
public partial class JobStatus
{
public JobStatus()
{
Job = new HashSet<Job>();
}
public long Id { get; set; }
public string Name { get; set; }
public string Description { get; set; }
public ICollection<Job> Job { get; set; }
}
Example code that is causing the problem
var jobs = _context.Set<Job>()
.Where(job => job.Id == 1)
.Include(job => job.Status);
One way to avoid the Job collection from being populated is to explicitly select the columns that you want returned, either through a defined or anonymous type:
var jobs = _context.Set<Job>()
.Where(job => job.Id == 1)
.Include(job => job.Status)
.Select(job => new
{
Id = job.Id,
StatusName = job.Status.Name
});
Add a "virtual" keyword. Any virtual ICollections will be lazy-loaded unless you specifically mark them otherwise.
public virtual ICollection<Job> Job { get; set; }
Here is my sample code
class Sample
{
public int Id{ get;set; }
public int AssociatedSiteId { get; set; }
[ForeignKey("AssociatedSiteId")]
public virtual SiteA Site { get; set; }
[ForeignKey("AssociatedSiteId")]
public virtual SiteB SiteB { get; set; }
}
I m Inserting the data as
using (var dbcontext = new sampleEntities)
{
var sample = new Sample();
sample.Id=1;
sample.AssociateId = siteInfo.SiteId; // This Id from SiteA Table
dbcontext.Sample.Add(sample);
dbcontext.SaveChanges();
}
where I try to Insert Data into Sample, I m getting the error as:
The INSERT statement conflicted with the FOREIGN KEY constraint.
I m new to EF Can any one please help me?
I am assuming you need some guidance :)
I have a question: Why Site and SiteB are using the same foreign key variable? Aren't them different entities? (SiteA and SiteB)
If you have
Sample.SiteA => MySiteA
Sample.SiteB => MySiteB
they cannot use the same foreignh key. Even if they are the same type, that key is used to locate the foreign entity and you would be overwriting one with the another... Check your code with mine.
Try the following:
class Sample
{
public int Id{ get;set; }
public int AssociatedSiteAId { get; set; } // Optional but sometimes useful if you don't use [ForegnKey("...")]
[ForeignKey("AssociatedSiteAId")]
public virtual SiteA Site { get; set; }
public int AssociatedSiteBId { get; set; } // Optional but sometimes useful if you don't use [ForegnKey("...")]
[ForeignKey("AssociatedSiteBId")]
public virtual SiteB SiteB { get; set; }
}
On your code,
using (db = new DbContext())
{
var site_a = db.SitesA.Find(123); // 1 should be the key of site a
var site_b = db.SitesB.Find(456); // 2 should be the key of site b
Sample sample = new Sample()
{
SiteA = site_a,
SiteB = site_b
};
db.Samples.Add(sample);
db.SaveChanges();
}
I have 2 classes that reference each other. It's a weird situation that our CRM needs.
I have an Organization and EmAddress tables and classes. The Organization inherits from Subscriber, which also has a table. I think this could be my problem, or the fact that I can't set Inverse on these because there is no "HasMany"...
The order of insert/update is ..
INSERT Email
INSERT Org
UPDATE Email to set Email.Subscriber
Email.Subscriber needs to be "NOT NULL", so this doesn't work. How can I change the order, I can't use Inverse because there is no list. Just 2 references.
public class Organization : Subscriber
{
public override string Class { get { return "Organization"; } }
EmAddress PrimaryEmailAddress {get;set;}
}
public class OrganizationMap : SubclassMap<Organization>
{
public OrganizationMap()
{
Table("Organization");
KeyColumn("Organization");
References(x => x.PrimaryEmail,"PrimaryEmailAddress").Cascade.SaveUpdate();
}
}
public EmAddressMap()
{
Id(x => x.Id, "EmAddress");
Map(x => x.EmailAddress, "eMailAddress");
References<Subscriber>(x => x.Subscriber,"Subscriber").LazyLoad().Fetch.Select().Not.Nullable();
/*not.nullable() throw s error. NHibernate INSERTS email, INSERTS org, UPDATES email. */
}
public class EmAddress
{
public virtual Guid Id { get; set; }
public virtual string EmailAddress { get; set; }
public virtual Subscriber Subscriber { get; set; }
}
//Implementation
var session = NHIbernateHelper.GetSession();
using(var tx = session.BeginTransaction())
{
var org = new Organization();
org.PrimaryEmail = new EmAddress(){Subscriber = org};
session.Save(org);
tx.commit();
}
This post might help:
http://ayende.com/blog/3960/nhibernate-mapping-one-to-one
Have only one side use many-to-one (Fluent: "References") and the other side uses one-to-one (Fluent: "HasOne").