We Keep Coding

EF core "cannot insert explicit value for identity column" when using BYTE as identity column type in Sql Server

I am having trouble seeding data into a SQL database using EF Core during program startup.
I have an EF Core table defined like this:
migrationBuilder.CreateTable(
name: "PoolStyles",
columns: table => new
{
Id = table.Column<byte>(type: "tinyint", nullable: false)
.Annotation("SqlServer:Identity", "1,1"),
Name = table.Column<string>(type: "nvarchar(256)", maxLength: 256),
Code = table.Column<string>(type: "nvarchar(40)", maxLength: 40),
Description = table.Column<string>(type: "nvarchar(MAX)", nullable: true),
IsRestricted = table.Column<bool>(type: "bit"),
Priority = table.Column<byte>(type: "tinyint", nullable: true, defaultValue: 0),
Icon = table.Column<string>(type: "nvarchar(200)", maxLength: 200, nullable: true),
},
constraints: table =>
{
table.PrimaryKey("PK_PoolStyles", x => x.Id);
});
and here is the matching entity class:
public class PoolStyle
{
public byte Id { get; set; }
public string Name { get; set; }
public string Code { get; set; }
public bool IsRestricted { get; set; }
public byte Priority { get; set; }
public string Icon { get; set; }
public string Description { get; set; }
}
During the start up of my application I attempt to seed some data into my database like so:
if (!context.PoolStyles.Any())
{
context.PoolStyles.Add(new PoolStyle { Code = "SRV", Description = "Survival Pool", Icon = "", IsRestricted = false, Name = "Survival" });
await context.SaveChangesAsync();
}
However I keep getting the following error:
An error occurred while migrating or seeding the database.
Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while updating the entries. See the inner exception for details.
Microsoft.Data.SqlClient.SqlException (0x80131904): Cannot insert explicit value for identity column in table 'PoolStyles' when IDENTITY_INSERT is set to OFF.
The error is preceded by the SQL EF Core tries to run:
fail: Microsoft.EntityFrameworkCore.Database.Command[20102]
Failed executing DbCommand (11ms) [Parameters=[#p0='?' (Size = 1) (DbType = Byte), #p1='?' (Size = 4000), #p2='?' (Size = 4000),
#p3='?' (Size = 4000), #p4='?' (DbType = Boolean), #p5='?' (Size =
4000), #p6='?' (Size = 1) (DbType = Byte)], CommandType='Text',
CommandTimeout='30']
SET NOCOUNT ON;
INSERT INTO [PoolStyles] ([Id], [Code], [Description], [Icon], [IsRestricted], [Name], [Priority])
VALUES (#p0, #p1, #p2, #p3, #p4, #p5, #p6);
PROBLEM: you can see that EF Core is adding the [Id] and [Priority] columns to the executed SQL but those values ARE NOT included in the object that I try to pass to the SaveChangesAsync() function.
Also, I am not using annotations so can't use those to solve this problem, I "hand code" the migrations because of the complexity of what I need.
QUESTION: can someone help me figure out why EF Core is trying to add these values to the insert statement?

The issue was using byte as the data type for the identity column. I am not sure why as I know that I can manually create an identity column in sql server with type byte. Something in EF Core is choking.
THE SOLUTION:
I changed the identity column data type to int and the insert works properly with auto incrementing id values.
Maybe I missed something else but its working and these table will not have many records so I will just use int and continue on, haha.

Is it possible to use Database First EF 6 with SKIP LOCKED command?

I'm using a SQL table as a job queue very similar to the article here: https://vladmihalcea.com/database-job-queue-skip-locked/
My problem is that I'm using Entity Framework 6 with Database First code and from what I can tell, EF6 doesnt' support the skip locked command. Here is my table class and I'm using each computer as a worker to handle the task I'm passing it.
public partial class InProgress
{
public int ID { get; set; }
public string Task { get; set; }
public string Computer { get; set; }
public DateTime Date { get; set; }
}
Does anyone have any C# code they can share so I can make sure that no other computer can work on the same task as another computer at the same time?
UPDATE: I want to clarify that I'm not doing a traditional queue where you constantly add and remove to the queue or in this case table. I have a table that contains a task list and I'm constantly having the tasks worked on by multiple computers and when they are finished, they update the Date column with the finished time. I work on the tasks that have the oldest Date first.
Here is some pseudo code of what I'm trying to do based on the info provided
create procedure usp_enqueuePending
#date datetime,
#task varchar(50),
#computer varchar(50)
as
set nocount on;
insert into InProgresses(Date, Task, Computer)
values (#date, #task, #computer);
go
create procedure usp_dequeuePending
as
set nocount on;
declare #now datetime;
set #now = getutcdate();
with cte as (
select top(1)
Task
from InProgresses with (rowlock, updlock, readpast)
where Date < #now
order by Date)
delete from cte
output deleted.Task;
go
using var context = new context();
var dequeuedItem = context.usp_dequeuePending(); // not sure how to convert this back to an InProgress class
// do work here I'm guessing
// add to the queue when finished with it??
context.usp_enqueuePending(DateTime.UtcNow, task, computer);

You can write custom queries in EF Core, see here. So you could do something like this:
dbContext.InProgress
.FromSqlRaw("SELECT * FROM model.InProgress WITH (rowlock, updlock, readpast)")
.Where(...) // do other LINQ stuff here
It's not super pretty, but I don't know of a better solution at the moment.

EF ignores DatabaseGeneratedOption.None

I am using EF 6.1.3 Code First, but without migrations as the database already exists. I have an entity SRReports with the following property:
[Key, Required]
[DatabaseGeneratedAttribute(DatabaseGeneratedOption.None)]
public int FRID { get; set; }
Entity Framework ignores the DatabaseGeneratedOption.None and sends TSQL to the server assuming FRID is autogenerated. The entity is assigned a value for FRID but entity framework ignores it and assumes that is autogenerated. (this is confirmed by checking a TRACE of what is sent to the server). The exception message is:
Message=Cannot insert the value NULL into column 'FRID', table 'RevLogon.dbo.SCIREPORTS'; column does not allow nulls. INSERT fails.
The statement has been terminated.
I tried using the fluent API instead of (and in addition to) the annotations.
modelBuilder.Entity<SCIREPORTS>()
.HasKey(e => e.FRID);
modelBuilder.Entity<SCIREPORTS>()
.Property(e => e.FRID).HasDatabaseGeneratedOption(DatabaseGeneratedOption.None);
But then I get the error when I query the table:
SCIREPORTS sr = mydb.SCIREPORTS.Where(s => s.FRID == FRID ).FirstOrDefault();
I get the following error:
System.MissingMethodException was caught
HResult=-2146233069
Message=Method not found: 'System.Data.Entity.ModelConfiguration.Configuration.PrimitivePropertyConfiguration System.Data.Entity.ModelConfiguration.Configuration.PrimitivePropertyConfiguration.HasDatabaseGeneratedOption(System.Nullable`1)'.*
I have uninstalled and reinstalled my EF package but haven't been able to solve this problem.
How can I get EF to insert the record with the ID?
Here is the basic code that inserts the record:
public SRUserComp(string myemail, short mycounter, int myFRID, string myreptype, string mypassword)
{ email = myemail;
Counter = mycounter;
reptype = myreptype;
password = mypassword;
FRID = myFRID; }
public bool CreateSRRecord(DateTime repduedate,
short repnumber)
{BSRModel mydb = new BSRModel();
sr = new SCIREPORTS();
sr.FRID = FRID;
sr.Counter = Counter;
sr.Final = true;
sr.RepDueDate = repduedate;
mydb.SCIREPORTS.Add(sr);
mydb.SaveChanges();
return true; }
After the saveChanges statement I get the following statements from the sql profiler on the server:
INSERT [dbo].[SCIREPORTS]([Counter], [RepDueDate], [RepArrivalDate],
[SciAppDate], [Adminappdate], [legacyDate], [RepNumber], [Final],
[RepReminderID], [Notes], [HaimSaw], [ResAuthApprovDate])
VALUES (#0, #1, NULL, NULL, NULL, NULL, #2, #3, NULL, NULL, NULL, NULL)
SELECT [FRID]
FROM [dbo].[SCIREPORTS]
WHERE ##ROWCOUNT > 0 AND [FRID] = scope_identity()

How to avoid "Violation of UNIQUE KEY constraint" when doing LOTS of concurrent INSERTs

I am performing MANY concurrent SQL INSERT statements which are colliding on a UNIQUE KEY constraint, even though I am also checking for existing records for the given key inside of a single transaction. I am looking for a way to eliminate, or minimize, the amount of collisions I am getting without hurting the performance (too much).
Background:
I am working on an ASP.NET MVC4 WebApi project which receives A LOT of HTTP POST requests to INSERT records. It gets about 5K - 10K requests a second. The project's sole responsibility is de-duplicating and aggregating records. It is very write heavy; it has a relatively small amount of read requests; all of which use a Transaction with IsolationLevel.ReadUncommitted.
Database schema
Here is the DB table:
CREATE TABLE [MySchema].[Records] (
Id BIGINT IDENTITY NOT NULL,
RecordType TINYINT NOT NULL,
UserID BIGINT NOT NULL,
OtherID SMALLINT NULL,
TimestampUtc DATETIMEOFFSET NOT NULL,
CONSTRAINT [UQ_MySchemaRecords_UserIdRecordTypeOtherId] UNIQUE CLUSTERED (
[UserID], [RecordType], [OtherID]
),
CONSTRAINT [PK_MySchemaRecords_Id] PRIMARY KEY NONCLUSTERED (
[Id] ASC
)
)
Repository Code
Here is the code for the Upsert method which is causing the Exception:
using System;
using System.Data;
using System.Data.SqlClient;
using System.Linq;
using Dapper;
namespace MyProject.DataAccess
{
public class MyRepo
{
public void Upsert(MyRecord record)
{
var dbConnectionString = "MyDbConnectionString";
using (var connection = new SqlConnection(dbConnectionString))
{
connection.Open();
using (var transaction = connection.BeginTransaction(IsolationLevel.ReadCommitted))
{
try
{
var existingRecord = FindByByUniqueKey(transaction, record.RecordType, record.UserID, record.OtherID);
if (existingRecord == null)
{
const string sql = #"INSERT INTO [MySchema].[Records]
([UserID], [RecordType], [OtherID], [TimestampUtc])
VALUES (#UserID, #RecordType, #OtherID, #TimestampUtc)
SELECT CAST(SCOPE_IDENTITY() AS BIGINT";
var results = transaction.Connection.Query<long>(sql, record, transaction);
record.Id = results.Single();
}
else if (existingRecord.TimestampUtc <= record.TimestampUtc)
{
// UPDATE
}
transaction.Commit();
}
catch (Exception e)
{
transaction.Rollback();
throw e;
}
}
}
}
// all read-only methods use explicit transactions with IsolationLevel.ReadUncommitted
private static MyRecord FindByByUniqueKey(SqlTransaction transaction, RecordType recordType, long userID, short? otherID)
{
const string sql = #"SELECT * from [MySchema].[Records]
WHERE [UserID] = #UserID
AND [RecordType] = #RecordType
AND [OtherID] = #OtherID";
var paramz = new {
UserID = userID,
RecordType = recordType,
OtherID = otherID
};
var results = transaction.Connection.Query<MyRecord>(sql, paramz, transaction);
return results.SingleOrDefault();
}
}
public class MyRecord
{
public long ID { get; set; }
public RecordType RecordType { get; set; }
public long UserID { get; set; }
public short? OtherID { get; set; }
public DateTimeOffset TimestampUtc { get; set; }
}
public enum RecordType : byte
{
TypeOne = 1,
TypeTwo = 2,
TypeThree = 3
}
}
The Problem
When the server is under heavy enough load, I am seeing many of these Exceptions occurring:
Violation of UNIQUE KEY constraint 'UQ_MySchemaRecords_UserIdRecordTypeOtherId'. Cannot insert duplicate key in object 'MySchema.Records'. The duplicate key value is (1234567890, 1, 123). The statement has been terminated.
This Exception occurs often, as many as 10 times in a minute.
What I have tried
I tried changing the IsolationLevel to Serializable. The Exception occured much less often but still occured. Also, the performance of the code suffered greatly; the system could only handle 2K requests a second. I suspect that this decrease in throughput was actually the cause of the reduced Exceptions so I concluded that this didn't solve my problem.
I have considered using the UPDLOCK Table Hint but I don't fully understand how it cooperates with isolation levels or how to apply it to my code. It does seem like it might be the best solution though, from my current understanding.
I also tried adding the initial SELECT statement (for existing records) to be part of the INSERT statement, like shown here but this attempt still had the same problem.
I tried implementing my Upsert method by using the SQL MERGE statement but this also suffered from the same problem.
My Question(s)
Is there anything I can do to prevent this type of UNIQUE key constraint collisions?
If I should be using the UPDLOCK table hint (or any other table hint for that matter), how would I add that to my code? Would I add it to the INSERT? The SELECT? Both?

Make the validating read take a lock:
FROM SomeTable WITH (UPDLOCK, ROWLOCK, HOLDLOCK)
This serializes accesses on a single key, allowing for concurrency on all others.
HOLDLOCK ( = SERIALIZABLE) protects a range of values. This ensures a row that doesn't exist continues to not exist so the INSERT succeeds.
UPDLOCK ensures any existing row is not changed or deleted by another concurrent transaction so the UPDATE succeeds.
ROWLOCK encourages the engine to take a row-level lock.
These changes may increase the chances of a deadlock.

It may be faster to permit and suppress the errors in your scenario than to attempt to eliminate them. If you're consolidating multiple sources synchronously with overlapping data you will need to create a bottleneck somewhere to manage the race condition.
You could create a singleton manager class that held the unique constraints of the records in a hashset so you would automatically drop duplicates when they're added to the set. Records get added prior to submitting to the DB and removed upon statement completion. That way either the hashset eats the duplicate or the existing record check you do at the top of your try detects the committed duplicate record.

AFAIK, the only solution is to check for duplication before insert. It demands at least one round-trip to DB results in poor performance.
You can do SELECT on a table and hold the lock to prevent other parallel threads to SELECT and getting the same value. Here is the detailed solution: Pessimistic locking in EF code first
PS:
Based on Aron's comment and it's nice work-around, I should say my proposed solution is based on this assumption that you don't want to use buffer or queue.

Loops introduced by ORM Reduces Performance drastically

I have a table as shown below. It has accounts of type Fixed and Savings. I need to update the status of all accounts of user 1. There are 10000 accounts for this user. Essentially the logic would be as shown in the following SQL Stored Procedure Script. The script takes only less than 1 second to execute (83 milli seconds).
But when I converted it to a ORM using LINQ to SQL it takes more than 3 minutes (204814 milli seconds). It is at least 240,000% slower.
Is there a pattern in LINQ to SQL (or other ORM) that will help to overcome this performance hit?
What can force it to do a update in one go to database?
Note: I am aware of calling stored procedures from LINQ. I don’t see that as ORM and not an option for me.
Manual Stored Procedure Script
DECLARE #UserID INT
DECLARE #StatusForFixed VARCHAR(50)
DECLARE #StatusForSavings VARCHAR(50)
SET #UserID = 1
SET #StatusForFixed = 'FrozenFA11'
SET #StatusForSavings = 'FrozenSB22'
UPDATE BankAccount
SET Status =
CASE
WHEN BankAccount.AccountType='Fixed' THEN #StatusForFixed
WHEN BankAccount.AccountType='Savings' THEN #StatusForSavings
END
WHERE AccountOwnerID=#UserID
LINQ Generated Code Sample
Note: This type of statements happen 10000 times
UPDATE [dbo].[BankAccount]
SET [Status] = #p3
WHERE [BankAccountID] = #p0
-- #p0: Input Int (Size = -1; Prec = 0; Scale = 0) [3585]
-- #p3: Input NChar (Size = 10; Prec = 0; Scale = 0) [FrozenSB]
CODE after applying ORM
public class BankAccountAppService
{
public RepositoryLayer.ILijosBankRepository AccountRepository { get; set; }
public void FreezeAllAccountsForUser(int userId)
{
IEnumerable<DBML_Project.BankAccount> accounts = AccountRepository.GetAllAccountsForUser(userId);
foreach (DBML_Project.BankAccount acc in accounts)
{
acc.Freeze();
}
AccountRepository.UpdateAccount();
}
}
public class LijosSimpleBankRepository : ILijosBankRepository
{
public System.Data.Linq.DataContext Context
{
get;
set;
}
public List<DBML_Project.BankAccount> GetAllAccountsForUser(int userID)
{
IQueryable<DBML_Project.BankAccount> queryResultEntities = Context.GetTable<DBML_Project.BankAccount>().Where(p => p.AccountOwnerID == userID);
return queryResultEntities.ToList();
}
public List<T> GetAllAccountsofType<T>() where T : DBML_Project.BankAccount
{
var query = from p in Context.GetTable<DBML_Project.BankAccount>().OfType<T>()
select p;
List<T> typeList = query.ToList();
return typeList;
}
public virtual void UpdateAccount()
{
Context.SubmitChanges();
}
}
namespace DBML_Project
{
public partial class BankAccount
{
//Define the domain behaviors
public virtual void Freeze()
{
//Do nothing
}
}
public class FixedBankAccount : BankAccount
{
public override void Freeze()
{
this.Status = "FrozenFA";
}
}
public class SavingsBankAccount : BankAccount
{
public override void Freeze()
{
this.Status = "FrozenSB";
}
}
}
REFERENCE
Pass List as XElement to be used as XML Datatype parameter

You are comparing two wildly different scenarios:
1: running a script locally on the SQL server, a single set-based UPDATE
2: fetching 10,000 records over the network, updating each, submitting each individually
You can improve 2 a bit by deferring the SubmitChanges() into one single batch of 10,000 rather than 10,000 batches of 1 (just: don't call SubmitChanges() until the end), but that still involves sending the details of 10,000 records in two directions, plus all the overheads (for example, SubmitChanges() might still choose to do that via 10,000 individual calls).
Basically, object-based tools are not intended for bulk updates against records. If the SP works, use the SP. Maybe call the SP via a data-context, just for convenience of it adding the method/parameters/etc.

You can still execute your stored procedure / custom SQL script from your application. You can even map the procedure in your Linq-to-sql model so that you don't need to open connection and create command manually.
I'm not exactly sure if Linq-to-sql always executes each modification command in separate roundtrip to database but I guess it does (at least in most cases). EF does it always. NHibernate has better support for such operations because it has command batching.
What you showed here is not batch update (single command updating a lot of records) - most ORMs will always update each record separately - that is how these tools work. If you load records and modify each of them in loop the relation to original query used to load records is lost. You now have 10.000 loaded records in your application which must be updated. The bulk update is not possible because you must move 10.000 changes from your application to the database.
If you want to do bulk update you should either use direct SQL or implement some logic which will make the update from Linq-to-sql instead of loading records and updating them in the application. Check this article or simply search for Bulk / Batch updates in Linq-to-sql.

This is because Linq to SQL First loads the data from server and then update each record individually which includes data query/transfer to client, update request for each record. Whereas in SP case there is just a call to SP which executes the Update query on the server directly and it does not include data fetch and update of each record. It updates the record in bulk

Another approach I did is passing the object values to the stored procedure as XML datatype. But there comes a timeout exception (after some 25 seconds) when the record count is more than 1000. Is it due to huge xml file?
Note: It takes around 5 seconds for 1000 records
public virtual void UpdateBankAccountUsingParseXML_SP(System.Xml.Linq.XElement inputXML)
{
string connectionstring = "Data Source=.;Initial Catalog=LibraryReservationSystem;Integrated Security=True;Connect Timeout=600";
var myDataContext = new DBML_Project.MyDataClassesDataContext(connectionstring);
myDataContext.ParseXML(inputXML);
}
public void FreezeAllAccountsForUser(int userId)
{
List<DTOLayer.BankAccountDTOForStatus> bankAccountDTOList = new List<DTOLayer.BankAccountDTOForStatus>();
IEnumerable<DBML_Project.BankAccount> accounts = AccountRepository.GetAllAccountsForUser(userId);
foreach (DBML_Project.BankAccount acc in accounts)
{
string typeResult = Convert.ToString(acc.GetType());
string baseValue = Convert.ToString(typeof(DBML_Project.BankAccount));
if (String.Equals(typeResult, baseValue))
{
throw new Exception("Not correct derived type");
}
acc.Freeze();
DTOLayer.BankAccountDTOForStatus presentAccount = new DTOLayer.BankAccountDTOForStatus();
presentAccount.BankAccountID = acc.BankAccountID;
presentAccount.Status = acc.Status;
bankAccountDTOList.Add(presentAccount);
}
IEnumerable<System.Xml.Linq.XElement> el = bankAccountDTOList.Select(x =>
new System.Xml.Linq.XElement("BankAccountDTOForStatus",
new System.Xml.Linq.XElement("BankAccountID", x.BankAccountID),
new System.Xml.Linq.XElement("Status", x.Status)
));
System.Xml.Linq.XElement root = new System.Xml.Linq.XElement("root", el);
AccountRepository.UpdateBankAccountUsingParseXML_SP(root);
//AccountRepository.Update();
}
Stored Procedure
ALTER PROCEDURE [dbo].[ParseXML] (#InputXML xml)
AS
BEGIN
DECLARE #MyTable TABLE (RowNumber int, BankAccountID int, StatusVal varchar(max))
INSERT INTO #MyTable(RowNumber, BankAccountID,StatusVal)
SELECT ROW_NUMBER() OVER(ORDER BY c.value('BankAccountID[1]','int') ASC) AS Row,
c.value('BankAccountID[1]','int'),
c.value('Status[1]','varchar(32)')
FROM
#inputXML.nodes('//BankAccountDTOForStatus') T(c);
DECLARE #Count INT
SET #Count = 0
DECLARE #NumberOfRows INT
SELECT #NumberOfRows = COUNT(*) FROM #MyTable
WHILE #Count < #NumberOfRows
BEGIN
SET #Count = #Count + 1
DECLARE #BankAccID INT
DECLARE #Status VARCHAR(MAX)
SELECT #BankAccID = BankAccountID
FROM #MyTable
WHERE RowNumber = #Count
SELECT #Status = StatusVal
FROM #MyTable
WHERE RowNumber = #Count
UPDATE BankAccount
SET Status= #Status
WHERE BankAccountID = #BankAccID
END
END
GO