We Keep Coding

EF Core 5 Concurrency Issue in WPF Application with SQLite

I have been trying to troubleshoot this for a little longer than I would like to admit and admittedly this is my first time using EF Core with SQLite, but I cannot figure out what is going on. This is a .NET Core (3.1) WPF app using EF Core 5. It is a single user application. I am trying to add a new entity and this one specific entity has an issue. All of the other entities in the context add and update without any issues. Here is the exception I am getting while trying to add a new entity.
Microsoft.EntityFrameworkCore.DbUpdateConcurrencyException: Database operation expected to affect 1 row(s) but actually affected 0 row(s). Data may have been modified or deleted since entities were loaded.
Here is my code for the entity:
public class SourceDataElement
{
public int Id { get; set; }
public int ColumnIndex { get; set; }
// Foreign key & navigation to SourceDataRow
public int RowId { get; set; }
public SourceDataRow Row { get; set; }
public string StringValue { get; set; }
public int? IntegerValue { get; set; }
public bool? BooleanValue { get; set; }
public DateTime? DateTimeValue { get; set; }
public Double? DoubleValue { get; set; }
// Foreign key & navigation to SourceDataElementDataType
public int SourceDataElementDataTypeId { get; set; }
public SourceDataElementDataType SourceDataElementDataType { get; set; }
public DateTime? LastUpdated { get; set; }
[Timestamp] public byte[] Timestamp { get; set; }
}
I created a small tester application to isolate just this entity and try different things, but I have had no luck. I can manually insert into the table in the database, but there is something with EF Core.
Here is the code I am using to create a new entity:
var dateTimeStamp = DateTime.Now;
var newSourceDataElement = new SourceDataElement()
{
ColumnIndex = 0,
RowId = 1,
StringValue = "Test Data",
IntegerValue = null,
BooleanValue = null,
DateTimeValue = null,
DoubleValue = null,
SourceDataElementDataTypeId = 1,
LastUpdated = dateTimeStamp
};
_context.Add(newSourceDataElement);
await _context.SaveChangesAsync();
Here is the SQL that EF Core is creating. It seems okay to me.
dbug: 10/18/2021 10:41:45.714 RelationalEventId.CommandExecuting[20100] (Microsoft.EntityFrameworkCore.Database.Command)
Executing DbCommand [Parameters=[#p0=NULL, #p1='0' (DbType = String), #p2=NULL, #p3=NULL, #p4=NULL, #p5='2021-10-18T10:41:45.4850016-04:00' (Nullable = true) (DbType = String), #p6='1' (DbType = String), #p7='1' (DbType = String), #p8='Test Data' (Size = 9)], CommandType='Text', CommandTimeout='30']
INSERT INTO "SourceDataElements" ("BooleanValue", "ColumnIndex", "DateTimeValue", "DoubleValue", "IntegerValue", "LastUpdated", "RowId", "SourceDataElementDataTypeId", "StringValue")
VALUES (#p0, #p1, #p2, #p3, #p4, #p5, #p6, #p7, #p8);
SELECT "Id", "Timestamp"
FROM "SourceDataElements"
WHERE changes() = 1 AND "rowid" = last_insert_rowid();
info: 10/18/2021 10:41:45.720 RelationalEventId.CommandExecuted[20101] (Microsoft.EntityFrameworkCore.Database.Command)
Executed DbCommand (8ms) [Parameters=[#p0=NULL, #p1='0' (DbType = String), #p2=NULL, #p3=NULL, #p4=NULL, #p5='2021-10-18T10:41:45.4850016-04:00' (Nullable = true) (DbType = String), #p6='1' (DbType = String), #p7='1' (DbType = String), #p8='Test Data' (Size = 9)], CommandType='Text', CommandTimeout='30']
INSERT INTO "SourceDataElements" ("BooleanValue", "ColumnIndex", "DateTimeValue", "DoubleValue", "IntegerValue", "LastUpdated", "RowId", "SourceDataElementDataTypeId", "StringValue")
VALUES (#p0, #p1, #p2, #p3, #p4, #p5, #p6, #p7, #p8);
SELECT "Id", "Timestamp"
FROM "SourceDataElements"
WHERE changes() = 1 AND "rowid" = last_insert_rowid();
I tried adding the [Concurrency] attribute to the LastUpdated column, I added the timestamp with insert and update triggers to populate it, but nothing seems to help. Not sure what's going on. Any suggestions?
Thanks,
Mike

Table-valued parameter insert performing poorly

I have a TVP+SP insert strategy implemented as i need to insert big amounts of rows (probably concurrently) while being able to get some info in return like Id and stuff. Initially I'm using EF code first approach to generate the DB structure. My entities:
FacilityGroup
public class FacilityGroup
{
public int Id { get; set; }
[Required]
public string Name { get; set; }
public string InternalNotes { get; set; }
public virtual List<FacilityInstance> Facilities { get; set; } = new List<FacilityInstance>();
}
FacilityInstance
public class FacilityInstance
{
public int Id { get; set; }
[Required]
[Index("IX_FacilityName")]
[StringLength(450)]
public string Name { get; set; }
[Required]
public string FacilityCode { get; set; }
//[Required]
public virtual FacilityGroup FacilityGroup { get; set; }
[ForeignKey(nameof(FacilityGroup))]
[Index("IX_FacilityGroupId")]
public int FacilityGroupId { get; set; }
public virtual List<DataBatch> RelatedBatches { get; set; } = new List<DataBatch>();
public virtual HashSet<BatchRecord> BatchRecords { get; set; } = new HashSet<BatchRecord>();
}
BatchRecord
public class BatchRecord
{
public long Id { get; set; }
//todo index?
public string ItemName { get; set; }
[Index("IX_Supplier")]
[StringLength(450)]
public string Supplier { get; set; }
public decimal Quantity { get; set; }
public string ItemUnit { get; set; }
public string EntityUnit { get; set; }
public decimal ItemSize { get; set; }
public decimal PackageSize { get; set; }
[Index("IX_FamilyCode")]
[Required]
[StringLength(4)]
public string FamilyCode { get; set; }
[Required]
public string Family { get; set; }
[Index("IX_CategoryCode")]
[Required]
[StringLength(16)]
public string CategoryCode { get; set; }
[Required]
public string Category { get; set; }
[Index("IX_SubCategoryCode")]
[Required]
[StringLength(16)]
public string SubCategoryCode { get; set; }
[Required]
public string SubCategory { get; set; }
public string ItemGroupCode { get; set; }
public string ItemGroup { get; set; }
public decimal PurchaseValue { get; set; }
public decimal UnitPurchaseValue { get; set; }
public decimal PackagePurchaseValue { get; set; }
[Required]
public virtual DataBatch DataBatch { get; set; }
[ForeignKey(nameof(DataBatch))]
public int DataBatchId { get; set; }
[Required]
public virtual FacilityInstance FacilityInstance { get; set; }
[ForeignKey(nameof(FacilityInstance))]
[Index("IX_FacilityInstance")]
public int FacilityInstanceId { get; set; }
[Required]
public virtual Currency Currency { get; set; }
[ForeignKey(nameof(Currency))]
public int CurrencyId { get; set; }
}
DataBatch
public class DataBatch
{
public int Id { get; set; }
[Required]
public string Name { get; set; }
public DateTime DateCreated { get; set; }
public BatchStatus BatchStatus { get; set; }
public virtual List<FacilityInstance> RelatedFacilities { get; set; } = new List<FacilityInstance>();
public virtual HashSet<BatchRecord> BatchRecords { get; set; } = new HashSet<BatchRecord>();
}
And then my SQL Server related code, TVP Structure:
CREATE TYPE dbo.RecordImportStructure
AS TABLE (
ItemName VARCHAR(MAX),
Supplier VARCHAR(MAX),
Quantity DECIMAL(18, 2),
ItemUnit VARCHAR(MAX),
EntityUnit VARCHAR(MAX),
ItemSize DECIMAL(18, 2),
PackageSize DECIMAL(18, 2),
FamilyCode VARCHAR(4),
Family VARCHAR(MAX),
CategoryCode VARCHAR(MAX),
Category VARCHAR(MAX),
SubCategoryCode VARCHAR(MAX),
SubCategory VARCHAR(MAX),
ItemGroupCode VARCHAR(MAX),
ItemGroup VARCHAR(MAX),
PurchaseValue DECIMAL(18, 2),
UnitPurchaseValue DECIMAL(18, 2),
PackagePurchaseValue DECIMAL(18, 2),
FacilityCode VARCHAR(MAX),
CurrencyCode VARCHAR(MAX)
);
Insert stored procedure:
CREATE PROCEDURE dbo.ImportBatchRecords (
#BatchId INT,
#ImportTable dbo.RecordImportStructure READONLY
)
AS
SET NOCOUNT ON;
DECLARE #ErrorCode int
DECLARE #Step varchar(200)
--Clear old stuff?
--TRUNCATE TABLE dbo.BatchRecords;
INSERT INTO dbo.BatchRecords (
ItemName,
Supplier,
Quantity,
ItemUnit,
EntityUnit,
ItemSize,
PackageSize,
FamilyCode,
Family,
CategoryCode,
Category,
SubCategoryCode,
SubCategory,
ItemGroupCode,
ItemGroup,
PurchaseValue,
UnitPurchaseValue,
PackagePurchaseValue,
DataBatchId,
FacilityInstanceId,
CurrencyId
)
OUTPUT INSERTED.Id
SELECT
ItemName,
Supplier,
Quantity,
ItemUnit,
EntityUnit,
ItemSize,
PackageSize,
FamilyCode,
Family,
CategoryCode,
Category,
SubCategoryCode,
SubCategory,
ItemGroupCode,
ItemGroup,
PurchaseValue,
UnitPurchaseValue,
PackagePurchaseValue,
#BatchId,
--FacilityInstanceId,
--CurrencyId
(SELECT TOP 1 f.Id from dbo.FacilityInstances f WHERE f.FacilityCode=FacilityCode),
(SELECT TOP 1 c.Id from dbo.Currencies c WHERE c.CurrencyCode=CurrencyCode)
FROM #ImportTable;
And finally my quick, test only solution to execute this stuff on .NET side.
public class BatchRecordDataHandler : IBulkDataHandler<BatchRecordImportItem>
{
public async Task<int> ImportAsync(SqlConnection conn, SqlTransaction transaction, IEnumerable<BatchRecordImportItem> src)
{
using (var cmd = new SqlCommand())
{
cmd.CommandText = "ImportBatchRecords";
cmd.Connection = conn;
cmd.Transaction = transaction;
cmd.CommandType = CommandType.StoredProcedure;
cmd.CommandTimeout = 600;
var batchIdParam = new SqlParameter
{
ParameterName = "#BatchId",
SqlDbType = SqlDbType.Int,
Value = 1
};
var tableParam = new SqlParameter
{
ParameterName = "#ImportTable",
TypeName = "dbo.RecordImportStructure",
SqlDbType = SqlDbType.Structured,
Value = DataToSqlRecords(src)
};
cmd.Parameters.Add(batchIdParam);
cmd.Parameters.Add(tableParam);
cmd.Transaction = transaction;
using (var res = await cmd.ExecuteReaderAsync())
{
var resultTable = new DataTable();
resultTable.Load(res);
var cnt = resultTable.AsEnumerable().Count();
return cnt;
}
}
}
private IEnumerable<SqlDataRecord> DataToSqlRecords(IEnumerable<BatchRecordImportItem> src)
{
var tvpSchema = new[] {
new SqlMetaData("ItemName", SqlDbType.VarChar, SqlMetaData.Max),
new SqlMetaData("Supplier", SqlDbType.VarChar, SqlMetaData.Max),
new SqlMetaData("Quantity", SqlDbType.Decimal),
new SqlMetaData("ItemUnit", SqlDbType.VarChar, SqlMetaData.Max),
new SqlMetaData("EntityUnit", SqlDbType.VarChar, SqlMetaData.Max),
new SqlMetaData("ItemSize", SqlDbType.Decimal),
new SqlMetaData("PackageSize", SqlDbType.Decimal),
new SqlMetaData("FamilyCode", SqlDbType.VarChar, SqlMetaData.Max),
new SqlMetaData("Family", SqlDbType.VarChar, SqlMetaData.Max),
new SqlMetaData("CategoryCode", SqlDbType.VarChar, SqlMetaData.Max),
new SqlMetaData("Category", SqlDbType.VarChar, SqlMetaData.Max),
new SqlMetaData("SubCategoryCode", SqlDbType.VarChar, SqlMetaData.Max),
new SqlMetaData("SubCategory", SqlDbType.VarChar, SqlMetaData.Max),
new SqlMetaData("ItemGroupCode", SqlDbType.VarChar, SqlMetaData.Max),
new SqlMetaData("ItemGroup", SqlDbType.VarChar, SqlMetaData.Max),
new SqlMetaData("PurchaseValue", SqlDbType.Decimal),
new SqlMetaData("UnitPurchaseValue", SqlDbType.Decimal),
new SqlMetaData("PackagePurchaseValue", SqlDbType.Decimal),
new SqlMetaData("FacilityInstanceId", SqlDbType.VarChar, SqlMetaData.Max),
new SqlMetaData("CurrencyId", SqlDbType.VarChar, SqlMetaData.Max),
};
var dataRecord = new SqlDataRecord(tvpSchema);
foreach (var importItem in src)
{
dataRecord.SetValues(importItem.ItemName,
importItem.Supplier,
importItem.Quantity,
importItem.ItemUnit,
importItem.EntityUnit,
importItem.ItemSize,
importItem.PackageSize,
importItem.FamilyCode,
importItem.Family,
importItem.CategoryCode,
importItem.Category,
importItem.SubCategoryCode,
importItem.SubCategory,
importItem.ItemGroupCode,
importItem.ItemGroup,
importItem.PurchaseValue,
importItem.UnitPurchaseValue,
importItem.PackagePurchaseValue,
importItem.FacilityCode,
importItem.CurrencyCode);
yield return dataRecord;
}
}
}
Import entity structure:
public class BatchRecordImportItem
{
public string ItemName { get; set; }
public string Supplier { get; set; }
public decimal Quantity { get; set; }
public string ItemUnit { get; set; }
public string EntityUnit { get; set; }
public decimal ItemSize { get; set; }
public decimal PackageSize { get; set; }
public string FamilyCode { get; set; }
public string Family { get; set; }
public string CategoryCode { get; set; }
public string Category { get; set; }
public string SubCategoryCode { get; set; }
public string SubCategory { get; set; }
public string ItemGroupCode { get; set; }
public string ItemGroup { get; set; }
public decimal PurchaseValue { get; set; }
public decimal UnitPurchaseValue { get; set; }
public decimal PackagePurchaseValue { get; set; }
public int DataBatchId { get; set; }
public string FacilityCode { get; set; }
public string CurrencyCode { get; set; }
}
Please don't mind useless reader at the end, doesn't really do much. So without the reader inserting 2.5kk rows took around 26 minutes while SqlBulkCopy took around 6+- minutes. Is there something I'm doing fundamentally wrong? I’m using IsolationLevel.Snapshot if this matters. Using SQL Server 2014, free to change DB structure and indices.
UPD 1
Done a couple of adjustments/improvement attempts described by #Xedni, specifically:
Limited all string fields that didn't have a max length to some fixed length
Changed all TVP members from VARCHAR(MAX) to VARCHAR(*SomeValue*)
Added a unique index to FacilityInstance->FacilityCode
Added a unique index to Curreency->CurrencyCode
Tried adding WITH RECOMPILE to my SP
Tried using DataTable instead of IEnumerable<SqlDataRecord>
Tried batchinng data into smaller buckets, 50k and 100k per SP execution instead of 2.5kk
My structure is now like this:
CREATE TYPE dbo.RecordImportStructure
AS TABLE (
ItemName VARCHAR(4096),
Supplier VARCHAR(450),
Quantity DECIMAL(18, 2),
ItemUnit VARCHAR(2048),
EntityUnit VARCHAR(2048),
ItemSize DECIMAL(18, 2),
PackageSize DECIMAL(18, 2),
FamilyCode VARCHAR(16),
Family VARCHAR(512),
CategoryCode VARCHAR(16),
Category VARCHAR(512),
SubCategoryCode VARCHAR(16),
SubCategory VARCHAR(512),
ItemGroupCode VARCHAR(16),
ItemGroup VARCHAR(512),
PurchaseValue DECIMAL(18, 2),
UnitPurchaseValue DECIMAL(18, 2),
PackagePurchaseValue DECIMAL(18, 2),
FacilityCode VARCHAR(450),
CurrencyCode VARCHAR(4)
);
So far no noticeable performance gains unfortunately, 26-28 min as before
UPD 2
Checked the execution plan - indices are my bane?
UPD 3
Added OPTION (RECOMPILE); at the end of my SP, gained a minor boost, now sitting at ~25m for 2.5kk

You could set traceflag 2453:
FIX: Poor performance when you use table variables in SQL Server 2012 or SQL Server 2014
When you use a table variable in a batch or procedure, the query is compiled and optimized for the initial empty state of table variable. If this table variable is populated with many rows at runtime, the pre-compiled query plan may no longer be optimal. For example, the query may be joining a table variable with nested loop since it is usually more efficient for small number of rows. This query plan can be inefficient if the table variable has millions of rows. A hash join may be a better choice under such condition. To get a new query plan, it needs to be recompiled. Unlike other user or temporary tables, however, row count change in a table variable does not trigger a query recompile. Typically, you can work around this with OPTION (RECOMPILE), which has its own overhead cost.
The trace flag 2453 allows the benefit of query recompile without OPTION (RECOMPILE). This trace flag differs from OPTION (RECOMPILE) in two main aspects.
(1) It uses the same row count threshold as other tables. The query does not need to be compiled for every execution unlike OPTION (RECOMPILE). It would trigger recompile only when the row count change exceeds the predefined threshold.
(2) OPTION (RECOMPILE) forces the query to peek parameters and optimize the query for them. This trace flag does not force parameter peeking.
You can turn on trace flag 2453 to allow a table variable to trigger recompile when enough number of rows are changed. This may allow the query optimizer to choose a more efficient plan

Try with the following stored procedure:
CREATE PROCEDURE dbo.ImportBatchRecords (
#BatchId INT,
#ImportTable dbo.RecordImportStructure READONLY
)
AS
SET NOCOUNT ON;
DECLARE #ErrorCode int
DECLARE #Step varchar(200)
CREATE TABLE #FacilityInstances
(
Id int NOT NULL,
FacilityCode varchar(512) NOT NULL UNIQUE WITH (IGNORE_DUP_KEY=ON)
);
CREATE TABLE #Currencies
(
Id int NOT NULL,
CurrencyCode varchar(512) NOT NULL UNIQUE WITH (IGNORE_DUP_KEY = ON)
)
INSERT INTO #FacilityInstances(Id, FacilityCode)
SELECT Id, FacilityCode FROM dbo.FacilityInstances
WHERE FacilityCode IS NOT NULL AND Id IS NOT NULL;
INSERT INTO #Currencies(Id, CurrencyCode)
SELECT Id, CurrencyCode FROM dbo.Currencies
WHERE CurrencyCode IS NOT NULL AND Id IS NOT NULL
INSERT INTO dbo.BatchRecords (
ItemName,
Supplier,
Quantity,
ItemUnit,
EntityUnit,
ItemSize,
PackageSize,
FamilyCode,
Family,
CategoryCode,
Category,
SubCategoryCode,
SubCategory,
ItemGroupCode,
ItemGroup,
PurchaseValue,
UnitPurchaseValue,
PackagePurchaseValue,
DataBatchId,
FacilityInstanceId,
CurrencyId
)
OUTPUT INSERTED.Id
SELECT
ItemName,
Supplier,
Quantity,
ItemUnit,
EntityUnit,
ItemSize,
PackageSize,
FamilyCode,
Family,
CategoryCode,
Category,
SubCategoryCode,
SubCategory,
ItemGroupCode,
ItemGroup,
PurchaseValue,
UnitPurchaseValue,
PackagePurchaseValue,
#BatchId,
F.Id,
C.Id
FROM
#FacilityInstances F RIGHT OUTER HASH JOIN
(
#Currencies C
RIGHT OUTER HASH JOIN #ImportTable IT
ON C.CurrencyCode = IT.CurrencyCode
)
ON F.FacilityCode = IT.FacilityCode
This enforces the execution plan to use hash match joins instead of nested loops. I think the culprit of bad performance is the first nested loop that performs an index scan for each row in #ImportTable
I don't know if CurrencyCode is unique in Currencies table, so I create the temporal table #Currencies with unique currency codes.
I don't know if FacilityCode is unique in Facilities table, so I create the temporal table #FacilityInstances with unique facility codes.
If they are unique you don't need the temporal tables, you can use the permanent tables directly.
Assuming CurrencyCode and FacilityCode are unique the following stored procedure would be better because it doesn't create unnecessary temporary tables:
CREATE PROCEDURE dbo.ImportBatchRecords (
#BatchId INT,
#ImportTable dbo.RecordImportStructure READONLY
)
AS
SET NOCOUNT ON;
DECLARE #ErrorCode int
DECLARE #Step varchar(200)
INSERT INTO dbo.BatchRecords (
ItemName,
Supplier,
Quantity,
ItemUnit,
EntityUnit,
ItemSize,
PackageSize,
FamilyCode,
Family,
CategoryCode,
Category,
SubCategoryCode,
SubCategory,
ItemGroupCode,
ItemGroup,
PurchaseValue,
UnitPurchaseValue,
PackagePurchaseValue,
DataBatchId,
FacilityInstanceId,
CurrencyId
)
OUTPUT INSERTED.Id
SELECT
ItemName,
Supplier,
Quantity,
ItemUnit,
EntityUnit,
ItemSize,
PackageSize,
FamilyCode,
Family,
CategoryCode,
Category,
SubCategoryCode,
SubCategory,
ItemGroupCode,
ItemGroup,
PurchaseValue,
UnitPurchaseValue,
PackagePurchaseValue,
#BatchId,
F.Id,
C.Id
FROM
dbo.FacilityInstances F RIGHT OUTER HASH JOIN
(
dbo.Currencies C
RIGHT OUTER HASH JOIN #ImportTable IT
ON C.CurrencyCode = IT.CurrencyCode
)
ON F.FacilityCode = IT.FacilityCode

I would guess your proc could use some love. Without seeing an execution plan its hard to say for sure, but here are some thoughts.
A table variable (which a table-valued-parameter essentially is) is always assumed by SQL Server to contain exactly 1 row (even if it doesn't). This is irrelevant for many cases, but you have two correlated subqueries in your insert list which is where I'd focus my attention. It's more than likely hammering that poor table variable with a bunch of nested loop joins because of the cardinality estimate. I would consider putting the rows from your TVP into a temp table, updating the temp table with the IDs from FacilityInstances and Currencies then do your final insert from that.

Well... why not just use SQL Bulk Copy?
There's plenty of solutions out there that help you convert a collection of entities into a IDataReader object that can be handed directly to SqlBulkCopy.
This is a good start...
https://github.com/matthewschrager/Repository/blob/master/Repository.EntityFramework/EntityDataReader.cs
Then it becomes as simple as...
SqlBulkCopy bulkCopy = new SqlBulkCopy(connection);
IDataReader dataReader = storeEntities.AsDataReader();
bulkCopy.WriteToServer(dataReader);
I've used this code, the one caveat is that you need to be quite careful about the definition of your entity. The order of the properties in the entity determines the order of the columns exposed by the IDataReader and this needs to correlate with the order of the columns in the table that you are bulk copying to.
Alternatively there's other code here..
https://www.codeproject.com/Tips/1114089/Entity-Framework-Performance-Tuning-Using-SqlBulkC

I know there is an accepted answer, but I can't resist. I believe you can improve the performance 20-50% over the accepted answer.
The key is to SqlBulkCopy to the final table dbo.BatchRecords directly.
To make this happen you need FacilityInstanceId and CurrencyId before to SqlBulkCopy. To get them, load SELECT Id, FacilityCode FROM FacilityIntances and SELECT Id, CurrencyCode FROM Currencies into collections, then build a dictionary:
var facilityIdByFacilityCode = facilitiesCollection.ToDictionary(x => x.FacilityCode, x => x.Id);
var currencyIdByCurrencyCode = currenciesCollection.ToDictionnary(x => x.CurrencyCode, x => x.Id);
Once you have the dictionaries, getting the id's from the codes is constant time cost. This is equivalent and very similar to HASH MATCH JOIN in SQL Server, but at the client side.
The other barrier you need to tear down is to get the Id column of new inserted rows in dbo.BatchRecords table. Actually can you get the Ids before inserting them.
Make the Id column "sequence driven":
CREATE SEQUENCE BatchRecords_Id_Seq START WITH 1;
CREATE TABLE BatchRecords
(
Id int NOT NULL CONSTRAINT DF_BatchRecords_Id DEFAULT (NEXT VALUE FOR BatchRecords_Id_Seq),
.....
CONSTRAINT PK_BatchRecords PRIMARY KEY (Id)
)
One you have the BatchRecords collection, you know how many records are in it. You can then reserve a contiguous range of sequences. Execute the following T-SQL:
DECLARE #BatchCollectionCount int = 2500 -- Replace with the actual value
DECLARE #range_first_value sql_variant
DECLARE #range_last_value sql_variant
EXEC sp_sequence_get_range
#sequence_name = N'BatchRecords_Id_Seq',
#range_size = #BatchCollectionCount,
#range_first_value = #range_first_value OUTPUT,
#range_last_value = #range_last_value OUTPUT
SELECT
CAST(#range_first_value AS INT) AS range_first_value,
CAST(#range_last_value AS int) as range_last_value
This returns range_first_value and range_last_value. You can now assign BatchRecord.Id to each record:
int id = range_first_value;
foreach (var record in batchRecords)
{
record.Id = id++;
}
Next, you can SqlBulkCopy the batch record collection directly into the final table dbo.BatchRecords.
To get a DataReader from an IEnumerable<T> to feed SqlBulkCopy.WriteToServer you can use code like this which is part of EntityLite, a micro ORM I developed.
You can make it even faster if you cache facilityIdByFacilityCode and currencyIdByCurrencyCode. To be sure these dictionaries are up to date you can use SqlDependencyor techniques like this one.

Unnecessary conversion to bigint

I have employee table with bigint primary key field in database and entity data model with database first approach. Employee class have this structure
public partial class Employee
{
public long Emp_No { get; set; }
public string Name { get; set; }
public string Family { get; set; }
...
}
I write this basic query with Entity Framework
List<long> ids = new List<long>() {1,2,3,4,5,6}
database.Employees.Where(q => ids.Contain(q.Emp_No)).ToList();
It Generate query as the following:
SELECT
[Extent1].[Emp_No] AS [Emp_No],
[Extent1].[Name] AS [Name],
[Extent1].[Family] AS [Family],
...
FROM [dbo].[Employee] AS [Extent1]
WHERE [Extent1].[Emp_No] IN (cast(0 as bigint),
cast(1 as bigint),
cast(2 as bigint),
cast(3 as bigint),
cast(4 as bigint),
cast(5 as bigint),
cast(6 as bigint))
As you can see there is unnecessary cast to bigint in query while both type of Emp_No and ids array are long, It causes bad execution times specially whenever ids array has many elements.
How can I remove this redundant cast?

There is virtually no cost in the conversion cast(0 as bigint) and because Emp_No is also a bigint if you did not have the cast there the int would still need to be promoted to a bigint to be able to do the IN comparision so the cast would still happen, just behind the scenes.
Run the non cast version of the query yourself in management studio and get the actual execution plan and you will still see the conversion in the query plan.

Not really sure what you're asking for here but..
Change your long to int and the query should make you an int instead of bigint.
public partial class Employee
{
public int Emp_No { get; set; }
public string Name { get; set; }
public string Family { get; set; }
....
}
Long is the equivalent of bigint.
You can read more here: What is the equivalent of bigint in C#?

C# Entity Framework Error Converting nvarchar to int by enum

I have an issue receiving the following error:
Conversion failed when converting the nvarchar value 'HolidayRequest' to data type int.
I have an inheritance structure within Entity Framework. My base Model is as follows:
public abstract class Request
{
public int ID { get; set; }
public int EmployeeID { get; set; }
public string Subject { get; set; }
[Display(Name = "Date Requested")]
public DateTime DateRequested { get; set; }
[Display(Name = "Start Date")]
public DateTime StartDateTime { get; set; }
[Display(Name = "Return To Work Date")]
public DateTime EndDateTime { get; set; }
public RequestStatus Status { get; set; }
public string Notes { get; set; }
public int? ResponseID { get; set; }
public RequestDiscriminator Discriminator { get; set; }
[ForeignKey("EmployeeID")]
public virtual Employee Employee { get; set; }
[ForeignKey("ResponseID")]
public virtual Response Response { get; set; }
}
When I attempt to retrieve data from the database as follows:
public IQueryable<Request> GetPendingHolidayRequestsByEmployeeId(int employeeId)
{
var list = _context.Requests.Where(p => p.Discriminator == RequestDiscriminator.Holiday && p.EmployeeID == employeeId && p.Status == RequestStatus.NotProcessed);
return list;
}
This is where the error occurs.
Given the context of the error I believe that it is an issue with this enum public RequestDiscriminator Discriminator { get; set; }
The structure of the Enum is:
public enum RequestDiscriminator
{
Holiday,
Absence
}
The SQL statement that is being sent by entity framework is:
SELECT
[Extent1].[ID] AS [ID],
CASE WHEN ([Extent1].[Discriminator1] = N'AbsenceRequest') THEN '0X0X' ELSE '0X1X' END AS [C1],
[Extent1].[EmployeeID] AS [EmployeeID],
[Extent1].[Subject] AS [Subject],
[Extent1].[DateRequested] AS [DateRequested],
[Extent1].[StartDateTime] AS [StartDateTime],
[Extent1].[EndDateTime] AS [EndDateTime],
[Extent1].[Status] AS [Status],
[Extent1].[Notes] AS [Notes],
[Extent1].[ResponseID] AS [ResponseID],
[Extent1].[Discriminator] AS [Discriminator]
CASE WHEN ([Extent1].[Discriminator1] = N'AbsenceRequest') THEN [Extent1].[ReasonCode] END AS [C2],
CASE WHEN ([Extent1].[Discriminator1] = N'AbsenceRequest') THEN [Extent1].[TakenPaid] END AS [C3],
CASE WHEN ([Extent1].[Discriminator1] = N'AbsenceRequest') THEN CAST(NULL AS float) ELSE [Extent1].[TotalDays] END AS [C4],
CASE WHEN ([Extent1].[Discriminator1] = N'AbsenceRequest') THEN [Extent1].[Employee_EmployeeID] END AS [C5],
CASE WHEN ([Extent1].[Discriminator1] = N'AbsenceRequest') THEN CAST(NULL AS int) ELSE [Extent1].[Employee_EmployeeID1] END AS [C6]
FROM [dbo].[Request] AS [Extent1]
WHERE ([Extent1].[Discriminator1] IN (N'AbsenceRequest',N'HolidayRequest')) AND (0 = [Extent1].[Discriminator]) AND (0 = [Extent1].[Status])
Any help would be greatly appreciated as i am completely stumped.
UPDATE
As per the suggestion in the answer below, I have removed both Discriminator columns from the database and readded mine. However, the SQL query still appears to be looking for the Discriminator1 column that now doesnt exist.

I have managed to solve this. The answer was to remove the discriminator column i created, and allow EF to determine the type by including the type in the linq query as follows:
var test = _context.Requests.Where(r => r.EmployeeID == employeeId).OfType<AbsenceRequest>();
The OfType<>() seems to tell EF to look at the Discriminator column for this type name and only return data of that type.

It looks like you have an extra column in there "Discriminator1". Did you originally have that field as a string, then tried to convert it to an enum? The best option might be to try dropping the field altogether and make sure it drops both Discriminator and Discriminator1. Then try adding it back. This can occur in when having custom mappings, so something may be off there.

System.Data.SqlClient.SqlException Thrown by Dapper When Query Result Has More Than 1000 Records

The method call below fails with the message "The conversion of a varchar data type to a datetime data type resulted in an out-of-range value.":
public IEnumerable<SomeResult> GetResults(SqlConnection connection, string attribute)
{
var sql = string.Format(#"
SELECT TOP 2000
r.Id
,r.LastName
,r.FirstName
,r.Ssn
,r.CurrentId
,BeginDate = case when isdate(rli.BeginDate) = 1 then convert(datetime, rli.BeginDate) else NULL end
,EndDate = case when isdate(rli.EndDate) = 1 then convert(datetime, rli.EndDate) else NULL end
,rli.LcknTyCd
,rli.ProvId
FROM
[dbo].[Span] rli
INNER JOIN [dbo].Recipient r
ON rli.SysId = r.SysId
INNER JOIN [dbo].ValidRecipient lc
ON r.SysId = lc.SysId
WHERE
BeginDate <= GETDATE()
AND EndDate >= GETDATE()
AND rli.LcknTyCd = #LcknTyCd);
return connection.Query<SomeResult>(sql, new { LcknTyCd = attribute}).ToList();
}
public struct SomeResult
{
public string Id{ get; set; }
public string LastName { get; set; }
public string FirstName { get; set; }
public string Ssn { get; set; }
public string CurrentId{ get; set; }
public DateTime? BeginDate { get; set; }
public DateTime? EndDate { get; set; }
public string LcknTyCd{ get; set; }
public string ProvId{ get; set; }
}
If the result set contains 1000 (or fewer) records, the code works correctly. When I execute the query in SQL Server Management Studio (2014 edition), I don't get an error either. Even when I remove the TOP from the select and execute it in SSMS, no error occurs (12,000+ records are returned, as expected).
What should I be doing instead of the above implementation to successfully retrieve result sets with more than 1000 rows? Would a stored procedure be more appropriate in this case?

Sounds like your date fields are stored in a varchar column.
Ideally, you should change those to datetime fields.
If that's not an option, change your WHERE clause to look like this:
WHERE
case when isdate(rli.BeginDate) = 1 then convert(datetime, rli.BeginDate) else NULL end <= GETDATE()
AND case when isdate(rli.EndDate) = 1 then convert(datetime, rli.EndDate) else NULL end >= GETDATE()
AND rli.LcknTyCd = #LcknTyCd);
The reason it succeeded on your top 1000 query is likely because the top 1000 records found all contained valid dates.

That is a database server error: dapper doesn't know about varchar and doesn't take in terms of varchar - it talks about .net Strings. So: one of your dates-stored-as-varchar is broken and does not contain a valid value.
Basically: try this query in SSMS: I expect it will break there too!
Changing to a stored procedure will not change this at all. What needs to change is the broken data - and (more importantly) the bad choice of storing date/time data in a text-based column.