I got following issue while running below code
Transaction (Process ID 54) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. Entity framework
string[] to = new string[] { "A", "B" };
Parallel.ForEach(to, code =>
{
DeleteHouses(code);
});
private static void DeleteHouses(string code)
{
using var ctx = new BloggingContext();
using var trans = ctx.Database.BeginTransaction(System.Data.IsolationLevel.ReadUncommitted);
var todel = ctx.dbsHouses.Where(d => d.Code == code).ToList();
if (todel != null)
{
ctx.dbsHouses.RemoveRange(todel);
ctx.SaveChanges();//Exception
}
trans.Commit();
}
I tried Read Read Uncommitted IsolationLevel and using
dataContext.ExecuteCommand("SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED");
But none of them working for me
Related
I've seen a lot of posts about this, but I'm still not quite sure what the answer is. I inherited some code that is handling items one by one. It does a bunch of stuff (it's somewhat CPU intensive), then needs to save data to the database, and update the status of the item. It uses EF.
Here is the existing method:
private void CheckForThingsToDo(IContainer container)
{
var finished = false;
while (!finished)
{
using (var scope = container.BeginLifetimeScope("Console Request"))
{
var queries = scope.Resolve<IQueries>();
var commands = scope.Resolve<ICommands>();
var context = scope.Resolve<IContext>();
var nextItem = queries.GetNextItem();
finished = (nextItem == null);
if (finished) continue;
using (var transaction = context.BeginTransaction())
{
if (nextItem == null) return;
try
{
commands.ProcessItem(nextItem.Id); // this is somewhat CPU intensive
transaction.Commit();
}
catch(Exception ex)
{
_logger.Error(ex.Message, ex);
}
}
}
}
}
I would like to be able to run these in parallel because it's maxing out one core and the rest of the server is sitting there.
private void CheckForThingsToDoParallel(IContainer container)
{
using (var scope = container.BeginLifetimeScope("Console Request"))
{
var context0 = new EntityFrameworkRecordkeepingContext();
var queries = new AccountTransactionQueries(context0, mapper);
var items = queries.GetItems();
Parallel.ForEach(items,
new ParallelOptions { MaxDegreeOfParallelism = 4 },
() => new EntityFrameworkRecordkeepingContext(),
(item, parallelLoopState, context) =>
{
try
{
var queries = new Queries(context);
var repo = new Repository(context);
var commands = new Commands(repo, queries);
using (var transaction = context.BeginTransaction())
{
commands.ProcessItem(nextBatchItem);
transaction.Commit();
}
}
catch (Exception ex)
{
_logger.Error(ex.Message, ex);
}
return context;
},
(context) =>
{
context.Dispose();
});
}
}
Should something like that work? It works some of the time. But I end up getting this error:
DbUpdateConcurrencyException: Store update, insert, or delete statement affected an unexpected number of rows (0). Entities may have been modified or deleted since entities were loaded
Each item is a separate thing, so any inserts done in relation to one item should not impact another set of inserts from a different item.
I feel like I might be missing something obvious. Some of the parallel code is me trying to figure stuff out. Any thoughts? I need a push in the right direction.
I'm using Entity Framework 6.0 and SQL Server 2016 for my ASP.Net Website. I recently found a problem with concurrency at one of my function. The function is used for processing unpaid order and sometimes this function is executed multiple times for the same key and the same time (because multiple user access it together).
Here is what it looks like.
public void PaidOrder(string paymentCode)
{
using (MyEntities db = new MyEntities())
{
using (DbContextTransaction trans = db.Database.BeginTransaction())
{
try
{
Order_Payment_Code payment = db.Order_Payment_Code.Where(item => item.PaymentCode == paymentCode).FirstOrDefault();
if(payment.Status == PaymentStatus.NotPaid)
{
//This Scope can be executed multiple times
payment.Status = PaymentStatus.Paid;
db.Entry(payment).State = EntityState.Modified;
db.SaveChanges();
//Continue processing Order
trans.Commit();
}
}
catch (Exception ex)
{
trans.Rollback();
}
}
}
}
What I don't understand is why scope inside my if statement can be executed multiple time even it is inside a transaction? Isn't transaction suppose to be isolating the data? Or my understanding of transaction is wrong? If so, then what is the correct way to make the scope inside my if statement only executed once?
A simple and reliable way to serialize an EF SQL Server transaction is to use an Application Lock.
Add this method to your DbContext:
public void GetAppLock(string lockName)
{
var sql = "exec sp_getapplock #lockName, 'exclusive';";
var pLockName = new SqlParameter("#lockName", SqlDbType.NVarChar, 255);
pLockName.Value = lockName;
this.Database.ExecuteSqlCommand(sql, pLockName);
}
And call it just after you start your transaction.
public void PaidOrder(string paymentCode)
{
using (MyEntities db = new MyEntities())
{
using (DbContextTransaction trans = db.Database.BeginTransaction())
{
db.GetAppLock("PaidOrder");
Order_Payment_Code payment = db.Order_Payment_Code.Where(item => item.PaymentCode == paymentCode).FirstOrDefault();
if(payment.Status == PaymentStatus.NotPaid)
{
//This Scope can be executed multiple times
payment.Status = PaymentStatus.Paid;
db.Entry(payment).State = EntityState.Modified;
db.SaveChanges();
//Continue processing Order
}
trans.Commit();
}
}
}
Then only one instance of that transaction can run at a time, even if you have multiple front-end servers. So this is like a Mutex that works across all the clients that access the same database.
I'm using EF and saving my POCO objects using this function:
public void SaveAll(IList<CoreEntity> entitaCoreList)
{
bool result = false;
using (var context = new NSTEntities())
{
//context.Configuration.AutoDetectChangesEnabled = false;
foreach (var entitaCore in entitaCoreList)
{
TRACCIAVEICOLO_T500 tracciamentoVeicoliEF = new TRACCIAVEICOLO_T500();
tracciamentoVeicoliEF.C_IDTRACCIAMENTOVEICOLO = tracciaVeicolo.Id;
CultureInfo ci = CultureInfo.CreateSpecificCulture("en-EN");
tracciamentoVeicoliEF.Z_COORD = System.Data.Spatial.DbGeography.PointFromText(
"POINT (" + tracciaVeicolo.Longitudine.ToString(ci) + " " + tracciaVeicolo.Latitudine.ToString(ci) + ")", 4326);
tracciamentoVeicoliEF.D_DATARILEVAZIONE = tracciaVeicolo.DataRilevazione;
tracciamentoVeicoliEF.C_CODICEWEBFLEET = tracciaVeicolo.CodiceVeicoloWebfleet;
tracciamentoVeicoliEF.S_POSITIONSTRING = tracciaVeicolo.posString;
tracciamentoVeicoliEF.P_TIPOMESSAGGIO = (int) tracciaVeicolo.TipoMessaggio;
tracciamentoVeicoliEF.V_VELOCITA = tracciaVeicolo.Velocita;
tracciamentoVeicoliEF.V_DIREZIONE = tracciaVeicolo.Direzione;
tracciamentoVeicoliEF.S_GPSSTATUS = tracciaVeicolo.GpsStatus;
tableSet.Add(tracciamentoVeicoliEF);
}
context.SaveChanges();
}
}
}
But it's very slow, it takes nearly 25 seconds for 1000 records.
I tried using a raw query like this:
public void SaveRaw(List<TracciaVeicolo> v)
{
StringBuilder query = new StringBuilder();
query.Append(#"INSERT INTO [dbo].[TRACCIAMENTOVEICOLI_T500]([Z_COORD],[C_CODICEWEBFLEET],[D_DATARILEVAZIONE],[S_POSITIONSTRING],[P_TIPOMESSAGGIO],[V_VELOCITA],[V_DIREZIONE],[S_GPSSTATUS])VALUES ");
bool first = true;
foreach(var o in v)
{
if (!first)
{
query.Append(",");
}
query.AppendFormat("(geography::Point(47.65100, -122.34900, 4326),'{0}','{1}','{2}',{3},{4},{5},'{6}')"
, o.CodiceVeicoloWebfleet
,o.DataRilevazione.ToString("yyyy-dd-MM HH:mm:ss")
,o.posString
, (int)o.TipoMessaggio
, o.Velocita
, o.Direzione
, o.GpsStatus);
first = false;
}
using (var context = new NSTEntities())
{
context.Database.ExecuteSqlCommand(query.ToString());
}
}
And it takes 5 seconds. Am I using EF wrong? I've also tried using context.Configuration.AutoDetectChangesEnabled = false; (as you can see in the first code snippet) but it doesn't change anything
The query EF is running is like this:
declare #p3 sys.geography
set #p3=convert(sys.geography,0xE6100000010CE9297288B82F44404DF4F92823263240)
exec sp_executesql N'insert [dbo].[TRACCIAMENTOVEICOLI_T500]([Z_COORD], [C_CODICEWEBFLEET], [D_DATARILEVAZIONE], [S_POSITIONSTRING], [P_TIPOMESSAGGIO], [V_VELOCITA], [V_DIREZIONE], [S_GPSSTATUS])
values (#0, #1, #2, #3, #4, #5, #6, #7)
select [C_IDTRACCIAMENTOVEICOLO]
from [dbo].[TRACCIAMENTOVEICOLI_T500]
where ##ROWCOUNT > 0 and [C_IDTRACCIAMENTOVEICOLO] = scope_identity()',N'#0 [geography],#1 nvarchar(20),#2 datetime2(7),#3 nvarchar(256),#4 int,#5 float,#6 int,#7 char(1)',#0=#p3,#1=N'1-83645-666EE1173',#2='2016-02-29 15:34:57',#3=N'Vicino a Lecce, 1a Lecce Centro-1B ',#4=0,#5=8,3333333333333339,#6=50,#7='A'
You can try executing it by combining several operations into one transaction. This will save you a lot of time which goes into network latency when you perform single operations.
using (var context = new NSTEntities())
{
using (var dbContextTransaction = context.Database.BeginTransaction())
{
try
{
[... foreach ... tableSet.Add(...) ...]
context.SaveChanges();
dbContextTransaction.Commit();
}
catch (Exception exception)
{
dbContextTransaction.Rollback();
// Log exception (never ignore)
}
}
}
You can also log the SQL-operations to determine what is happening.
For example using something like: context.Database.Log = s => Debug.WriteLine(s);
As you noticed, Entity Framework SaveChanges is very slow since a database round trip is made for every changes.
They have some best practices you can use like using AddRange instead of "Add" but at the end, you will still have some performance issue because 1000 database round trips will be performed.
Disclaimer: I'm the owner of the project Entity Framework Extensions
One way to dramatically improve performance is by using a third party library which allow to use bulk operations.
public void SaveAll(IList<CoreEntity> entitaCoreList)
{
bool result = false;
using (var context = new NSTEntities())
{
// ... code ...
context.BulkSaveChanges();
}
}
I've looked around but couldn't find an appropriate (or one that is satisfactory to me) on how to address an issue we are having.
I use Dapper and ODP.NET 12 Managed drivers. The problem is not encountered when TransactionScope is not used.
When performing commands under a transaction scope, I get an error "Transaction has aborted" via the TransactionAbortedException thrown.
Observed behaviors:
1) TransactionAbortedException is thrown if and only if the transaction is completed and the TransactionScope is disposed. The point at when the exception is thrown is during dispose.
2) Despite the exception, the transaction concept actually works! After Complete() is invoked, the changes are committed into the database.
Below is the code snippet.
// Conn string: "Data Source=OraDB;Persist Security Info=True;User ID=userxxx;Password=passwordxxx;" providerName="Oracle.ManagedDataAccess.Client
// Note: GetDbFactory().Create() returns a DbConnection object
using (TransactionScope scope = new TransactionScope(TransactionScopeOption.RequiresNew, new TransactionOptions { IsolationLevel = System.Transactions.IsolationLevel.ReadCommitted }))
using (var dbConn = GetDbFactory().Create())
{
foreach (MyDTO dto in dtoList)
{
var tableDAO= new TableDAO(dbConn);
MyEntity entity = new MyEntity()
{
Field1 = dto.Field1,
Field2 = dto.Field2
};
tableDAO.AddOrUpdate(entity);
}
// Commit changes
scope.Complete();
}
// This method is under the DAO class
public void AddOrUpdate(MyEntity entity)
{
// Verify arguments
entity.AsArgumentThrowExceptionIfNull("entity");
// build param
OracleDynamicParameters parameters = new OracleDynamicParameters();
parameters.Add("P_FIELD1", entity.Field1);
parameters.Add("P_FIELD2", entity.Field2);
// execute SP
dbConnection.Execute("PRC_MY_ENTITY_ADDORUPDATE", parameters, commandType: CommandType.StoredProcedure);
}//-- end AddOrUpdate()
==================================================================
UPDATE (09-Apr-15)
I have changed my approach and use the following pattern for now for Oracle. Our code deals with connections in both Oracle and SQL Server so I'd much prefer that the coding pattern is consistent, but until a solution is found with using Oracle+TransactionScope, we'll use the pattern below for Oracle command execution:
using (var dbConnection = dbConnFactory.Create())
{
// Open db connection
dbConnection.Open();
using (var trans = dbConnection.BeginTransaction())
{
bool isSuccess = false;
try
{
// Perform DB operations here
trans.Commit();
isSuccess = true;
}
finally
{
if(!isSuccess) trans.Rollback();
}
}
}
First, the exception is documented as possible:
A call to the Dispose method marks the end of the transaction scope. Exceptions that occur after calling this method may not affect the transaction.
This is in the documentation on the TransactionScope class (https://msdn.microsoft.com/en-us/library/system.transactions.transactionscope%28v=vs.110%29.aspx).
One other thing I noticed is that the complete on the transaction is being before the connection is closed. I'd change this to close the connection and then complete the transaction.
using (TransactionScope scope = new TransactionScope(TransactionScopeOption.RequiresNew, new TransactionOptions { IsolationLevel = System.Transactions.IsolationLevel.ReadCommitted }))
{
using (var dbConn = GetDbFactory().Create())
{
foreach (MyDTO dto in dtoList)
{
var tableDAO= new TableDAO(dbConn);
MyEntity entity = new MyEntity()
{
Field1 = dto.Field1,
Field2 = dto.Field2
};
tableDAO.AddOrUpdate(entity);
}
}
// Commit changes
scope.Complete();
}
I don't see anything wrong with that code. However, if that loop runs long enough, the transaction is going to timeout. You'll then get the exception in question the next time you do an operation against the database. I would try increasing the timeout - Timeout is a property on the TransactionScopeOption class.
I am resorting to using BeginTransaction() as the final approach (refer to my update in my original post). I have read more about why TransactionScope() was failing.
1) ODP.Net promotes to distributed transaction even when using a single DB connection when connecting to Oracle 10g and below (source). Lo and behold, the database I'm connecting to is indeed 10g.
2) You'll need Oracle MTS Service installed. This I didn't have setup on my dev machine.
I have two tables which need to be inserted when my application run.
Let's say that I have tables as followed
tbl_FirstTable and tbl_SecondTable
My problem is data volume.
I need to insert over 10,000 rows to tbl_FirstTable and over 500,000 rows to tbl_SecondTable.
So fristly, I use entity framework as follow.
public bool Save_tbl_FirstTable_Vs_tbl_SecondTable(List<tbl_FirstTable> List_tbl_FirstTable, List<tbl_SecondTable> List_tbl_SecondTable)
{
bool IsSuccessSave = false;
try
{
using (DummyDBClass_ObjectContext _DummyDBClass_ObjectContext = new DummyDBClass_ObjectContext())
{
foreach (tbl_FirstTable _tbl_FirstTable in List_tbl_FirstTable)
{
_DummyDBClass_ObjectContext.tbl_FirstTable.InsertOnSubmit(_tbl_FirstTable);
}
foreach (tbl_SecondTable _tbl_SecondTable in List_tbl_SecondTable)
{
_DummyDBClass_ObjectContext.tbl_SecondTable.InsertOnSubmit(_tbl_SecondTable);
}
_DummyDBClass_ObjectContext.SubmitChanges();
IsSuccessSave = true;
}
}
catch (Exception ex)
{
Log4NetWrapper.WriteError(string.Format("{0} : {1} : Exception={2}",
this.GetType().FullName,
(new StackTrace(new StackFrame(0))).GetFrame(0).GetMethod().Name.ToString(),
ex.Message.ToString()));
if (ex.InnerException != null)
{
Log4NetWrapper.WriteError(string.Format("{0} : {1} : InnerException Exception={2}",
this.GetType().FullName,
(new StackTrace(new StackFrame(0))).GetFrame(0).GetMethod().Name.ToString(),
ex.InnerException.Message.ToString()));
}
}
return IsSuccessSave;
}
That is the place I face error Time out exception.
I think that exception will be solved If I use below code.
DummyDBClass_ObjectContext.CommandTimeout = 1800; // 30 minutes
So I used it. It solved but I face another error OutOfMemory Exception.
So I searched the solutions, fortunately, I found below articles.
Problem with Bulk insert using Entity Framework
Using Transactions with SqlBulkCopy
Performing a Bulk Copy Operation in a Transaction
According to that articles, I change my code from Entity Framework to Classic ADO.net code.
public bool Save_tbl_FirstTable_Vs_tbl_SecondTable(DataTable DT_tbl_FirstTable, DataTable DT_tbl_SecondTable)
{
bool IsSuccessSave = false;
SqlTransaction transaction = null;
try
{
using (DummyDBClass_ObjectContext _DummyDBClass_ObjectContext = new DummyDBClass_ObjectContext())
{
var connectionString = ((EntityConnection)_DummyDBClass_ObjectContext.Connection).StoreConnection.ConnectionString;
using (SqlConnection connection = new SqlConnection(connectionString))
{
connection.Open();
using (transaction = connection.BeginTransaction())
{
using (SqlBulkCopy bulkCopy_tbl_FirstTable = new SqlBulkCopy(connection, SqlBulkCopyOptions.KeepIdentity, transaction))
{
bulkCopy_tbl_FirstTable.BatchSize = 5000;
bulkCopy_tbl_FirstTable.DestinationTableName = "dbo.tbl_FirstTable";
bulkCopy_tbl_FirstTable.ColumnMappings.Add("ID", "ID");
bulkCopy_tbl_FirstTable.ColumnMappings.Add("UploadFileID", "UploadFileID");
bulkCopy_tbl_FirstTable.ColumnMappings.Add("Active", "Active");
bulkCopy_tbl_FirstTable.ColumnMappings.Add("CreatedUserID", "CreatedUserID");
bulkCopy_tbl_FirstTable.ColumnMappings.Add("CreatedDate", "CreatedDate");
bulkCopy_tbl_FirstTable.ColumnMappings.Add("UpdatedUserID", "UpdatedUserID");
bulkCopy_tbl_FirstTable.ColumnMappings.Add("UpdatedDate", "UpdatedDate");
bulkCopy_tbl_FirstTable.WriteToServer(DT_tbl_FirstTable);
}
using (SqlBulkCopy bulkCopy_tbl_SecondTable = new SqlBulkCopy(connection, SqlBulkCopyOptions.KeepIdentity, transaction))
{
bulkCopy_tbl_SecondTable.BatchSize = 5000;
bulkCopy_tbl_SecondTable.DestinationTableName = "dbo.tbl_SecondTable";
bulkCopy_tbl_SecondTable.ColumnMappings.Add("ID", "ID");
bulkCopy_tbl_SecondTable.ColumnMappings.Add("UploadFileDetailID", "UploadFileDetailID");
bulkCopy_tbl_SecondTable.ColumnMappings.Add("CompaignFieldMasterID", "CompaignFieldMasterID");
bulkCopy_tbl_SecondTable.ColumnMappings.Add("Value", "Value");
bulkCopy_tbl_SecondTable.ColumnMappings.Add("Active", "Active");
bulkCopy_tbl_SecondTable.ColumnMappings.Add("CreatedUserID", "CreatedUserID");
bulkCopy_tbl_SecondTable.ColumnMappings.Add("CreatedDate", "CreatedDate");
bulkCopy_tbl_SecondTable.ColumnMappings.Add("UpdatedUserID", "UpdatedUserID");
bulkCopy_tbl_SecondTable.ColumnMappings.Add("UpdatedDate", "UpdatedDate");
bulkCopy_tbl_SecondTable.WriteToServer(DT_tbl_SecondTable);
}
transaction.Commit();
IsSuccessSave = true;
}
connection.Close();
}
}
}
catch (Exception ex)
{
if (transaction != null)
transaction.Rollback();
Log4NetWrapper.WriteError(string.Format("{0} : {1} : Exception={2}",
this.GetType().FullName,
(new StackTrace(new StackFrame(0))).GetFrame(0).GetMethod().Name.ToString(),
ex.Message.ToString()));
if (ex.InnerException != null)
{
Log4NetWrapper.WriteError(string.Format("{0} : {1} : InnerException Exception={2}",
this.GetType().FullName,
(new StackTrace(new StackFrame(0))).GetFrame(0).GetMethod().Name.ToString(),
ex.InnerException.Message.ToString()));
}
}
return IsSuccessSave;
}
Finally, It perform insert process in less than 15 seconds for over 500,000 rows.
There is two reasons why I post this scenario.
I would like to share what I found out.
As I am not perfect, I still need to get more suggestion from you.
So, every better solution will be appreciated.
1) Use EF6.x, which has much better performance than EF5.x
Here are more suggestions (from Bulk insert with EF)
2) Keep the active Context Graph small by using a new context for each Unit of Work
3) Turn off AutoDetechChangesEnabled - context.Configuration.AutoDetectChangesEnabled = false;
4) Batching, in your loop, Call SaveChanges periodically
I use payed Entity Framework extension from ZZZ Projects which is developer friendly because of fluent API (extenssion methods, functional approach). This is not an andvertisment, I use it in business projects for several years and it is great. If You want to use something for free and You have Oracle database the Oracle Managed Data Access Oracle.ManagedDataAccess.Core has implementation of bulk operations.
Bulk Operations are not really what ORMs are meant for. For bulk insert operations, I send xml to the stored procedure, and I shred it and bulk insert/update or merge from there.
So even when I use an ORM, I create a Domain Library that is not EF (or NHibernate) dependent.so I have a "safety valve" to bypass the ORM in certain situations.
You should look at using the System.Data.SqlClient.SqlBulkCopy for this. Here's the documentation- http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlbulkcopy.aspx, and of course there are plenty of tutorials online.
In case we want EF to bulk insert records, would suggest below points to improve performance
Call SaveChanges() after for example 100 records and dispose the context and create a new one.
Disable change detection
Example:
using (TransactionScope scope = new TransactionScope())
{
MyDbContext context = null;
try
{
context = new MyDbContext();
context.Configuration.AutoDetectChangesEnabled = false;
int count = 0;
foreach (var entityToInsert in someCollectionOfEntitiesToInsert)
{
++count;
context = AddToContext(context, entityToInsert, count, 100, true);
}
context.SaveChanges();
}
finally
{
if (context != null)
context.Dispose();
}
scope.Complete();
}
private MyDbContext AddToContext(MyDbContext context,
Entity entity, int count, int commitCount, bool recreateContext)
{
context.Set<Entity>().Add(entity);
if (count % commitCount == 0)
{
context.SaveChanges();
if (recreateContext)
{
context.Dispose();
context = new MyDbContext();
context.Configuration.AutoDetectChangesEnabled = false;
}
}
return context;
}
For the performance it is important to call SaveChanges() after "many" records ("many" around 100 or 1000). It also improves the performance to dispose the context after SaveChanges and create a new one.
This clears the context from all entites, SaveChanges doesn't do that, the entities are still attached to the context in state Unchanged. It is the growing size of attached entities in the context what slows down the insertion step by step.
So, it is helpful to clear it after some time.
AutoDetectChangesEnabled = false; on the DbContext.
It also has a big additional performance effect: Why is inserting entities in EF 4.1 so slow compared to ObjectContext?.
below combination increase speed well enough in EF.
context.Configuration.AutoDetectChangesEnabled = false;
context.Configuration.ValidateOnSaveEnabled = false;