I have a 3 layer structure in my solution, where the two lower ones are the logic layer and the database layer. The database layer consists of all CRUD methods for each entity in my database. And in the logic layer we have CRUD services for each entity database CRUD method (EntityService.Add/Get/Update/Delete -> EntityRepository.Add/Get/Update/Delete).
So here I am trying to do a big save method in services that saves several entities. The method calls several Add methods in the logic layer for each entity, which in their turn calls the database Add method. I might add that all database CRUD methods ends with SaveChanges(). Of course I want the save method to be Rollbacked if anything goes wrong so our database don't get unconnected rows, so I have added BeginTransaction() to the save method.
Maybe worth mentioning is that after every CRUD we check so that nothing has gone wrong, and if it has we do a return with the returncode and whatnot. These returns happens before transaction.Commit()
The save method is structured like this: (Logic Layer)
public void SaveMethod(...)
{
using (var context = _unitOfWork.EntityRepository.GetContext()) //Gets the dbContext
{
using (var transaction = context.Database.BeginTransaction())
{
try
{
Service.Entity1.AddEntity1();
Service.Entity2.AddEntity2();
//And so on...
transaction.Commit();
}
catch{
//Exception handling...
}
}
}
}
The method that gets the context to the Save method transaction: (database layer)
public AniPlanContext GetContext()
{
var dbBuilder = new DbContextOptionsBuilder<MyContext>();
var dbConn = _configuration.GetConnectionString("dbContextConnectionstring"); //Gets the db connection string
var options = dbBuilder.UseSqlServer(dbConn).Options;
return new AniPlanContext(options);
}
How an AddEntity method looks like: (Logic layer)
public Entity AddClinic(...)
{
try
{
//Validation....
Entity entity = _unitOfWork.EntityRepository.Add(
new Entity
{
//Set the attributes...
});
return entity;
}
catch (Exception ex)
{
//Exception
}
}
How a database Add looks like: (Database layer)
public TEntity Add(TEntity entity)
{
if (entity == null)
{
throw new ArgumentNullException($"{nameof(Add)} entity must not be null");
}
try
{
_context.Add(entity);
_context.SaveChanges();
return entity;
}
catch (Exception)
{
throw new Exception($"{nameof(entity)} could not be saved");
}
}
Anyhow, my issue is that BeginTransaction Commits the things that has done their AddEntity() even if the save method crashes or fails in any way. Can you help me understand why this is and how I can fix this?
I have tried with TransactionScope and it works but when reading several blogposts it sounds like BeginTransaction() is the save and more reliant way to go. Of what I have understood, both of these should be disposed and rollbacked if it does not go through transaction.Commit or scope.Complete, is that correct?
So to summarize or clarify: I'd like to use BeginTransaction() to save several entities to the database but that also rollback / dispose the transaction if anything has gone wrong.
As #PanagiotisKanavos and #Tomas Chabada agreed upon, it is not the error that is the problem but my solution structure.
This link provides (for me at least) a new way of seeing the UnitOfWork / Repository structure. Apparently they are not needed, which is completely mindblowing. It is worth reading if you ever happen upon the same issue as I had above. It might not be the transaction that is wrong but how we use EF Core and the DBContext and their transactions.
I have an application that reads data from one database, and transforms that data into a new form and writes it into a new database. Some of the tables in the new database are made from multiple tables in the old database so there is a large amount of reading and writing going on. Here is the basic concept of the system:
public void TransferData()
{
OldEntities oldContext = new OldEntities()
NewEntities newContext = new NewEntities()
using(var transaction = newContext.Database.BeginTransaction())
{
try{
TransferTable(oldContext, newContext);
} catch (Exception e) {
transaction.Rollback();
}
}
}
public void TransferTable(OldEntities oldContext, NewEntities newContext)
{
List<Entity1> mainTable = oldContext.Where();
Parallel.ForEach(mainTable, (row) =>
{
using(NewEntities anotherNewContext = new NewContext())
{
anotherNewContext.Database.UseTransaction(newContext.Database.CurrentTransaction.UnderlyingTransaction);
// Do Work
}
});
}
This causes the following exception:
The transaction passed in is not associated with the current connection. Only transactions associated with the current connection may be used.
How can I get around this. The transaction will always be coming from a different EF context but I need them all to share the same transaction. I couldn't find a way to create the new context as a "child" of the original and I am trying to avoid creating a transaction entirely separate from the EF context. Any suggestions?
There is an excellent overview of transactions here which explains how to use transactions in a variety of contexts some of which are similar to yours. Rather than trying to fix your code as is it may be that a modified approach will help.
I assume you are using EF6
i'm stuck within a problem that is going to happen to everyone in ADO.NET mixed with Entity Framework contexts.
I've got a big procedure that handles and saves data into database by using multiple ways like ADO.NET dataadapters and direct CRUD commands against DB. All the procedure is wrapped by 2 using() blocks that creates and releases a DBConnection/DBTransaction and a try/catch block to commit or rollback the transaction. Unfortunately, in the middle of this routine, i have to recall a saving procedure implemented by using Entity Framework. This leads me to a problem:
According to the official documentation, Entity Framework 5 allows me to pass a connection with an transaction associated transaction (it should work, in debug mode, when i call SaveChanges() i don't receive any TimeOutException due to deadlocks, conversely if i pass a new connection it does), but unfortunately after SaveChanges() kicks in the connection is closed and associated transaction committed! Even if i set the flag 'contextOwnsConnection'!
As far as i know, if i migrate EF5 to EF6, things should work (am i right?), but unfortunately i can't, because the project i'm working on is very large and involves a lot of dependecies and it would take a large amount of time.
How can i make it work with EF5? Is there any trick or pattern to achieve the desired result? Am i right about the behavior of EF6? Does it worth EF6 migration?
Here you are a simple example of how does my code looks like.
For privacy reasons i can't post the original code but just imagine a situation like this with a lot of more complexity:
using(DbConnection conn = DBProvider.CreateConnection()){
//Open the created connection
conn.Open();
//Create a new transaction
using(DbTransaction tr = DBProvider.CreateTransaction()){
//Begin a new transaction
tr.Begin();
bool saveOk;
try{
//Updates customers by using dataadapter
dataAdapterCustomers.InsertCommand.Transaction = tr;
dataAdapterCustomers.UpdateCommand.Transaction = tr;
dataAdapterCustomers.DeleteCommand.Transaction = tr;
dataAdapterCustomers.Update();
//Updates stock items by using dataadapter
stockAdapterCustomers.InsertCommand.Transaction = tr;
stockAdapterCustomers.UpdateCommand.Transaction = tr;
stockAdapterCustomers.DeleteCommand.Transaction = tr;
stockAdapterCustomers.Update();
//...Many other DB accessing here...
//Updates stock quantity by using simple DBCommand
quantityUpdateCmd.Transaction = tr;
quantityUpdateCmd.ExecuteNonQuery();
//Updates stock statistics by using a simple DBCommand
updateStockStatsCmd.Transaction = tr;
updateStockStatsCmd.ExecuteNonQuery();
//...Many other DB accessing here...
//HERE:
//Creates a new activity and save it using EF.
//I use a UnitOfWork and i pass to it my connection and 'false' as contextOwnsConnection parameter
//(it 'll be used by the DBContext contained in my Unit of work)
using(ActivityUoW uow = new ActivityUoW(conn, false)){
Activity act = new Activity();
act.Name = "Saving activity";
act.Description = "Done by user";
act.Date = DateTime.Now;
uow.Activities.Add(act);
uow.SaveChanges();
}
//Based on activity result, launch a store procedure that makes other complex things.
UNFORTUNATELY THE CONNECTION HAS BEEN CLOSED AND TRANSACTION COMMITTED, SO THE FOLLOWING INSTRUCTION WILL FAIL.
launchActivityUpdateSpCmd.Transaction = tr;
launchActivityUpdateSpCmd.ExecuteNonQuery();
//...Many other DB accessing here...
//Data saved correctly
saveOk = true;
}
catch(Exception ex){
//There was an error during save
saveOk = false;
}
//Commit or rollback transaction according to save procedure result
if(saveOk)
tr.Commit();
else
tr.Rollback();
}
}
I didn't quite follow your question(s) and wasn't sure if your issue was related to how to handle transactions or if you had a question about EF5 to EF6 migrations. That being said, you have an interesting mixture of data access code.
Regarding transactions - I would look into using the TransactionScope which is part of the System.Transactions namespace.
For example:
try
{
using (var scope = new TransactionScope())
{
using (var conn = new SqlConnection("your connection string"))
{
conn.Open();
// your EF and ADO.NET code
}
scope.Complete();
}
}
catch (TransactionAbortedException ex)
{
}
catch (ApplicationException ex)
{
}
I have two tables which need to be inserted when my application run.
Let's say that I have tables as followed
tbl_FirstTable and tbl_SecondTable
My problem is data volume.
I need to insert over 10,000 rows to tbl_FirstTable and over 500,000 rows to tbl_SecondTable.
So fristly, I use entity framework as follow.
public bool Save_tbl_FirstTable_Vs_tbl_SecondTable(List<tbl_FirstTable> List_tbl_FirstTable, List<tbl_SecondTable> List_tbl_SecondTable)
{
bool IsSuccessSave = false;
try
{
using (DummyDBClass_ObjectContext _DummyDBClass_ObjectContext = new DummyDBClass_ObjectContext())
{
foreach (tbl_FirstTable _tbl_FirstTable in List_tbl_FirstTable)
{
_DummyDBClass_ObjectContext.tbl_FirstTable.InsertOnSubmit(_tbl_FirstTable);
}
foreach (tbl_SecondTable _tbl_SecondTable in List_tbl_SecondTable)
{
_DummyDBClass_ObjectContext.tbl_SecondTable.InsertOnSubmit(_tbl_SecondTable);
}
_DummyDBClass_ObjectContext.SubmitChanges();
IsSuccessSave = true;
}
}
catch (Exception ex)
{
Log4NetWrapper.WriteError(string.Format("{0} : {1} : Exception={2}",
this.GetType().FullName,
(new StackTrace(new StackFrame(0))).GetFrame(0).GetMethod().Name.ToString(),
ex.Message.ToString()));
if (ex.InnerException != null)
{
Log4NetWrapper.WriteError(string.Format("{0} : {1} : InnerException Exception={2}",
this.GetType().FullName,
(new StackTrace(new StackFrame(0))).GetFrame(0).GetMethod().Name.ToString(),
ex.InnerException.Message.ToString()));
}
}
return IsSuccessSave;
}
That is the place I face error Time out exception.
I think that exception will be solved If I use below code.
DummyDBClass_ObjectContext.CommandTimeout = 1800; // 30 minutes
So I used it. It solved but I face another error OutOfMemory Exception.
So I searched the solutions, fortunately, I found below articles.
Problem with Bulk insert using Entity Framework
Using Transactions with SqlBulkCopy
Performing a Bulk Copy Operation in a Transaction
According to that articles, I change my code from Entity Framework to Classic ADO.net code.
public bool Save_tbl_FirstTable_Vs_tbl_SecondTable(DataTable DT_tbl_FirstTable, DataTable DT_tbl_SecondTable)
{
bool IsSuccessSave = false;
SqlTransaction transaction = null;
try
{
using (DummyDBClass_ObjectContext _DummyDBClass_ObjectContext = new DummyDBClass_ObjectContext())
{
var connectionString = ((EntityConnection)_DummyDBClass_ObjectContext.Connection).StoreConnection.ConnectionString;
using (SqlConnection connection = new SqlConnection(connectionString))
{
connection.Open();
using (transaction = connection.BeginTransaction())
{
using (SqlBulkCopy bulkCopy_tbl_FirstTable = new SqlBulkCopy(connection, SqlBulkCopyOptions.KeepIdentity, transaction))
{
bulkCopy_tbl_FirstTable.BatchSize = 5000;
bulkCopy_tbl_FirstTable.DestinationTableName = "dbo.tbl_FirstTable";
bulkCopy_tbl_FirstTable.ColumnMappings.Add("ID", "ID");
bulkCopy_tbl_FirstTable.ColumnMappings.Add("UploadFileID", "UploadFileID");
bulkCopy_tbl_FirstTable.ColumnMappings.Add("Active", "Active");
bulkCopy_tbl_FirstTable.ColumnMappings.Add("CreatedUserID", "CreatedUserID");
bulkCopy_tbl_FirstTable.ColumnMappings.Add("CreatedDate", "CreatedDate");
bulkCopy_tbl_FirstTable.ColumnMappings.Add("UpdatedUserID", "UpdatedUserID");
bulkCopy_tbl_FirstTable.ColumnMappings.Add("UpdatedDate", "UpdatedDate");
bulkCopy_tbl_FirstTable.WriteToServer(DT_tbl_FirstTable);
}
using (SqlBulkCopy bulkCopy_tbl_SecondTable = new SqlBulkCopy(connection, SqlBulkCopyOptions.KeepIdentity, transaction))
{
bulkCopy_tbl_SecondTable.BatchSize = 5000;
bulkCopy_tbl_SecondTable.DestinationTableName = "dbo.tbl_SecondTable";
bulkCopy_tbl_SecondTable.ColumnMappings.Add("ID", "ID");
bulkCopy_tbl_SecondTable.ColumnMappings.Add("UploadFileDetailID", "UploadFileDetailID");
bulkCopy_tbl_SecondTable.ColumnMappings.Add("CompaignFieldMasterID", "CompaignFieldMasterID");
bulkCopy_tbl_SecondTable.ColumnMappings.Add("Value", "Value");
bulkCopy_tbl_SecondTable.ColumnMappings.Add("Active", "Active");
bulkCopy_tbl_SecondTable.ColumnMappings.Add("CreatedUserID", "CreatedUserID");
bulkCopy_tbl_SecondTable.ColumnMappings.Add("CreatedDate", "CreatedDate");
bulkCopy_tbl_SecondTable.ColumnMappings.Add("UpdatedUserID", "UpdatedUserID");
bulkCopy_tbl_SecondTable.ColumnMappings.Add("UpdatedDate", "UpdatedDate");
bulkCopy_tbl_SecondTable.WriteToServer(DT_tbl_SecondTable);
}
transaction.Commit();
IsSuccessSave = true;
}
connection.Close();
}
}
}
catch (Exception ex)
{
if (transaction != null)
transaction.Rollback();
Log4NetWrapper.WriteError(string.Format("{0} : {1} : Exception={2}",
this.GetType().FullName,
(new StackTrace(new StackFrame(0))).GetFrame(0).GetMethod().Name.ToString(),
ex.Message.ToString()));
if (ex.InnerException != null)
{
Log4NetWrapper.WriteError(string.Format("{0} : {1} : InnerException Exception={2}",
this.GetType().FullName,
(new StackTrace(new StackFrame(0))).GetFrame(0).GetMethod().Name.ToString(),
ex.InnerException.Message.ToString()));
}
}
return IsSuccessSave;
}
Finally, It perform insert process in less than 15 seconds for over 500,000 rows.
There is two reasons why I post this scenario.
I would like to share what I found out.
As I am not perfect, I still need to get more suggestion from you.
So, every better solution will be appreciated.
1) Use EF6.x, which has much better performance than EF5.x
Here are more suggestions (from Bulk insert with EF)
2) Keep the active Context Graph small by using a new context for each Unit of Work
3) Turn off AutoDetechChangesEnabled - context.Configuration.AutoDetectChangesEnabled = false;
4) Batching, in your loop, Call SaveChanges periodically
I use payed Entity Framework extension from ZZZ Projects which is developer friendly because of fluent API (extenssion methods, functional approach). This is not an andvertisment, I use it in business projects for several years and it is great. If You want to use something for free and You have Oracle database the Oracle Managed Data Access Oracle.ManagedDataAccess.Core has implementation of bulk operations.
Bulk Operations are not really what ORMs are meant for. For bulk insert operations, I send xml to the stored procedure, and I shred it and bulk insert/update or merge from there.
So even when I use an ORM, I create a Domain Library that is not EF (or NHibernate) dependent.so I have a "safety valve" to bypass the ORM in certain situations.
You should look at using the System.Data.SqlClient.SqlBulkCopy for this. Here's the documentation- http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlbulkcopy.aspx, and of course there are plenty of tutorials online.
In case we want EF to bulk insert records, would suggest below points to improve performance
Call SaveChanges() after for example 100 records and dispose the context and create a new one.
Disable change detection
Example:
using (TransactionScope scope = new TransactionScope())
{
MyDbContext context = null;
try
{
context = new MyDbContext();
context.Configuration.AutoDetectChangesEnabled = false;
int count = 0;
foreach (var entityToInsert in someCollectionOfEntitiesToInsert)
{
++count;
context = AddToContext(context, entityToInsert, count, 100, true);
}
context.SaveChanges();
}
finally
{
if (context != null)
context.Dispose();
}
scope.Complete();
}
private MyDbContext AddToContext(MyDbContext context,
Entity entity, int count, int commitCount, bool recreateContext)
{
context.Set<Entity>().Add(entity);
if (count % commitCount == 0)
{
context.SaveChanges();
if (recreateContext)
{
context.Dispose();
context = new MyDbContext();
context.Configuration.AutoDetectChangesEnabled = false;
}
}
return context;
}
For the performance it is important to call SaveChanges() after "many" records ("many" around 100 or 1000). It also improves the performance to dispose the context after SaveChanges and create a new one.
This clears the context from all entites, SaveChanges doesn't do that, the entities are still attached to the context in state Unchanged. It is the growing size of attached entities in the context what slows down the insertion step by step.
So, it is helpful to clear it after some time.
AutoDetectChangesEnabled = false; on the DbContext.
It also has a big additional performance effect: Why is inserting entities in EF 4.1 so slow compared to ObjectContext?.
below combination increase speed well enough in EF.
context.Configuration.AutoDetectChangesEnabled = false;
context.Configuration.ValidateOnSaveEnabled = false;
I am new to Entity Framework so please bear with me.
I have a program that I want to select multiple records from a table and store it in a queue:
private Queue<RecordsToProcess> getRecordsToProcess()
{
Queue<RecordsToProcess> results = new Queue<RecordsToProcess>();
using (MyEntity context = new MyEntity())
{
var query = from v in context.RecordsToProcess
where v.Processed == false
select v;
foreach (RecordsToProcess record in query)
{
results.Enqueue(record);
}
}
}
Then I spin up multiple worker threads. Each worker thread takes one of the items in queue, processes it, and then saves it to the database.
private void processWorkerThread(object stateInfo)
{
while (workQueue.Count > 0)
{
RecordToProcess record = new RecordToProcess;
lock(workQueue)
{
if (workQueue.Count > 0)
RecordToProcess = workQueue.Dequeue();
else
break;
}
//Do the record processing here
//How do I save that record here???
}
}
My understanding is that to save changes back to the database you just call context.SaveChanges() but I can't do that in this situation can I?
Any help is appreciated.
Thanks!
Since you are disposing your MyEntity context in the first method (by wrapping it in a using statement), the entities that are enqueued will be in a "detached" state. That means, among other things, that changes done to the entity will not be tracked and you will not be able to lazy load navigation properties.
It is perfectly fine to dequeue these entities, "attaching" them to a different context, update them, and then call SaveChanges to persist the changes.
You can read about Attaching and Detaching Objects and Add/Attach and Entity States
It might be safer if you save off the primary key in the queue instead and retrieve the entities again. This way you are more likely avoid any data concurrency issues.