We Keep Coding

Trade-offs between longer data context or transaction outside of context?

I am writing a Web API Rest service that does operations on a distinct set of entities. I have broken them down like this:
db = new DBEntities();
using (var dbContextTransaction = db.Database.BeginTransaction())
{
try
{
ProcessClient();
ProcessClientPerson();
ProcessGuardian();
ProcessAddress();
ProcessEmail();
ProcessPhones();
ProcessChildren();
dbContextTransaction.Commit();
}
catch (Exception ex)
{
dbContextTransaction.Rollback();
etc.
Following the advice that data contexts should live as short as possible, each of the methods creates its own data context, calls SaveChanges(), and disposes of it at the end:
private ProcessClient()
{
db = new DBEntities();
....
This obviously does not work - a transaction context created this way is tied to the data context. If something goes wrong in one of the entity operations, only that operation is rolled back (implicitly), but the overarching transaction is not.
I found this approach for creating a transaction outside of EF, but I am wondering if I should follow it or if I should just let my data context live for the duration of the transaction and keep the transaction inside of EF!?
I am not looking for an opinion, but for data around stability, performance, etc.

There is no immediate need to keep contexts short-lived. You can do that but you don't have to.
Over time entities will accumulate in a context. If you risk running out of memory it can be necessary to let go of a context.
Otherwise, the usual procedure is to keep the context alive for the duration of the logical unit of work. Here, that UOW is all those methods in their entirety.
This also makes transaction management easier (as you already found out).
dbContextTransaction.Rollback();
This is an anti-pattern. Simply don't commit in case of error.

I have mixed feelings about this. I am working against a legacy database that has no foreign key constraints, and I am inserting, updating, and deleting between 20 and 30 objects in one of these service calls.
The problem is that I need to call SaveChanges() frequently to get the identity column values that will become foreign keys.
On the other hand, I have to be able to roll back everything if there is a problem three layers down, so a single large transaction is needed.
For some reason that I have not been able to determine, calling SaveChanges repeatedly on the same data context would result in errors that the connection state is open. So I ended up giving each method its own data context anyway:
var scope = new TransactionScope(TransactionScopeOption.RequiresNew,
new TransactionOptions() { IsolationLevel = IsolationLevel.ReadUncommitted);
using (scope)
{
try
{
ProcessClient();
ProcessClientPerson();
ProcessGuardian();
ProcessAddress();
ProcessEmail();
ProcessPhones();
ProcessChildren();
scope.Complete();
}
catch (System.Data.Entity.Validation.
DbEntityValidationException ex)
{
[...] handle validation errors etc [...]
}
}
with each section doing basically this, once stripped down to the bare essentials:
private void ProcessClient() {
{
using (MyDBEntities db = new MyDBEntities())
{
[...] doing client stuff [...]
aClient.LastUpdated = DateTime.Now;
db.AddOrUpdate(db, db.Clients, aClient, aClient.ClientID);
db.SaveChanges();
ClientId = aClient.ClientID; // now I can use this to form FKs
}
}
Mixed feelings about locking, because on my development VM the transaction runs for 1-2 seconds and this is a production database with office staff and online customers doing CRUD transactions through web applications at the same time.
Unrelated, but helpful for my AddOrUpdate method was this blog post.

MSDTC getting invoked. But why?

For my data access I use TransactionScopes at the API level to wrap entire operations in a single transaction so that my SQL operations can be somewhat composable. I have a web project that hosts an API and a separate service library that is the implementation and calls to SQL. At the beginning of an Operation (an API entry-point) I open the TransactionScope. Whenever a SqlConnection is needed within the processing of the Operation, ask for the AmbientConnection instead of directly making a new connection. AmbientConnection finds or creates a new SqlConnection for the current transation. Doing this is supposed to allow for good composibility but also avoid the invocation of the MSDTC because it should keep using the same connection for the each suboperation within the transaction. When the transaction is completed (with scope.complete()), the connection is automatically closed.
The problem is that every once in a while the MSDTC is still getting invoked and I cannot figure out why. I've used this before sucessfully and I believe I never got an MSDTC invoked. Two things seem different to me this time though: 1) I'm using SQL Server 2008 R1 (10.50.4000) - not my choice - and I'm aware that the MSDTC behavior changed beginning with this version and perhaps not all the kinks were worked out until later versions. 2) The use of async-await is new and I believe I'm having to use TransactionScopeAsyncFlowOption.Enabled to accommodate this new feature in case some part of the implementation is async. Perhaps more measures are necessary.
I tried Pooling=false in the connection string in case it was MSDTC getting invoked because of two independent logical connections handled errantly under a single pooled connection. But that didn't work.
API Operation
// Exposed API composing multiple low-level operations within a single TransactionScope
// independent of any database platform specifics.
[HttpPost]
public async Task<IHttpActionResult> GetMeTheTwoThings()
{
using (var scope = new TransactionScope(TransactionScopeOption.Required, TransactionScopeAsyncFlowOption.Enabled))
{
var result = new TwoThings(
await serviceLayer.GetThingOne(),
await serviceLayer.GetThingTwo());
scope.Complete();
return Ok(result);
}
}
Service layer implementation
public async Task<ThingOne> GetThingOne()
{
using (var cmd = connManagement.AmbientConnection.CreateCommand())
{
cmd.CommandType = System.Data.CommandType.StoredProcedure;
cmd.CommandText = "dbo.GetThingOne";
return (ThingOne)(await cmd.ExecuteScalarAsync());
}
}
public async Task<ThingTwo> GetThingTwo()
{
using (var cmd = connManagement.AmbientConnection.CreateCommand())
{
cmd.CommandType = System.Data.CommandType.StoredProcedure;
cmd.CommandText = "dbo.GetThingTwo";
return (ThingTwo)(await cmd.ExecuteScalarAsync());
}
}
AmbientConnection implementation
internal class SQLConnManagement
{
readonly string connStr;
readonly ConcurrentDictionary<Transaction, SqlConnection> txConnections = new ConcurrentDictionary<Transaction, SqlConnection>();
private SqlConnection CreateConnection(Transaction tx)
{
var conn = new SqlConnection(this.connStr);
// When the transaction completes, close the connection as well
tx.TransactionCompleted += (s, e) =>
{
SqlConnection closing_conn;
if (txConnections.TryRemove(e.Transaction, out closing_conn))
{
closing_conn.Dispose(); // closing_conn == conn
}
};
conn.Open();
return conn;
}
internal SqlConnection AmbientConnection
{
get
{
var txCurrent = Transaction.Current;
if (txCurrent == null) throw new InvalidOperationException("An ambient transaction is required.");
return txConnections.GetOrAdd(txCurrent, CreateConnection);
}
}
public SQLConnManagement(string connStr)
{
this.connStr = connStr;
}
}
Not to overcomplicate the post, but this might be relevant because it seems to me that every time MSDTC has been invoked the logged stack trace shows that this next mechanism has been involved. Certain data I cache with the built in ObjetCache because it doesn't change often and so I just get it at most once per minute or whatever. This is a little fancy, but I don't see why the Lazy generator would be treated any differently from a more typical call and why this specifically would cause the MSSDTC to sometimes be invoked. I've tried LazyThreadSafetyMode.ExecutionAndPublication too just in case but that doesn't help anyway (and then the exception just keeps getting delivered as the cached result for subsequent requests before the expiration, of course, and that's not desirable).
/// <summary>
/// Cache element that gets the item by key, or if it is missing, creates, caches, and returns the item
/// </summary>
static T CacheGetWithGenerate<T>(ObjectCache cache, string key, Func<T> generator, DateTimeOffset offset) where T : class
{
var generatorWrapped = new Lazy<T>(generator, System.Threading.LazyThreadSafetyMode.PublicationOnly);
return ((Lazy<T>)cache.AddOrGetExisting(
key,
generatorWrapped,
offset))?.Value ?? generatorWrapped.Value;
}
public ThingTwo CachedThingTwo
{
get
{
return CacheGetWithGenerate(
MemoryCache.Default,
"Services.ThingTwoData",
() => GetThingTwo(), // ok, GetThingTwo isn't async this time, fudged example
DateTime.Now.Add(TimeSpan.FromMinutes(1)));
}
}
Do you know why MSDTC is being invoked?

PublicationOnly means that two connections can be created and one thrown away. I'm surprised you made this bug because you explicitly stated PublicationOnly (as opposed to the default safety mode which is safe). You explicitly allowed this bug.
For some reason I did not see that you tried ExecutionAndPublication already. Since not using it is a bug please fix the code in the question.
CreateConnection is also broken in the sense that in case of exception on open the connection object is not getting disposed. Probably harmless but you never know.
Also, audit this code for thread aborts which can happen when ASP.NET times out a request. You are doing very dangerous and brittle things here.
The pattern that I use is to use an IOC container to inject a connection that is shared for the entire request. The first client for that connection opens it. The request end event closes it. Simple, and does away with all that nasty shared, mutable, multi-threaded state.
Why are you using a cache for data that you do not want to lose? This is probably the bug. Don't do that.
What is ?.Value ?? generatorWrapped.Value about? The dictionary can never return null. Delete that code. If it could return null then forcing the lazy value would create a second connection so that's a logic bug as well.

Snapshot isolation in SQL / code blocking reads

I suspect I don't understand fully what's going on or something weird is happening. (The first case is more likely I guess.)
The big picture:
I'm trying to have a web-service perform certain operations asynchronously as they can be time consuming and I don't want the client to wait for the operations to finish (just query for the results every now and again to see the operation is done).
The async code is wrapped in a transaction - in case something goes wrong, I want to be able to rollback any changes.
Unfortunately the last step of the async code is to call a DIFFERENT service which queries the same database.
Despite wrapping the whole thing in a Snapshot transaction, the last step fails as the service cannot read from the database.
For that matter, while the async operation is underway I cannot perform a simple SELECT statements from the database either.
Here's a sample of the code which I'm currently using to test out the transactions (using Entity Framework 5 model first):
using (var transaction = new System.Transactions.TransactionScope(System.Transactions.TransactionScopeOption.RequiresNew, new System.Transactions.TransactionOptions() { IsolationLevel = System.Transactions.IsolationLevel.Snapshot }))
{
var db = new DataModelContainer();
Log test = new Log();
test.Message = "TEST";
test.Date = DateTime.UtcNow;
test.Details = "asd";
test.Type = "test";
test.StackTrace = "asd";
db.LogSet.Add(test);
db.SaveChanges();
using (var suppressed = new System.Transactions.TransactionScope(System.Transactions.TransactionScopeOption.Suppress))
{
var newDb = new DataModelContainer();
var log = newDb.LogSet.ToArray(); //deadlock here... WHY?
}
test = db.LogSet.Where(l => l.Message == "TEST").Single();
db.LogSet.Remove(test);
db.SaveChanges();
transaction.Complete();
}
The code creates a simple Log entry in the database (yeah, I'm playing around at the moment so the values are rubbish). I've set the SQL database to allow snapshot isolation, and to my knowledge reads should still be permitted (these are being tested in this code by using a new, suppressed transaction and a new DataModelContainer). However, I cannot query the LogSet in the suppressed transaction or in SQL Management Studio - the whole table is locked!
So... why? Why is it locked if the transaction scope is defined as such? I've also tried other isolation levels (like ReadUncommited) and I still cannot query the table.
Could someone please provide an explanation for this behavior?

In SSMS, set your current isolation level to SNAPSHOT and see if that corrects your problem - it's probably set to READ COMMITTED and would therefore still block due to pending updates.
Update:
You can allow READ COMMITTED to access versioned rows DB-wide by altering the following option (and avoiding having to constantly set the current isolation level to SNAPSHOT):
ALTER DATABASE MyDatabase
SET READ_COMMITTED_SNAPSHOT ON

Attempt at an Asynchronous method is failing

I have an MVC3/.NET 4 application which uses Entity Framework (4.3.1 Code First)
I have wrapped EF into a Repository/UnitOfWork pattern as described here…
http://www.asp.net/mvc/tutorials/getting-started-with-ef-using-mvc/implementing-the-repository-and-unit-of-work-patterns-in-an-asp-net-mvc-application
Typically, as it explains in the article, when I require the creation of a new record I’ve been doing this…
public ActionResult Create(Course course)
{
unitOfWork.CourseRepository.Add(course);
unitOfWork.Save();
return RedirectToAction("Index");
}
However, when more than simply saving a record to a database is required I wrap the logic into what I’ve called an IService. For example…
private ICourseService courseService;
public ActionResult Create(Course course)
{
courseService.ProcessNewCourse(course);
return RedirectToAction("Index");
}
In one of my services I have something like the following…
public void ProcessNewCourse(Course course)
{
// Save the course to the database…
unitOfWork.CourseRepository.Add(course);
unitOfWork.Save();
// Generate a PDF that email some people about the new course being created, which requires more use of the unitOfWork…
var someInformation = unitOfWork.AnotherRepository.GetStuff();
var myPdfCreator = new PdfCreator();
IEnumerable<People> people = unitOfWork.PeopleRepository.GetAllThatWantNotifiying(course);
foreach(var person in people)
{
var message = “Hi ” + person.FullName;
var attachment = myPdfCreator.CreatePdf();
etc...
smtpClient.Send();
}
}
The above isn’t the actual code (my app has nothing to do with courses, I’m using view models, and I have separated the PDF creation and email message out into other classes) but the gist of what is going on is as above!
My problem is that the generation of the PDF and emailing it out is taking some time. The user just needs to know that the record has been saved to the database so I thought I would put the code below the unitOfWork.Save(); into an asynchronous method. The user can then be redirected and the server can happily take its time processing the emails, and attachments and whatever else I require it to do post save.
This is where I’m struggling.
I’ve tried a few things, the current being the following in ICourseService…
public class CourseService : ICourseService
{
private delegate void NotifyDelegate(Course course);
private NotifyDelegate notifyDelegate;
public CourseService()
{
notifyDelegate = new NotifyDelegate(this.Notify);
}
public void ProcessNewCourse(Course course)
{
// Save the course to the database…
unitOfWork.CourseRepository.Add(course);
unitOfWork.Save();
notifyDelegate.BeginInvoke(course);
}
private void Notify(Course course)
{
// All the stuff under unitOfWork.Save(); moved here.
}
}
My Questions/Problems
I’m randomly getting the error: "There is already an open DataReader associated with this Command which must be closed first." in the Notify() method.
Is it something to do with the fact that I’m trying to share the unitOrWork and therefore a dbContext across threads?
If so, can someone be kind enough to explain why this is a problem?
Should I be giving a new instance of unitOfWork to the Notify method?
Am I using the right patterns/classes to invoke the method asynchronously? Or should I be using something along the lines of....
new System.Threading.Tasks.Task(() => { Notify(course); }).Start();
I must say I've become very confused with the terms asynchronous, parallel, and concurrent!!
Any links to articles (c# async for idiots) would be appreciated!!
Many thanks.
UPDATE:
A little more digging got me to this SO page: https://stackoverflow.com/a/5491978/192999 which says...
"Be aware though that EF contexts are not thread safe, i.e. you cannot use the same context in more than one thread."
...so am I trying to achieve the impossible? Does this mean I should be creating a new IUnitOfWork instance for my new thread?

You could create a polling background thread that does the lengthy operation separately from your main flow. This thread could scan the database for new items (or items marked to process). This solution is pretty simple and ensures that jobs get done even if you application crashes (it will be picked up when the polling thread is started again).
You could also use a Synchronised Queue if it's not terrible if the request is 'lost', in the case your application crashes after the doc is requested and before it's generated/sent.
One thing is almost sure - as rikitikitik said - you will need to use a new unit of work, which means a separate transaction.
You could also look at Best threading queue example / best practice .

How to prevent EntityFramework deadlock when concurrently running these two statements

Calls into my web service use the following code to ensure that the caller has a valid session. If a valid session found then it updates the session details and saves the changes. All simple enough and works fine.
// Create the Entity Framework context
using(MyContext ctx = CreateMyContext())
{
// Get the user session for the client session
UserSession session = (from us in context.UserSessions.Include("UserEntity")
where us.SessionId = callerSessionId
select us).FirstOrDefault<UserSession>();
if (session == null)
return false;
else
{
// Update session details
session.Calls++;
session.LastAccessed = DateTime.Now.Ticks;
Console.WriteLine("Call by User:{0}", session.UserEntity.Name);
// Save session changes back to the server
ctx.SaveChanges();
return true;
}
}
All works fine until the same caller, and hence the same session, makes multiple concurrent calls (which is perfectly valid to happen). In this case I sometimes get a deadlock. Using SQL Server Profiler I can see the following is happening.
Caller A performs the select and acquires a shared lock on the user session. Caller B performs the select and acquires a shared lock on the same user session. Caller A cannot perform its update because of Caller B's shared lock. Caller B cannot perform its update because of caller A's shared lock. Deadlock.
This seems like a simple and classic deadlock scenario and there must be a simple method to resolve it. Surely almost all real world applications have this same problem.But none of the Entity Frameworks books I have mention anything about deadlocks.

I found an article that talks about this HERE. It basically sounds like you can start and stop a transaction that surrounds your EF call... The block gives the following code example so credit goes to Diego B Vega... The blog post also links to another blog with additional information.
using (var scope = new TransactionScope(TransactionScopeOption.Required, new
TransactionOptions { IsolationLevel= IsolationLevel.Snapshot }))
{
// do something with EF here
scope.Complete();
}

Will the following work for you?
using(MyContext ctx = CreateMyContext())
{
ctx.Database.ExecuteSqlCommand("SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;");
// Get the user session for the client session
...
}

You probably have each session using a transaction maybe? in which case they will deadlock as the transactions both try to upgrade from shared lock to exclusive lock when they attempt to save. This seems to be not well documented since EF favours optimistic concurrency.
One way out of this is to provide an updatelock hint using something like this:
return context.TestEntities
.SqlQuery("SELECT TOP 1 Id, Value FROM TestEntities WITH (UPDLOCK)")
.Single();
}
see: entity framework 6 and pessimistic concurrency