The back end of my application runs on NetCore as a Web API. The front end can be considered unrelated.
Most of my application runs on Entity framework. However I've recently had to implement some custom filtering into one of our DB calls. This Db call is called up to 3 times at once when a page on the web UI is loaded.
I'm having issues with the following code, which is throwing the following error:
System.InvalidOperationException: 'The connection was not closed. The connection's current state is open.'
The code:
foreach (var filter in dbFilters)
{
var cacheKey = $"matching_filters_id:{matchingConfigId}_filter:{filter.FilterFunctionName}";
var ids = new List<int>();
if (!_cache.TryGetValue(cacheKey, out ids))
{
ids = new List<int>();
using (var conn = _dbContext.Database.GetDbConnection())
{
conn.Open();
using (var command = conn.CreateCommand())
{
command.CommandText =
SqlHelper.BuildGetMatchingFilterIdsForIntersect(filter.FilterFunctionName);
using (var result = command.ExecuteReader())
{
while (!result.IsClosed && result.Read())
{
ids.Add((int) result["ItemId"]);
}
}
}
conn.Close();
}
var cacheEntryOptions = new MemoryCacheEntryOptions()
.SetSlidingExpiration(TimeSpan.FromSeconds(30));
_cache.Set(cacheKey, ids, cacheEntryOptions);
}
// Joins filtered list against each filter to make the list smaller.
firstFilter = firstFilter.Join(ids, o => o.SourceKey, id => id, (o, id) => o).ToArray();
}
What i'm attempting to do, is filter the list firstFilter which is a POCO containing a few properties. I'm attempting to filter this by joining it to one or more list of filtered ItemIds being returned from multiple different SQL queries. Not the most elegant solution but its what i have to work with.
What i understand to be happening is that as the client is calling this method more then once, the methods are executing concurrently, causing the DB connection to be opened twice at once, causing the connection to fail and the error to be thrown.
I've attempted to stop this from happening by removing the async keyword form my method: public JsonResult GetMatches(int matchingConfigId...... ) but without success. I've also tried to intorduce session into the application, which I've read might cause calls to the same method from one client 'session' to happen in sequence rather then concurrently. But this has not seemed to have any effect.
Of course i might be barking up the entirely wrong tree.. but I've been bashing my head against said tree for most of the day now and I've reached my limit. Any help would be greatly appreciated
So it turns out that the variable dbFilters, which came from entity framework, had not been resolved with a ToList() or ToListAsync() yet, so the connection to the DB for that call was still open, causing the subsequent calls in the above piece of code to fail.
Related
I am using Entity Framework 6, C#, and MySQL InnoDB as our db engine.
I have the following code to "insert on duplicate update" a record:
public async Task AddHostIdToGroup(HostsToGroup hostToGroup)
{
using (var context = new MaintDbContext())
{
HostsToGroup htg = context.HostsToGroup.SingleOrDefault(hs => hs.HostId == hostToGroup.HostId);
if (htg != null)
{
htg.GroupId = hostToGroup.GroupId;
await context.SaveChangesAsync();
}
else
{
context.HostsToGroup.Add(hostToGroup);
await context.SaveChangesAsync();
}
}
}
The code itself looks fine for insert on duplicate update.
Still, on our production server I occasionally see duplicate errors.
My initial thought was that it's a race condition issue.
What can I do to prevent these errors, or how should I handle them?
Your problem is not necessarily the code but what triggers that code. Say the code is triggered by an ASP.net app button, if a user double clicks on the button then it could shoot off two simultaneous requests that results in double entity creation.
So either fix your front end/entry point to eliminate the double create scenario, push everything into a synchronous service/queue that will allow you to deduplicate or use Where instead of Single and handle the duplication in code.
We are using the ExecutionStrategy and have this helper method in our db context:
public Task<T> ExecuteWithinTransactionAsync<T>(Func<IDbContextTransaction, Task<T>> operation, string operationInfo)
{
int counter = 0;
return Database.CreateExecutionStrategy().ExecuteAsync(RunOperationWithinTransaction);
async Task<T> RunOperationWithinTransaction()
{
counter++;
if (counter > 1)
{
Logger.Log(LogLevel.Warn, $"Executing ({counter}. time) transaction for {operationInfo}.");
ClearChangeTracker();
}
using (var transaction = await Database.BeginTransactionAsync(IsolationLevel.Serializable))
{
return await operation.Invoke(transaction);
}
}
}
We than use ExecuteWithinTransactionAsync when calling complex/fragile business logic which should be executed in a serializable transaction reliably. We are using Postgres so it can happen that our transaction will be aborted due to serialization issues. The execution strategy detects it and retries the operation. That works nicely. But EF still keeps the old cache from the previous execution. That's why we introduced ClearChangeTracker which looks like this:
private void ClearChangeTracker()
{
ChangeTracker.DetectChanges();
foreach (var entity in ChangeTracker.Entries().ToList())
{
entity.State = EntityState.Detached;
}
}
And this seemed to have worked properly, until we found a case where it didn't work anymore. When we add new entities to a navigation property list, these entities won't be removed on the next try. For instance
var parent = context.Parents.FirstOrDefault(p => p.Id == 1);
if (parent.Children.Any())
{
throw new Exception("Parent already has a child"); // This exception is thrown on the second try
}
parent.Children.Add(new Child());
context.SaveChangesAsync();
So if the last line context.SaveChangesAsync() fails, and the whole operation is re-run, parent.Children already contains the new child added in parent.Children.Add(new Child()); and I didn't find any way to remove that item from EF.
However, if we remove the check (if (parent.Children.Any())), if the item already exists or not, and just try adding it a second time, it's only stored once in the DB.
I was trying to figure out how to clear the DbContext properly, but most of the time, the answer was just to create a new DbContext. However, that's not an option, since the DbContext is needed for the ExecutionStrategy. That's why I wanted to know, what's the suggested way to used the ExecutionStrategy and having a clean DbContext on every retry.
Further technical details
EF Core version: 1.1.2
Database Provider: Npgsql.EntityFrameworkCore.PostgreSQL (1.1.1)
Operating system: Windows 10, Dockerized in Linux
In ef-core 2.0.0, this new feature DbContext pooling was introduced. For it to work properly, DbContext instances are now able to reset their internal state, so they can be handed out as "new". The reset method can be called like so (inside your DbContext):
((IDbContextPoolable)this).ResetState();
So if you can upgrade to ef-core 2.0.0, go for it. Not only to benefit from this new feature, it's more mature in many ways.
Disclaimer: this method is intended for internal use, so the API may change in the future.
For my data access I use TransactionScopes at the API level to wrap entire operations in a single transaction so that my SQL operations can be somewhat composable. I have a web project that hosts an API and a separate service library that is the implementation and calls to SQL. At the beginning of an Operation (an API entry-point) I open the TransactionScope. Whenever a SqlConnection is needed within the processing of the Operation, ask for the AmbientConnection instead of directly making a new connection. AmbientConnection finds or creates a new SqlConnection for the current transation. Doing this is supposed to allow for good composibility but also avoid the invocation of the MSDTC because it should keep using the same connection for the each suboperation within the transaction. When the transaction is completed (with scope.complete()), the connection is automatically closed.
The problem is that every once in a while the MSDTC is still getting invoked and I cannot figure out why. I've used this before sucessfully and I believe I never got an MSDTC invoked. Two things seem different to me this time though: 1) I'm using SQL Server 2008 R1 (10.50.4000) - not my choice - and I'm aware that the MSDTC behavior changed beginning with this version and perhaps not all the kinks were worked out until later versions. 2) The use of async-await is new and I believe I'm having to use TransactionScopeAsyncFlowOption.Enabled to accommodate this new feature in case some part of the implementation is async. Perhaps more measures are necessary.
I tried Pooling=false in the connection string in case it was MSDTC getting invoked because of two independent logical connections handled errantly under a single pooled connection. But that didn't work.
API Operation
// Exposed API composing multiple low-level operations within a single TransactionScope
// independent of any database platform specifics.
[HttpPost]
public async Task<IHttpActionResult> GetMeTheTwoThings()
{
using (var scope = new TransactionScope(TransactionScopeOption.Required, TransactionScopeAsyncFlowOption.Enabled))
{
var result = new TwoThings(
await serviceLayer.GetThingOne(),
await serviceLayer.GetThingTwo());
scope.Complete();
return Ok(result);
}
}
Service layer implementation
public async Task<ThingOne> GetThingOne()
{
using (var cmd = connManagement.AmbientConnection.CreateCommand())
{
cmd.CommandType = System.Data.CommandType.StoredProcedure;
cmd.CommandText = "dbo.GetThingOne";
return (ThingOne)(await cmd.ExecuteScalarAsync());
}
}
public async Task<ThingTwo> GetThingTwo()
{
using (var cmd = connManagement.AmbientConnection.CreateCommand())
{
cmd.CommandType = System.Data.CommandType.StoredProcedure;
cmd.CommandText = "dbo.GetThingTwo";
return (ThingTwo)(await cmd.ExecuteScalarAsync());
}
}
AmbientConnection implementation
internal class SQLConnManagement
{
readonly string connStr;
readonly ConcurrentDictionary<Transaction, SqlConnection> txConnections = new ConcurrentDictionary<Transaction, SqlConnection>();
private SqlConnection CreateConnection(Transaction tx)
{
var conn = new SqlConnection(this.connStr);
// When the transaction completes, close the connection as well
tx.TransactionCompleted += (s, e) =>
{
SqlConnection closing_conn;
if (txConnections.TryRemove(e.Transaction, out closing_conn))
{
closing_conn.Dispose(); // closing_conn == conn
}
};
conn.Open();
return conn;
}
internal SqlConnection AmbientConnection
{
get
{
var txCurrent = Transaction.Current;
if (txCurrent == null) throw new InvalidOperationException("An ambient transaction is required.");
return txConnections.GetOrAdd(txCurrent, CreateConnection);
}
}
public SQLConnManagement(string connStr)
{
this.connStr = connStr;
}
}
Not to overcomplicate the post, but this might be relevant because it seems to me that every time MSDTC has been invoked the logged stack trace shows that this next mechanism has been involved. Certain data I cache with the built in ObjetCache because it doesn't change often and so I just get it at most once per minute or whatever. This is a little fancy, but I don't see why the Lazy generator would be treated any differently from a more typical call and why this specifically would cause the MSSDTC to sometimes be invoked. I've tried LazyThreadSafetyMode.ExecutionAndPublication too just in case but that doesn't help anyway (and then the exception just keeps getting delivered as the cached result for subsequent requests before the expiration, of course, and that's not desirable).
/// <summary>
/// Cache element that gets the item by key, or if it is missing, creates, caches, and returns the item
/// </summary>
static T CacheGetWithGenerate<T>(ObjectCache cache, string key, Func<T> generator, DateTimeOffset offset) where T : class
{
var generatorWrapped = new Lazy<T>(generator, System.Threading.LazyThreadSafetyMode.PublicationOnly);
return ((Lazy<T>)cache.AddOrGetExisting(
key,
generatorWrapped,
offset))?.Value ?? generatorWrapped.Value;
}
public ThingTwo CachedThingTwo
{
get
{
return CacheGetWithGenerate(
MemoryCache.Default,
"Services.ThingTwoData",
() => GetThingTwo(), // ok, GetThingTwo isn't async this time, fudged example
DateTime.Now.Add(TimeSpan.FromMinutes(1)));
}
}
Do you know why MSDTC is being invoked?
PublicationOnly means that two connections can be created and one thrown away. I'm surprised you made this bug because you explicitly stated PublicationOnly (as opposed to the default safety mode which is safe). You explicitly allowed this bug.
For some reason I did not see that you tried ExecutionAndPublication already. Since not using it is a bug please fix the code in the question.
CreateConnection is also broken in the sense that in case of exception on open the connection object is not getting disposed. Probably harmless but you never know.
Also, audit this code for thread aborts which can happen when ASP.NET times out a request. You are doing very dangerous and brittle things here.
The pattern that I use is to use an IOC container to inject a connection that is shared for the entire request. The first client for that connection opens it. The request end event closes it. Simple, and does away with all that nasty shared, mutable, multi-threaded state.
Why are you using a cache for data that you do not want to lose? This is probably the bug. Don't do that.
What is ?.Value ?? generatorWrapped.Value about? The dictionary can never return null. Delete that code. If it could return null then forcing the lazy value would create a second connection so that's a logic bug as well.
I suspect I don't understand fully what's going on or something weird is happening. (The first case is more likely I guess.)
The big picture:
I'm trying to have a web-service perform certain operations asynchronously as they can be time consuming and I don't want the client to wait for the operations to finish (just query for the results every now and again to see the operation is done).
The async code is wrapped in a transaction - in case something goes wrong, I want to be able to rollback any changes.
Unfortunately the last step of the async code is to call a DIFFERENT service which queries the same database.
Despite wrapping the whole thing in a Snapshot transaction, the last step fails as the service cannot read from the database.
For that matter, while the async operation is underway I cannot perform a simple SELECT statements from the database either.
Here's a sample of the code which I'm currently using to test out the transactions (using Entity Framework 5 model first):
using (var transaction = new System.Transactions.TransactionScope(System.Transactions.TransactionScopeOption.RequiresNew, new System.Transactions.TransactionOptions() { IsolationLevel = System.Transactions.IsolationLevel.Snapshot }))
{
var db = new DataModelContainer();
Log test = new Log();
test.Message = "TEST";
test.Date = DateTime.UtcNow;
test.Details = "asd";
test.Type = "test";
test.StackTrace = "asd";
db.LogSet.Add(test);
db.SaveChanges();
using (var suppressed = new System.Transactions.TransactionScope(System.Transactions.TransactionScopeOption.Suppress))
{
var newDb = new DataModelContainer();
var log = newDb.LogSet.ToArray(); //deadlock here... WHY?
}
test = db.LogSet.Where(l => l.Message == "TEST").Single();
db.LogSet.Remove(test);
db.SaveChanges();
transaction.Complete();
}
The code creates a simple Log entry in the database (yeah, I'm playing around at the moment so the values are rubbish). I've set the SQL database to allow snapshot isolation, and to my knowledge reads should still be permitted (these are being tested in this code by using a new, suppressed transaction and a new DataModelContainer). However, I cannot query the LogSet in the suppressed transaction or in SQL Management Studio - the whole table is locked!
So... why? Why is it locked if the transaction scope is defined as such? I've also tried other isolation levels (like ReadUncommited) and I still cannot query the table.
Could someone please provide an explanation for this behavior?
In SSMS, set your current isolation level to SNAPSHOT and see if that corrects your problem - it's probably set to READ COMMITTED and would therefore still block due to pending updates.
Update:
You can allow READ COMMITTED to access versioned rows DB-wide by altering the following option (and avoiding having to constantly set the current isolation level to SNAPSHOT):
ALTER DATABASE MyDatabase
SET READ_COMMITTED_SNAPSHOT ON
I'm in troubles with TransactionPropagation.NotSupported. I believed that this propagation causes that the code is executed within no transaction. Means that when I marked the specific method, the current transaction will be suspended and the code will be executed without any transaction.
The current version of spring.net creates new transaction instead. See the following code:
[Test]
public void A() {
TransactionTemplate template = new TransactionTemplate(TransactionManager) {
PropagationBehavior = TransactionPropagation.NotSupported
};
template.Execute(delegate {
Assert.AreEqual(0,
SessionFactory.GetCurrentSession().Linq<XXX>().
Where(t => t.Id.Equals(YYY)).ToList().Count);
return null;
});
}
I hoped that this notation causes that linq query is executed without transaction and it'll throw the new exception. But the log showed that it creates both new session and transaction automatically.
I've find out this issue when I marked any method by mentioned annotation and despite the annotation the LINQ query inside was correctly executed.
The question is: how can I mark the method to it's doesn't use the transaction at all? I don't want to use propagation never as I want the current transaction would be suspended.
My project has the business code flow, there is transaction handling, and I want to mark any parts to be certainly non-transactional.
You mention being able to tell from the log that a new transaction is started. What log, the database or the application? What database are you using? Some databases won't let you run a query outside a transaction at all, so would just start one internally for you...
Update:
Your issues looks similar to this one:
https://jira.springframework.org/browse/SPRNET-1307?page=com.atlassian.jira.plugin.ext.bamboo%3Abamboo-build-results-tabpanel#issue-tabs
I would make sure you are running the version of Spring.NET that has this fix in it (looks like v 1.3.1 or greater?)
Also, you could try manually suppressing the transaction and see if that fixes the behavior:
using(var tx = new TransactionScope(TransactionScopeOption.Suppress))
{
// make DB call...
}