I was following the MVC music store tutorial and I came to the part where they are using database connections (DbConnection is a child of DbContext). I was taught to create methods like this (wrapping with using):
public class StoreManagerController : Controller
{
//
// GET: /StoreManager/
public ActionResult Index()
{
using(var db = new DbConnection())
{
var albums = db.Albums.Include(a => a.Genre).Include(a => a.Artist);
return View(albums.ToList());
}
}
...
}
but Visual Studio generated me a controller which looked like this:
public class StoreManagerController : Controller
{
private DbConnection db = new DbConnection();
//
// GET: /StoreManager/
public ActionResult Index()
{
var albums = db.Albums.Include(a => a.Genre).Include(a => a.Artist);
return View(albums.ToList());
}
...
}
I assume, Visual Studio isn't wrong, but why was I told to wrap each method with using to make the connections as short as possible and also the users to use separate connections?
I assume, Visual Studio isn't wrong, but why was I told to wrap each method with using
using(var db = new DbConnection())
{
var albums = db.Albums.Include(a => a.Genre).Include(a => a.Artist);
return View(albums.ToList());
}
The scope of db remains only within the curly braces. This is perhaps another purpose that using keyword serves in C#. It defines the scope of a variable, here in the above case it is your db object.
Now, if you debug the code that visual studio generated for you, then you notice that there is a Dispose method being invoked each and every time an object of a controller class is made or in other words, an action method is called within the corresponding Controller.
The DBContext instance is always disposed because of the following -
As you load more objects and their references into memory, the memory consumption of the context may increase rapidly. This may cause performance issues.
If an exception causes the context to be in an unrecoverable state, the whole application may terminate.
The chances of running into concurrency-related issues increase as the gap between the time when the data is queried and updated grows.
For more info - Reference
This might depend on the usability of your app; whether or not you need a persistent connection, and the cost of creating one (and a myriad of other factors).
But for starters, you should always dispose of the connection (as in the first pattern, not the one suggested by Visual Studio) and then move to other patterns based on new requirements or performance-related issues.
The biggest issue I see with the Visual Studio-suggested options is that you have no way of controlling the lifetime of the DbConncetion object, and are leaving it up to the garbage collector to eventually dispose of it. This could leave the connection resources in use for an rather undetermined period of time.
Related
We are using the ExecutionStrategy and have this helper method in our db context:
public Task<T> ExecuteWithinTransactionAsync<T>(Func<IDbContextTransaction, Task<T>> operation, string operationInfo)
{
int counter = 0;
return Database.CreateExecutionStrategy().ExecuteAsync(RunOperationWithinTransaction);
async Task<T> RunOperationWithinTransaction()
{
counter++;
if (counter > 1)
{
Logger.Log(LogLevel.Warn, $"Executing ({counter}. time) transaction for {operationInfo}.");
ClearChangeTracker();
}
using (var transaction = await Database.BeginTransactionAsync(IsolationLevel.Serializable))
{
return await operation.Invoke(transaction);
}
}
}
We than use ExecuteWithinTransactionAsync when calling complex/fragile business logic which should be executed in a serializable transaction reliably. We are using Postgres so it can happen that our transaction will be aborted due to serialization issues. The execution strategy detects it and retries the operation. That works nicely. But EF still keeps the old cache from the previous execution. That's why we introduced ClearChangeTracker which looks like this:
private void ClearChangeTracker()
{
ChangeTracker.DetectChanges();
foreach (var entity in ChangeTracker.Entries().ToList())
{
entity.State = EntityState.Detached;
}
}
And this seemed to have worked properly, until we found a case where it didn't work anymore. When we add new entities to a navigation property list, these entities won't be removed on the next try. For instance
var parent = context.Parents.FirstOrDefault(p => p.Id == 1);
if (parent.Children.Any())
{
throw new Exception("Parent already has a child"); // This exception is thrown on the second try
}
parent.Children.Add(new Child());
context.SaveChangesAsync();
So if the last line context.SaveChangesAsync() fails, and the whole operation is re-run, parent.Children already contains the new child added in parent.Children.Add(new Child()); and I didn't find any way to remove that item from EF.
However, if we remove the check (if (parent.Children.Any())), if the item already exists or not, and just try adding it a second time, it's only stored once in the DB.
I was trying to figure out how to clear the DbContext properly, but most of the time, the answer was just to create a new DbContext. However, that's not an option, since the DbContext is needed for the ExecutionStrategy. That's why I wanted to know, what's the suggested way to used the ExecutionStrategy and having a clean DbContext on every retry.
Further technical details
EF Core version: 1.1.2
Database Provider: Npgsql.EntityFrameworkCore.PostgreSQL (1.1.1)
Operating system: Windows 10, Dockerized in Linux
In ef-core 2.0.0, this new feature DbContext pooling was introduced. For it to work properly, DbContext instances are now able to reset their internal state, so they can be handed out as "new". The reset method can be called like so (inside your DbContext):
((IDbContextPoolable)this).ResetState();
So if you can upgrade to ef-core 2.0.0, go for it. Not only to benefit from this new feature, it's more mature in many ways.
Disclaimer: this method is intended for internal use, so the API may change in the future.
For my data access I use TransactionScopes at the API level to wrap entire operations in a single transaction so that my SQL operations can be somewhat composable. I have a web project that hosts an API and a separate service library that is the implementation and calls to SQL. At the beginning of an Operation (an API entry-point) I open the TransactionScope. Whenever a SqlConnection is needed within the processing of the Operation, ask for the AmbientConnection instead of directly making a new connection. AmbientConnection finds or creates a new SqlConnection for the current transation. Doing this is supposed to allow for good composibility but also avoid the invocation of the MSDTC because it should keep using the same connection for the each suboperation within the transaction. When the transaction is completed (with scope.complete()), the connection is automatically closed.
The problem is that every once in a while the MSDTC is still getting invoked and I cannot figure out why. I've used this before sucessfully and I believe I never got an MSDTC invoked. Two things seem different to me this time though: 1) I'm using SQL Server 2008 R1 (10.50.4000) - not my choice - and I'm aware that the MSDTC behavior changed beginning with this version and perhaps not all the kinks were worked out until later versions. 2) The use of async-await is new and I believe I'm having to use TransactionScopeAsyncFlowOption.Enabled to accommodate this new feature in case some part of the implementation is async. Perhaps more measures are necessary.
I tried Pooling=false in the connection string in case it was MSDTC getting invoked because of two independent logical connections handled errantly under a single pooled connection. But that didn't work.
API Operation
// Exposed API composing multiple low-level operations within a single TransactionScope
// independent of any database platform specifics.
[HttpPost]
public async Task<IHttpActionResult> GetMeTheTwoThings()
{
using (var scope = new TransactionScope(TransactionScopeOption.Required, TransactionScopeAsyncFlowOption.Enabled))
{
var result = new TwoThings(
await serviceLayer.GetThingOne(),
await serviceLayer.GetThingTwo());
scope.Complete();
return Ok(result);
}
}
Service layer implementation
public async Task<ThingOne> GetThingOne()
{
using (var cmd = connManagement.AmbientConnection.CreateCommand())
{
cmd.CommandType = System.Data.CommandType.StoredProcedure;
cmd.CommandText = "dbo.GetThingOne";
return (ThingOne)(await cmd.ExecuteScalarAsync());
}
}
public async Task<ThingTwo> GetThingTwo()
{
using (var cmd = connManagement.AmbientConnection.CreateCommand())
{
cmd.CommandType = System.Data.CommandType.StoredProcedure;
cmd.CommandText = "dbo.GetThingTwo";
return (ThingTwo)(await cmd.ExecuteScalarAsync());
}
}
AmbientConnection implementation
internal class SQLConnManagement
{
readonly string connStr;
readonly ConcurrentDictionary<Transaction, SqlConnection> txConnections = new ConcurrentDictionary<Transaction, SqlConnection>();
private SqlConnection CreateConnection(Transaction tx)
{
var conn = new SqlConnection(this.connStr);
// When the transaction completes, close the connection as well
tx.TransactionCompleted += (s, e) =>
{
SqlConnection closing_conn;
if (txConnections.TryRemove(e.Transaction, out closing_conn))
{
closing_conn.Dispose(); // closing_conn == conn
}
};
conn.Open();
return conn;
}
internal SqlConnection AmbientConnection
{
get
{
var txCurrent = Transaction.Current;
if (txCurrent == null) throw new InvalidOperationException("An ambient transaction is required.");
return txConnections.GetOrAdd(txCurrent, CreateConnection);
}
}
public SQLConnManagement(string connStr)
{
this.connStr = connStr;
}
}
Not to overcomplicate the post, but this might be relevant because it seems to me that every time MSDTC has been invoked the logged stack trace shows that this next mechanism has been involved. Certain data I cache with the built in ObjetCache because it doesn't change often and so I just get it at most once per minute or whatever. This is a little fancy, but I don't see why the Lazy generator would be treated any differently from a more typical call and why this specifically would cause the MSSDTC to sometimes be invoked. I've tried LazyThreadSafetyMode.ExecutionAndPublication too just in case but that doesn't help anyway (and then the exception just keeps getting delivered as the cached result for subsequent requests before the expiration, of course, and that's not desirable).
/// <summary>
/// Cache element that gets the item by key, or if it is missing, creates, caches, and returns the item
/// </summary>
static T CacheGetWithGenerate<T>(ObjectCache cache, string key, Func<T> generator, DateTimeOffset offset) where T : class
{
var generatorWrapped = new Lazy<T>(generator, System.Threading.LazyThreadSafetyMode.PublicationOnly);
return ((Lazy<T>)cache.AddOrGetExisting(
key,
generatorWrapped,
offset))?.Value ?? generatorWrapped.Value;
}
public ThingTwo CachedThingTwo
{
get
{
return CacheGetWithGenerate(
MemoryCache.Default,
"Services.ThingTwoData",
() => GetThingTwo(), // ok, GetThingTwo isn't async this time, fudged example
DateTime.Now.Add(TimeSpan.FromMinutes(1)));
}
}
Do you know why MSDTC is being invoked?
PublicationOnly means that two connections can be created and one thrown away. I'm surprised you made this bug because you explicitly stated PublicationOnly (as opposed to the default safety mode which is safe). You explicitly allowed this bug.
For some reason I did not see that you tried ExecutionAndPublication already. Since not using it is a bug please fix the code in the question.
CreateConnection is also broken in the sense that in case of exception on open the connection object is not getting disposed. Probably harmless but you never know.
Also, audit this code for thread aborts which can happen when ASP.NET times out a request. You are doing very dangerous and brittle things here.
The pattern that I use is to use an IOC container to inject a connection that is shared for the entire request. The first client for that connection opens it. The request end event closes it. Simple, and does away with all that nasty shared, mutable, multi-threaded state.
Why are you using a cache for data that you do not want to lose? This is probably the bug. Don't do that.
What is ?.Value ?? generatorWrapped.Value about? The dictionary can never return null. Delete that code. If it could return null then forcing the lazy value would create a second connection so that's a logic bug as well.
We have an issue with some of our ASP.Net applications. Some of our apps claim a large amount of memory from start as their working set.
On our 2 webfarm-servers (4gb of RAM each) run multiple applications. We have a stable environment with about 1.2gb of memory free.
Then we add an MVC5 + WebApi v2 + Entity Framework app that instantly claims 1+gb as working set memory, while only actually using about 300mb. This causes other applications to complain that there is not enough memory left.
We already tried setting limit for virtual memory and the limit for private memory, without any avail. If we set this to about 500mb, the app still uses more or less the same amount of memory (way over 500) and does not seem to respect the limits put in place.
For reference, I tested this with an empty MVC5 project (VS2013 template) and this already claims 300mb of memory, while only using about 10mb.
Setting the app as a 32-bit app seems to have some impact in reducing the size of the working set.
Is there any way to reduce the size of the working set, or to enforce a hard limit on the size of it?
Edit:
In the case of the huge memory use for the project using Web Api v2 and Entity Framework, my API controllers look like this:
namespace Foo.Api
{
public class BarController : ApiController
{
private FooContext db = new FooContext();
public IQueryable<Bar> GetBar(string bla)
{
return db.Bar.Where(f => f.Category.Equals(bla)).OrderBy(f => f.Year);
}
}
as they look in most tutorials I could find (including the ones from microsoft). Using using here does not work because of LINQ deferred loading. It could work if I added a ToList (not tested) everywhere, but does this have any other impact?
edit2:
It works if I do
namespace Foo.Api
{
public class BarController : ApiController
{
public List<Bar> GetBar(string bla)
{
using(FooContext db = new FooContext){
return db.Bar.Where(f => f.Category.Equals(bla)).OrderBy(f => f.Year).ToList();
}
}
}
Does the ToList() have any implications on the performance of the api? (I know I can not continue querying cheaply as with an IQueryable)
Edit3:
I notice that its the private working set of the app that is quite high. Is there a way to limit this? (Without causing constant recycles)
Edit4:
As far as I know I have a Dispose on each and every APIController. My front-end is just some simple MVC controllers but for the large part .cshtml and javascript (angular) files.
We have another app, just regular mvc, with two models and some simple views (no database, or other external resources that could be leaked) and this also consumes up to 4-500mb of memory. If I profile it, I can't see anything that indicates memory leaks, I do see that only 10 or 20 mb is actually used, the rest is unmanaged memory that is unassigned (but part of the private memory working set, so claimed by this app and unusable by any other).
I had a similar problem with some of my applications. I was able to solve the problem by properly closing the disposable database resources by wrapping them in using clauses.
For Entity Framework, that would mean to ensure you always close your context after each request. Connections should be disposed between requests.
using (var db = new MyEFContext())
{
// Execute queries here
var query = from u as db.User
where u.UserId = 1234
select u.Name;
// Execute the query.
return query.ToList();
// This bracket will dispose the context properly.
}
You may need to wrap the context into a service that request-caches your context in order to keep it alive throughout the request, and disposes of it when complete.
Or, if using the pattern of having a single context for the entire controller like in the MSDN examples, make sure you override the Dispose(bool) method, like the example here.
protected override void Dispose(bool disposing)
{
if (disposing)
{
db.Dispose();
}
base.Dispose(disposing);
}
So your controller (from above) should look like this:
namespace Foo.Api
{
public class BarController : ApiController
{
private FooContext db = new FooContext();
public IQueryable<Bar> GetBar(string bla)
{
return db.Bar.Where(f => f.Category.Equals(bla)).OrderBy(f => f.Year);
}
// WebApi 2 will call this automatically after each
// request. You need this to ensure your context is disposed
// and the memory it is using is freed when your app does garbage
// collection.
protected override void Dispose(bool disposing)
{
if (disposing)
{
db.Dispose();
}
base.Dispose(disposing);
}
}
}
The behavior I saw was that the application would consume a lot of memory, but it could garbage collect enough memory to keep it from ever getting an OutOfMemoryException. This made it difficult to find the problem, but disposing the database resources solved it. One of the applications used to hover at around 600 MB of RAM usage, and now it hovers around 75 MB.
But this advice doesn't just apply to database connections. Any class that implements IDisposable should be suspect if you are running into memory leaks. But since you mentioned you are using EntityFramework, it is the most likely suspect.
Removing all Telerik Kendo MVC references (dll and such) fixed our problems. If we run the application without, all our memory problems are gone and we see normal memory use.
Basically: it was an external library causing high memory use.
I have an MVC3/.NET 4 application which uses Entity Framework (4.3.1 Code First)
I have wrapped EF into a Repository/UnitOfWork pattern as described here…
http://www.asp.net/mvc/tutorials/getting-started-with-ef-using-mvc/implementing-the-repository-and-unit-of-work-patterns-in-an-asp-net-mvc-application
Typically, as it explains in the article, when I require the creation of a new record I’ve been doing this…
public ActionResult Create(Course course)
{
unitOfWork.CourseRepository.Add(course);
unitOfWork.Save();
return RedirectToAction("Index");
}
However, when more than simply saving a record to a database is required I wrap the logic into what I’ve called an IService. For example…
private ICourseService courseService;
public ActionResult Create(Course course)
{
courseService.ProcessNewCourse(course);
return RedirectToAction("Index");
}
In one of my services I have something like the following…
public void ProcessNewCourse(Course course)
{
// Save the course to the database…
unitOfWork.CourseRepository.Add(course);
unitOfWork.Save();
// Generate a PDF that email some people about the new course being created, which requires more use of the unitOfWork…
var someInformation = unitOfWork.AnotherRepository.GetStuff();
var myPdfCreator = new PdfCreator();
IEnumerable<People> people = unitOfWork.PeopleRepository.GetAllThatWantNotifiying(course);
foreach(var person in people)
{
var message = “Hi ” + person.FullName;
var attachment = myPdfCreator.CreatePdf();
etc...
smtpClient.Send();
}
}
The above isn’t the actual code (my app has nothing to do with courses, I’m using view models, and I have separated the PDF creation and email message out into other classes) but the gist of what is going on is as above!
My problem is that the generation of the PDF and emailing it out is taking some time. The user just needs to know that the record has been saved to the database so I thought I would put the code below the unitOfWork.Save(); into an asynchronous method. The user can then be redirected and the server can happily take its time processing the emails, and attachments and whatever else I require it to do post save.
This is where I’m struggling.
I’ve tried a few things, the current being the following in ICourseService…
public class CourseService : ICourseService
{
private delegate void NotifyDelegate(Course course);
private NotifyDelegate notifyDelegate;
public CourseService()
{
notifyDelegate = new NotifyDelegate(this.Notify);
}
public void ProcessNewCourse(Course course)
{
// Save the course to the database…
unitOfWork.CourseRepository.Add(course);
unitOfWork.Save();
notifyDelegate.BeginInvoke(course);
}
private void Notify(Course course)
{
// All the stuff under unitOfWork.Save(); moved here.
}
}
My Questions/Problems
I’m randomly getting the error: "There is already an open DataReader associated with this Command which must be closed first." in the Notify() method.
Is it something to do with the fact that I’m trying to share the unitOrWork and therefore a dbContext across threads?
If so, can someone be kind enough to explain why this is a problem?
Should I be giving a new instance of unitOfWork to the Notify method?
Am I using the right patterns/classes to invoke the method asynchronously? Or should I be using something along the lines of....
new System.Threading.Tasks.Task(() => { Notify(course); }).Start();
I must say I've become very confused with the terms asynchronous, parallel, and concurrent!!
Any links to articles (c# async for idiots) would be appreciated!!
Many thanks.
UPDATE:
A little more digging got me to this SO page: https://stackoverflow.com/a/5491978/192999 which says...
"Be aware though that EF contexts are not thread safe, i.e. you cannot use the same context in more than one thread."
...so am I trying to achieve the impossible? Does this mean I should be creating a new IUnitOfWork instance for my new thread?
You could create a polling background thread that does the lengthy operation separately from your main flow. This thread could scan the database for new items (or items marked to process). This solution is pretty simple and ensures that jobs get done even if you application crashes (it will be picked up when the polling thread is started again).
You could also use a Synchronised Queue if it's not terrible if the request is 'lost', in the case your application crashes after the doc is requested and before it's generated/sent.
One thing is almost sure - as rikitikitik said - you will need to use a new unit of work, which means a separate transaction.
You could also look at Best threading queue example / best practice .
I've seen some posts here that look like mine, however I couldn't find the answer to my specific problem.
The thing is that I'm trying to insert an entity into the DB using a nav. prop.
however I'm constantly getting: The ObjectContext instance has been disposed and can no longer be used for operations that require a connection.
using (EntitiesDatabase context = new EntitiesDatabase())
{
if (UserHandler.Instance.User is Admin)
{
((Admin)UserHandler.Instance.User).ProjectManagers.Add(
new ProjectManager(firstNameTextBox.Text, lastNameTextBox.Text, usernameTextBox.Text, passwordTextBox.Text));
}
else if (UserHandler.Instance.User is ProjectManager)
{
((ProjectManager)UserHandler.Instance.User).Developers.Add(
new Developer(firstNameTextBox.Text, lastNameTextBox.Text, usernameTextBox.Text, passwordTextBox.Text));
}
context.SaveChanges();
}
The error exception is thrown at line 5-6 of the code (u.ProjectManagers.Add(new ProjectManager(...))
For clarity:
UserHandler is a singelton that holds the reference to the logged on user.
The user has a navigation property
public virtual ICollection<ProjectManager> ProjectManagers { get; set; }
That should actually insert the new project manager to the DB after I do SaveChanges() on the context.
I've done some simple tests on smaller projects (like the ones in the PluralSight tutorials) and it did work... for some reason these same lines do not work.
It really seems as if the new project manager I'm creating is being disposed before I try to insert it to the nav. prop. because I'm never getting to the save changes line...
From other answers I've seen, I guessed that the using statement for some reason disposes of things before it has reached the end bracket. I don't get why...
Could someone point me in the right way?
BTW this is not homework guys, it's a project of mine in which I am trying to learn C#, EF and other technologies.
It looks like the context you get the User instance from are fetched in a different context, which are disposed.
You have two options either use the same context, or attach the entity to the new context using ObjectContext.Attach like this:
using (EntitiesDatabase context = new EntitiesDatabase())
{
context.Attach(UserHandler.Instance.User);
if (UserHandler.Instance.User is Admin)
{
((Admin)UserHandler.Instance.User).ProjectManagers.Add(
new ProjectManager(firstNameTextBox.Text, lastNameTextBox.Text, usernameTextBox.Text, passwordTextBox.Text));
}
else if (UserHandler.Instance.User is ProjectManager)
{
((ProjectManager)UserHandler.Instance.User).Developers.Add(
new Developer(firstNameTextBox.Text, lastNameTextBox.Text, usernameTextBox.Text, passwordTextBox.Text));
}
context.SaveChanges();
}
My guess would be that the data context related to UserHandler.Instance.User isn't alive in the current calling context, and UserHandler.Instance.User isn't attached to the context more recently instantiated.