We Keep Coding

EF 6 Connection pool and stored query strings RAM leak

I have 2 problems.
First one is that Connection pool and (OracleConnectionImpl) is growing steadily over time, till it reaches out of memory exception. As you can see on the image, it has 90 MB over 3-4 hours of running.
I am using short lived contexts everywhere, but it keeps on growing and never clears itself. Is there any way for me to clear it?
Second one is that EF stores too much duplicates of string queries over time.
It mostly stores those which come from .Reload() function, because it is not paramterized, it hardcode the ID into the query.
And then there are strings like "ID" which is somewhere cached 2947x.
Is there any way for the .Reload() function to make it parametrized, or to clear all of those stored strings?
This app is refreshing warehouse jobs and palletes every few seconds, to keep it in sync for all machines and I am not aware of better way than .Reload() because of WPF bindings.
Edit 1
I have simple helper function for reloading many intities at once, even one extension. It doesnt matter that it is passed as object, because the same problem remains even with the last example.
public static void ReloadEntities(bool dispatch, params IEnumerable<object>[] entities)
{
using (var ctx = new eWMSEntities())
{
if (dispatch)
{
Application.Current.Dispatcher.Invoke(DispatcherPriority.Background, (SendOrPostCallback)delegate
{
entities.SelectMany(x => x.Select(s => s)).ToList().ForEach(entity =>
{
ctx.Set(entity.GetType()).Attach(entity);
ctx.Entry(entity).Reload();
ctx.Entry(entity).State = EntityState.Detached;
});
}, null);
}
else
{
entities.SelectMany(x => x.Select(s => s)).ToList().ForEach(entity =>
{
ctx.Set(entity.GetType()).Attach(entity);
ctx.Entry(entity).Reload();
ctx.Entry(entity).State = EntityState.Detached;
});
}
ctx.Dispose();
}
}
public static void ReloadEntity(this object entity, bool dispatch)
{
using (var ctx = new eWMSEntities())
{
ctx.Set(entity.GetType()).Attach(entity);
if (dispatch)
{
Application.Current.Dispatcher.Invoke(DispatcherPriority.Background, (SendOrPostCallback)delegate
{
ctx.Entry(entity).Reload();
}, null);
}
else
{
ctx.Entry(entity).Reload();
}
ctx.Entry(entity).State = EntityState.Detached;
ctx.Dispose();
}
}
while (true && JobLines.Contains(line))
{
using (var ctx = new eWMSEntities())
{
ctx.T_JOB_LINES.attach(line);
ctx.entry(line).Reload();
}
await Task.Delay(3000);
}
This is snapshot after 3 days of running

I'm not recommend but if there are no any other solution...
Maybe you can try
GC.Collect();
Let system force to collect something didn't important in your memory.

Use the "Unit of work" pattern - it will solve many problems. I did not find for WPF, but found for ASP.NET MVC

Handle multiple similar requests in webapi

In my WebApi controller I have the following (pseudo) code that receives update notifications from Instagrams real-time API:
[HttpPost]
public void Post(InstagramUpdate instagramUpdate)
{
var subscriptionId = instagramUpdate.SubscriptionId;
var lastUpdate = GetLastUpdate(subscriptionId);
// To avoid breaking my Instagram request limit, do not fetch new images too often.
if (lastUpdate.AddSeconds(5) < DateTime.UtcNow)
{
// More than 5 seconds ago since last update for this subscription. Get new images
GetNewImagesFromInstagram(subscriptionId);
UpdateLastUpdate(subscriptionId, DateTime.UtcNow);
}
}
This won't work very well if I receive two update notifications for the same subscription almost simultaneously, since lastUpdate won't have been updated until after the first request has been processed.
What would be the best way to tackle this problem? I'm thinking of using some kind of cache, but I'm not sure how. Is there some kind of best practices for these kind of things? I'm guessing it's a common problem: "receive notification, do something if something hasn't been done recently..."

Thanks to this answer I went with the following approach, using MemoryCache
[HttpPost]
public void Post(IEnumerable<InstagramUpdate> instagramUpdates)
{
foreach (var instagramUpdate in instagramUpdates)
{
if (WaitingToProcessSubscriptionUpdate(instagramUpdate.Subscription_id))
{
// Ongoing request, do nothing
}
else
{
// Process update
}
}
}
private bool WaitingToProcessSubscriptionUpdate(string subscriptionId)
{
// Check in the in memory cache if this subscription is in queue to be processed. Add it otherwise
var queuedRequest = _cache.AddOrGetExisting(subscriptionId, string.Empty, new CacheItemPolicy
{
// Automatically expire this item after 1 minute (if update failed for example)
AbsoluteExpiration = DateTime.Now.AddMinutes(1)
});
return queuedRequest != null;
}

I am afraid that it is awful idea, but ... Maybe it worth to add lock to this method ? Like
private List<int> subscriptions = new List<int>();
and then
int subscriptinId = 1;//add calculation here
int subscriptionIdIndex = subscriptions.IndexOf(subscriptinId);
lock (subscriptions[subscriptionIdIndex])
{
//your method code
}
Feel free to criticize this approach )

Returning IQueryable from within a using block. Need better design

I have created a PhoneBook style application; on my phonebook object I have a local member _site which is used as a filter, since there are approximately 1000 phone numbers, split across 12 sites within my organisation. Only one site will be retrieved at a time using this method.
This was my original method. The GUI has several methods for reordering the data, so I left it as an IQueryable because I would like to defer SQL to allow for filtering to be done on the SQL server rather than on the client PC.
Works
public IQueryable<PhoneNumber> GetPhoneDirectory()
{
PhoneBookDataContext db = new PhoneBookDataContext())
return db.PhoneNumbers.Where(d => d.Site == _site);
}
However, I am also trying to keep to 'best practise' in terms of using statements.
Doesn't Work
public IQueryable<PhoneNumber> GetPhoneDirectory()
{
using (PhoneBookDataContext db = new PhoneBookDataContext())
{
return db.PhoneNumbers.Where(d => d.Site == _site);
}
}
Now as pointed out by #justanotheruseryoumay, this will cause an exception because the datacontext is disposed by the time the objects are accessed.
I guess what I am asking is, how can I make sure my data context is disposed nicely, when I cannot use a 'using' statement and don't strictly know when the context is done with.

If you want to return IQueryable you can make your class that contains the GetPhoneDirectory disposable, make the PhoneBookDataContext a field, and dispose it in your dispose method.
You will then put the onus on the caller to dispose his instance of your class.
E.g.
public class MyClass : IDisposable
{
PhoneBookDataContext db;
public MyClass()
{
db = new PhoneBookDataContext();
}
public IQueryable<PhoneNumber> GetPhoneDirectory()
{
return db.PhoneNumbers.Where(d => d.Site == _site);
}
public void Dispose()
{
if (db != null)
{
db.Dispose();
db = null;
}
}
}
// Caller
using(var myClass = new MyClass())
{
var queryable = myClass.GetPhoneDirectory();
...
}

The execution of the query will still be deferred and the PhoneBookDataContext will still be properly Disposed because using is interpreted by the compile as a try/finally. When you actually execute the query it will result in a runtime error because the PhoneBookDataContext no longer exists. I would suggest doing a .ToList() on your query and returning it that way. If you want to change the order after you return it then you can still do LINQ on it as you please.
EDIT:
Another thing you could do is to create the using with the PhoneBookDataContext in the calling method and pass in the context. The context is really going to be used in that method anyway and you can keep it around as long as you need it and stick with the good using format.

Yes; It is bad design because your IQueryable<PhoneNumber> will be evaluated only when you call a method that cause it to be evaluated, like ToList() or when you iterate it with foreach.
in your code you are returning a IQueryable<PhoneNumber> which is not evaluated yet, and before the caller get any chance to execute it, it's internals that has the responsibility for yielding the records to you (db); is already disposed.
Just as a suggestion:
public IEnumerable<PhoneNumber> GetPhoneDirectory()
{
using (PhoneBookDataContext db = new PhoneBookDataContext())
{
return db.PhoneNumbers.Where(d => d.Site == _site).ToList();
}
}
Or relocate the db object to somewhere else in your design (Unit Of Work and Repository are nice patterns to get a look at IMHO).

When calling "context.AttachTo" - The ObjectContext instance has been disposed and can no longer be used for operations that require a connection

I'm having difficulty making a change to an entity object through a new context. I've had this work plenty of times before, but in this instance I'm getting the old "The ObjectContext instance has been disposed" exception.
Here's my quick edit/save code:
private void SaveChanges()
{
using (var context = new Manticore.ManticoreContext(Global.ManticoreClient))
{
**context.AttachTo("Users", Global.LoggedInUser);**
Global.LoggedInUser.FirstName = this.FirstNameText.Text;
Global.LoggedInUser.LastName = this.LastNameText.Text;
Global.LoggedInUser.Email = this.EmailText.Text;
context.SaveChanges();
}
}
The Global.LoggedInUser property (which is instantiated):
public static Manticore.User LoggedInUser
{
get
{
return HttpContext.Current.Session["LoggedInUser"] as Manticore.User;
}
set
{
HttpContext.Current.Session["LoggedInUser"] = value;
}
}
And the kicker is here's a quick unit test which works (no assert right now, but no exception being thrown):
private User _testUser;
private TestInstanceBucket _testBucket;
[TestInitialize]
public void TestInitialize()
{
using (var context = new Manticore.ManticoreContext())
{
this._testBucket = new TestInstanceBucket(context);
this._testUser = this._testBucket.TestUser;
context.AddObject("Users", this._testUser);
context.SaveChanges();
}
}
[TestMethod]
public void User_ShouldBeAbleToChangeDetails()
{
using (var context = new ManticoreContext())
{
context.AttachTo("Users", this._testUser);
this._testUser.FirstName = "New";
context.SaveChanges();
}
}
Like I say, I've done code like this before and it's been fine. Have I been lucking out, or could storing the entity in the session be causing problems?
Update
I've moved the code from global to a pagebase class which the page using SaveChanges() inherits. Same problem so it rules our static classes/methods and storing the entity in the session somehow causing problems.
Update
So, after several hours of banging my head against a wall, I have a fix that's fairly simple if annoying. After the initial fetch of the user I now call
context.Detach(user);
I can only assume it's something to do with how fast the context is being disposed with garbage collection in ASP.NET compared to in the test environment.

In my EF experience this issue always seems to occur when the entity is still linked to the old context. I believe it is safest to always just re-load the entity using the new context (query on PK). I know it is not the most efficient but it avoids this problem.
I took a look at this page before answering, if you haven't tried some of its suggestions you might give them a go: EF Add/Attach.

Method that adds elements to a DbSet from multiple threads

static Object LockEx=new Object();
public void SaveMyData(IEnumerable<MyData> list)
{
lock (LockEx)
{
using (PersistencyContext db = new PersistencyContext())
{
foreach (var el in list)
{
try
{
db.MyData.Add(el);
db.SaveChanges();
}
catch (DbUpdateException)
{
db.Entry(el).State = EntityState.Modified;
db.SaveChanges();
}
}
}
}
}
This methods is called from multiple threads. Right now I use a static lock to avoid 2 threads to save data at the same time. Though this is wrong because I only want to save data. The catch is used to create an update query in case the insert (Add) fails because the entry already exists.
What happens if I remove the lock. How will the SaveChanges work? How should my code look like? Thanks

I would remove the lock because the database already handles concurrency anyway by design, then I will also verify if the record exists before trying to add it, then I would do the add or update depending on this result. Just to avoid exceptions because they are performance killers.

Building on Davide's answer, you could also call SaveChanges once after you added all the new entities. That should be faster.