I'm encountering (I hope) a deadlocking issue with a WCF service I'm trying to write.
I have the following lock on a function that "locates" a particular item im the list:
CIPRecipe FindRecipe_ByUniqueID(string uniqueID)
{
lock (_locks[LOCK_RECIPES])
{
foreach (var r in _recipes.Keys)
{
if (_recipes[r].UniqueID == uniqueID)
{
return _recipes[r];
}
}
}
return null;
}
However, various functions reiterate through this list and always apply the same LOCK for example ....
lock (_locks[LOCK_RECIPES_NO_ADD_OR_REMOVE])
{
foreach (var r in _recipes)
{
r.Value.UpdateSummary();
summaries.Add((RecipeSummary)r.Value.Summary);
}
}
What I suspect is, an item in _recipes in the above example has suddenly called a function which ultimately calls the first function - "CIPRecipe FindRecipe_ByUniqueID(string uniqueID)" and this is causing a deadlock when it is reached in the iteration.
I need to stop this list changing whilst I'm iterating through it. Can someone advise me the best practice?
Thanks
What you want is to use a ReaderWriterLockSlim, this will let unlimited concurrent readers through but only a single writer through and block all readers while the writer is writing.
This assumes _locks has been chagned from a object[] to a ReaderWriterSlim[]
//On Read
CIPRecipe FindRecipe_ByUniqueID(string uniqueID)
{
var lockObj = _locks[LOCK_RECIPES];
lockObj.EnterReadLock();
try
{
foreach (var r in _recipes.Keys)
{
if (_recipes[r].UniqueID == uniqueID)
{
return _recipes[r];
}
}
}
finally
{
lockObj.ExitReadLock();
}
return null;
}
//On write
var lockObject = _locks[LOCK_RECIPES]; //Note this now uses the same lock object as the other method.
lockObj.EnterWriteLock();
try
{
foreach (var r in _recipes)
{
r.Value.UpdateSummary();
summaries.Add((RecipeSummary)r.Value.Summary);
}
}
finally
{
lockObj.ExitWriteLock();
}
I don't know if it will solve your deadlock issue, if it is caused by you allowing reads during a write it may.
Perhaps a ConcurrentDictionary is called for here?
Related
In a method returning IEnumerable<>, I'm opening and looping over a resource (e.g. a database row reader). Once the loop finished, the resource is closed again.
However, it may happen that the caller decides not to finish the enumeration. This leaves the resource open.
Example:
IEnumerable<Foo> Bar ()
{
using (var r = OpenResource()) {
while (r.Read ()) {
yield return r;
}
}
}
// OK - this closes the resource again
foreach (var foo in Bar()) {
Console.WriteLine (foo);
}
// Not OK - resource stays open!
Console.WriteLine (Bar().First());
How would I solve this? Can I easily cancel an enumeration, i.e. tell it to skip over the rest of the loop, or dispose it (putting the cleanup code in Dispose)?
I considered returning a Func<Result, bool> so the user can have it return false if he's done with iterating. Similarly, some kind of cancel token could be used, too. But both approaches seem cumbersome to me.
Normally it is the IEnumerator<> that implements the IDisposable, and if you look at the definition of IEnumerator<> you'll see that:
public interface IEnumerator<out T> : IDisposable, IEnumerator
The foreach statement correctly Dispose() the IEnumerator<> that receives from the IEnumerable<>, so that:
IEnumerable<SomeClass> res = SomeQuery();
foreach (SomeClass sc in res)
{
if (something)
break;
}
upon exiting the foreach in any way (the break, an exception, naturally finishing res), the Dispose() of the IEnumerator<> should be called. See https://msdn.microsoft.com/en-us/library/aa664754(v=vs.71).aspx for an example of how the foreach is implemented (a try... finally... with a Dispose() inside the finally)
Note that the C# will produce "correct" code for using used inside a yield function. See for example here: http://goo.gl/Igzmiz
public IEnumerable<Foo> Bar()
{
using (var r = OpenResource())
{
while (r.Read ())
{
yield return new Foo();
}
}
}
is converted to something that
void IDisposable.Dispose()
{
int num = this.<>1__state;
if (num == -3 || num == 1)
{
try
{
}
finally
{
this.<>m__Finally1();
}
}
}
The Dispose() method of IEnumerator<> will call a m__Finally1 method that will (IDisposable)this.<r>5__1.Dispose(); (where 5__1 is the r returned from OpenResource()). The m__Finally is even called if the code simply "exits" the while (r.Read ()):
if (!this.<r>5__1.Read())
{
this.<>m__Finally1();
and/or if there is an exception.
catch
{
this.System.IDisposable.Dispose();
If i iterate over a list of Weakreferences, how can i be sure, the reference still exists, after proofing via _ref.IsAlive?
For example i have this piece of code, where scopeReferences is a of Weakreferences:
foreach (var _ref in scopeReferences)
{
if (_ref.IsAlive)
{
if (_ref.Target is ScriptScope)
{
// Is it alive any more?
((ScriptScope)_ref.Target).SetVariable(name, value);
}
}
}
Maybe some one knows the answer, i just don't want to create any problems due to the fact, i don't know what's going on in this part. Thank you all a lot!
You could copy it to a variable, after which you either have it or you don't, and you can safely test it:
foreach (var _ref in scopeReferences)
{
ScriptScope tmp = _ref.Target as ScriptScope;
if (tmp != null)
{
tmp.SetVariable(name, value);
}
}
I'm trying to take all items in one fell swoop from a ConcurrentBag. Since there's nothing like TryEmpty on the collection, I've resorted to using Interlocked.Exchange in the same fashion as described here: How to remove all Items from ConcurrentBag?
My code looks like this:
private ConcurrentBag<Foo> _allFoos; //Initialized in constructor.
public bool LotsOfThreadsAccessingThisMethod(Foo toInsert)
{
this._allFoos.Add(toInsert);
return true;
}
public void SingleThreadProcessingLoopAsALongRunningTask(object state)
{
var token = (CancellationToken) state;
var workingSet = new List<Foo>();
while (!token.IsCancellationRequested)
{
if (!workingSet.Any())
{
workingSet = Interlocked.Exchange(ref this._allFoos, new ConcurrentBag<Foo>).ToList();
}
var processingCount = (int)Math.Min(workingSet.Count, TRANSACTION_LIMIT);
if (processingCount > 0)
{
using (var ctx = new MyEntityFrameworkContext())
{
ctx.BulkInsert(workingSet.Take(processingCount));
}
workingSet.RemoveRange(0, processingCount);
}
}
}
The problem is that this sometimes misses items that are added to the list. I've written a test application that feeds data to my ConcurrentBag.Add method and verified that it is sending all of the data. When I set a breakpoint on the Add call and check the count of the ConcurrentBag after, it's zero. The item just isn't being added.
I'm fairly positive that it's because the Interlocked.Exchange call doesn't use the internal locking mechanism of the ConcurrentBag so it's losing data somewhere in the swap, but I have no knowledge of what's actually happening.
How can I just grab all the items out of the ConcurrentBag at one time without resorting to my own locking mechanism? And why does Add ignore the item?
I think taking all the items from the ConcurentBag is not needed. You can achieve exactly the same behavior you are trying to implement simply by changing the processing logic as follows (no need for own synchronization or interlocked swaps):
public void SingleThreadProcessingLoopAsALongRunningTask(object state)
{
var token = (CancellationToken)state;
var buffer = new List<Foo>(TRANSACTION_LIMIT);
while (!token.IsCancellationRequested)
{
Foo item;
if (!this._allFoos.TryTake(out item))
{
if (buffer.Count == 0) continue;
}
else
{
buffer.Add(item);
if (buffer.Count < TRANSACTION_LIMIT) continue;
}
using (var ctx = new MyEntityFrameworkContext())
{
ctx.BulkInsert(buffer);
}
buffer.Clear();
}
}
I have the following code:
public class EmailJobQueue
{
private EmailJobQueue()
{
}
private static readonly object JobsLocker = new object();
private static readonly Queue<EmailJob> Jobs = new Queue<EmailJob>();
private static readonly object ErroredIdsLocker = new object();
private static readonly List<long> ErroredIds = new List<long>();
public static EmailJob GetNextJob()
{
lock (JobsLocker)
{
lock (ErroredIdsLocker)
{
// If there are no jobs or they have all errored then get some new ones - if jobs have previously been skipped then this will re get them
if (!Jobs.Any() || Jobs.All(j => ErroredIds.Contains(j.Id)))
{
var db = new DBDataContext();
foreach (var emailJob in db.Emailing_SelectSend(1))
{
// Dont re add jobs that exist
if (Jobs.All(j => j.Id != emailJob.Id) && !ErroredIds.Contains(emailJob.Id))
{
Jobs.Enqueue(new EmailJob(emailJob));
}
}
}
while (Jobs.Any())
{
var curJob = Jobs.Dequeue();
// Check the job has not previously errored - if they all have then eventually we will exit the loop
if (!ErroredIds.Contains(curJob.Id))
return curJob;
}
return null;
}
}
}
public static void ReInsertErrored(long id)
{
lock (ErroredIdsLocker)
{
ErroredIds.Add(id);
}
}
}
I then start 10 threads which do this:
var email = EmailJobQueue.GetNextJob();
if (email != null)
{
// Breakpoint here
}
The thing is that if I put a breakpoint where the comment is and add one item to the queue then the breakpoint gets hit multiple times. Is this an issue with my code or a peculiarity with VS debugger?
Thanks,
Joe
It appears as if you are getting your jobs from the database:
foreach (var emailJob in db.Emailing_SelectSend(1))
Is that database call marking the records as unavailable for section in future queries? If not, I believe that's why you're hitting the break point multiple times.
For example, if I replace that call to the database with the following, I see your behavior.
// MockDB is a static configured as `MockDB.Enqueue(new EmailJob{Id = 1})`
private static IEnumerable<EmailJob> GetJobFromDB()
{
return new List<EmailJob>{MockDB.Peek()};
}
However, if I actually Dequeue from the mock db, it only hits the breakpoint once.
private static IEnumerable<EmailJob> GetJobFromDB()
{
var list = new List<EmailJob>();
if (MockDB.Any())
list.Add(MockDB.Dequeue());
return list;
}
This is a side effect of debugging a multi-threaded piece of your application.
You are seeing the breakpoint being hit on each thread. Debugging a multi-threaded piece of the application is tricky because you're actually debugging all threads at the same time. In fact, at times, it will jump between classes while you're stepping through because it's doing different things on all of those threads, depending on your application.
Now, to address whether or not it's thread-safe. That really depends on how you're using the resources on those threads. If you're just reading, it's likely that it's thread-safe. But if you're writing, you'll need to leverage at least the lock operation on shared objects:
lock (someLockObject)
{
// perform the write operation
}
With the new ConcurrentBag<T> in .NET 4, how do you remove a certain, specific object from it when only TryTake() and TryPeek() are available?
I'm thinking of using TryTake() and then just adding the resulting object back into the list if I don't want to remove it, but I feel like I might be missing something. Is this the correct way?
The short answer: you can't do it in an easy way.
The ConcurrentBag keeps a thread local queue for each thread and it only looks at other threads' queues once its own queue becomes empty. If you remove an item and put it back then the next item you remove may be the same item again. There is no guarantee that repeatedly removing items and putting them back will allow you to iterate over the all the items.
Two alternatives for you:
Remove all items and remember them, until you find the one you want to remove, then put the others back afterwards. Note that if two threads try to do this simultaneously you will have problems.
Use a more suitable data structure such as ConcurrentDictionary.
You can't. Its a bag, it isn't ordered. When you put it back, you'll just get stuck in an endless loop.
You want a Set. You can emulate one with ConcurrentDictionary. Or a HashSet that you protect yourself with a lock.
The ConcurrentBag is great to handle a List where you can add items and enumerate from many thread, then eventually throw it away as its name is suggesting :)
As Mark Byers told, you can re-build a new ConcurrentBag that does not contains the item you wish to remove, but you have to protect this against multiple threads hits using a lock. This is a one-liner:
myBag = new ConcurrentBag<Entry>(myBag.Except(new[] { removedEntry }));
This works, and match the spirit in which the ConcurrentBag has been designed for.
As you mention, TryTake() is the only option. This is also the example on MSDN. Reflector shows no other hidden internal methods of interest either.
Mark is correct in that the ConcurrentDictionary is will work in the way you are wanting. If you wish to still use a ConcurrentBag the following, not efficient mind you, will get you there.
var stringToMatch = "test";
var temp = new List<string>();
var x = new ConcurrentBag<string>();
for (int i = 0; i < 10; i++)
{
x.Add(string.Format("adding{0}", i));
}
string y;
while (!x.IsEmpty)
{
x.TryTake(out y);
if(string.Equals(y, stringToMatch, StringComparison.CurrentCultureIgnoreCase))
{
break;
}
temp.Add(y);
}
foreach (var item in temp)
{
x.Add(item);
}
public static void Remove<T>(this ConcurrentBag<T> bag, T item)
{
while (bag.Count > 0)
{
T result;
bag.TryTake(out result);
if (result.Equals(item))
{
break;
}
bag.Add(result);
}
}
This is my extension class which I am using in my projects. It can a remove single item from ConcurrentBag and can also remove list of items from bag
public static class ConcurrentBag
{
static Object locker = new object();
public static void Clear<T>(this ConcurrentBag<T> bag)
{
bag = new ConcurrentBag<T>();
}
public static void Remove<T>(this ConcurrentBag<T> bag, List<T> itemlist)
{
try
{
lock (locker)
{
List<T> removelist = bag.ToList();
Parallel.ForEach(itemlist, currentitem => {
removelist.Remove(currentitem);
});
bag = new ConcurrentBag<T>();
Parallel.ForEach(removelist, currentitem =>
{
bag.Add(currentitem);
});
}
}
catch (Exception ex)
{
Debug.WriteLine(ex.Message);
}
}
public static void Remove<T>(this ConcurrentBag<T> bag, T removeitem)
{
try
{
lock (locker)
{
List<T> removelist = bag.ToList();
removelist.Remove(removeitem);
bag = new ConcurrentBag<T>();
Parallel.ForEach(removelist, currentitem =>
{
bag.Add(currentitem);
});
}
}
catch (Exception ex)
{
Debug.WriteLine(ex.Message);
}
}
}
public static ConcurrentBag<String> RemoveItemFromConcurrentBag(ConcurrentBag<String> Array, String Item)
{
var Temp=new ConcurrentBag<String>();
Parallel.ForEach(Array, Line =>
{
if (Line != Item) Temp.Add(Line);
});
return Temp;
}
how about:
bag.Where(x => x == item).Take(1);
It works, I'm not sure how efficiently...