Are there any data structures in the C# Collections library where modification of the structure does not invalidate iterators?
Consider the following:
List<int> myList = new List<int>();
myList.Add( 1 );
myList.Add( 2 );
List<int>.Enumerator myIter = myList.GetEnumerator();
myIter.MoveNext(); // myIter.Current == 1
myList.Add( 3 );
myIter.MoveNext(); // throws InvalidOperationException
Yes, take a look at the System.Collections.Concurrent namespace in .NET 4.0.
Note that for some of the collections in this namespace (e.g., ConcurrentQueue<T>), this works by only exposing an enumerator on a "snapshot" of the collection in question.
From the MSDN documentation on ConcurrentQueue<T>:
The enumeration represents a
moment-in-time snapshot of the
contents of the queue. It does not
reflect any updates to the collection
after GetEnumerator was called. The
enumerator is safe to use concurrently
with reads from and writes to the
queue.
This is not the case for all of the collections, though. ConcurrentDictionary<TKey, TValue>, for instance, gives you an enumerator that maintains updates to the underlying collection between calls to MoveNext.
From the MSDN documentation on ConcurrentDictionary<TKey, TValue>:
The enumerator returned from the
dictionary is safe to use concurrently
with reads and writes to the
dictionary, however it does not
represent a moment-in-time snapshot of
the dictionary. The contents exposed
through the enumerator may contain
modifications made to the dictionary
after GetEnumerator was called.
If you don't have 4.0, then I think the others are right and there is no such collection provided by .NET. You can always build your own, however, by doing the same thing ConcurrentQueue<T> does (iterate over a snapshot).
According to this MSDN article on IEnumerator the invalidation behaviour you have found is required by all implementations of IEnumerable.
An enumerator remains valid as long as the collection remains unchanged. If
changes are made to the collection, such as adding, modifying, or deleting
elements, the enumerator is irrecoverably invalidated and the next call to
MoveNext or Reset throws an InvalidOperationException. If the collection is
modified between MoveNext and Current, Current returns the element that it is
set to, even if the enumerator is already invalidated.
Supporting this behavior requires some pretty complex internal handling, so most of the collections don't support this (I'm not sure about the Concurrent namespace).
However, you can very well simulate this behavior using immutable collections. They don't allow you to modify the collection by design, but you can work with them in a slightly different way and this kind of processing allows you to use enumerator concurrently without complex handling (implemented in Concurrent collections).
You can implement a collection like that easily, or you can use FSharpList<T> from FSharp.Core.dll (not a standard part of .NET 4.0 though):
open Microsoft.FSharp.Collections;
// Create immutable list from other collection
var list = ListModule.OfSeq(anyCollection);
// now we can use `GetEnumerable`
var en = list.GetEnumerable();
// To modify the collection, you create a new collection that adds
// element to the front (without actually copying everything)
var added = new FSharpList<int>(42, list);
The benefit of immutable collections is that you can work with them (by creating copies) without affecting the original one, and so the behavior you wanted is "for free". For more information, there is a great series by Eric Lippert.
The only way to do this is to make a copy of the list before you iterate it:
var myIter = new List<int>(myList).GetEnumerator();
No, they do not exist. ALl C# standard collections invalidate the numerator when the structure changes.
Use a for loop instead of a foreach, and then you can modify it. I wouldn't advise it though....
Related
I have a question about the thread-safety of the List<T> collection.
Here is my test class:
Test t = new Test();
t.a = 100;
t.b = 20;
t.c = 10;
Then let's say 10 instance of above have been created and added to the List as below.
List<Test> tCollection = new List<Test>();
tCollection.add(t);
Later I iterate through the objects of test in tCollection.
foreach(Test t in tCollection)
{
// do calculation
}
Is adding objects to List<Test> and iteration through List<Test> is thread safe?
The answer is No.
Usual collections are all non thread-safe.
You have to use the thread-safe analogues of collections*. read more about these collections on MSDN.
The default .net collections are not thread-safe.
This means that they contain no additional code that deals with multi-threaded access, since such code would decrease their performance in single-thread scenarios.
As Alexander Galkin said in the other answer, Microsoft provides some collections that are tailor-made for multi-threaded access. However you will note that there is no ConcurrentList<T> that works exactly like a List. This is because creating a thread-safe list with with all the properties of a list such as random access, insertion and removal and good performance is practically impossible.
The closest equivalent in your case would be a System.Collections.Concurrent.ConcurrentQueue<T> or ConcurrentStack<T>.
However, there may be an easier way in your case:
Read-only access to a List is always thread-safe. You can iterate over your list, or access random elements, from as many threads as you want, provided that you do not change the list.
Adding elements to the list is not thread-safe. This will be obvious if you iterate over the list while adding elements, you will get an Exception: "Collection was modified after the enumerator was instantiated".
But even adding elements from multiple threads without iterating over the list at the same time is not safe. You will not get such an obvious error, instead it will sometimes work and sometimes not.
If you only add elements once after the program starts, then you can get by with a normal list. Just make sure that no reader threads are started yet while you add elements to the list in a single writer thread. Once you added all elements to the list, you can iterate over it from as many threads as you want.
For more complex scenarios it is usually better to switch to ConcurrentQueue or ConcurrentStack, instead of implementing your own thread-safe methods using lock.
What happens when a thread is adding or removing elements of a ConcurrentBag<T> while another thread is enumerating this bag? Will the new elements also show up in the enumeration and will the removed elements not show up?
One can read the fine manual to discover:
ConcurrentBag<T>.GetEnumerator Method
The enumeration represents a moment-in-time snapshot of the contents of the bag. It does not reflect any updates to the collection after GetEnumerator was called. The enumerator is safe to use concurrently with reads from and writes to the bag.
Emphasis mine.
Justin Etheredge has a blog post explaining the features of the ConcurrentBag class:
In order to implement the enumerable as thread-safe, the GetEnumerator method returns a moment-in-time snapshot of the ConcurrentBag as it was when you started iterating over it. In this way, any items added after the enumeration started won’t be seen while iterating over the data structure.
This means: When starting to enumerate the ConcurrentBag<T>, a snapshot of the current state is created. The enumeration will only show the elements that were present in the bag at the time the enumeration started.
Other threads can still add and remove elements as they like but this will not change the set of elements seen by the enumeration.
I'm working with large collections of objects and sequential reads of them.
I found most questions along these lines refer to multi-threading, but I am more concerned with errors within the thread itself due to misuse of a distributable library.
A system within the library manages a potentially large collection of objects, at one point it performs a sequential read of this collection performing an operation on each element.
Depending on the element implementation, which can be extended outside the library, an object may attempt to remove itself from the collection.
I would like that to be an option, but if this happens when the collection is being sequentially read this can lead to errors. I would like to be able to lock the contents of the collection while its being read and put any removal request on a schedule to be executed after the sequential read has finished.
The removal request has to go through the system since objects do not have public access to the collection, I could just go with an isReading flag but I wonder if there is a more elegant construct.
Does C# or .NET provide a tool to do this? perhaps to lock the list contents so I can intercept removal requests during sequential reads? or would I have to implement that behavior from scratch for this scenario?
You may want to look into using the SynchronizedCollection<T> class in .NET 2.0+.
Alternatively, have a look at the answer to this question: What is the difference between SynchronizedCollection<T> and the other concurrent collections?
You can use the next trick
List<T> collection;
for(int index = collection; index >= 0; --index)
{
var item = collection[index];
if(MUST BE DELETED)
{
collection.RemoveAt(index); // this is faster
OR
collection.Remove(item);
}
}
this code will not crash at collection modified and will process each item of collection
I have a question that I am surprised hasn't already been asked in exactly this format.
If I have an IEnumerable that is generated based on iterating through a source of data, (and using a yield return statement), how can I detect when there has been a modification to the source after an access via an Enumerator that was generated via a GetEnumerator call?
Here is the strange part: I'm not multi-threading. I think my question has a flaw in it somewhere, because this should be simple. . . I just want to know when the source has changed and the iterator is out of date.
Thank you so much.
You would need to handle creating the enumerator yourself in order to track this information, or, at a minimum use yield return; with your own type of modification tracking in place.
Most of the framework collection classes, for example, keep a "version" number. When they make an enumerator, they keep a snapshot of that version number, and check it during MoveNext(). You could make the same check before calling yield return XXX;
Most collection classes in the .NET BCL uses a version attribute for change tracking. That is: the enumerator is constructed with a version number (integer) and checks the original source of the version number is still the same each iteration (when movenext is called). The collection in turn increments the version attribute each time a modification is made. This tracking mechanism is simple and effective.
2 other ways i've seen are:
Having the collection hold an internal collection containing weak references to outstanding enumerators. and each time a modification is made to the collection, it makes each enumerator which is still alive invalid.
Or implementing events in the collection ( INotifyCollectionChanged ) and simply register on that event in the enumerator. And if raised, mark the enumerator as invalid. This method is relatively easy to implement, generic and comes without to much overhead but requires your collection to support events
Microsoft suggests any modification to an IEnumerable collection should void any existing IEnumerator objects, but that policy is seldom particularly helpful and can sometimes be a nuisance. There is no reason why the author of an IEnumerable/IEnumerator should feel a need to throw an exception if a collection is modified in a way that will not prevent the IEnumerator from returning the same data as it would have returned without such modification. I would go further and suggest that it should be considered desirable, when possible, to have an enumerator remain functional if it can obey the following constraints:
Items which are in the collection throughout the duration of enumeration must be returned exactly once.
Each items which is added or deleted during enumeration may be returned zero or one times, but no more than one. If an object is removed from the collection and re-added, it may be regarded as having been originally housed in one item but put into a new one, so enumeration may legitimately return the old one, the new one, both, or neither.
The VisualBasic.Collection class behaves according to the above constraints; such behavior can be very useful, making it possible to enumerate through the class and remove items meeting a particular criterion.
Of course, designing a collection to behave sensibly if it's modified during enumeration may not necessarily be easier than throwing an exception, but for collections of reasonable size such semantics may be obtained by having the enumerator convert the collection to a list and enumerate the contents of the list. If desired, and especially if thread safety is not required, it may be helpful to have the collection keep a strong or weak reference to the list returned by its enumerator, and void such reference any time it is modified. Another option would be to have a "real" reference to the collection be held in a wrapper class, and have the inner class keep a count of how many enumerators exist (enumerators would get a reference to the real collection). If an attempt is made to modify the collection while enumerators exist, replace the collection instance with a copy and then make the modifications on that (the copy would start with a reference count of zero). Such a design would avoid making redundant copies of the list except in the scenario where an IEnumerator is abandoned without being Dispose'd; even in that scenario, unlike scenarios involving WeakReferences or events, no objects would be kept alive any longer than necessary.
I haven't found an answer, but as a work around I have just been catching the exception like this (WPF example):
while (slideShowOn)
{
if (this.Model.Images.Count < 1)
{
break;
}
var _bitmapsEnumerator = this.Model.Images.GetEnumerator();
try
{
while (_bitmapsEnumerator.MoveNext())
{
this.Model.SelectedImage = _bitmapsEnumerator.Current;
Dispatcher.Invoke(new Action(() => { }), DispatcherPriority.ContextIdle, null);
Thread.Sleep(41);
}
}
catch (System.InvalidOperationException ex)
{
// Scratch this bit: the error message isn't restricted to English
// if (ex.Message == "Collection was modified; enumeration operation may not execute.")
// {
//
// }
// else throw ex;
}
}
I have a main thread that populates a List<T>. Further I create a chain of objects that will execute on different threads, requiring access to the List. The original list will never be written to after it's generated. My thought was to pass the list as IEnumerable<T> to the objects executing on other threads, mainly for the reason of not allowing those implementing those objects to write to the list by mistake. In other words if the original list is guaranteed not be written to, is it safe for multiple threads to use .Where or foreach on the IEnumerable?
I am not sure if the iterator in itself is thread safe if the original collection is never changed.
IEnumerable<T> can't be modified. So what can be non thread safe with it? (If you don't modify the actual List<T>).
For non thread safety you need writing and reading operations.
"Iterator in itself" is instantiated for each foreach.
Edit: I simplified my answer a bit, but #Eric Lippert added valuable comment. IEnumerable<T> doesn't define modifying methods, but it doesn't mean that access operators are thread safe (GetEnumerator, MoveNext and etc.) Simplest example: GetEnumerator implemented as this:
Every time returns same instance of IEnumerator
Resets it's position
More sophisticated example is caching.
This is interesting point, but fortunately I don't know any standard class that has not thread-safe implementation of IEnumerable.
Each thread that calls Where or foreach gets its own enumerator - they don't share one enumerator object for the same list. So since the List isn't being modified, and since each thread is working with its own copy of an enumerator, there should be no thread safety issues.
You can see this at work in one thread - Just create a List of 10 objects, and get two enumerators from that List. Use one enumerator to enumerate through 5 items, and use the other to enumerate through 5 items. You will see that both enumerators enumerated through only the first 5 items, and that the second one did not start where the first enumerator left off.
As long as you are certain that the List will never be modified then it will be safe to read from multiple threads. This includes the use of the IEnumerator instances it provides.
This is going to be true for most collections. In fact, all collections in the BCL should be stable during enumeration. In other words, the enumerator will not modify the data structure. I can think of some obscure cases, like a splay-tree, were enumerating it might modify the structure. Again, none of the BCL collections do that.
If you are certain that the list will not be modified after creation, you should guarantee that by converting it to a ReadOnlyCollection<T>. Of course if you keep the original list that the read only collection uses you can modify it, but if you toss the original list away you're effectively making it permentantly read only.
From the Thread Safety section of the collection:
A ReadOnlyCollection can support multiple readers concurrently, as long as the collection is not modified.
So if you don't touch the original list again and stop referencing it, you can ensure that multiple threads can read it without worry (so long as you don't do anything wacky with trying to modify it again).
In other words if the original list is guaranteed not be written to, is it safe for multiple threads to use .Where or foreach on the IEnumerable?
Yes it's only a problem if the list gets mutated.
But note than IEnumerable<T> can be cast back to a list and then modified.
But there is another alternative: wrap your list into a ReadOnlyCollection<T> and pass that around. If you now throw away the original list you basically created a new immutable list.
If you are using net framework 4.5 or greater, this could be a great soulution
http://msdn.microsoft.com/en-us/library/dd997305(v=vs.110).aspx
(microsoft already implemented a thread safe enumerable)