I'm writing a C# library that needs to treat a List at high speed via multiple Timers.
I ran into very erratic error, where I try to remove an element that I know for sure is contained into the List but the program returns the following error :
System.IndexOutOfRangeException : 'index was outside the bounds of the array.'
I've made a simple example to reproduce this behaviour. Because of that issue's randomness, I've pushed hard on List operations so it throws the error right away. So this example is necessary "weird".
I've made a public repo on here : Issue Example Repo
Basically, here's what I'm dealing with:
list = new List<DummyElement>();
for (int i = 0; i < 1000; i++)
{
Timer addTimer = new Timer(0.01f);
addTimer.Start();
addTimer.Elapsed += AddItem;
Timer removeTimer = new Timer(0.01f);
removeTimer.Start();
removeTimer.Elapsed += RemoveItem;
}
void AddItem(object source, ElapsedEventArgs e)
{
list.Add(new DummyElement());
}
void RemoveItem(object source, ElapsedEventArgs e)
{
int listCount = list.Count;
if (listCount > 0) // This condition is successfully passed, so there is at least one element on the list
{
list.RemoveAt(0); // This line throw an IndexOutOfRangeException error
}
}
I believe it is a thread related issue, as if the list count was changing AFTER the condition was successfully passed.
I know nothing about thread, how can I deal with this issue?
In the For loop that goes upto 1000 - you are creating about 1000 Timers that add an item into list and 1000 timers that remove first item.
Since you haven't used any synchronization here's what happens:- Say there is 1 item in List and 2 RemoveItems are executing. both see listCount > 0 as True then one of them goes ahead and removes the Item at 0th Index while the other gets an Exception cause there is no item to remove now.
Now I cant suggest a solution to this by just looking at the code. I also need to understand the intent.
This is a Text Book Producer Consumer problem so the Text book advice here is using Lock construct:
Assume you have a class member like:
private object _lockMe = new object();
void RemoveItem(object source, ElapsedEventArgs e)
{
lock(_lockMe)
{
int listCount = list.Count;
if (listCount > 0) // This condition is successfully passed, so there is at least one element on the list
{
list.RemoveAt(0); // This line throw an IndexOutOfRangeException error
}
}
}
Related
I am implementing a code using List<>. Each list has an Class Type item. One of the members of this class is a System.Timer.Timer variable called "TimeOut". The TimeOut.Interval value is 10 and TimeOut.Elapsed Event should be triggered every TimeOut.Interval (which is 10).
The Event is to delete that item from the list. The Event is overloaded such that
TimeOut.Elapsed += (sender, e) => DeleteEntry(sender, e, i)
where i is the index of the item in the list.
For the Event method,
public void DeleteEntry(object sender, System.Timers.ElapsedEventArgs e, int i)
{
//Delete Item from List
listTable.RemoveAt(i);
}
My problem is if two items started their timer together and decided to call the EventHandler after the same specified interval, the one added to the list first is deleted, and then when the second one is called immediately., it throws an error telling me that the index is out of bounds. This happens because one of the items is deleted, the lists shifted its index up and and the "i" sent to the DeleteEntry Method is the old one which is no longer existent.
In other cases when I have more than two items in the list, it deletes the wrong one since the method uses a wrong index.
Using Arrays is not an option for me. Too much work to de-fragment the array every deletion of an element.
How can I solve this? Do I need to do something with the ThreadPool, setting a flag or something like that? Can SynchronizingObject property help in anyway?
Thank you.
Solution:
From what your saying it seems that your problem is the list shifting and you don't know at what point the item is. If that is the case you will want to access the Count property of the list which tells you the amount of items in the list. Be careful working with it as it is does not account for the indexing of 0.
Solution:
From what your saying it seems that your problem is the list shifting and you don't know at what point the item is. If that is the case you will want to access the Count property of the list which tells you the amount of items in the list. Be careful working with it as it is does not account for the indexing of 0.
Edit:
Example:
private static List<int> Numbers = new List<int>();
private static readonly object NumbersLock = new object();
static void Main(string[] args)
{
Thread X = new Thread(() => {
AddNumbers();
});
Thread Y = new Thread(() => {
AddNumbers();
});
X.Start();
Y.Start();
}
private static void AddNumbers() {
Random r = new Random(DateTime.UtcNow.Millisecond);
while (true) {
lock (NumbersLock) {
Numbers.Add(r.Next(0, 100));
}
}
}
I have these codes in my windows form C# application:
private void button7_Click(object sender, EventArgs e)
{
ThreadStart starter = delegate { thread_func(2, 1000000); };
thread1_thread = new Thread(starter);
starter = delegate { thread_func(1000000, 2000000); };
thread2_thread = new Thread(starter);
starter = delegate { thread_func(2000000, 3000000); };
thread3_thread = new Thread(starter);
starter = delegate { thread_func(3000000, 4000000); };
thread4_thread = new Thread(starter);
thread1_thread.Start();
thread2_thread.Start();
thread3_thread.Start();
thread4_thread.Start();
}
void thread_func(decimal input1,decimal input2)
{
for (; input1 < input2; input1++)
{
threadNumbers_list.Add(input1);
if (input1 % 2 != 0)
{
if (isPrime_func(input1))
{
PrimeNumbers_decimal_list.Add(input1);
}
}
}
}
public static Boolean isPrime_func(decimal number)
{
decimal boundary = (decimal)Math.Floor(Math.Sqrt((double)number));
if (number == 1) return false;
if (number == 2) return true;
for (decimal i = 2; i <= boundary; ++i)
{
if (number % i == 0) return false;
}
return true;
}
Every time I run click that button I get different results. I have tried many things but could not figure out why this happens. Even for lower ranges it happens. Just in range of 100 numbers for example it gives the same result always.
Some time my list count reaches 283138 and sometimes 283131 and other near numbers.
Another weird this is that when I comment checking even numbers, operation takes shorter time than this mode. What's wrong?
When multiple threads access a list, that list have to be thread safe or otherwise you are going to have a lot of problems.
.NET provides some thread-safe collections like the ConcurrentQueue<T> class.
Side note: Please consider using Tasks instead of threads. Also, the .NET framework supports data parallelism via the Parallel class. Consider using such class instead.
Regarding the performance when you don't check if the number is even, I tested this locally and I got the following numbers:
It takes ~76 seconds when I don't check if the number is even.
It takes ~66 seconds when I do check if the number is even.
So this does not match your measurements. It might be caused by the way you measure. I measure with a Stopwatch like this:
//...
Stopwatch sw = Stopwatch.StartNew();
thread1_thread.Start();
thread2_thread.Start();
thread3_thread.Start();
thread4_thread.Start();
thread1_thread.Join();
thread2_thread.Join();
thread3_thread.Join();
thread4_thread.Join();
long result = sw.ElapsedMilliseconds;
//...
By the way, here is something that you can do that might save some execution time for you:
Create a normal List<T> instance for each thread inside the thread_func method so that you don't have multi-threading issues. Then after the loop finishes, you can update the master list from the local list. Only updating the master list has to be thread safe. In this case I would prefer that the master list is a normal List<T> and that you use the lock keyword to synchronize access to it because you only need to update it 4 times (the number of threads).
I have an WinRT application that fires notifications everytime it recieves data from a device. I also have a UI control that is databound to an observable collection which I wish to add the new data to
While I have made it capable of updating the observable collection, it causes the UI to become very laggy, as the amount of data generated it fast. It would therefore be better to batch the update, maybe every few hundred milliseconds.
Below shows a snippet of the code. First I create the periodic timer
TimerElapsedHandler f = new TimerElapsedHandler(batchUpdate);
CreatePeriodicTimer(f, new TimeSpan(0, 0, 3));
Below is my event handler for when new data comes in, along with the temporary list that stores the information
List<FinancialStuff> lst = new List<FinancialStuff>();
async void myData_ValueChanged(GattCharacteristic sender, GattValueChangedEventArgs args)
{
var data = new byte[args.CharacteristicValue.Length];
DataReader.FromBuffer(args.CharacteristicValue).ReadBytes(data);
lst.Add(new FinancialStuff() { Time = "DateTime.UtcNow.ToString("mm:ss.ffffff")", Amount = data[0] });
}
Then my batch update, which is called peroidically
private void batchUpdate(ThreadPoolTimer source)
{
AddItem<FinancialStuff>(financialStuffList, lst);
}
Then finally, for testing I want to clear the observable collection and items.
public async void AddItem<T>(ObservableCollection<T> oc, List<T> items)
{
lock (items)
{
if (Dispatcher.HasThreadAccess)
{
foreach (T item in items)
oc.Add(item);
}
else
{
Dispatcher.RunAsync(CoreDispatcherPriority.Low, () =>
{
oc.Clear();
for (int i = 0; i < items.Count; i++)
{
items.Count());
oc.Add(items[i]);
}
lst.Clear();
});
}
}
}
While this seems to work, after a few updates the UI locks up and it updates very slowly/if not at all. For testing, it's only getting a few hundred items in the list by the time the timer is fired.
Can anybody enlighten me as to why as to why this is happening - I'm presuming my design is very poor.
Thanks
You're not locking your list in the event handler
// "lst" is never locked in your event handler
List<FinancialStuff> lst = new List<FinancialStuff>();
lst.Add(new FinancialStuff() { Time = "DateTime.UtcNow.ToString("mm:ss.ffffff")", Amount = data[0] });
Passing "lst" above to your async method
AddItem<FinancialStuff>(financialStuffList, lst);
You're locking "items" below, which is really "lst" above. However, you're adding to the list while your processing it. I assume the event handler has a higher priority so your processing is slower than your add. This can lead to "i < items.Count" being true forever.
public async void AddItem<T>(ObservableCollection<T> oc, List<T> items)
{
// "lst" reference is locked here, but it wasn't locked in the event handler
lock (items)
{
if (Dispatcher.HasThreadAccess)
{
foreach (T item in items)
oc.Add(item);
}
else
{
Dispatcher.RunAsync(CoreDispatcherPriority.Low, () =>
{
oc.Clear();
// This may never exit the for loop
for (int i = 0; i < items.Count; i++)
{
items.Count());
oc.Add(items[i]);
}
lst.Clear();
});
}
}
}
EDIT:
Do you need to view every piece of data? There is going to be some overhead when using a lock. If you're getting data quicker than the speed of how fast you can render it, you'll eventually be backed up and/or have a very large collection to render, which might also cause some problems. I suggest you do some filtering to only draw the last x number of items (say 100). Also, I'm not sure why you need the if (Dispatcher.HasThreadAccess) condition either.
Try the following:
public async void AddItem<T>(ObservableCollection<T> oc, List<T> items)
{
// "lst" reference is locked here, but it wasn't locked in the event handler
lock (items)
{
// Change this to what you want
const int maxSize = 100;
// Make sure it doesn't index out of bounds
int startIndex = Math.Max(0, items.Count - maxSize);
int length = items.Count - startIndex;
List<T> itemsToRender = items.GetRange(startIndex, length);
// You can clear it here in your background thread. The references to the objects
// are now in the itemsToRender list.
lst.Clear();
// Dispatcher.RunAsync(CoreDispatcherPriority.Low, () =>
// Please verify this is the correct syntax
Dispatcher.Run(() =>
{
// At second look, this might need to be locked too
// EDIT: This probably will just add overhead now that it's not running async.
// You can probably remove this lock
lock(oc)
{
oc.Clear();
for (int i = 0; i < itemsToRender.Count; i++)
{
// I didn't notice it before, but why are you checking the count again?
// items.Count());
oc.Add(itemsToRender[i]);
}
}
});
}
}
EDIT2:
Since your AddItem method is already on a background thread, I don't think you need to run the Dispatcher.RunAsync. Instead, I think it might be desirable for it to block so you don't end up with multiple calls to that section of code. Try using Dispatcher.Run instead. I've updated the code example above to show the changes. You shouldn't need the lock on the oc anymore since the lock on the items is good enough. Also, verify the syntax for Dispatcher.Run is correct.
I know there are MANY similiar questions, but I can't seem to get to the bottom of this.
In my program I execute a verification method which should compare two ascii HEX files with eachother (one is local, the other is read from a USB device). Some code:
private void buttonVerify_Click(object sender, EventArgs e)
{
onlyVerifying = true;
Thread t = new Thread(verifyProgram);
}
private void verifyProgram()
{
verifying = true;
externalFlashFile.Clear();
// After this method is finished, the returned data will end up in
// this.externalFlashFile since this listen to the usb's returned data
hexFile.readExternalFlashForVerify(usbDongle, autoEvent);
externalFlashFile.RemoveAt(0);
//externalFlashFile.RemoveAt(externalFlashFile.Count - 1);
hexFile.verifyProgram(externalFlashFile);
}
public void verifyProgram(List<string> externalProgram)
{
byte[] originalFile = null; // Will be modified later with given size
byte[] externalFile = new byte[4096];
int k = 0, errors = 0;
// Remove last line which contains USB command data
externalProgram.RemoveAt(externalProgram.Count - 1);
foreach (String currentLine in externalProgram)
{
for (int i = 0; i < 64; i += 2)
{
string currentDataByte = currentLine.Substring(i, 2);
externalFile[k] = Convert.ToByte(currentDataByte, 16);
k++;
}
progress += steps;
}
//... compare externalFile and originalFile
When executing the readExternalFlashForVerify the USB is responding with requested data. This data is parsed and calls an eventhandler:
public void usbDongle_OnDataParsed(object sender, EventArgs e)
{
if (verifying)
{
usbDongle.receivedBytesString.Trim();
externalFlashFile.Add(usbDongle.receivedBytesString.Substring(2, 32 * 2));
// Allow hexFile continue its thread processing
autoEvent.Set();
}
}
The first run is always completes correctly. The following executions, at the third or fourth iteration of the foreach, I get an extra element in externalProgram. This is not a global variable (argument in function call) and the function is not called anywhere else. This ofcourse throws an exception.
I tried adding .ToList() to externalProgram in the foreach but that didn't do any difference. How can my externalProgram be modified during this execution?
EDIT: I never found the cause of this, but replacing the foreach with a hard-coded for-loop solved the issue at hand. Not an optimal solution, but don't have much time on this.
// The list should never be larger than 128 items
for (int j = 0; j < 0x7f ; j++)
{
string currentLine = externalProgram[j];
// ...
Usually when you receive an exception with a message like that it is caused by multiple accesses from different threads to a list.
What I suggest you is to use a lock when you add and remove items from that list, so you're sure the indexes to that collection are not changing. You have to think what would happen if you try to remove the last element (of index 3, for example) of a collection when someone else removes a previous item (changing the lenght of the collection to 3...).
This example: Properly locking a List<T> in MultiThreaded Scenarios? describes better what I mean.
Probably this line is a problem:
externalProgram.RemoveAt(externalProgram.Count - 1);
If verifyProgram is called multiple times, it will remove more and more lines from externalProgram list passed by reference
Perhaps someone can point me in the correct direction, because I'm completely stumped on this.
I have a function that simply prints out a LinkedList of classes:
LinkedList<Component> components = new LinkedList<Component>();
...
private void PrintComponentList()
{
Console.WriteLine("---Component List: " + components.Count + " entries---");
foreach (Component c in components)
{
Console.WriteLine(c);
}
Console.WriteLine("------");
}
The Component object actually has a custom ToString() call as such:
int Id;
...
public override String ToString()
{
return GetType() + ": " + Id;
}
This function typically works fine - however I've run into the issue that when it builds to about 30 or so entries in the list, the PrintcomplentList foreach statement comes back with an InvalidOperationException: Collection was modified after the enumerator was instantiated.
Now as you can see I'm not modifying the code within the for loop, and I haven't explicitly created any threads, although this is within an XNA environment (if it matters). It should be noted that the printout is frequent enough that the Console output is slowing down the program as a whole.
I'm completely stumped, has anyone else out there run into this?
I suspect the place to start looking will be at any places where you manipulate the list - i.e. insert/remove/re-assign items. My suspicion is that there will be a callback/even-handler somewhere that is getting fired asynchronously (perhaps as part of the XNA paint etc loops), and which is editing the list - essentially causing this problem as a race condition.
To check if this is the case, put some debug/trace output around the places that manipulate the list, and see if it ever (and in particular, just before the exception) runs the manipulation code at the same time as your console output:
private void SomeCallback()
{
Console.WriteLine("---Adding foo"); // temp investigation code; remove
components.AddLast(foo);
Console.WriteLine("---Added foo"); // temp investigation code; remove
}
Unfortunately, such things are often a pain to debug, as changing the code to investigate it often changes the problem (a Heisenbug).
One answer would be to synchronize access; i.e. in all the places that edit the list, use a lock around the complete operation:
LinkedList<Component> components = new LinkedList<Component>();
readonly object syncLock = new object();
...
private void PrintComponentList()
{
lock(syncLock)
{ // take lock before first use (.Count), covering the foreach
Console.WriteLine("---Component List: " + components.Count
+ " entries---");
foreach (Component c in components)
{
Console.WriteLine(c);
}
Console.WriteLine("------");
} // release lock
}
and in your callback (or whatever)
private void SomeCallback()
{
lock(syncLock)
{
components.AddLast(foo);
}
}
In particular, a "complete operation" might include:
check the count and foreach/for
check for existance and insert/remove
etc
(i.e. not the individual/discrete operations - but units of work)
Instead of foreach, I use while( collection.count >0) then use collection[i].
I don't know if this is relevant to the OP but I had the same error and found this thread during a google search. I was able to solve it by adding a break after removing an element in the loop.
foreach( Weapon activeWeapon in activeWeapons ){
if (activeWeapon.position.Z < activeWeapon.range)
{
activeWeapons.Remove(activeWeapon);
break; // Fixes error
}
else
{
activeWeapon.position += activeWeapon.velocity;
}
}
}
If you leave out the break, you will get the error "InvalidOperationException: Collection was modified after the enumerator was instantiated."
Using Break could be a way but it may impact your series of operation.
What I do in that case in simply convert the foreach to traditional for loop
for(i=0; i < List.count; i++)
{
List.Remove();
i--;
}
This works without any issues.