I'm working on a simple irc chat bot (specifically for twitch.tv streams), and I'm using a List to keep a list of all the users in the channel. When someone leaves or joins, I add or remove them from the list. Then I have a thread that runs every minute that checks if the stream is online, and if it is, it hands out "currency" to all the people in my user list.
I'm sure you can already see where my problem is. If someone leaves or joins while my program is looping through the users in my list, then I get a Collection Modified exception. Currently, as a workaround, I just make a temp list and copy the real list into it, then loop through the temp list instead, but I was just curious if there was a "better" way to do it?
Quick psuedocode:
private List<string> users = new List<string>();
private void IrcInitialize(){
//connect to irc stuff
//blah
//blah
//blah
Thread workThread = new Thread(new ThreadStart(doWork());
workThread.Start();
}
private void ircListener(){
parseIRCMessage(StreamReader.ReadLine());
}
private void parseIRCMessage(msg){
if (msgType == "JOIN"){
users.Add(user);
}
else if (msgType == "PART"){
users.Remove(user);
}
}
private void doWork(){
while (true) {
if (streamOnline() && handOutTime()){
handOutCurrency();
}
Thread.Sleep(60000);
}
}
private void handOutCurrency(){
List<string> temp = users; //This is what I'm currently doing
foreach (String user in temp) {
database.AddCurrency(user, 1);
}
}
Any other suggestions?
I suggest using a ConcurrentBag<string> for the users.
This allows multi-threaded access to the users even while it is being enumerated.
The big plus is that you do not have to worry about locking.
There are two ways to solve this problem:
Using a lock to synchronize access between two threads, or
Doing all access from a single thread.
The first way is simple: add lock(users) {...} block around the code that reads or modifies the users list.
The second way is slightly more involved: define two concurrent queues, toAdd and toRemove in your class. Instead of adding or removing users directly from the users list, add them to the toAdd and toRemove queues. When the sleeping thread wakes up, it should first empty both queues, performing the modifications as necessary. Only then it should hand out the currency.
ConcurrentQueue<string> toAdd = new ConcurrentQueue<string>();
ConcurrentQueue<string> toRemove = new ConcurrentQueue<string>();
private void parseIRCMessage(msg){
if (msgType == "JOIN"){
toAdd.Enqueue(user);
}
else if (msgType == "PART"){
toRemove.Enqueue(user);
}
}
private void doWork(){
while (true) {
string user;
while (toAdd.TryDequeue(out user)) {
users.Add(user);
}
while (toRemove.TryDequeue(out user)) {
users.Remove(user);
}
if (streamOnline() && handOutTime()){
handOutCurrency();
}
Thread.Sleep(60000);
}
}
The suggestions from dasblinkenlight's answer are good. Another option is to do something similar to what you suggested: work with a immutable copy of the list. Except with normal List, you would need to make sure that it's not changed while you're copying it (and you would actually need to copy the list, not just a reference to it, like your code suggested).
A better version of this approach would be to use ImmutableList from the immutable collections library. With that, each modification creates a new collection (but sharing most parts with the previous version to improve efficiency). This way, you could have one thread that modifies the list (actually, creates new lists based on the old one) and you could also read the list from another thread at the same time. This will work, because new changes won't be reflected in an old copy of the list.
With that, your code would look something like this:
private ImmutableList<string> users = ImmutableList<string>.Empty;
private void ParseIRCMessage(string msg)
{
if (msgType == "JOIN")
{
users = users.Add(user);
}
else if (msgType == "PART")
{
users = users.Remove(user);
}
}
private void HandOutCurrency()
{
foreach (String user in users)
{
database.AddCurrency(user, 1);
}
}
You need to lock on the list during all reads, writes, and iterations of the list.
private void parseIRCMessage(msg){
lock(users)
{
if (msgType == "JOIN"){
users.Add(user);
}
else if (msgType == "PART"){
users.Remove(user);
}
}
}
private void doWork(){
while (true) {
if (streamOnline() && handOutTime()){
handOutCurrency();
}
Thread.Sleep(60000);
}
}
private void handOutCurrency(){
lock(users)
{
foreach (String user in users) {
database.AddCurrency(user, 1);
}
}
}
etc...
Related
I want to know if the following code is thread safe, which I assume it is not. And how I could possibly make it thread safe?
Basically I have a ConcurrentDictionary which acts as a cache for a database table. I want to query the DB every 10 seconds and update the db cache. There will be other threads querying this dictionary the whole time.
I can't just use TryAdd as there may also me elements which have been removed. So I decided instead of searching through the entire dictionary to possibly update, add or remove. I would just reinitialize the dictionary. Please do tell me if this is a silly idea.
My concern is that when I reinitialize the dictionary the querying threads will not longer by thread safe for the instance when the initialization takes place. For that reason I have used a lock for the dictionary when updating it, However I am not sure if this is correct as the object changes in the lock?
private static System.Timers.Timer updateTimer;
private static volatile Boolean _isBusyUpdating = false;
private static ConcurrentDictionary<int, string> _contactIdNames;
public Constructor()
{
// Setup Timers for data updater
updateTimer = new System.Timers.Timer();
updateTimer.Interval = new TimeSpan(0, 0, 10, 0).TotalMilliseconds;
updateTimer.Elapsed += OnTimedEvent;
// Start the timer
updateTimer.Enabled = true;
}
private void OnTimedEvent(Object source, System.Timers.ElapsedEventArgs e)
{
if (!_isBusyUpdating)
{
_isBusyUpdating = true;
// Get new data values and update the list
try
{
var tmp = new ConcurrentDictionary<int, string>();
using (var db = new DBEntities())
{
foreach (var item in db.ContactIDs.Select(x => new { x.Qualifier, x.AlarmCode, x.Description }).AsEnumerable())
{
int key = (item.Qualifier * 1000) + item.AlarmCode;
tmp.TryAdd(key, item.Description);
}
}
if (_contactIdNames == null)
{
_contactIdNames = tmp;
}
else
{
lock (_contactIdNames)
{
_contactIdNames = tmp;
}
}
}
catch (Exception e)
{
Debug.WriteLine("Error occurred in update ContactId db store", e);
}
_isBusyUpdating = false;
}
}
/// Use the dictionary from another Thread
public int GetIdFromClientString(string Name)
{
try
{
int pk;
if (_contactIdNames.TryGetValue(Name, out pk))
{
return pk;
}
}
catch { }
//If all else fails return -1
return -1;
}
You're right your code is not thread safe.
You need to lock _isBusyUpdating variable.
You need to lock _contactIdNames every time, not only when its not null.
Also this code is similar to singleton pattern and it has the same problem with initialization. You can solve it with Double checked locking. However you also need double checked locking when accessing entries.
In the case when you updating whole dictionary at once you need to lock current value every time when accessing. Otherwise you can access it while it's still changing and get error. So you either need to lock variable each time or use Interlocked.
As MSDN says volatile should do the trick with _isBusyUpdating, it should be thread safe.
If you don't want to keep track of _contactIdNames thread safety try to implement update of each entry on the same dictionary. The problem will be in difference detection between DB and current values (what entries have been removed or added, others can be simple rewritten), but not in thread safety, since ConcurrentDictionary is already thread safe.
You seem to be making a lot of work for yourself. Here's how I would tackle this task:
public class Constructor
{
private volatile Dictionary<int, string> _contactIdNames;
public Constructor()
{
Observable
.Interval(TimeSpan.FromSeconds(10.0))
.StartWith(-1)
.Select(n =>
{
using (var db = new DBEntities())
{
return db.ContactIDs.ToDictionary(
x => x.Qualifier * 1000 + x.AlarmCode,
x => x.Description);
}
})
.Subscribe(x => _contactIdNames = x);
}
public string TryGetValue(int key)
{
string value = null;
_contactIdNames.TryGetValue(key, out value);
return value;
}
}
I'm using Microsoft's Reactive Extensions (Rx) Framework - NuGet "Rx-Main" - for the timer to update the dictionary.
The Rx should be fairly straightforward. If you haven't seen it before in very simple terms it's like LINQ meets events.
If you don't like Rx then just go with your current timer model.
All this code does is create a new dictionary every 10 seconds from the DB. I'm just using a plain dictionary since it is only being created from one thread. Since reference assignment is atomic then you can just re-assign the dictionary when you like with complete thread-safety.
Multiple threads can safely read from a dictionary as long as the elements don't change.
I want to know if the following code is thread safe, which I assume it
is not. And how I could possibly make it thread safe?
I believe it's not. First of all i'd create property for ConcurrentDictionary and check if update is underway inside get method, and if it is, i'd return the previous version of object :
private object obj = new object();
private ConcurrentDictionary<int, string> _contactIdNames;
private ConcurrentDictionary<int, string> _contactIdNamesOld;
private volatile bool _isBusyUpdating = false;
public ConcurrentDictionary<int, string> ContactIdNames
{
get
{
if (!_isBusyUpdating) return _contactIdNames;
return _contactIdNamesOld;
}
private set
{
if(_isBusyUpdating) _contactIdNamesOld =
new ConcurrentDictionary<int, string>(_contactIdNames);
_contactIdNames = value;
}
}
And your method can be :
private static void OnTimedEvent(Object source, System.Timers.ElapsedEventArgs e)
{
if (_isBusyUpdating) return;
lock (obj)
{
_isBusyUpdating = true;
// Get new data values and update the list
try
{
ContactIdNames = new ConcurrentDictionary<int, string>();
using (var db = new DBEntities())
{
foreach (var item in db.ContactIDs.Select(x => new { x.Qualifier, x.AlarmCode, x.Description }).AsEnumerable())
{
int key = (item.Qualifier * 1000) + item.AlarmCode;
_contactIdNames.TryAdd(key, item.Description);
}
}
}
catch (Exception e)
{
Debug.WriteLine("Error occurred in update ContactId db store", e);
_contactIdNames = _contactIdNamesOld;
}
finally
{
_isBusyUpdating = false;
}
}
}
P.S.
My concern is that when I reinitialize the dictionary the querying
threads will not longer by thread safe for the instance when the
initialization takes place. For that reason I have used a lock for the
dictionary when updating it, However I am not sure if this is correct
as the object changes in the lock?
It's ConcurrentDictionary<T> type is threadsafe and not the instance of it, so even if you create new instance and change the reference to it - it's not something to worry about.
I want to create a thread-safe collection that can be modified while being enumerated.
The sample ActionSet class stores Action handlers. It has the Add method that adds a new handler to the list and the Invoke method that enumerates and invokes all of the collected action handlers. The intended working scenarios include very frequent enumerations with occasional modifications while enumerating.
Normal collections throw exception if you modify them using the Add method while the enumeration is not over.
There is an easy, but slow solution to the problem: Just clone the collection before enumeration:
class ThreadSafeSlowActionSet {
List<Action> _actions = new List<Action>();
public void Add(Action action) {
lock(_actions) {
_actions.Add(action);
}
}
public void Invoke() {
lock(_actions) {
List<Action> actionsClone = _actions.ToList();
}
foreach (var action in actionsClone ) {
action();
}
}
}
The problem with this solution is the enumeration overhead and I want enumeration to be very fast.
I've created a rather fast "recursion-safe" collection that allows adding new values even while enumerating. If you add new values while the main _actions collection is being enumerated, the values are added to the temporary _delta collection instead of the main one. After all enumerations are finished, the _delta values are added to the _actions collection. If you add some new values while the main _actions collection is being enumerated (creating the _delta collection) and then re-enter the Invoke method again we have to create a new merged collection (_actions + _delta) and replace _actions with it.
So, this collection looks "recursion-safe", but I want to make it thread-safe. I think that I need to use the Interlocked.* constructs, classes from System.Threading and other synchronization primitives to make this collection thread-safe, but I don't have a good idea on how to do that.
How to make this collection thread-safe?
class RecursionSafeFastActionSet {
List<Action> _actions = new List<Action>(); //The main store
List<Action> _delta; //Temporary buffer for storing added values while the main store is being enumerated
int _lock = 0; //The number of concurrent Invoke enumerations
public void Add(Action action) {
if (_lock == 0) { //_actions list is not being enumerated and can be modified
_actions.Add(action);
} else { //_actions list is being enumerated and cannot be modified
if (_delta == null) {
_delta = new List<Action>();
}
_delta.Add(action); //Storing the new values in the _delta buffer
}
}
public void Invoke() {
if (_delta != null) { //Re-entering Invoke after calling Add: Invoke->Add,Invoke
Debug.Assert(_lock > 0);
var newActions = new List<Action>(_actions); //Creating a new list for merging delta
newActions.AddRange(_delta); //Merging the delta
_delta = null;
_actions = newActions; //Replacing the original list (which is still being iterated)
}
_lock++;
foreach (var action in _actions) {
action();
}
_lock--;
if (_lock == 0 && _delta != null) {
_actions.AddRange(_delta); //Merging the delta
_delta = null;
}
}
}
Update: Added the ThreadSafeSlowActionSet variant.
A simpler approach (used, for example, by ConcurrentBag) is to have GetEnumerator() return an enumerator over a snapshot of the collection's contents. In your case this might look like:
public IEnumerator<Action> GetEnumerator()
{
lock(sync)
{
return _actions.ToList().GetEnumerator();
}
}
If you do this, you don't need a _delta field and the complexity it adds.
Here is your class modified for thread safety:
class SafeActionSet
{
Object _sync = new Object();
List<Action> _actions = new List<Action>(); //The main store
List<Action> _delta = new List<Action>(); //Temporary buffer for storing added values while the main store is being enumerated
int _lock = 0; //The number of concurrent Invoke enumerations
public void Add(Action action)
{
lock(sync)
{
if (0 == _lock)
{ //_actions list is not being enumerated and can be modified
_actions.Add(action);
}
else
{ //_actions list is being enumerated and cannot be modified
_delta.Add(action); //Storing the new values in the _delta buffer
}
}
}
public void Invoke()
{
lock(sync)
{
if (0 < _delta.Count)
{ //Re-entering Invoke after calling Add: Invoke->Add,Invoke
Debug.Assert(0 < _lock);
var newActions = new List<Action>(_actions); //Creating a new list for merging delta
newActions.AddRange(_delta); //Merging the delta
_delta.Clear();
_actions = newActions; //Replacing the original list (which is still being iterated)
}
++_lock;
}
foreach (var action in _actions)
{
action();
}
lock(sync)
{
--_lock;
if ((0 == _lock) && (0 < _delta.Count))
{
_actions.AddRange(_delta); //Merging the delta
_delta.Clear();
}
}
}
}
I made a few other tweaks, for the following reason:
reversed IF expressions to have constant value first, so if I do a
typo and put "=" instead of "==" or "!=" etc., the compiler will
instantly tell me of the typo.
(: a habit I got into because my brain and fingers are often out of sync :)
preallocated _delta, and called .Clear() instead of setting it to null,
because I find it is easier to read.
the various lock(_sync) {...} give you your thread safety on all instance variable access.
:( with the exception of your access to _action in the enumeration itself. ):
Since I actually also needed to delete items from the collection, the implementation that I ultimately used was based on a rewritten LinkedList that locks adjacent nodes on deletion/insertion and doesn't complain if the collection was changed during enumeration.
I also added a Dictionary to make the element search fast.
I have a function like this:
foreach (ListViewItem item in getListViewItems(listView2)) //for proxy
{
if (reader.Peek() == -1)
{
break;
}
lock (reader)
{
line = reader.ReadLine();
}
//proxy code
List<string> mylist = new List<string>();
if (item != null)
{
for (int s = 0; s < 3; s++)
{
if (item.SubItems[s].Text != null)
{
mylist.Add(item.SubItems[s].Text);
}
else
{
mylist.Add("");
}
}
}
else
{
break;
}
//end proxy code
//some other code including the threadpool
}
and the delegate code:
private delegate ListView.ListViewItemCollection GetItems(ListView lstview);
private ListView.ListViewItemCollection getListViewItems(ListView lstview)
{
ListView.ListViewItemCollection temp = new ListView.ListViewItemCollection(new ListView());
if (!lstview.InvokeRequired)
{
foreach (ListViewItem item in lstview.CheckedItems)
{
temp.Add((ListViewItem)item.Clone());
}
return temp;
}
else
{
return (ListView.ListViewItemCollection)this.Invoke(new GetItems(getListViewItems), new object[] { lstview });
}
}
EDIT:
I wanna replace that foreach loop in the main function with a conditional function:
if (reader.Peek() == -1)
{
break;
}
lock (reader)
{
line = reader.ReadLine();
}
if (use_proxy == true)
{
mylist2 = get_current_proxy();
}
//some other code including the threadpool
private List<string> get_current_proxy()
{
//what shall I add here?
}
How can I make that function do the same as foreach loop but using for loop? I mean getting the proxies one by one ...
I see multiple questions revolving around an idea of scraping a website for emails then spamming. You have very cool tools for that already, no need for a new one.
Anyway - I don't understand your question, and it seems that I'm not the only one here, but the thing you'll have to KNOW before anything else is:
Having ANYTHING in Windows run in multiple threads will ultimately have to be synchronized when you do Invoke() which HAVE TO wait until it all passes through ONE thread and that's the one that holds a message loop. So you can try to read from or write to ListView from multiple threads, but to do each read/write you'll have to Invoke() (you probably tried it directly and BAAAAM) and every Invoke() has only ONE hole to go through, and all your threads will have to wait their turn.
Next: having ListView to be a CONTAINER for your data is so BAD I can't even comment any further. Consider something as a
class MyData
{
public string Name;
public string URL;
// ...
}
and
List<MyData> _myData;
to hold your data. You'll be able to access it from multiple threads, if you take care of some low-key sync issues.
Lastly, how come you ask us questions about .net C# programming if you don't even know the syntax. Well, it's rhetorical, ...
I'm making server-client application. Client can manage server files (rename/copy/delete etc.) When I select more than one like (n) files to copy/paste it starts (n) threads of pasteC
I need a way to make them in a queue. How do I do that???
private void Paste()
{
foreach (ListViewItem item in copiedItems)
{
Thread pasteC = new Thread(unused => fmc.PasteFromCopy(item.Text, somePath));
pasteC.Start();
}
}
the code down is in fmc class..
public void PasteFromCopy(string source,string dest)
{
if (IsFolder(source))
{
CopyDirectory(source, dest);
}
else
{
CopyStream(source, dest);
}
}
I think there is no reason to use threads here. You're just making your code more complicated and consume too much memory (each thread needs 1 MB of memory). Do something like this and you won't have to worry about queues or locks:
private void Paste()
{
foreach (ListViewItem item in copiedItems)
{
fmc.PasteFromCopy(item.Text, somePath);
}
}
EDIT:
If you need to use thread other than the main thread, because you don't want to lock that up, you can use one thread that does all the copying. Something like:
private void Paste()
{
var thread = new Thread(() =>
{
foreach (ListViewItem item in copiedItems)
{
fmc.PasteFromCopy(item.Text, somePath);
}
});
thread.Start();
}
Still no need to use locks, and much more memory efficient than your solution with many threads.
i've figured it out
public void PasteFromCopy(string source,string dest)
{
lock(this)
{
if (IsFolder(source))
{
CopyDirectory(source, dest);
}
else
{
CopyStream(source, dest);
}
}
}
depending on the do work method my result could either be a List of Strings or a list of byte[]
How can we check the RunWorkerCompletedEventArgs e -
if (e is List<String>)
is this the correct way to check?
No, this is not the right way.
The correct way is to use this:
if(e.Result is List<string>)
{
//...
}
else if(e.Result is List<byte[]>)
{
//...
}
else
{
//...
}
e will always be of type RunWorkerCompletedEventArgs. But this class contains a property Result that contains the result of your DoWork event handler. That's the one, you need to check.
Yes, that's one possible way to do it.
If you only have two types it would be quite easy:
if(e.Result is List<string>)
{
}
else if(e.Result is List<byte[]>)
{
}
else
{
}
But the problem comes in to play if you have to support more than just two or three. In that case i'm going to create a Dictionary<Type, Action<object>> and write individual functions for each type. Something like this:
var supportedTypes = new Dictionary<Type, Action<object>>();
supportedTypes.Add(typeof(List<string>), ComputeListOfStrings);
supportedTypes.Add(typeof(List<byte[]>), ComputeListOfByteArrays);
private void ComputeListOfString(object listOfStrings)
{
var list = (List<string>)listOfStrings;
}
private void ComputeListOfByteArrays(object listOfByteArrays)
{
var list = (List<byte[]>)listOfByteArrays;
}
This makes it more simple to support new types and also stays to be O(1) while the if-else-if runs into the order-matters problem.
Used will this in your background worker as followed:
worker.OnRunWorkerCompleted += (sender, e) =>
{
Action<object> supportedAction;
supportedTypes.TryGetValue(e.Result.GetType(), out supportedAction);
if(supportedAction != null)
{
supportedAction();
}
};
the e.Result is the property with your results, so to get the type you can do:
if(e.Result.GetType().Equals(typeof(List<String>)))