We Keep Coding

Threads monitoring a Queue<Actions>

I doing a small project to map a network (routers only) using SNMP. In order to speed things up, I´m trying to have a pool of threads responsible for doing the jobs I need, apart from the first job which is done by the main thread.
At this time I have two jobs, one takes a parameter the other doesn´t:
UpdateDeviceInfo(NetworkDevice nd)
UpdateLinks() *not defined yet
What I´m trying to achieve is to have those working threads waiting for a job to
appear on a Queue<Action> and wait while it is empty. The main thread will add the first job and then wait for all workers, which might add more jobs, to finish before starting adding the second job and wake up the sleeping threads.
My problem/questions are:
How to define the Queue<Actions> so that I can insert the methods and the parameters if any. If not possible I could make all functions accept the same parameter.
How to launch the working threads indefinitely. I not sure where should I create the for(;;).
This is my code so far:
public enum DatabaseState
{
Empty = 0,
Learning = 1,
Updating = 2,
Stable = 3,
Exiting = 4
};
public class NetworkDB
{
public Dictionary<string, NetworkDevice> database;
private Queue<Action<NetworkDevice>> jobs;
private string _community;
private string _ipaddress;
private Object _statelock = new Object();
private DatabaseState _state = DatabaseState.Empty;
private readonly int workers = 4;
private Object _threadswaitinglock = new Object();
private int _threadswaiting = 0;
public Dictionary<string, NetworkDevice> Database { get => database; set => database = value; }
public NetworkDB(string community, string ipaddress)
{
_community = community;
_ipaddress = ipaddress;
database = new Dictionary<string, NetworkDevice>();
jobs = new Queue<Action<NetworkDevice>>();
}
public void Start()
{
NetworkDevice nd = SNMP.GetDeviceInfo(new IpAddress(_ipaddress), _community);
if (nd.Status > NetworkDeviceStatus.Unknown)
{
database.Add(nd.Id, nd);
_state = DatabaseState.Learning;
nd.Update(this); // The first job is done by the main thread
for (int i = 0; i < workers; i++)
{
Thread t = new Thread(JobRemove);
t.Start();
}
lock (_statelock)
{
if (_state == DatabaseState.Learning)
{
Monitor.Wait(_statelock);
}
}
lock (_statelock)
{
if (_state == DatabaseState.Updating)
{
Monitor.Wait(_statelock);
}
}
foreach (KeyValuePair<string, NetworkDevice> n in database)
{
using (System.IO.StreamWriter file = new System.IO.StreamWriter(n.Value.Name + ".txt")
{
file.WriteLine(n);
}
}
}
}
public void JobInsert(Action<NetworkDevice> func, NetworkDevice nd)
{
lock (jobs)
{
jobs.Enqueue(item);
if (jobs.Count == 1)
{
// wake up any blocked dequeue
Monitor.Pulse(jobs);
}
}
}
public void JobRemove()
{
Action<NetworkDevice> item;
lock (jobs)
{
while (jobs.Count == 0)
{
lock (_threadswaitinglock)
{
_threadswaiting += 1;
if (_threadswaiting == workers)
Monitor.Pulse(_statelock);
}
Monitor.Wait(jobs);
}
lock (_threadswaitinglock)
{
_threadswaiting -= 1;
}
item = jobs.Dequeue();
item.Invoke();
}
}
public bool NetworkDeviceExists(NetworkDevice nd)
{
try
{
Monitor.Enter(database);
if (database.ContainsKey(nd.Id))
{
return true;
}
else
{
database.Add(nd.Id, nd);
Action<NetworkDevice> action = new Action<NetworkDevice>(UpdateDeviceInfo);
jobs.Enqueue(action);
return false;
}
}
finally
{
Monitor.Exit(database);
}
}
//Job1 - Learning -> Update device info
public void UpdateDeviceInfo(NetworkDevice nd)
{
nd.Update(this);
try
{
Monitor.Enter(database);
nd.Status = NetworkDeviceStatus.Self;
}
finally
{
Monitor.Exit(database);
}
}
//Job2 - Updating -> After Learning, create links between neighbours
private void UpdateLinks()
{
}
}

Your best bet seems like using a BlockingCollection instead of the Queue class. They behave effectively the same in terms of FIFO, but a BlockingCollection will let each of your threads block until an item can be taken by calling GetConsumingEnumerable or Take. Here is a complete example.
http://mikehadlow.blogspot.com/2012/11/using-blockingcollection-to-communicate.html?m=1
As for including the parameters, it seems like you could use closure to enclose the NetworkDevice itself and then just enqueue Action instead of Action<>

Threads conflict in some cases, in loop conditions

I am working on a project that uses Threads. In some cases, I have these problems:
Here is some piece of my code :
List<EmailAddress> lstEmailAddress = new List<EmailAddress>();
private void TimerCheckInternetConnection_Tick(object sender, EventArgs e)
{
lock (TicketLock)
{
if (UtilityManager.CheckForInternetConnection())
{
if (ApplicationRunStatus == Enum_ApplicationRunStatus.UnknownDisconnect || ApplicationRunStatus == Enum_ApplicationRunStatus.IsReady)
{
// Connect
ThreadPool.QueueUserWorkItem((o) =>
{
for (int i = 0; i < lstEmailAddress.Count; i++)
{
lstEmailAddress[i].IsActive = lstEmailAddress[i].Login();
}
this.BeginInvoke(new Action(() =>
{
// some code
}));
});
}
}
}
}
and this is EmailAddress class :
class EmailAddress
{
private Imap4Client imap = new Imap4Client();
private object objectLock = new object();
public bool IsActive;
public string Address;
public string Password;
public string RecieveServerAddress;
public int RecieveServerPort;
public bool Login()
{
lock (objectLock)
{
try
{
imap.ConnectSsl(RecieveServerAddress, RecieveServerPort);
}
catch (Exception)
{
}
try
{
imap.Login(Address, Password);
return true;
}
catch (Exception)
{
return false;
}
}
}
}
And my problem is this:
When I want to use Login procedure that belongs to EmailAddress Class, it has some conflict. As you can see, I used Lock but any thing changed.
For more details:
If I have 3 items in lstEmailAddress , the Login procedure has to be called 3 times by this code. but every time, the login procedure will work on same username and password. So all my emails cannot login correctly.
If I remove threadpool, it will be ok.

Your code is very confusing:
If you add the lock in your code, it will run synchroniously, only one thread at the time, which will lead to performance loss.
If you queue work via QueueUserWorkItem - it will run in other thread, and not inside TicketLock
You should incapsulate locks inside your class, and should not lock entire logic in your program.
You start work for a loop variable i, which is being closured for it's last value, which lead for a problem you state in last sentence.
lock object in Email class isn't static so it's being created for each instance, and doesn't actually lock anithing.
As you are using Invoke method, your code is being started from UI, and you need to pass the synchronization context. I suggest you to use TPL code for this, and do not directly work with ThreadPool
So I suggest you this solution:
List<EmailAddress> lstEmailAddress = new List<EmailAddress>();
private void TimerCheckInternetConnection_Tick(object sender, EventArgs e)
{
// remove this lock as we have another in Email class
//lock (TicketLock)
if (UtilityManager.CheckForInternetConnection())
{
if (ApplicationRunStatus == Enum_ApplicationRunStatus.UnknownDisconnect
|| ApplicationRunStatus == Enum_ApplicationRunStatus.IsReady)
{
for (int i = 0; i < lstEmailAddress.Count; i++)
{
// use local variable to store index
int localIndex = i;
// Connect
ThreadPool.QueueUserWorkItem((o) =>
{
// if you add a lock here, this will run synchroniosly,
// and you aren't really need the ThreadPool
//lock (TicketLock)
lstEmailAddress[localIndex].IsActive = lstEmailAddress[localIndex].Login();
this.BeginInvoke(new Action(() =>
{
// some code
}));
});
}
}
}
}
class EmailAddress
{
// if you have to login only for one user simultaneosly
// use static variables here, other wise simply remove the lock as it is useless
private static Imap4Client imap;
private static object objectLock;
// static constructor for only one initialization for a static fields
static EmailAddress()
{
objectLock = new object();
imap = new Imap4Client();
}
public bool IsActive;
public string Address;
public string Password;
public string RecieveServerAddress;
public int RecieveServerPort;
public bool Login()
{
// aquire a static lock
lock (objectLock)
{
try
{
imap.ConnectSsl(RecieveServerAddress, RecieveServerPort);
}
catch (Exception)
{
// STORE THE EXCEPTION!!!
// return as you haven't connected
return false;
}
try
{
imap.Login(Address, Password);
return true;
}
catch (Exception)
{
// STORE THE EXCEPTION!!!
return false;
}
}
}
}

Change your Code as and try . you code is queing item from lstEmailAddress where it will always go and hit last item from the list. change your code to inquie each item in threadpool. that should fix. it.
for (int i = 0; i < lstEmailAddress.Count; i++)
{
ThreadPool.QueueUserWorkItem((o) =>
{
lstEmailAddress[i].IsActive = lstEmailAddress[i].Login();
}
}

How to monitor/wait on array of objects?

I am using concurrent bag to store a set of objects. I want to implement something like
if(an object is present)
return it
else wait until one get free, if it does not get free in a specific time throw an exception.
if(object has been returned)
add to bag
I was thinking to use monitors but monitor can wait on a specific object. I want to wait till any of them is free. How can I implement it?

Extending the msdn example found here:
public class FiniteObjectPool<T>: IDisposable
{
System.Threading.AutoResetEvent m_Wait = new System.Threading.AutoResetEvent(false);
private ConcurrentBag<T> _objects;
public FiniteObjectPool()
{
_objects = new ConcurrentBag<T>();
}
public T GetObject()
{
T item;
while(!_objects.TryTake(out item))
{
m_Wait.WaitOne(); //an object was not available, wait until one is
}
return item;
}
public void PutObject(T item)
{
_objects.Add(item);
m_Wait.Set(); //signal a waiting thread that object may now be available
}
public void Dispose()
{
m_Wait.Dispose();
}
}
EDIT - example usage with 'Context' idiom wrapper
class Program
{
public class FiniteObjectPoolContext<T>: IDisposable
{
FiniteObjectPool<T> m_Pool = new FiniteObjectPool<T>();
public T Value { get; set; }
public FiniteObjectPoolContext(FiniteObjectPool<T> pool)
{
m_Pool = pool;
Value = pool.GetObject(); //take an object out - this will block if none is available
}
public void Dispose()
{
m_Pool.PutObject(Value); //put the object back because this context is finished
}
}
static void Main(string[] args)
{
FiniteObjectPool<int> pool = new FiniteObjectPool<int>();
for (int i = 0; i < 10; i++)
{
pool.PutObject(i);
}
List<Task> tasks = new List<Task>();
for (int i = 0; i < 20; i++)
{
int id = i;
tasks.Add(Task.Run(() =>
{
Console.WriteLine("Running task " + id);
using (var con = new FiniteObjectPoolContext<int>(pool))
{
Console.WriteLine("Task " + id + " got object from pool: " + con.Value);
System.Threading.Thread.Sleep(5000);
Console.WriteLine("Task " + id + " is finished with pool object: " + con.Value);
}
}));
}
Task.WaitAll(tasks.ToArray());
Console.WriteLine("DONE");
Console.ReadLine();
}
}
Notice the latency injected by the thread synchronization mechanisms.

Try Semaphores for the operations that you want to do. .NET has two implementations for Semaphores. Semaphore and SemaphoreSlim, both can be used to implement many threads trying to access a pool of resources.

working with threads - add to collection

List<int> data=new List<int>();
foreach(int id in ids){
var myThread=new Thread(new ThreadStart(Work));
myThread.Start(id);
}
Work(){
}
Method Work does some processing on the received id and then adds the result to the data list? How can I add data to the collection from each thread? How would my code look like? thanks

If you're using .NET 4, I strongly suggest you use Parallel Extensions instead. For example:
var list = ids.AsParallel()
.Select(Work)
.ToList();
where Work is:
public int Work(int id)
{
...
}
so that it can receive the id appropriately. If you're not keen on the method conversion, you could add a lambda expression:
var list = ids.AsParallel()
.Select(id => Work(id))
.ToList();
Either way, this will avoid creating more threads than you really need, and deal with the thread safety side of things without you having to manage the locks yourself.

First of all, you need to protect your multithreaded access with a lock. Second, you need to pass the parameter to your thread (or use lambda which can capture the local variable; beware that if you capture loop variable, it will change the value during the loop, so you ought to have a local copy).
object collectionLock = new object();
List<int> data = new List<int>();
foreach (int id in ids)
{
Thread t = new Thread(Worker);
t.Start(id);
}
void Worker(object o)
{
int id = (int)o;
lock(collectionLock)
{
data.Add(id);
}
}

you can pass and retrieve data (using callbacks) from threads. See MSDN article.
Example:
public class SomeClass
{
public static List<int> data = new List<int>();
public static readonly object obj = new object();
public void SomeMethod(int[] ids)
{
foreach (int id in ids)
{
Work w = new Work();
w.Data = id;
w.callback = ResultCallback;
var myThread = new Thread(new ThreadStart(w.DoWork));
myThread.Start();
}
}
public static void ResultCallback(int d)
{
lock (obj)
{
data.Add(d);
}
}
}
public delegate void ExampleCallback(int data);
class Work
{
public int Data { get; set; }
public ExampleCallback callback;
public void DoWork()
{
Console.WriteLine("Instance thread procedure. Data={0}", Data);
if (callback != null)
callback(Data);
}
}

Get list of object's from collection being updated by thread

I have a Dictionary of items that a thread is updating. I want to have a method get the updated list of items using another thread.
Like so:
internal List<string> GetListOfEntities()
{
List<string> listOfEntities = new List<string>();
foreach (string entityName in ModelFacade._totalListOfStkObjects.Keys)
{
listOfEntities.Add(entityName);
}
return listOfEntities;
}
ModelFacade._totalListOfStkObjects is the collection being updated by the thread. I keep getting the exception: "Collection was modified; enumeration operation may not execute."; I have tried copying _totalListOfStkObjects to a local collection and iterating over that in GetListOfEntities().. but I get the same error..?
Any help ?
WulfgarPro

There isn't going to be a guaranteed thread-safe way to access the dictionary. Your best bet is to either change your code so that you're not sharing the collection or to to lock the dictionary when accessing:
object dictLock = new object();
internal List<string> GetListOfEntities()
{
lock (dictLock)
{
return ModelFacade._totalListOfStkObjects.Keys.ToList();
}
}
Make sure you also lock the dictionary when modifying it in another thread.

Change your Dictionary to ConcurrentDictionary if you are using .NET 4. Here is an easy example to simulate your question and resolve it.
class DataItem
{
public int Data { get; set; }
public bool IsDirty { get; set; }
}
var data = new ConcurrentDictionary<string, DataItem>();
Thread addingItems = new Thread(() =>
{
for (int i = 0; i < 10000; i++)
{
data.TryAdd("data " + i, new DataItem { Data = i, IsDirty = true });
Thread.Sleep(100);
}
});
Thread fetchingItems = new Thread(() =>
{
int count = 0;
while (count < 100)
{
foreach (var item in data)
{
if (item.Value.IsDirty)
{
Console.WriteLine(item.Key + " " + item.Value);
item.Value.IsDirty = false;
count++;
}
}
}
});
addingItems.Start();
fetchingItems.Start();

You can wrap the dictionary up in a thread-safe singleton class. This should provide all of the functionality of ConcurrentDictionary to Dictionary. Referencing the Dictionary should only require one additional layer of indirection.
Reference:
Singleton.Instance.myDictionary.Add(1, "Hello World");
Declaration:
public sealed class Singleton
{
private static volatile Singleton instance;
private static object syncRoot = new Object();
public Dictionary<int, string> myDictionary = new Dictionary<int, string>();
private Singleton() {}
public static Singleton Instance
{
get
{
if (instance == null)
{
lock (syncRoot)
{
if (instance == null)
instance = new Singleton();
}
}
return instance;
}
}
}
Look here for more information on the Singleton Pattern in C#. Note that there is only one difference between the pattern on this link and my example code.