We Keep Coding

Threads monitoring a Queue<Actions>

I doing a small project to map a network (routers only) using SNMP. In order to speed things up, I´m trying to have a pool of threads responsible for doing the jobs I need, apart from the first job which is done by the main thread.
At this time I have two jobs, one takes a parameter the other doesn´t:
UpdateDeviceInfo(NetworkDevice nd)
UpdateLinks() *not defined yet
What I´m trying to achieve is to have those working threads waiting for a job to
appear on a Queue<Action> and wait while it is empty. The main thread will add the first job and then wait for all workers, which might add more jobs, to finish before starting adding the second job and wake up the sleeping threads.
My problem/questions are:
How to define the Queue<Actions> so that I can insert the methods and the parameters if any. If not possible I could make all functions accept the same parameter.
How to launch the working threads indefinitely. I not sure where should I create the for(;;).
This is my code so far:
public enum DatabaseState
{
Empty = 0,
Learning = 1,
Updating = 2,
Stable = 3,
Exiting = 4
};
public class NetworkDB
{
public Dictionary<string, NetworkDevice> database;
private Queue<Action<NetworkDevice>> jobs;
private string _community;
private string _ipaddress;
private Object _statelock = new Object();
private DatabaseState _state = DatabaseState.Empty;
private readonly int workers = 4;
private Object _threadswaitinglock = new Object();
private int _threadswaiting = 0;
public Dictionary<string, NetworkDevice> Database { get => database; set => database = value; }
public NetworkDB(string community, string ipaddress)
{
_community = community;
_ipaddress = ipaddress;
database = new Dictionary<string, NetworkDevice>();
jobs = new Queue<Action<NetworkDevice>>();
}
public void Start()
{
NetworkDevice nd = SNMP.GetDeviceInfo(new IpAddress(_ipaddress), _community);
if (nd.Status > NetworkDeviceStatus.Unknown)
{
database.Add(nd.Id, nd);
_state = DatabaseState.Learning;
nd.Update(this); // The first job is done by the main thread
for (int i = 0; i < workers; i++)
{
Thread t = new Thread(JobRemove);
t.Start();
}
lock (_statelock)
{
if (_state == DatabaseState.Learning)
{
Monitor.Wait(_statelock);
}
}
lock (_statelock)
{
if (_state == DatabaseState.Updating)
{
Monitor.Wait(_statelock);
}
}
foreach (KeyValuePair<string, NetworkDevice> n in database)
{
using (System.IO.StreamWriter file = new System.IO.StreamWriter(n.Value.Name + ".txt")
{
file.WriteLine(n);
}
}
}
}
public void JobInsert(Action<NetworkDevice> func, NetworkDevice nd)
{
lock (jobs)
{
jobs.Enqueue(item);
if (jobs.Count == 1)
{
// wake up any blocked dequeue
Monitor.Pulse(jobs);
}
}
}
public void JobRemove()
{
Action<NetworkDevice> item;
lock (jobs)
{
while (jobs.Count == 0)
{
lock (_threadswaitinglock)
{
_threadswaiting += 1;
if (_threadswaiting == workers)
Monitor.Pulse(_statelock);
}
Monitor.Wait(jobs);
}
lock (_threadswaitinglock)
{
_threadswaiting -= 1;
}
item = jobs.Dequeue();
item.Invoke();
}
}
public bool NetworkDeviceExists(NetworkDevice nd)
{
try
{
Monitor.Enter(database);
if (database.ContainsKey(nd.Id))
{
return true;
}
else
{
database.Add(nd.Id, nd);
Action<NetworkDevice> action = new Action<NetworkDevice>(UpdateDeviceInfo);
jobs.Enqueue(action);
return false;
}
}
finally
{
Monitor.Exit(database);
}
}
//Job1 - Learning -> Update device info
public void UpdateDeviceInfo(NetworkDevice nd)
{
nd.Update(this);
try
{
Monitor.Enter(database);
nd.Status = NetworkDeviceStatus.Self;
}
finally
{
Monitor.Exit(database);
}
}
//Job2 - Updating -> After Learning, create links between neighbours
private void UpdateLinks()
{
}
}

Your best bet seems like using a BlockingCollection instead of the Queue class. They behave effectively the same in terms of FIFO, but a BlockingCollection will let each of your threads block until an item can be taken by calling GetConsumingEnumerable or Take. Here is a complete example.
http://mikehadlow.blogspot.com/2012/11/using-blockingcollection-to-communicate.html?m=1
As for including the parameters, it seems like you could use closure to enclose the NetworkDevice itself and then just enqueue Action instead of Action<>

Refactoring entity method to avoid concurrency problems

I'm nearing the end of a project for which I'm trying to use DDD, but have discovered a glaring bug that I'm not sure how to easily solve.
Here is my entity - I've reduced it for simplicity:
public class Contribution : Entity
{
protected Contribution()
{
this.Parts = new List<ContributionPart>();
}
internal Contribution(Guid id)
{
this.Id = id;
this.Parts = new List<ContributionPart>();
}
public Guid Id { get; private set; }
protected virtual IList<ContributionPart> Parts { get; private set; }
public void UploadParts(string path, IEnumerable<long> partLengths)
{
if (this.Parts.Count > 0)
{
throw new InvalidOperationException("Parts have already been uploaded.");
}
long startPosition = 0;
int partNumber = 1;
foreach (long partLength in partLengths)
{
this.Parts.Add(new ContributionPart(this.Id, partNumber, partLength));
this.Commands.Add(new UploadContributionPartCommand(this.Id, partNumber, path, startPosition, partLength));
startPosition += partLength;
partNumber++;
}
}
public void SetUploadResult(int partNumber, string etag)
{
if (etag == null)
{
throw new ArgumentNullException(nameof(etag));
}
ContributionPart part = this.Parts.SingleOrDefault(p => p.PartNumber == partNumber);
if (part == null)
{
throw new ContributionPartNotFoundException(this.Id, partNumber);
}
part.SetUploadResult(etag);
if (this.Parts.All(p => p.IsUploaded))
{
IEnumerable<PartUploadedResult> results = this.Parts.Select(p => new PartUploadedResult(p.PartNumber, p.ETag));
this.Events.Add(new ContributionUploaded(this.Id, results));
}
}
}
My bug occurs in the SetUploadResult method. Basically, multiple threads are performing uploads concurrently, and then call SetUploadResult at the end of the upload. But because the entity was loaded a few seconds beforehand, each thread will be calling SetUploadResult on a different instance of the entity, and so the test if (this.Parts.All(p => p.IsUploaded) will never evaluate to true.
I'm not sure how to easily resolve this. The idea behind adding multiple UploadContributionPartCommands to the Commands collection was so that each ContributionPart could be uploaded in parallel - my CommandBus ensures this - but with each part uploaded in parallel, it causes problems for my entity logic.

I think you can refactor the Contribution so that it will not handle the SetUploadResult. It will decouple the Contribution entity and the side effects of the SetUploadResult are isolated, keeping the technical concern out of the Contribution domain model.
Create a dispatcher class that contains what the SetUploadResult is doing.
Once the Contribution entity is finished carrying out its logic, the thread of execution will return to the application service. It is at this point that the events from the entity can be fed into the dispatcher.
If they are long running process, you can add them as collection of tasks and run them asynchronously. Then you can just await when all tasks are done. You can search in SO on how to do this.
var results = await Task.WhenAll(task1, task2,...taskN);

If several threads may call the SetUploadResult method simultaneously and you have a race condition your should protect the critical section using a synchronization mechanism such as a lock: https://msdn.microsoft.com/en-us/library/c5kehkcz.aspx.
If you make the lock field static it will be shared across all instances of your entity type, e.g.:
private static readonly object _lock = new object();
public void SetUploadResult(int partNumber, string etag)
{
if (etag == null)
{
throw new ArgumentNullException(nameof(etag));
}
ContributionPart part = this.Parts.SingleOrDefault(p => p.PartNumber == partNumber);
if (part == null)
{
throw new ContributionPartNotFoundException(this.Id, partNumber);
}
part.SetUploadResult(etag);
lock (_lock) //Only one thread at a time can enter this critical section.
//The second thread will wait here until the first thread leaves the critical section.
{
if (this.Parts.All(p => p.IsUploaded))
{
IEnumerable<PartUploadedResult> results = this.Parts.Select(p => new PartUploadedResult(p.PartNumber, p.ETag));
this.Events.Add(new ContributionUploaded(this.Id, results));
}
}
}

How to ensure thread safe ASP.net page to access static list of objects

In my web application i am having following common objectList for all online users.
public static List<MyClass> myObjectList = new List<MyClass>();
so when multiple online users try to read data from this object myObjectList then are there any chances of thread synchronization issue.
In another scenario multiple users are reading from myObjectList and few of them are writing also but every user is writing on a different index of List . Every user may add a new item to this list . So now I think there are chances of synchronization issue.
How to write thread safe utility class that can read and write data from this object in safer way.
Suggestions are highly welcome
Code suggested by Angelo looks like this
using System;
using System.Collections.Concurrent;
using System.Threading;
using System.Threading.Tasks;
namespace ObjectPoolExample
{
public class ObjectPool<T>
{
private ConcurrentBag<T> _objects;
private Func<T> _objectGenerator;
public ObjectPool(Func<T> objectGenerator)
{
if (objectGenerator == null) throw new ArgumentNullException("objectGenerator");
_objects = new ConcurrentBag<T>();
_objectGenerator = objectGenerator;
}
public T GetObject()
{
T item;
if (_objects.TryTake(out item)) return item;
return _objectGenerator();
}
public void PutObject(T item)
{
_objects.Add(item);
}
}
class Program
{
static void Main(string[] args)
{
CancellationTokenSource cts = new CancellationTokenSource();
// Create an opportunity for the user to cancel.
Task.Factory.StartNew(() =>
{
if (Console.ReadKey().KeyChar == 'c' || Console.ReadKey().KeyChar == 'C')
cts.Cancel();
});
ObjectPool<MyClass> pool = new ObjectPool<MyClass> (() => new MyClass());
// Create a high demand for MyClass objects.
Parallel.For(0, 1000000, (i, loopState) =>
{
MyClass mc = pool.GetObject();
Console.CursorLeft = 0;
// This is the bottleneck in our application. All threads in this loop
// must serialize their access to the static Console class.
Console.WriteLine("{0:####.####}", mc.GetValue(i));
pool.PutObject(mc);
if (cts.Token.IsCancellationRequested)
loopState.Stop();
});
Console.WriteLine("Press the Enter key to exit.");
Console.ReadLine();
}
}
// A toy class that requires some resources to create.
// You can experiment here to measure the performance of the
// object pool vs. ordinary instantiation.
class MyClass
{
public int[] Nums {get; set;}
public double GetValue(long i)
{
return Math.Sqrt(Nums[i]);
}
public MyClass()
{
Nums = new int[1000000];
Random rand = new Random();
for (int i = 0; i < Nums.Length; i++)
Nums[i] = rand.Next();
}
}
}
I think i can go with this approach.

If you are using .NET 4.0 you are better off changing to one of the thread-safe collections already supported by the runtime, like for example a ConcurrentBag.
The concurrent bag however does not support access by index if I recall correctly so you may need to resort to a ConcurrentDictionary if you need access to an object by a given key.
If .NET 4.0 is not an option you should read the following blog post:
Why are thread safe collections so hard?

working with threads - add to collection

List<int> data=new List<int>();
foreach(int id in ids){
var myThread=new Thread(new ThreadStart(Work));
myThread.Start(id);
}
Work(){
}
Method Work does some processing on the received id and then adds the result to the data list? How can I add data to the collection from each thread? How would my code look like? thanks

If you're using .NET 4, I strongly suggest you use Parallel Extensions instead. For example:
var list = ids.AsParallel()
.Select(Work)
.ToList();
where Work is:
public int Work(int id)
{
...
}
so that it can receive the id appropriately. If you're not keen on the method conversion, you could add a lambda expression:
var list = ids.AsParallel()
.Select(id => Work(id))
.ToList();
Either way, this will avoid creating more threads than you really need, and deal with the thread safety side of things without you having to manage the locks yourself.

First of all, you need to protect your multithreaded access with a lock. Second, you need to pass the parameter to your thread (or use lambda which can capture the local variable; beware that if you capture loop variable, it will change the value during the loop, so you ought to have a local copy).
object collectionLock = new object();
List<int> data = new List<int>();
foreach (int id in ids)
{
Thread t = new Thread(Worker);
t.Start(id);
}
void Worker(object o)
{
int id = (int)o;
lock(collectionLock)
{
data.Add(id);
}
}

you can pass and retrieve data (using callbacks) from threads. See MSDN article.
Example:
public class SomeClass
{
public static List<int> data = new List<int>();
public static readonly object obj = new object();
public void SomeMethod(int[] ids)
{
foreach (int id in ids)
{
Work w = new Work();
w.Data = id;
w.callback = ResultCallback;
var myThread = new Thread(new ThreadStart(w.DoWork));
myThread.Start();
}
}
public static void ResultCallback(int d)
{
lock (obj)
{
data.Add(d);
}
}
}
public delegate void ExampleCallback(int data);
class Work
{
public int Data { get; set; }
public ExampleCallback callback;
public void DoWork()
{
Console.WriteLine("Instance thread procedure. Data={0}", Data);
if (callback != null)
callback(Data);
}
}

Get list of object's from collection being updated by thread

I have a Dictionary of items that a thread is updating. I want to have a method get the updated list of items using another thread.
Like so:
internal List<string> GetListOfEntities()
{
List<string> listOfEntities = new List<string>();
foreach (string entityName in ModelFacade._totalListOfStkObjects.Keys)
{
listOfEntities.Add(entityName);
}
return listOfEntities;
}
ModelFacade._totalListOfStkObjects is the collection being updated by the thread. I keep getting the exception: "Collection was modified; enumeration operation may not execute."; I have tried copying _totalListOfStkObjects to a local collection and iterating over that in GetListOfEntities().. but I get the same error..?
Any help ?
WulfgarPro

There isn't going to be a guaranteed thread-safe way to access the dictionary. Your best bet is to either change your code so that you're not sharing the collection or to to lock the dictionary when accessing:
object dictLock = new object();
internal List<string> GetListOfEntities()
{
lock (dictLock)
{
return ModelFacade._totalListOfStkObjects.Keys.ToList();
}
}
Make sure you also lock the dictionary when modifying it in another thread.

Change your Dictionary to ConcurrentDictionary if you are using .NET 4. Here is an easy example to simulate your question and resolve it.
class DataItem
{
public int Data { get; set; }
public bool IsDirty { get; set; }
}
var data = new ConcurrentDictionary<string, DataItem>();
Thread addingItems = new Thread(() =>
{
for (int i = 0; i < 10000; i++)
{
data.TryAdd("data " + i, new DataItem { Data = i, IsDirty = true });
Thread.Sleep(100);
}
});
Thread fetchingItems = new Thread(() =>
{
int count = 0;
while (count < 100)
{
foreach (var item in data)
{
if (item.Value.IsDirty)
{
Console.WriteLine(item.Key + " " + item.Value);
item.Value.IsDirty = false;
count++;
}
}
}
});
addingItems.Start();
fetchingItems.Start();

You can wrap the dictionary up in a thread-safe singleton class. This should provide all of the functionality of ConcurrentDictionary to Dictionary. Referencing the Dictionary should only require one additional layer of indirection.
Reference:
Singleton.Instance.myDictionary.Add(1, "Hello World");
Declaration:
public sealed class Singleton
{
private static volatile Singleton instance;
private static object syncRoot = new Object();
public Dictionary<int, string> myDictionary = new Dictionary<int, string>();
private Singleton() {}
public static Singleton Instance
{
get
{
if (instance == null)
{
lock (syncRoot)
{
if (instance == null)
instance = new Singleton();
}
}
return instance;
}
}
}
Look here for more information on the Singleton Pattern in C#. Note that there is only one difference between the pattern on this link and my example code.