We Keep Coding

Two threads one core

I'm playing around with a simple console app that creates one thread and I do some inter thread communication between the main and the worker thread.
I'm posting objects from the main thread to a concurrent queue and the worker thread is dequeueing that and does some processing.
What strikes me as odd, is that when I profile this app, even despite I have two cores.
One core is 100% free and the other core have done all the work, and I see that both threads have been running in that core.
Why is this?
Is it because I use a wait handle that sets when I post a message and releases when the processing is done?
This is my sample code, now using 2 worker threads.
It still behaves the same, main, worker1 and worker2 is running in the same core.
Ideas?
[EDIT]
It sort of works now, atleast, I get twice the performance compared to yesterday.
the trick was to slow down the consumer just enough to avoid signaling using the AutoResetEvent.
public class SingleThreadDispatcher
{
public long Count;
private readonly ConcurrentQueue<Action> _queue = new ConcurrentQueue<Action>();
private volatile bool _hasMoreTasks;
private volatile bool _running = true;
private int _status;
private readonly AutoResetEvent _signal = new AutoResetEvent(false);
public SingleThreadDispatcher()
{
var thread = new Thread(Run)
{
IsBackground = true,
Name = "worker" + Guid.NewGuid(),
};
thread.Start();
}
private void Run()
{
while (_running)
{
_signal.WaitOne();
do
{
_hasMoreTasks = false;
Action task;
while (_queue.TryDequeue(out task) && _running)
{
Count ++;
task();
}
//wait a short while to let _hasMoreTasks to maybe be set to true
//this avoids the roundtrip to the AutoResetEvent
//that is, if there is intense pressure on the pool, we let some new
//tasks have the chance to arrive and be processed w/o signaling
if(!_hasMoreTasks)
Thread.Sleep(5);
Interlocked.Exchange(ref _status, 0);
} while (_hasMoreTasks);
}
}
public void Schedule(Action task)
{
_hasMoreTasks = true;
_queue.Enqueue(task);
SetSignal();
}
private void SetSignal()
{
if (Interlocked.Exchange(ref _status, 1) == 0)
{
_signal.Set();
}
}
}

Is it because I use a wait handle that sets when I post a message and releases when the processing is done?
Without seeing your code it is hard to say for sure, but from your description it appears that the two threads that you wrote act as co-routines: when the main thread is running, the worker thread has nothing to do, and vice versa. It looks like .NET scheduler is smart enough to not load the second core when this happens.
You can change this behavior in several ways - for example
by doing some work on the main thread before waiting on the handle, or
by adding more worker threads that would compete for the tasks that your main thread posts, and could both get a task to work on.

OK, I've figured out what the problem is.
The producer and consumer is pretty much just as fast in this case.
This results in the consumer finishing all its work fast and then looping back to wait for the AutoResetEvent.
The next time the producer sends a task, it has to touch the AutoresetEvent and set it.
The solution was to add a very very small delay in the consumer, making it slightly slower than the producer.
This results in when the producer sends a task, it notices that the consumer is already active and it just has to post to the worker queue w/o touching the AutoResetEvent.
The original behavior resulted in a sort of ping-pong effect, that can be seen on the screenshot.

Dasblinkelight (probably) has the right answer.
Apart from that, it would also be the correct behaviour when one of your threads is I/O bound (that is, it's not stuck on the CPU) - in that case, you've got nothing to gain from using multiple cores, and .NET is smart enough to just change contexts on one core.
This is often the case for UI threads - it has very little work to do, so there usually isn't much of a reason for it to occupy a whole core for itself. And yes, if your concurrent queue is not used properly, it could simply mean that the main thread waits for the worker thread - again, in that case, there's no need to switch cores, since the original thread is waiting anyway.

You should use BlockingCollection rather than ConcurrentQueue. By default, BlockingCollection uses a ConcurrentQueue under the hood, but it has a much easier to use interface. In particular, it does non-busy waits. In addition, BlockingCollection supports cancellation, so your consumer becomes very simple. Here's an example:
public class SingleThreadDispatcher
{
public long Count;
private readonly BlockingCollection<Action> _queue = new BlockingCollection<Action>();
private readonly CancellationTokenSource _cancellation = new CancellationTokenSource();
public SingleThreadDispatcher()
{
var thread = new Thread(Run)
{
IsBackground = true,
Name = "worker" + Guid.NewGuid(),
};
thread.Start();
}
private void Run()
{
foreach (var task in _queue.GetConsumingEnumerable(_cancellation.Token))
{
Count++;
task();
}
}
public void Schedule(Action task)
{
_queue.Add(task);
}
}
The loop with GetConsumingEnumerable will do a non-busy wait on the queue. There's no need to do it with a separate event. It will wait for an item to be added to the queue, or it will exit if you set the cancellation token.
To stop it normally, you just call _queue.CompleteAdding(). That tells the consumer that no more items will be added to the queue. The consumer will empty the queue and then exit.
If you want to quit early, then just call _cancellation.Cancel(). That will cause GetConsumingEnumerable to exit.
In general, you shouldn't ever have to use ConcurrentQueue directly. BlockingCollection is easier to use and provides equivalent performance.

Section of code with up to N threads executing in FIFO order

I have a section of code which should be executed by a maximum number of threads lower than N and also the order in which threads are calling someFunction() should be reflected in the order in which they are entering the section, that is FIFO order.
If I use the Semaphore I have no control over the order in which threads are entering the section.
"There is no guaranteed order, such as FIFO or LIFO, in which blocked
threads enter the semaphore."
The initial attempt:
class someClass
{
static volatile Semaphore semaphore;
...
someClass()
{
semaphore = new Semaphore(N,N)
}
someType someFunction(InputType input)
{
try
{
semaphore.WaitOne();
/* Section Begins */
var response = someHeavyJob(input); // submitted to the server
return response;
/* Section Ends */
}
finally
{
semaphore.Release();
}
}
}
If I combine a Semaphore and a ConcurrentQueue as follows thread may come back with a response to the request brought by other thread what would require significant changes in other parts of code.
What is the .NET 4.5 solution for the following problem:
Allow for maximum number of threads lower than N in the section of code
The order in which threads are entering the section is FIFO
Threads will get the response for the request they brought (and not the response to the request brought by other threads)
class someClass
{
static volatile ConcurrentQueue<someType> cqueue;
static volatile Semaphore semaphore;
...
someClass()
{
cqueue = new ConcurrentQueue<someType>();
semaphore = new Semaphore(N,N)
}
someType someFunction(Request request)
{
try
{
cqueue.enqueue(request);
semaphore.WaitOne();
Request newrequest;
cqueue.TryDequeue(out newrequest);
/* Section Begins */
var response = someHeavyJob(Request newrequest); // submitted to the server
return response;
/* Section Ends */
}
finally
{
semaphore.Release();
}
}
}
UPDATE:
I am clarifying my question:
SomeHeavyJobs() funciton is a blocking call to the server on which this job is being processed.
UPDATE2:
Thank you all for answers. For the record: I ended up using the FIFO Semaphore

'If I combine a Semaphore and a ConcurrentQueue as follows thread may come back with a response to the request brought by other thread what would require significant changes in other parts of code.'
I hate to say it, but I would suggest 'changes in other parts of code', even though I don't know how much 'significance' this would have.
Typicaly, such a requirement is met as you suggested, by queueing messages that contain a reference to the originating class instance so that responses can be 'returned' to the object that requested them. If the originators are all descended from some 'messagehandler' class, that makes it easier on the thread that will call the function, (which should be a member of messagehandler). Once the thread/s have performed the function, they can call a 'onCompletion' method of the messagehandler. 'onCompletion' could either signal an event that the originator is waiting on, (synchronous), or queue something to a private P-C queue of the originator, (asynchronous).
So, a BlockingCollection, one consumer thread and judicious use of C++/C# inheritance/polymorphism should do the job.
Strangely, this is almost exactly what I am being forced into with my current embedded ARM project. The command-line interface thread used for config/debug/log is now so large that it needs a massive 600 words of stack, even in 'Thumb, Optimize of size' mode. It can no longer be permitted to call the SD filesystem directly and must now queue itself to the thread that runs the SD card, (which has the largest stack in the system to run FAT32), and wait on a semaphore for the SD thread to call its methods and signal the semaphore when done.
This is the classic way of ensuring that the calls are made sequentially and will cetainly work. It's basicaly a threadpool with only one thread.
Like the other posters have written, any other approach is likely to be, err.. 'brave'.

also the order in which threads are calling someFunction() should be
reflected in the order in which they are entering the section, that is
FIFO order
This is not possible by principle.
semaphore.WaitOne(); //#1
var response = someHeavyJob(input); //#2
Even is Semaphore was strictly FIFO, the following could happen:
All threads enter the section in FIFO order (1)
All threads get descheduled from the CPU (between 1 and 2)
All threads get rescheduled in random order or even in LIFO order (between 1 and 2)
All thread start entering someHeavyJob in arbitrary order (2)
You can never ensure that the threads will "enter" the function in a specific order.
As for a FIFO semaphore, you can build a Semaphore yourself using a lock and a Queue. Looks like your already did that and posted the code. This approach is correct as far as I can tell.

Have you looked at Smart Thread Pool?
[Edit]
If I am still getting the problem right, as I've stated in the comments, I don't believe that a multithreaded solution is feasible for this problem.
If a task k cannot be started before task k-1 has finished, then you only need a single thread to execute them. If you are allowed to execute some combinations of tasks in parallel, then you need to specify the rules exactly.

Can I optimise this concurrency better?

I've recently begun my first multi-threading code, and I'd appreciate some comments.
It delivers video samples from a buffer that is filled in the background by a stream parser (outside the scope of this question). If the buffer is empty, it needs to wait until the buffer level becomes acceptable and then continue.
Code is for Silverlight 4, some error-checking removed:
// External class requests samples - can happen multiple times concurrently
protected override void GetSampleAsync()
{
Interlocked.Add(ref getVideoSampleRequestsOutstanding, 1);
}
// Runs on a background thread
void DoVideoPumping()
{
do
{
if (getVideoSampleRequestsOutstanding > 0)
{
PumpNextVideoSample();
// Decrement the counter
Interlocked.Add(ref getVideoSampleRequestsOutstanding, -1);
}
else Thread.Sleep(0);
} while (!this.StopAllBackgroundThreads);
}
void PumpNextVideoSample()
{
// If the video sample buffer is empty, tell stream parser to give us more samples
bool MyVidBufferIsEmpty = false; bool hlsClientIsExhausted = false;
ParseMoreSamplesIfMyVideoBufferIsLow(ref MyVidBufferIsEmpty, ref parserAtEndOfStream);
if (parserAtEndOfStream) // No more data, start running down buffers
this.RunningDownVideoBuffer = true;
else if (MyVidBufferIsEmpty)
{
// Buffer is empty, wait for samples
WaitingOnEmptyVideoBuffer = true;
WaitOnEmptyVideoBuffer.WaitOne();
}
// Buffer is OK
nextSample = DeQueueVideoSample(); // thread-safe, returns NULL if a problem
// Send the sample to the external renderer
ReportGetSampleCompleted(nextSample);
}
The code seems to work well. However, I'm told that using Thread.Wait(...) is 'evil': when no samples are being requested, my code loops unnecessarily, eating up CPU time.
Can my code be further optimised? Since my class is designed for an environment where samples WILL be requested, does the potential 'pointless loop' scenario outweigh the simplicity of its current design?
Comments much appreciated.

This looks like the classic producer/consumer pattern. The normal way to solve this is with what is known as a blocking queue.
Version 4.0 of .net introduced a set of efficient, well-designed, concurrent collection classes for this very type of problem. I think BlockingCollection<T> will serve your present needs.
If you don't have access to .net 4.0 then there are many websites containing implementations of blocking queues. Personally my standard reference is Joe Duffy's book, Concurrent Programming on Windows. A good start would be Marc Gravell's blocking queue presented here in Stack Overflow.
The first advantage of using a blocking queue is that you stop using busy wait loops, hacky calls to Sleep() etc. Using a blocking queue to avoid this sort of code is always a good idea.
However, I perceive a more important benefit to using a blocking queue. At the moment your code to produce work items, consume them, and handle the queue is all intermingled. If you use a blocking queue correctly then you will end up with much better factored code which keeps separate various components of the algorithm: queue, producer and consumer.

You have one main problem: Thread.Sleep()
It has a granularity of ~20ms, that is kind of crude for video. In addition Sleep(0) has issues of possible starvation of lower-priority threads [].
The better approach is waiting on a Waithandle, preferably built into a Queue.

Blocking queue is a good and simple example of a blocking queue.
The main key is that the threads need to be coordinated with signals and not by checking the value of a counter or the state of a data structure. Any checking takes ressources (CPU) and thus you need signals (Monitor.Wait and Monitor.Pulse).

You could use an AutoResetEvent rather than a manual thread.sleep. It's fairly simple to do so:
AutoResetEvent e;
void RequestSample()
{
Interlocked.Increment(ref requestsOutstanding);
e.Set(); //also set this when StopAllBackgroundThreads=true!
}
void Pump()
{
while (!this.StopAllBackgroundThreads) {
e.WaitOne();
int leftOver = Interlocked.Decrement(ref requestsOutstanding);
while(leftOver >= 0) {
PumpNextVideoSample();
leftOver = Interlocked.Decrement(ref requestsOutstanding);
}
Interlocked.Increment(ref requestsOutstanding);
}
}
Note that it's probably even more attractive to implement a semaphore. Basically; synchronization overhead is liable to be almost nil anyhow in your scenario, and a simpler programming model is worth it. With a semaphore, you'd have something like this:
MySemaphore sem;
void RequestSample()
{
sem.Release();
}
void Pump()
{
while (true) {
sem.Acquire();
if(this.StopAllBackgroundThreads) break;
PumpNextVideoSample();
}
}
...I'd say the simplicity is worth it!
e.g. a simple implemenation of a semaphore:
public sealed class SimpleSemaphore
{
readonly object sync = new object();
public int val;
public void WaitOne()
{
lock(sync) {
while(true) {
if(val > 0) {
val--;
return;
}
Monitor.Wait(sync);
}
}
}
public void Release()
{
lock(sync) {
if(val==int.MaxValue)
throw new Exception("Too many releases without waits.");
val++;
Monitor.Pulse(sync);
}
}
}
On one trivial benchmark this trivial implementation needs ~1.7 seconds where Semaphore needs 7.5 and SemaphoreSlim needs 1.1; suprisingly reasonable, in other words.

How to elegantly access thread-safe collection and use AutoResetEvent

I have two methods, ProcessQueue and AddToQueue, which happen on different threads. Sometimes I will attempt to Process the Queue before an item is added to a queue, at which point I want to wait for an item to be added to a queue. I also want to make sure that I will never get a situation where I wait, after the Queue is evaluated as being empty and then after the Queue is added to on a different thread. Below is my attempt at doing this, but a deadlock is created because the Auto Reset Event waits with a lock still in force.
There has to be a more elegant way of doing this. Any suggestions?
private readonly object m_Locker = new object();
private readonly Queue<int> m_Queue = new Queue<int>();
private readonly AutoResetEvent m_AutoResetEvent = new AutoResetEvent(false);
void ProcessQueue()
{
lock (m_Locker)
{
if (m_Queue.Count == 0)
{
// nothing is happening, so wait for it to happen
m_AutoResetEvent.WaitOne();
}
}
Console.WriteLine("Processed {0}", m_Queue.Dequeue());
}
// on another thread
void AddToQueue(int i)
{
lock (m_Locker)
{
m_Queue.Enqueue(i);
m_AutoResetEvent.Set();
}
}

You must release the lock on the queue m_locker before you issue the wait. You could do that manually with a Monitor, reacquire and recheck after your wait is satisfied. This way you only hold the lock while you are checking for non-zero element count.
If you are on .Net 4 you can use BlockingCollection<T> or ConcurrentQueue<T> instead, from System.Collections.Concurrent. There's really no reason to build this by hand any more.
This code won't work if you have > 1 concurrent consumer - you'd need a Semaphore instead of AutoResetEvent in that case to ensure the correct number of consumers get signaled.
Since you can't use .Net 4, there are guidelines for this scenario here. Note that the comments on that article include some approaches you can use to make this bulletproof.
The following example demonstrates
thread synchronization between the
primary thread and two worker threads
using the lock keyword, and the
AutoResetEvent and ManualResetEvent
classes.

The problem is that you keep the queue locked in while you're waiting for the event.
This way the other process can't add to the queue because it is already locked. Try this:
int value = 0;
while (true)
{
lock (m_Locker)
{
if (m_Queue.Count > 0)
{
value = m_Queue.Dequeue();
break;
}
}
m_AutoResetEvent.WaitOne();
}
With the example above, you also dequeue in the lock, so you are sure that no other thread has a chance to dequeue between the moment you waited and the moment that you check the queue actually had an item.

Well, this is textbook deadlock example. The bottom line is you don't want to enter the Wait state on your AutoResetEvent while locking on m_locker in the ProcessQueue function.
Also, note that the generic Queue implementation in .NET is not thread-safe so you should also guard access to the Dequeue call in ProcessQueue.

Wouldn't you want to do:
// no lock up here
while (true)
{
// nothing is happening, so wait for it to happen
m_AutoResetEvent.WaitOne();
lock (m_locker)
{
// ProcessTheQueue(); // process the queue after the reset event is Set
}
}
and then:
lock (m_Locker)
{
m_Queue.Enqueue(i);
}
m_AutoResetEvent.Set();
?

If you are using .NET 4 the new BlockingCollection<T> provides the most elegant way to handle this.

Why bothering with the AutoResetEvent in the first place?
When you call the Process function, if it doesn't find anything than it should exit. I don't see the point in waiting since you'll probably just call it again after a while...

How to async add elements to Queue<T> in C#?

public void EnqueueTask(int[] task)
{
lock (_locker)
{
_taskQ.Enqueue(task);
Monitor.PulseAll(_locker);
}
}
So, here I'm adding elements to my queue and than threads do some work with them.How can I add items to my queue asynchronously?

If you using .net V4 have a look at the new thread safe collections, they are mostly none blocking so will properly avoid the need for an async add.

Since your using Queue<T> (recommended), Queue.Synchronized can't be used.
But besides that I would use the thread pool. But your EnqueueTask method kind of implies that the threading logic is handled outside of your "TaskQueue" class (your method implies that it is a Queue of tasks).
Your implementation also implies that it is not "Here" we wan't to add logic but rather in another place, the code you have there isn't really blocking for long so I would turn things upside down.
It also implies that the thing taking things off the queue is already on another thread since you use "PulseAll" to weak that thread up.
E.g.
public void StartQueueHandler()
{
new Thread(()=>StartWorker).Start();
}
private int[] Dequeue()
{
lock(_locker)
{
while(_taskQ.Count == 0) Monitor.Wait(_locker);
return _taskQ.Dequeue();
}
}
private void StartWorker(object obj)
{
while(_keepProcessing)
{
//Handle thread abort or have another "shot down" mechanism.
int[] work = Dequeue();
//If work should be done in parallel without results.
ThreadPool.QueueUserWorkItem(obj => DoWork(work));
//If work should be done sequential according to the queue.
DoWork(work);
}
}

Maybe something like this could work:
void AddToQueue(Queue queue, string mess) {
var t = new Thread(() => Queue.Synchronized(queue).Enqueue(mess));
t.Start();
}
The new thread ensures that your current thread does not block.
Queue.Syncronized handles all locking of the queue.
It could be replaced with your locker code, might be better performance.

The code from your question seems to indicate that you are attempting to implement a blocking queue. I make that obseration from the call to Monitor.PulseAll after the Queue<T>.Enqueue. This is the normal pattern for signalling the dequeuing thread. So if that is the case then the best option is to use the BlockingCollection class which is available in .NET 4.0.