We Keep Coding

How can I have two separate task schedulers?

I am writing a game, and using OpenGL I require that some work be offloaded to the rendering thread where an OpenGL context is active, but everything else is handled by the normal thread pool.
Is there a way I can force a Task to be executed in a special thread-pool, and any new tasks created from an async also be dispatched to that thread pool?
I want a few specialized threads for rendering, and I would like to be able to use async and await for example for creating and filling a vertex buffer.
If I just use a custom task scheduler and a new Factory(new MyScheduler()) it seems that any subsequent Task objects will be dispatched to the thread pool anyway where Task.Factory.Scheduler suddenly is null.
The following code should show what I want to be able to do:
public async Task Initialize()
{
// The two following tasks should run on the rendering thread pool
// They cannot run synchronously because that will cause them to fail.
this.VertexBuffer = await CreateVertexBuffer();
this.IndexBuffer = await CreateIndexBuffer();
// This should be dispatched, or run synchrounousyly, on the normal thread pool
Vertex[] vertices = CreateVertices();
// Issue task for filling vertex buffer on rendering thread pool
var fillVertexBufferTask = FillVertexBufffer(vertices, this.VertexBuffer);
// This should be dispatched, or run synchrounousyly, on the normal thread pool
short[] indices = CreateIndices();
// Wait for tasks on the rendering thread pool to complete.
await FillIndexBuffer(indices, this.IndexBuffer);
await fillVertexBufferTask; // Wait for the rendering task to complete.
}
Is there any way to achieve this, or is it outside the scope of async/await?

This is possible and basically the same thing what Microsoft did for the Windows Forms and WPF Synchronization Context.
First Part - You are in the OpenGL thread, and want to put some work into the thread pool, and after this work is done you want back into the OpenGL thread.
I think the best way for you to go about this is to implement your own SynchronizationContext. This thing basically controls how the TaskScheduler works and how it schedules the task. The default implementation simply sends the tasks to the thread pool. What you need to do is to send the task to a dedicated thread (that holds the OpenGL context) and execute them one by one there.
The key of the implementation is to overwrite the Post and the Send methods. Both methods are expected to execute the callback, where Send has to wait for the call to finish and Post does not. The example implementation using the thread pool is that Sendsimply directly calls the callback and Post delegates the callback to the thread pool.
For the execution queue for your OpenGL thread I am think a Thread that queries a BlockingCollection should do nicely. Just send the callbacks to this queue. You may also need some callback in case your post method is called from the wrong thread and you need to wait for the task to finish.
But all in all this way should work. async/await ensures that the SynchronizationContext is restored after a async call that is executed in the thread pool for example. So you should be able to return to the OpenGL thread after you did put some work off into another thread.
Second Part - You are in another thread and want to send some work into the OpenGL thread and await the completion of that work.
This is possible too. My idea in this case is that you don't use Tasks but other awaitable objects. In general every object can be awaitable. It just has to implement a public method getAwaiter() that returns a object implementing the INotifyCompletion interface. What await does is that it puts the remaining method into a new Action and sends this action to the OnCompleted method of that interface. The awaiter is expected to call the scheduled actions once the operation it is awaiting is done. Also this awaiter has to ensure that the SynchronizationContext is captured and the continuations are executed on the captured SynchronizationContext. That sounds complicated, but once you get the hang of it, it goes fairly easy. What helped me a lot is the reference source of the YieldAwaiter (this is basically what happens if you use await Task.Yield()). This is not what you need, but I think it is a place to start.
The method that returns the awaiter has to take care of sending the actual work to the thread that has to execute it (you maybe already have the execution queue from the first part) and the awaiter has to trigger once that work is done.
Conclusion
Make no mistake. That is a lot of work. But if you do all that you will have less problem down the line because you can seamless use the async/await pattern as if you would be working inside windows forms or WPF and that is a hue plus.

First, realize that await introduces the special behavior after the method is called; that is to say, this code:
this.VertexBuffer = await CreateVertexBuffer();
is pretty much the same as this code:
var createVertexBufferTask = CreateVertexBuffer();
this.VertexBuffer = await createVertexBufferTask;
So, you'll have to explicitly schedule code to execute a method within a different context.
You mention using a MyScheduler but I don't see your code using it. Something like this should work:
this.factory = new TaskFactory(CancellationToken.None, TaskCreationOptions.DenyChildAttach, TaskContinuationOptions.None, new MyScheduler());
public async Task Initialize()
{
// Since you mention OpenGL, I'm assuming this method is called on the UI thread.
// Run these methods on the rendering thread pool.
this.VertexBuffer = await this.factory.StartNew(() => CreateVertexBuffer()).Unwrap();
this.IndexBuffer = await this.factory.StartNew(() => CreateIndexBuffer()).Unwrap();
// Run these methods on the normal thread pool.
Vertex[] vertices = await Task.Run(() => CreateVertices());
var fillVertexBufferTask = Task.Run(() => FillVertexBufffer(vertices, this.VertexBuffer));
short[] indices = await Task.Run(() => CreateIndices());
await Task.Run(() => FillIndexBuffer(indices, this.IndexBuffer));
// Wait for the rendering task to complete.
await fillVertexBufferTask;
}
I would look into combining those multiple Task.Run calls, or (if Initialize is called on a normal thread pool thread) removing them completely.

AutoResetEvent + Background Worker conjunction with each other

I have a query regarding these two working in conjunction with one another as I am not convinced they go hand in hand.
I have some code which uses the background worker and within DoWork() it performs some logic. This logic simply consumes the TcpClient object and sends data to an external device.
My question is within DoWork() I call AutoResetEvent.WaitOne(). Does this stop the current thread until AutoResetEvent.Set() is called? If for example, AutoResetEvent.Set() is never called will the Background Worker simply halt and not perform any operations?
Not sure whether this is bad design by nature but supporting it unfortunately :(
EDIT some pseudo code
bool done = false;
while (!done)
{
//some logic here
done = System.DateTime.Now.Hour == 10;
if (!done)
{
AutoResetEvent.WaitOne();
//Question will the while loop fire again or does it wait for the Set() to be called
//before continuing?
}
}

within DoWork() I call AutoResetEvent.WaitOne(). Does this stop the current thread until AutoResetEvent.Set() is called? If for example, AutoResetEvent.Set() is never called will the Background Worker simply halt and not perform any operations?
The answer to this is yes as cited from MSDN WaitHandle.WaitOne Method will:
Blocks the current thread until the current WaitHandle receives a signal.
EDIT: for your added code, the while loop will wait for the Set()

How can I tell when a thread have finished when using a thread pool?

My thread:
public void main_news_thread(MainApplication main)
{
ThreadPool.QueueUserWorkItem(p => check_news(validrsslist, 0));
}
I call this thread every interval of time...
How can I know when the thread finishes so I can call two other methods which deal with the GUI? How can I refer to this threadpool thread?

Since you are talking about UI, you might want to look at BackgroundWorker, which offers a RunWorkerCompleted event that fires on the UI thread, and indicate success/failure/cancel etc.
http://msdn.microsoft.com/en-us/library/system.componentmodel.backgroundworker_events.aspx
Personally, though, I'd just run a callback method at the end of my worker code (remembering to switch back to the UI thread, via Dispatcher.Invoke in WPF or this.Invoke in winforms).

You can execute the methods in the thread itself (you have to take care of invoking yourself to access the gui thread):
ThreadPool.QueueUserWorkItem(p => {
check_news(validrsslist, 0);
//do something after the task is finished
});

How can I ensure a determenistic result for this multithreading problem?

Consider the following test snippet:
// act
AutoResetEvent workDoneEvent = new AutoResetEvent(false);
ThreadPool.QueueUserWorkItem(delegate
{
ProcessAndSignal(processor, workDoneEvent);
}, null);
// let worker thread have a go
workDoneEvent.WaitOne();
blockingFetcher.WaitForNextMessage = false;
// assert
Assert.That(processor.StopCause, Is.Null);
}
private static void ProcessAndSignal(MessageProcessor processor, AutoResetEvent workDoneEvent)
{
workDoneEvent.Set();
// this invocation will block until the WaitForNextMessageFlag is set
processor.ProcessMessages();
}
Ideal scenario:
ProcessAndSignalMethod is queued on the thread pool but does not start to execute.
The main thread blocks (autoResetEvent.WaitOne())
A worker thread starts to execute the "ProcessAndSignal" method
The worker threads has enough time to signal the flag and start execution of the ProcessMessages method
The main thread is spawned back into life and sets the property which will cause the ProcessAndSignal method to complete gracefully
Can the following scenario occur?
1) ProcessAndSignal() will start to execute before the main thread sets the AutoResetEvent to WaitOne() which will cause a deadlock (the processor.ProcessMessages() will go into an infinitive loop)

Yes, the scenario can occur. Yes it can deadlock if you don't declare the bool variable as volatile. Just don't use a bool, use an event like you did.
The logic looks weird, it smells like you are trying to let the main thread wait for the processing to be completed. The workDoneEvent doesn't actually signal that the work was done. Right now the main thread will check the assert before the worker is done, that can't be good. If the intention was that it signals that the worker is done then ProcessAndSignal should be the one calling Set(), at the end of the method. And the main thread should call WaitOne().
If this is at all accurate then you just should not use QUWI, just call ProcessAndSignal directly without using a thread. Far more efficient, zero odds for threading problems.

How do I block until a thread is returned to the pool?

As part of a windows service
I'm accepting incoming socket connection using
myListener.BeginAcceptSocket(acceptAsync, null)
The acceptAsync function executes on a seperate thread (just as expected).
When the service is requested to shutdown, I "signal" the threads that accepted and are currently working on the sockets, to finish up.
After signaling each thread to end,I need to block until they are all done. I have a list of threads, that I thought I could iterate through and Join each thread until they were all done.
Howerver it seems that these threads don't end, but return to the pool, so the Join will wait for ever.
How do I block until a thread is returned to the pool?

You shouldn't use Join in this case. Rather, you should use a series of WaitHandles (specifically, an AutoResetEvent or ManualResetEvent) which your threads will signal when they are done with their work.
You would then call the static WaitAll method on the WaitHandle class, passing all of the events to wait on.

The canonical pattern for doing this is to use a CountdownEvent. The main thread will increment the event to indicate that it is participating and the worker threads will do the same once they start. After the worker threads have finished they will decrement the event. When the main thread is ready to wait for completion it should decrement the event and then wait on it. If you are not using .NET 4.0 then you can get an implemention of a countdown event from part 4 of Joe Albahari's threading ebook.
public class Example
{
private CountdownEvent m_Finisher = new CountdownEvent(0);
public void MainThread()
{
m_Finisher.AddCount();
// Your stuff goes here.
// myListener.BeginAcceptSocket(OnAcceptSocket, null);
m_Finisher.Signal();
m_Finisher.Wait();
}
private void OnAcceptSocket(object state)
{
m_Finisher.AddCount()
try
{
// Your stuff goes here.
}
finally
{
m_Finisher.Signal();
}
}
}

The best way would be to change acceptAsync so that it signals on a semaphore, your main thread can then wait on that semaphore.
You don't have a lot of acces to or control over Threapool threads.