We Keep Coding

C# await tasks + infinite loop still freezing the UI

I am trying to get the proper 'structure' for monitoring the state of a game from external source(s) using (Tasks) async/await in order to run the tasks in an infinite loop, however the current way its written seems to just freeze up my UI.
What I have so far:
(in the "state machine" class)
// Start monitoring the game state for changes
public void Start()
{
tokenSource = new CancellationTokenSource();
CancellationToken token = tokenSource.Token;
IsRunning = true;
task = Task.Factory.StartNew(async () =>
{
while (true)
{
await Task.Run(()=>CheckForStateChange());
await Task.Delay(1000); // Pause 1 second before checking state again
}
}, token, TaskCreationOptions.LongRunning, TaskScheduler.FromCurrentSynchronizationContext());
}
Without the above "Task.Delay" line the UI completely freezes up. With the "Task.Delay" line it doesn't freeze up, but if I try to drag the window it skips back to where I began dragging it.
My assumption with the current code is that the 'await Task.Run()' executes and upon completion the 'await Task.Delay()' executes and then on completion returns to the beginning of the while(true) infinite loop. (ie. not running in parallel).
The CheckForStateChange() signature is as follows:
private void CheckForStateChange()
{
// ... A bunch of code to determine and update the current state value of the object
}
Nothing special there, simple non-async method. I have read through lots of examples / questions here on StackOverflow and I used to have CheckForStateChange as returning a Task (with awaitable actions inside the method) and many other iterations of code (with the same results).
Finally I call the Start() method from the main win32 form (button) as follows:
private void btnStartSW_Click(object sender, EventArgs e)
{
// Start the subscription of the event handler
if(!state.IsRunning)
{
state.StateChange += new SummonersWar.StateChangeHandler(OnGameStateChange);
state.Start();
}
}
I think the above code is the simplest form I have written the code structure in so far, but apparently its still not written 'properly'. Any help would be appreciated.
UPDATE:
The publisher side (state machine class):
// ------ Publisher of the event ---
public delegate void StateChangeHandler(string stateText);
public event StateChangeHandler StateChange;
protected void OnStateChange() // TODO pass text?
{
if (StateChange != null)
StateChange(StateText());
}
Where the StateText() method is just a temporary way of retrieving a 'text' representation of the current state (and is really a placeholder at this point until I organize it into a tidier struct)
IsRunning is purely a public bool.
And the handler in the UI thread:
private void OnGameStateChange(string stateText)
{
// Game State Changed (update the status bar)
labelGameState.Text = "State: " + stateText;
}

Why the UI freezes
In terms of the main question: you're already calling your CheckForStateChange via Task.Run, so there is no way that your CheckForStateChange will freeze the UI unless it includes calls which are marshalled back to the UI thread (i.e. Control.Invoke or SynchronizationContext.Post/Send used explicitly, or implicitly via a Task started on the UI TaskScheduler).
The best place to start looking is your StateChange handlers (i.e. StateChangeHandler). Also have a look at where the StateChange event is raised. You'll find thread marshalling code at one of these sites.
Other issues
You're passing the TaskScheduler pointing to the UI SynchronizationContext to the outer task. You're also passing in TaskCreationOptions.LongRunning. In simple terms you're telling the task factory to "start a task on a dedicated thread, and on the current thread". These two are mutually exclusive requirements and you can pretty safely drop them both.
If, as a result of the above, your outer task happens to execute on the UI thread, it won't really trip you up as the inner call is wrapped in Task.Run, but this probably isn't the behaviour you expect.
You are storing the result of Task.Factory.StartNew inside a task field or property. Note, however, that your Task.Factory.StartNew call returns a Task<Task>, so the saved Task instance will transition to completed state almost immediately unless you call Unwrap on it and get to the inner task. To avoid this entire mess, just use Task.Run to create the outer task (as it has Unwrap semantics built in). If you do that, you can ditch the inner Task.Run completely, like so:
public bool IsRunning
{
get
{
return task.Status == TaskStatus.Running;
}
}
public void Start()
{
tokenSource = new CancellationTokenSource();
CancellationToken token = tokenSource.Token;
task = Task.Run(async () =>
{
while (true)
{
CheckForStateChange(token);
token.ThrowIfCancellationRequested();
await Task.Delay(1000); // Pause 1 second before checking state again
}
}, token);
// Uncomment this and step through `CheckForStateChange`.
// When the execution hangs, you'll know what's causing the
// postbacks to the UI thread and *may* be able to take it out.
// task.Wait();
}
Since you have a CancellationToken you need to be passing it to CheckForStateChange, and checking it periodically - otherwise it only gets checked once, when the Task is started, and then never again.
Note that I have also provided a different IsRunning implementation. Volatile state is hard to get right. If the framework is giving it to you for free, you should use it.
Final word
Overall this entire solution feels like a bit of a crutch for something that should be done more reactively - but I can think of scenarios where this sort of design is valid. I'm just not convinced that yours is really one of them.
EDIT: how to find what's blocking the UI
I'll get downvoted to oblivion for this, but here goes:
The sure way to find what's causing postbacks to the UI thread is to deadlock with it. There's plenty of threads here on SO telling you how to avoid that, but in your case - we'll cause it on purpose and you'll know exactly what calls you need to avoid when you're polling for changes - although whether or not it will be possible to avoid these calls, remains to be seen.
I've put a task.Wait instruction at the end of my code snippet. Provided that you call Start on the UI thread, that should cause a deadlock with something inside your CheckForStateChange, and you will know what it is that you need to work around.

Synchronous I/O within an async/await-based Windows Service

Let's say I have a Windows Service which is doing some bit of work, then sleeping for a short amount of time, over and over forever (until the service is shut down). So in the service's OnStart, I could start up a thread whose entry point is something like:
private void WorkerThreadFunc()
{
while (!shuttingDown)
{
DoSomething();
Thread.Sleep(10);
}
}
And in the service's OnStop, I somehow set that shuttingDown flag and then join the thread. Actually there might be several such threads, and other threads too, all started in OnStart and shut down/joined in OnStop.
If I want to instead do this sort of thing in an async/await based Windows Service, it seems like I could have OnStart create cancelable tasks but not await (or wait) on them, and have OnStop cancel those tasks and then Task.WhenAll().Wait() on them. If I understand correctly, the equivalent of the "WorkerThreadFunc" shown above might be something like:
private async Task WorkAsync(CancellationToken cancel)
{
while (true)
{
cancel.ThrowIfCancellationRequested();
DoSomething();
await Task.Delay(10, cancel).ConfigureAwait(false);
}
}
Question #1: Uh... right? I am new to async/await and still trying to get my head around it.
Assuming that's right, now let's say that DoSomething() call is (or includes) a synchronous write I/O to some piece of hardware. If I'm understanding correctly:
Question #2: That is bad? I shouldn't be doing synchronous I/O within a Task in an async/await-based program? Because it ties up a thread from the thread pool while the I/O is happening, and threads from the thread pool are a highly limited resource? Please note that I might have dozens of such Workers going simultaneously to different pieces of hardware.
I am not sure I'm understanding that correctly - I am getting the idea that it's bad from articles like Stephen Cleary's "Task.Run Etiquette Examples: Don't Use Task.Run for the Wrong Thing", but that's specifically about it being bad to do blocking work within Task.Run. I'm not sure if it's also bad if I'm just doing it directly, as in the "private async Task Work()" example above?
Assuming that's bad too, then if I understand correctly I should instead utilize the nonblocking version of DoSomething (creating a nonblocking version of it if it doesn't already exist), and then:
private async Task WorkAsync(CancellationToken cancel)
{
while (true)
{
cancel.ThrowIfCancellationRequested();
await DoSomethingAsync(cancel).ConfigureAwait(false);
await Task.Delay(10, cancel).ConfigureAwait(false);
}
}
Question #3: But... what if DoSomething is from a third party library, which I must use and cannot alter, and that library doesn't expose a nonblocking version of DoSomething? It's just a black box set in stone that at some point does a blocking write to a piece of hardware.
Maybe I wrap it and use TaskCompletionSource? Something like:
private async Task WorkAsync(CancellationToken cancel)
{
while (true)
{
cancel.ThrowIfCancellationRequested();
await WrappedDoSomething().ConfigureAwait(false);
await Task.Delay(10, cancel).ConfigureAwait(false);
}
}
private Task WrappedDoSomething()
{
var tcs = new TaskCompletionSource<object>();
DoSomething();
tcs.SetResult(null);
return tcs.Task;
}
But that seems like it's just pushing the issue down a bit further rather than resolving it. WorkAsync() will still block when it calls WrappedDoSomething(), and only get to the "await" for that after WrappedDoSomething() has already completed the blocking work. Right?
Given that (if I understand correctly) in the general case async/await should be allowed to "spread" all the way up and down in a program, would this mean that if I need to use such a library, I essentially should not make the program async/await-based? I should go back to the Thread/WorkerThreadFunc/Thread.Sleep world?
What if an async/await-based program already exists, doing other things, but now additional functionality that uses such a library needs to be added to it? Does that mean that the async/await-based program should be rewritten as a Thread/etc.-based program?

Actually there might be several such threads, and other threads too, all started in OnStart and shut down/joined in OnStop.
On a side note, it's usually simpler to have a single "master" thread that will start/join all the others. Then OnStart/OnStop just deals with the master thread.
If I want to instead do this sort of thing in an async/await based Windows Service, it seems like I could have OnStart create cancelable tasks but not await (or wait) on them, and have OnStop cancel those tasks and then Task.WhenAll().Wait() on them.
That's a perfectly acceptable approach.
If I understand correctly, the equivalent of the "WorkerThreadFunc" shown above might be something like:
Probably want to pass the CancellationToken down; cancellation can be used by synchronous code, too:
private async Task WorkAsync(CancellationToken cancel)
{
while (true)
{
DoSomething(cancel);
await Task.Delay(10, cancel).ConfigureAwait(false);
}
}
Question #1: Uh... right? I am new to async/await and still trying to get my head around it.
It's not wrong, but it only saves you one thread on a Win32 service, which doesn't do much for you.
Question #2: That is bad? I shouldn't be doing synchronous I/O within a Task in an async/await-based program? Because it ties up a thread from the thread pool while the I/O is happening, and threads from the thread pool are a highly limited resource? Please note that I might have dozens of such Workers going simultaneously to different pieces of hardware.
Dozens of threads are not a lot. Generally, asynchronous I/O is better because it doesn't use any threads at all, but in this case you're on the desktop, so threads are not a highly limited resource. async is most beneficial on UI apps (where the UI thread is special and needs to be freed), and ASP.NET apps that need to scale (where the thread pool limits scalability).
Bottom line: calling a blocking method from an asynchronous method is not bad but it's not the best, either. If there is an asynchronous method, call that instead. But if there isn't, then just keep the blocking call and document it in the XML comments for that method (because an asynchronous method blocking is rather surprising behavior).
I am getting the idea that it's bad from articles like Stephen Cleary's "Task.Run Etiquette Examples: Don't Use Task.Run for the Wrong Thing", but that's specifically about it being bad to do blocking work within Task.Run.
Yes, that is specifically about using Task.Run to wrap synchronous methods and pretend they're asynchronous. It's a common mistake; all it does is trade one thread pool thread for another.
Assuming that's bad too, then if I understand correctly I should instead utilize the nonblocking version of DoSomething (creating a nonblocking version of it if it doesn't already exist)
Asynchronous is better (in terms of resource utilization - that is, fewer threads used), so if you want/need to reduce the number of threads, you should use async.
Question #3: But... what if DoSomething is from a third party library, which I must use and cannot alter, and that library doesn't expose a nonblocking version of DoSomething? It's just a black box set in stone that at some point does a blocking write to a piece of hardware.
Then just call it directly.
Maybe I wrap it and use TaskCompletionSource?
No, that doesn't do anything useful. That just calls it synchronously and then returns an already-completed task.
But that seems like it's just pushing the issue down a bit further rather than resolving it. WorkAsync() will still block when it calls WrappedDoSomething(), and only get to the "await" for that after WrappedDoSomething() has already completed the blocking work. Right?
Yup.
Given that (if I understand correctly) in the general case async/await should be allowed to "spread" all the way up and down in a program, would this mean that if I need to use such a library, I essentially should not make the program async/await-based? I should go back to the Thread/WorkerThreadFunc/Thread.Sleep world?
Assuming you already have a blocking Win32 service, it's probably fine to just keep it as it is. If you are writing a new one, personally I would make it async to reduce threads and allow asynchronous APIs, but you don't have to do it either way. I prefer Tasks over Threads in general, since it's much easier to get results from Tasks (including exceptions).
The "async all the way" rule only goes one way. That is, once you call an async method, then its caller should be async, and its caller should be async, etc. It does not mean that every method called by an async method must be async.
So, one good reason to have an async Win32 service would be if there's an async-only API you need to consume. That would cause your DoSomething method to become async DoSomethingAsync.
What if an async/await-based program already exists, doing other things, but now additional functionality that uses such a library needs to be added to it? Does that mean that the async/await-based program should be rewritten as a Thread/etc.-based program?
No. You can always just block from an async method. With proper documentation so when you are reusing/maintaining this code a year from now, you don't swear at your past self. :)

If you still spawn your threads, well, yes, it's bad. Because it will not give you any benefit as the thread is still allocated and consuming resources for the specific purpose of running your worker function. Running a few threads to be able to do work in parallel within a service has a minimal impact on your application.
If DoSomething() is synchronous, you could switch to the Timer class instead. It allows multiple timers to use a smaller amount of threads.
If it's important that the jobs can complete, you can modify your worker classes like this:
SemaphoreSlim _shutdownEvent = new SemaphoreSlim(0,1);
public async Task Stop()
{
return await _shutdownEvent.WaitAsync();
}
private void WorkerThreadFunc()
{
while (!shuttingDown)
{
DoSomething();
Thread.Sleep(10);
}
_shutdownEvent.Release();
}
.. which means that during shutdown you can do this:
var tasks = myServices.Select(x=> x.Stop());
Task.WaitAll(tasks);

A thread can only do one thing at a time. While it is working on your DoSomething it can't do anything else.
In an interview Eric Lippert described async-await in a restaurant metaphor. He suggests to use async-await only for functionality where your thread can do other things instead of waiting for a process to complete, like respond to operator input.
Alas, your thread is not waiting, it is doing hard work in DoSomething. And as long as DoSomething is not awaiting, your thread will not return from DoSomething to do the next thing.
So if your thread has something meaningful to do while procedure DoSomething is executing, it's wise to let another thread do the DoSomething, while your original thread is doing the meaningful stuff. Task.Run( () => DoSomething()) could do this for you. As long as the thread that called Task.Run doesn't await for this task, it is free to do other things.
You also want to cancel your process. DoSomething can't be cancelled. So even if cancellation is requested you'll have to wait until DoSomething is completed.
Below is your DoSomething in a form with a Start button and a Cancel button. While your thread is DoingSomething, one of the meaningful things your GUI thread may want to do is respond to pressing the cancel button:
void CancellableDoSomething(CancellationToken token)
{
while (!token.IsCancellationRequested)
{
DoSomething()
}
}
async Task DoSomethingAsync(CancellationToken token)
{
var task = Task.Run(CancellableDoSomething(token), token);
// if you have something meaningful to do, do it now, otherwise:
return Task;
}
CancellationTokenSource cancellationTokenSource = null;
private async void OnButtonStartSomething_Clicked(object sender, ...)
{
if (cancellationTokenSource != null)
// already doing something
return
// else: not doing something: start doing something
cancellationTokenSource = new CancellationtokenSource()
var task = AwaitDoSomethingAsync(cancellationTokenSource.Token);
// if you have something meaningful to do, do it now, otherwise:
await task;
cancellationTokenSource.Dispose();
cancellationTokenSource = null;
}
private void OnButtonCancelSomething(object sender, ...)
{
if (cancellationTokenSource == null)
// not doing something, nothing to cancel
return;
// else: cancel doing something
cancellationTokenSource.Cancel();
}

async Task - What actually happens on the CPU?

I've been reading about Tasks after asking this question and seeing that I completely misunderstood the concept. Answers such as the top answers here and here explain the idea, but I still don't get it.
So I've made this a very specific question: What actually happens on the CPU when a Task is executed?
This is what I've understood after some reading: A Task will share CPU time with the caller (and let's assume the caller is the "UI") so that if it's CPU-intensive - it will slow down the UI. If the Task is not CPU-intensive - it will be running "in the background". Seems clear enough …… until tested. The following code should allow the user to click on the button, and then alternately show "Shown" and "Button". But in reality: the Form is completely busy (-no user input possible) until the "Shown"s are all shown.
public Form1()
{
InitializeComponent();
Shown += Form1_Shown;
}
private async void Form1_Shown(object sender, EventArgs e)
{
await Doit("Shown");
}
private async Task Doit(string s)
{
WebClient client = new WebClient();
for (int i = 0; i < 10; i++)
{
client.DownloadData(uri);//This is here in order to delay the Text writing without much CPU use.
textBox1.Text += s + "\r\n";
this.Update();//textBox1.
}
}
private async void button1_Click(object sender, EventArgs e)
{
await Doit("Button");
}
Can someone please tell me what is actually happening on the CPU when a Task is executed (e.g. "When the CPU is not used by the UI, the Task uses it, except for when… etc.")?

The key to understanding this is that there are two kinds of tasks - one that executes code (what I call Delegate Tasks), and one that represents a future event (what I call Promise Tasks). Those two tasks are completely different, even though they're both represented by an instance of Task in .NET. I have some pretty pictures on my blog that may help understand how these types of task are different.
Delegate Tasks are the ones created by Task.Run and friends. They execute code on the thread pool (or possibly another TaskScheduler if you're using a TaskFactory). Most of the "task parallel library" documentation deals with Delegate Tasks. These are used to spread CPU-bound algorithms across multiple CPUs, or to push CPU-bound work off a UI thread.
Promise Tasks are the ones created by TaskCompletionSource<T> and friends (including async). These are the ones used for asynchronous programming, and are a natural fit for I/O-bound code.
Note that your example code will cause a compiler warning to the effect that your "asynchronous" method Doit is not actually asynchronous but is instead synchronous. So as it stands right now, it will synchronously call DownloadData, blocking the UI thread until the download completes, and then it will update the text box and finally return an already-completed task.
To make it asynchronous, you have to use await:
private async Task Doit(string s)
{
WebClient client = new WebClient();
for (int i = 0; i < 10; i++)
{
await client.DownloadDataTaskAsync(uri);
textBox1.Text += s + "\r\n";
this.Update();//textBox1.
}
}
Now it's returning an incomplete task when it hits the await, which allows the UI thread to return to its message processing loop. When the download completes, the remainder of this method will be queued to the UI thread as a message, and it will resume executing that method when it gets around to it. When the Doit method completes, then the task it returned earlier will complete.
So, tasks returned by async methods logically represent that method. The task itself is a Promise Task, not a Delegate Task, and does not actually "execute". The method is split into multiple parts (at each await point) and executes in chunks, but the task itself does not execute anywhere.
For further reading, I have a blog post on how async and await actually work (and how they schedule the chunks of the method), and another blog post on why asynchronous I/O tasks do not need to block threads.

As per your linked answers, Tasks and Threads are totally different concepts, and you are also getting confused with async / await
A Task is just a representation of some work to be done. It says nothing about HOW that work should be done.
A Thread is a representation of some work that is running on the CPU, but is sharing the CPU time with other threads that it can know nothing about.
You can run a Task on a Thread using Task.Run(). Your Task will run asynchronously and independently of any other code providing a threadpool thread is available.
You can also run a Task asynchronously on the SAME thread using async / await. Anytime the thread hits an await, it can save the current stack state, then travel back up the stack and carry on with other work until the awaited task has finished. Your Doit() code never awaits anything, so will run synchronously on your GUI thread until complete.

Tasks use the ThreadPool you can read extensively about what it is and how it works here
But in a nutshell, when a task is executed, the Task Scheduler looks in the ThreadPool to see if there is a thread available to run the action of the task. If not, it's going to be queued until one becomes available.
A ThreadPool is just a collection of already-instantiated threads made available so that multithreaded code can safely use concurrent programming without overwhelming the CPU with context-switching all the time.
Now, the problem with your code is that even though you return an object of type Task, you are not running anything concurrently - No separate thread is ever started!
In order to do that, you have two options, either you start yourDoit method as a Task, with
Option1
Task.Run(() => DoIt(s));
This will run the whole DoIt method on another thread from the Thread Pool, but it will lead to more problems, because in this method, you're trying to access UI-controls. therefore, you will need either to marshal those calls to the UI thread, or re-think your code so that the UI access is done directly on the UI thread after the asynchronous tasks completes.
Option 2 (preferred, if you can)
You use .net APIs which are already asynchronous, such as client.DownloadDataTaskAsync(); instead of client.DownloadData();
now, in your case, the problem is that you will need to have 10 calls, which are going to return 10 different objects of type Task<byte[]> and you want to await on the completion of all of them, not just one.
In order to do this, you will need to create a List<Task<byte[]>> returnedTasks and you will add to it all returned value from DownloadDataTaskAsync(). then, once this is done, you can use the following return value for your DoIt method.
return Task.WhenAll(returnedTasks);

How can I have two separate task schedulers?

I am writing a game, and using OpenGL I require that some work be offloaded to the rendering thread where an OpenGL context is active, but everything else is handled by the normal thread pool.
Is there a way I can force a Task to be executed in a special thread-pool, and any new tasks created from an async also be dispatched to that thread pool?
I want a few specialized threads for rendering, and I would like to be able to use async and await for example for creating and filling a vertex buffer.
If I just use a custom task scheduler and a new Factory(new MyScheduler()) it seems that any subsequent Task objects will be dispatched to the thread pool anyway where Task.Factory.Scheduler suddenly is null.
The following code should show what I want to be able to do:
public async Task Initialize()
{
// The two following tasks should run on the rendering thread pool
// They cannot run synchronously because that will cause them to fail.
this.VertexBuffer = await CreateVertexBuffer();
this.IndexBuffer = await CreateIndexBuffer();
// This should be dispatched, or run synchrounousyly, on the normal thread pool
Vertex[] vertices = CreateVertices();
// Issue task for filling vertex buffer on rendering thread pool
var fillVertexBufferTask = FillVertexBufffer(vertices, this.VertexBuffer);
// This should be dispatched, or run synchrounousyly, on the normal thread pool
short[] indices = CreateIndices();
// Wait for tasks on the rendering thread pool to complete.
await FillIndexBuffer(indices, this.IndexBuffer);
await fillVertexBufferTask; // Wait for the rendering task to complete.
}
Is there any way to achieve this, or is it outside the scope of async/await?

This is possible and basically the same thing what Microsoft did for the Windows Forms and WPF Synchronization Context.
First Part - You are in the OpenGL thread, and want to put some work into the thread pool, and after this work is done you want back into the OpenGL thread.
I think the best way for you to go about this is to implement your own SynchronizationContext. This thing basically controls how the TaskScheduler works and how it schedules the task. The default implementation simply sends the tasks to the thread pool. What you need to do is to send the task to a dedicated thread (that holds the OpenGL context) and execute them one by one there.
The key of the implementation is to overwrite the Post and the Send methods. Both methods are expected to execute the callback, where Send has to wait for the call to finish and Post does not. The example implementation using the thread pool is that Sendsimply directly calls the callback and Post delegates the callback to the thread pool.
For the execution queue for your OpenGL thread I am think a Thread that queries a BlockingCollection should do nicely. Just send the callbacks to this queue. You may also need some callback in case your post method is called from the wrong thread and you need to wait for the task to finish.
But all in all this way should work. async/await ensures that the SynchronizationContext is restored after a async call that is executed in the thread pool for example. So you should be able to return to the OpenGL thread after you did put some work off into another thread.
Second Part - You are in another thread and want to send some work into the OpenGL thread and await the completion of that work.
This is possible too. My idea in this case is that you don't use Tasks but other awaitable objects. In general every object can be awaitable. It just has to implement a public method getAwaiter() that returns a object implementing the INotifyCompletion interface. What await does is that it puts the remaining method into a new Action and sends this action to the OnCompleted method of that interface. The awaiter is expected to call the scheduled actions once the operation it is awaiting is done. Also this awaiter has to ensure that the SynchronizationContext is captured and the continuations are executed on the captured SynchronizationContext. That sounds complicated, but once you get the hang of it, it goes fairly easy. What helped me a lot is the reference source of the YieldAwaiter (this is basically what happens if you use await Task.Yield()). This is not what you need, but I think it is a place to start.
The method that returns the awaiter has to take care of sending the actual work to the thread that has to execute it (you maybe already have the execution queue from the first part) and the awaiter has to trigger once that work is done.
Conclusion
Make no mistake. That is a lot of work. But if you do all that you will have less problem down the line because you can seamless use the async/await pattern as if you would be working inside windows forms or WPF and that is a hue plus.

First, realize that await introduces the special behavior after the method is called; that is to say, this code:
this.VertexBuffer = await CreateVertexBuffer();
is pretty much the same as this code:
var createVertexBufferTask = CreateVertexBuffer();
this.VertexBuffer = await createVertexBufferTask;
So, you'll have to explicitly schedule code to execute a method within a different context.
You mention using a MyScheduler but I don't see your code using it. Something like this should work:
this.factory = new TaskFactory(CancellationToken.None, TaskCreationOptions.DenyChildAttach, TaskContinuationOptions.None, new MyScheduler());
public async Task Initialize()
{
// Since you mention OpenGL, I'm assuming this method is called on the UI thread.
// Run these methods on the rendering thread pool.
this.VertexBuffer = await this.factory.StartNew(() => CreateVertexBuffer()).Unwrap();
this.IndexBuffer = await this.factory.StartNew(() => CreateIndexBuffer()).Unwrap();
// Run these methods on the normal thread pool.
Vertex[] vertices = await Task.Run(() => CreateVertices());
var fillVertexBufferTask = Task.Run(() => FillVertexBufffer(vertices, this.VertexBuffer));
short[] indices = await Task.Run(() => CreateIndices());
await Task.Run(() => FillIndexBuffer(indices, this.IndexBuffer));
// Wait for the rendering task to complete.
await fillVertexBufferTask;
}
I would look into combining those multiple Task.Run calls, or (if Initialize is called on a normal thread pool thread) removing them completely.

Unexpected behaviour after returning from await

I know there are a lot of questions about async/await, but I couldn't find any answer to this.
I've encountered something I don't understand, consider the following code:
void Main()
{
Poetry();
while (true)
{
Console.WriteLine("Outside, within Main.");
Thread.Sleep(200);
}
}
async void Poetry()
{
//.. stuff happens before await
await Task.Delay(10);
for (int i = 0; i < 10; i++)
{
Console.WriteLine("Inside, after await.");
Thread.Sleep(200);
}
}
Obviously, on the await operator, the control returns to the caller, while the method being awaited, is running on the background. (assume an IO operation)
But after the control comes back to the await operator, the execution becomes parallel, instead of (my expectation) remaining single-threaded.
I'd expect that after "Delay" has been finished the thread will be forced back into the Poetry method, continues from where it left.
Which it does. The weird thing for me, is why the "Main" method keeps running? is that one thread jumping from one to the other? or are there two parallel threads?
Isn't it a thread-safety problem, once again?
I find this confusing. I'm not an expert. Thanks.

I have a description on my blog of how async methods resume after an await. In essence, await captures the current SynchronizationContext unless it is null in which case it captures the current TaskScheduler. That "context" is then used to schedule the remainder of the method.
Since you're executing a Console app, there is no SynchronizationContext, and the default TaskScheduler is captured to execute the remainder of the async method. That context queues the async method to the thread pool. It is not possible to return to the main thread of a Console app unless you actually give it a main loop with a SynchronizationContext (or TaskScheduler) that queues to that main loop.

Read It's All About the SynchronizationContext and I'm sure it'll become less confusing. The behavior you're seeing makes perfect sense. Task.Delay uses Win32 Kernel Timer APIs internally (namely, CreateTimerQueueTimer). The timer callback is invoked on a pool thread, different from your Main thread. That's where the rest of Poetry after await continues executing. This is how the default task scheduler works, in the absence of synchronization context on the original thread which initiated the await.
Because you don't do await the Poetry() task (and you can't unless you return Task instead of void), its for loop continues executing in parallel with the while loop in your Main. Why, and more importantly, how would you expect it to be "forced" back onto the Main thread? There has to be some explicit point of synchronization for this to happen, the thread cannot simply get interrupted in the middle of the while loop.
In a UI application, the core message loop may serve as such kind of synchronization point. E.g. for a WinForms app, WindowsFormsSynchronizationContext would make this happen. If await Task.Delay() is called on the main UI thread, the code after await would asynchronously continue on the main UI thread, upon some future iteration of the message loop run by Application.Run.
So, if it was a UI thread, the rest of the Poetry wouldn't get executed in parallel with the while loop following the Poetry() call. Rather, it would be executed when the control flow had returned to the message loop. Or, you might explicitly pump messages with Application.DoEvents() for the continuation to happen, although I wouldn't recommend doing that.
On a side note, don't use async void, rather use async Task, more info.

When you call an async routine the purpose of this is to allow the program to run a method while still allowing the calling routine, form or application to continue to respond to user input (in other words, continue execution normally). The "await" keyword pauses execution at the point it is used, runs the task using another thread then returns to that line when the thread completes.
So, in your case if you want the main routine to pause until the "Poetry" routine is done you need to use the await keyword something like this:
void async Main()
{
await Poetry();
while (true)
{
Console.WriteLine("Outside, within Main.");
Thread.Sleep(200);
}
}
You will also need to change the definition for Poetry to allow the await keyword to be used:
async Task Poetry()
Because this question really intrigued me I went ahead and wrote an example program you can actually compile. Just create a new console application and paste this example in. You can see the result of using "await" versus not using it.
class Program
{
static void Main(string[] args)
{
RunMain();
// pause long enough for all async routines to complete (10 minutes)
System.Threading.Thread.Sleep(10 * 60 * 1000);
}
private static async void RunMain()
{
// with await this will pause for poetry
await Poetry();
// without await this just runs
// Poetry();
for (int main = 0; main < 25; main++)
{
System.Threading.Thread.Sleep(10);
Console.WriteLine("MAIN [" + main + "]");
}
}
private static async Task Poetry()
{
await Task.Delay(10);
for (int i = 0; i < 10; i++)
{
Console.WriteLine("IN THE POETRY ROUTINE [" + i + "]");
System.Threading.Thread.Sleep(10);
}
}
}
Happy testing! Oh, and you can still read more information here.

I'd like to answer my own question here.
Some of you gave me great answers which all helped me understand better (and were thumbed up). Possibly no one gave me a full answer because I've failed to ask the full question. In any case someone will encounter my exact misunderstanding, I'd like this to be the first answer (but I'll recommend to look at some more answers below).
So, Task.Delay uses a Timer which uses the operating system to fire an event after N milliseconds. after this period a new pooled thread is created, which basically does almost nothing.
The await keyword means that after the thread has finished (and it's doing almost nothing) it should continue to whatever comes after the await keyword.
Here comes the synchronization context, as mentioned in other answers.
If there is no such context, the same newly-created-pooled-thread will continue running what ever comes after the await.
If there is a synchronizing context, the newly-created-pool-thread, will only push whatever comes after the await, into synchronizing context.
For the sake of it, here are a few points I didn't realize:
The async/await are not doing anything which wasn't (technologly speaking) possible before. Just maybe amazingly clumsy.
It's is just a language support for some of .NET 4.5 classes.
It's much like yield return. It may break your method into a few methods, and may even generate a class behind, and use some methods from the BCL, but nothing more.
Anyway, I recommend reading C# 5.0 In A Nutshell's chapter "Concurrency and Asynchrony". It helped me a lot. It is great, and actually explains the whole story.