We Keep Coding

Understanding single-threaded nature of Orleans grains

I have the following code snippet of a client and a grain in Orleans. (Although the recommended way to develop in Orleans is to await Tasks, the following code doesn't await at some points purely for experimentation purposes)
// client code
while(true)
{
Console.WriteLine("Client giving another request");
double temperature = random.NextDouble() * 40;
var grain = client.GetGrain<ITemperatureSensorGrain>(500);
Task t = sensor.SubmitTemperatureAsync((float)temperature);
Console.WriteLine("Client Task Status - "+t.Status);
await Task.Delay(5000);
}
// ITemperatureSensorGrain code
public async Task SubmitTemperatureAsync(float temperature)
{
long grainId = this.GetPrimaryKeyLong();
Console.WriteLine($"{grainId} outer received temperature: {temperature}");
Task x = SubmitTemp(temperature); // SubmitTemp() is another function in the same grain
x.Ignore();
Console.WriteLine($"{grainId} outer received temperature: {temperature} exiting");
}
public async Task SubmitTemp(float temp)
{
for(int i=0; i<1000; i++)
{
Console.WriteLine($"Internal function getting awaiting task {i}");
await Task.Delay(1000);
}
}
The output when I run the above code is the following:
Client giving another request
Client Task Status - WaitingForActivation
500 outer received temperature: 23.79668
Internal function getting awaiting task 0
500 outer received temperature: 23.79668 exiting
Internal function getting awaiting task 1
Internal function getting awaiting task 2
Internal function getting awaiting task 3
Internal function getting awaiting task 4
Client giving another request
Client Task Status - WaitingForActivation
500 outer received temperature: 39.0514
Internal function getting awaiting task 0 <------- from second call to SubmitTemp
500 outer received temperature: 39.0514 exiting
Internal function getting awaiting task 5 <------- from first call to SubmitTemp
Internal function getting awaiting task 1
Internal function getting awaiting task 6
Internal function getting awaiting task 2
Internal function getting awaiting task 7
Internal function getting awaiting task 3
Internal function getting awaiting task 8
Internal function getting awaiting task 4
Internal function getting awaiting task 9
The output makes sense from the perspective of a normal .Net application. If I can take help from this stackoverflow post, what is happening here is that:
Client makes a call to ITemperatureSendorGrain and proceeds ahead. When the await is hit, the client thread is returned to the threadpool.
SubmitTemperatureAsync receives the request and calls a local async function SubmitTemp.
SubmitTemp prints statement corresponding to i=0 after which point it hits await. Await causes the rest of the for loop to be scheduled as a continuation of the awaitable (Task.Delay) and the control returns to the calling function SubmitTemperatureAsync. Note here that the thread isn't returned to the threadpool when it encounters await in SubmitTemp function. The thread control is actually returned to the calling function SubmitTemperatureAsync. So, a turn, as defined in Orleans docs, ends when the top level method encounters an await. When a turn ends, the thread is returned to the threadpool.
The calling function doesn't wait for the task to finish and exits.
When the awaitable in SubmitTemp returns after 1s, it gets a thread from the threadpool and schedules the rest of the for loop on it.
When the awaitable in client code returns, it makes another call to the same grain and another round of for loop is scheduled corresponding to the second call to SubmitTemp.
My first question is have I correctly described what's happening in the code, especially about the thread not being returned to the thread pool when await is hit in the function SubmitTemp.
According to single-threaded nature of grains, at any time only one thread will be executing a grain's code. Also, once a request to a grain begins execution, it will be completed fully before the next request is taken up (called chunk based execution in the Orleans docs). On a high-level, this is true for the above code because the next call to SubmitTemperatureAsync will happen only when the current call to the method exits.
However, SubmitTemp was actually a sub-function of SubmitTemperatureAsync. Although SubmitTemperatureAsync exited, SubmitTemp is still executing and while it does that, Orleans allowed another call to SubmitTemperatureAsync to execute. Doesn't this violate the single-threaded nature of Orleans grain is my second question?
Consider that SubmitTemp in its for loop needs to access some data members of the grain class. So, the ExecutionContext will be captured when await is encountered and when Task.Delay(1000) returns, the captured ExecutionContext will be passed to the scheduling of the remainder for loop on a thread. Because ExecutionContext is passed, the remainder for loop will be able to access the data members in spite of running on a different thread. This is something which happens in any normal .Net asynchronous application.
My third question is about SynchronizationContext. I did a cursory search in the Orleans repository but couldn't find any implementation of SynchronizationContext.Post(), which leads me to believe that no SynchronizationContext is required to run Orleans methods. Can anyone confirm this? If this isn't true, and a SynchronizationContext is required, wouldn't the parallel execution of the various invocations of SubmitTemp (as shown in the above code), run the risk of ending in a deadlock (if someone holds on to the SynchronizationContext and doesn't release it)?

Question 1: Is the described execution flow an accurate representation of what's happening?
Your description looks roughly correct to me, but here's a few finer points:
Whether or not there's a thread pool is an implementation detail.
'Turns' are each synchronous portion of work scheduled on the activation's TaskScheduler.
Therefore, a turn ends whenever a execution has to be yielded back to the TaskScheduler.
That could be because an await was hit which wasn't synchronously completed, or maybe the user isn't using await at all and is programming using ContinueWith or custom awaitables.
A turn could be ended from a non-top-level method, eg if the code was changed to await SubmitTemp(x) instead of .Ignoring() it, then the turn would end when the Task.Delay(...) was hit inside SubmitTemp(x).
Question 2: Does the example program demonstrate a violation of the single-threadedness guarantee?
No, there is only ever a single thread executing the grain's code at a given time. However, that 'thread' has to split its time between the various tasks which are scheduled on the activation's TaskScheduler. i.e, there will never be a time where you suspend the process and discover that two threads are executing your grain's code at the same time.
As far as the runtime is concerned, the processing of your message ends when the Task (or other awaitable type) returned from the top level method completes. Until that occurs, no new messages will be scheduled for execution on your activation. Background tasks spawned from your methods are always allowed to interleave with other tasks.
.NET allows child tasks to be attached to their parent task. In that case, the parent task only completes when all child tasks complete. This is not the default behavior, however, and it's generally suggested that you avoid opting-in to this behavior (eg, by passing TaskCreationOptions.AttachedToParent to Task.Factory.StartNew).
If you did make use of that behavior (please don't), then you would see your activation loop on the first call to SubmitTemp() indefinitely, and no more messages will be processed.
Question 3: Does Orleans use SynchronizationContext?
Orleans does not use SynchronizationContext. Instead, it uses custom TaskScheduler implementations. See ActivationTaskScheduler.cs. Each activation has its own ActivationTaskScheduler and all messages are scheduler using that scheduler.
Regarding the follow-on question, the Task instances (each one representing a synchronous piece of work) which are scheduled against the activation are inserted into the same queue and so they are allowed to interleave, but the ActivationTaskScheduler is only executed by one thread at a time.

I understand that this is a contrived code snippet that was written to explore the execution guarantees of the Orleans runtime. I'm a little concerned that somebody might read this and misconstrued that this is a recommended pattern of how grain methods should be implemented.
That's why I'd like to stress that the recommended way of writing grain code is to await every Task in the call stack. In the above code it would mean awaiting x in the grain method and t in the client code. By default grains are non-reentrant, and that would prevent a second call from the client to start executing before execution of the first one is completed. Or one might choose to mark the grain class as [Reentrant] and allow interleaving of a second call. That would be much more clear and explicit than the background loop, and would make error handling possible.

How do I implement an async I/O bound operation from scratch?

I'm trying to understand how and when to use async programming and got to I/O bound operations, but I don't understand them. I want to implement them from scratch. How can I do that?
Consider the example below which is synchronous:
private void DownloadBigImage() {
var url = "https://cosmos-magazine.imgix.net/file/spina/photo/14402/180322-Steve-Full.jpg";
new WebClient().DownloadFile(url, "image.jpg");
}
How do I implement the async version by only having the normal synchronous method DownloadBigImage without using Task.Run since that will use a thread from the thread pool only for waiting - that's just being wasteful!
Also do not use the special method that's already async! This is the purpose of this question: how do I make it myself without relying on methods which are already async? So, NO things like:
await new WebClient().DownloadFileTaskAsync(url, "image.jpg");
Examples and documentation available are very lacking in this regard. I found only this:
https://learn.microsoft.com/en-us/dotnet/standard/async-in-depth
which says:
The call to GetStringAsync() calls through lower-level .NET libraries (perhaps calling other async methods) until it reaches a P/Invoke interop call into a native networking library. The native library may subsequently call into a System API call (such as write() to a socket on Linux). A task object will be created at the native/managed boundary, possibly using TaskCompletionSource. The task object will be passed up through the layers, possibly operated on or directly returned, eventually returned to the initial caller.
Basically I have to use a "P/Invoke interop call into a native networking library"... but how?

This is a great question which really isn't explained well in most texts about C# and async.
I searched for this for ages thinking I could and should maybe be implementing my own async I/O methods. If a method/library I was using didn't have async methods I thought I should somehow wrap these functions in code that made them asynchronous. It turns out that this isn't really feasible for most programmers. Yes, you can spawn a new thread using Thread.Start(() => {...}) and that does make your code asynchronous, but it also creates a new thread which is an expensive overhead for asynchronous operations. It can certainly free up your UI thread to ensure your app stays responsive, but it doesn't create a truly async operation the way that HttpClient.GetAsync() is a truly asynchronous operation.
This is because async methods in the .net libraries use something called "standard P/Invoke asynchronous I/O system in .NET" to call low level OS code that doesn't require a dedicated CPU thread while doing outbound IO (networking or storage). It actually doesn't dedicate a thread to its work and signals the .net runtime when it's done doing its stuff.
I'm not familiar with the details but this knowledge is enough to free me from trying to implement async I/O and make me focus on using the async methods already present in the .net libraries (such as HttpClient.GetAsync()). More interesting info can be found here (Microsoft async deep dive) and a nice description by Stephen Cleary here

I think this is a very interesting question and a fun learning exercise.
Fundamentally, you cannot use any existing API that is synchronous. Once it's synchronous there is no way to turn it truly asynchronous. You correctly identified that Task.Run and it's equivalents are not a solution.
If you refuse to call any async .NET API then you need to use PInvoke to call native APIs. This means that you need to call the WinHTTP API or use sockets directly. This is possible but I don't have the experience to guide you.
Rather, you can use async managed sockets to implement an async HTTP download.
Start with the synchronous code (this is a raw sketch):
using (var s = new Socket(...))
{
s.Connect(...);
s.Send(GetHttpRequestBytes());
var response = new StreamReader(new NetworkStream(s)).ReadToEnd();
}
This very roughly gets you an HTTP response as a string.
You can easily make this truly async by using await.
using (var s = new Socket(...))
{
await s.ConnectAsync(...);
await s.SendAsync(GetHttpRequestBytes());
var response = await new StreamReader(new NetworkStream(s)).ReadToEndAsync();
}
If you consider await cheating with respect to your exercise goals you would need to write this using callbacks. This is awful so I'm just going to write the connect part:
var s = new Socket(...)
s.BeginConnect(..., ar => {
//perform next steps here
}, null);
Again, this code is very raw but it shows the principle. Instead of waiting for an IO to complete (which happens implicitly inside of Connect) you register a callback that is called when the IO is done. That way your main thread continues to run. This turns your code into spaghetti.
You need to write safe disposal with callbacks. This is a problem because exception handling cannot span callbacks. Also, you likely need to write a read loop if you don't want to rely on the framework to do that. Async loops can be mind bending.

TLDR: Generally you can by using TaskCompletionSource.
If you only have blocking calls available then you cannot do this. But usually there are "old" asynchronous methods that do not use async nor Task, but rely instead on callbacks. In that case you can use a TaskCompletionSource to both create a Task that can be returned, and use it to set the Task to completed when the callback returns.
Example using the old .Net Framework 3.0 methods in WebClient (but programmed in later .Net that has Task):
public Task DownloadCallbackToAsync(string url, string filename)
{
using (var client = new WebClient())
{
TaskCompletionSource taskCreator = new TaskCompletionSource();
client.DownloadFileCompleted += (sender, args) => taskCreator.SetResult();
client.DownloadFileAsync(url, filename);
return taskCreator.Task;
}
}
Here you will imidiately initiate the call and return a Task. If you await the Task in the calling method you will not continue until the callback (DownloadFileCompleted) has occurred.
Note that this method itself is not async as it does not need to await a Task.

Create a new task that executes the synchronous code. The task will be executed by a thread of the threadpool.
private async Task DownloadBigImage()
{
await Task.Run(()=>
{
var url = "https://cosmos-magazine.imgix.net/file/spina/photo/14402/180322-Steve-Full.jpg";
new WebClient().DownloadFile(url, "image.jpg");
});
}

Synchronous I/O within an async/await-based Windows Service

Let's say I have a Windows Service which is doing some bit of work, then sleeping for a short amount of time, over and over forever (until the service is shut down). So in the service's OnStart, I could start up a thread whose entry point is something like:
private void WorkerThreadFunc()
{
while (!shuttingDown)
{
DoSomething();
Thread.Sleep(10);
}
}
And in the service's OnStop, I somehow set that shuttingDown flag and then join the thread. Actually there might be several such threads, and other threads too, all started in OnStart and shut down/joined in OnStop.
If I want to instead do this sort of thing in an async/await based Windows Service, it seems like I could have OnStart create cancelable tasks but not await (or wait) on them, and have OnStop cancel those tasks and then Task.WhenAll().Wait() on them. If I understand correctly, the equivalent of the "WorkerThreadFunc" shown above might be something like:
private async Task WorkAsync(CancellationToken cancel)
{
while (true)
{
cancel.ThrowIfCancellationRequested();
DoSomething();
await Task.Delay(10, cancel).ConfigureAwait(false);
}
}
Question #1: Uh... right? I am new to async/await and still trying to get my head around it.
Assuming that's right, now let's say that DoSomething() call is (or includes) a synchronous write I/O to some piece of hardware. If I'm understanding correctly:
Question #2: That is bad? I shouldn't be doing synchronous I/O within a Task in an async/await-based program? Because it ties up a thread from the thread pool while the I/O is happening, and threads from the thread pool are a highly limited resource? Please note that I might have dozens of such Workers going simultaneously to different pieces of hardware.
I am not sure I'm understanding that correctly - I am getting the idea that it's bad from articles like Stephen Cleary's "Task.Run Etiquette Examples: Don't Use Task.Run for the Wrong Thing", but that's specifically about it being bad to do blocking work within Task.Run. I'm not sure if it's also bad if I'm just doing it directly, as in the "private async Task Work()" example above?
Assuming that's bad too, then if I understand correctly I should instead utilize the nonblocking version of DoSomething (creating a nonblocking version of it if it doesn't already exist), and then:
private async Task WorkAsync(CancellationToken cancel)
{
while (true)
{
cancel.ThrowIfCancellationRequested();
await DoSomethingAsync(cancel).ConfigureAwait(false);
await Task.Delay(10, cancel).ConfigureAwait(false);
}
}
Question #3: But... what if DoSomething is from a third party library, which I must use and cannot alter, and that library doesn't expose a nonblocking version of DoSomething? It's just a black box set in stone that at some point does a blocking write to a piece of hardware.
Maybe I wrap it and use TaskCompletionSource? Something like:
private async Task WorkAsync(CancellationToken cancel)
{
while (true)
{
cancel.ThrowIfCancellationRequested();
await WrappedDoSomething().ConfigureAwait(false);
await Task.Delay(10, cancel).ConfigureAwait(false);
}
}
private Task WrappedDoSomething()
{
var tcs = new TaskCompletionSource<object>();
DoSomething();
tcs.SetResult(null);
return tcs.Task;
}
But that seems like it's just pushing the issue down a bit further rather than resolving it. WorkAsync() will still block when it calls WrappedDoSomething(), and only get to the "await" for that after WrappedDoSomething() has already completed the blocking work. Right?
Given that (if I understand correctly) in the general case async/await should be allowed to "spread" all the way up and down in a program, would this mean that if I need to use such a library, I essentially should not make the program async/await-based? I should go back to the Thread/WorkerThreadFunc/Thread.Sleep world?
What if an async/await-based program already exists, doing other things, but now additional functionality that uses such a library needs to be added to it? Does that mean that the async/await-based program should be rewritten as a Thread/etc.-based program?

Actually there might be several such threads, and other threads too, all started in OnStart and shut down/joined in OnStop.
On a side note, it's usually simpler to have a single "master" thread that will start/join all the others. Then OnStart/OnStop just deals with the master thread.
If I want to instead do this sort of thing in an async/await based Windows Service, it seems like I could have OnStart create cancelable tasks but not await (or wait) on them, and have OnStop cancel those tasks and then Task.WhenAll().Wait() on them.
That's a perfectly acceptable approach.
If I understand correctly, the equivalent of the "WorkerThreadFunc" shown above might be something like:
Probably want to pass the CancellationToken down; cancellation can be used by synchronous code, too:
private async Task WorkAsync(CancellationToken cancel)
{
while (true)
{
DoSomething(cancel);
await Task.Delay(10, cancel).ConfigureAwait(false);
}
}
Question #1: Uh... right? I am new to async/await and still trying to get my head around it.
It's not wrong, but it only saves you one thread on a Win32 service, which doesn't do much for you.
Question #2: That is bad? I shouldn't be doing synchronous I/O within a Task in an async/await-based program? Because it ties up a thread from the thread pool while the I/O is happening, and threads from the thread pool are a highly limited resource? Please note that I might have dozens of such Workers going simultaneously to different pieces of hardware.
Dozens of threads are not a lot. Generally, asynchronous I/O is better because it doesn't use any threads at all, but in this case you're on the desktop, so threads are not a highly limited resource. async is most beneficial on UI apps (where the UI thread is special and needs to be freed), and ASP.NET apps that need to scale (where the thread pool limits scalability).
Bottom line: calling a blocking method from an asynchronous method is not bad but it's not the best, either. If there is an asynchronous method, call that instead. But if there isn't, then just keep the blocking call and document it in the XML comments for that method (because an asynchronous method blocking is rather surprising behavior).
I am getting the idea that it's bad from articles like Stephen Cleary's "Task.Run Etiquette Examples: Don't Use Task.Run for the Wrong Thing", but that's specifically about it being bad to do blocking work within Task.Run.
Yes, that is specifically about using Task.Run to wrap synchronous methods and pretend they're asynchronous. It's a common mistake; all it does is trade one thread pool thread for another.
Assuming that's bad too, then if I understand correctly I should instead utilize the nonblocking version of DoSomething (creating a nonblocking version of it if it doesn't already exist)
Asynchronous is better (in terms of resource utilization - that is, fewer threads used), so if you want/need to reduce the number of threads, you should use async.
Question #3: But... what if DoSomething is from a third party library, which I must use and cannot alter, and that library doesn't expose a nonblocking version of DoSomething? It's just a black box set in stone that at some point does a blocking write to a piece of hardware.
Then just call it directly.
Maybe I wrap it and use TaskCompletionSource?
No, that doesn't do anything useful. That just calls it synchronously and then returns an already-completed task.
But that seems like it's just pushing the issue down a bit further rather than resolving it. WorkAsync() will still block when it calls WrappedDoSomething(), and only get to the "await" for that after WrappedDoSomething() has already completed the blocking work. Right?
Yup.
Given that (if I understand correctly) in the general case async/await should be allowed to "spread" all the way up and down in a program, would this mean that if I need to use such a library, I essentially should not make the program async/await-based? I should go back to the Thread/WorkerThreadFunc/Thread.Sleep world?
Assuming you already have a blocking Win32 service, it's probably fine to just keep it as it is. If you are writing a new one, personally I would make it async to reduce threads and allow asynchronous APIs, but you don't have to do it either way. I prefer Tasks over Threads in general, since it's much easier to get results from Tasks (including exceptions).
The "async all the way" rule only goes one way. That is, once you call an async method, then its caller should be async, and its caller should be async, etc. It does not mean that every method called by an async method must be async.
So, one good reason to have an async Win32 service would be if there's an async-only API you need to consume. That would cause your DoSomething method to become async DoSomethingAsync.
What if an async/await-based program already exists, doing other things, but now additional functionality that uses such a library needs to be added to it? Does that mean that the async/await-based program should be rewritten as a Thread/etc.-based program?
No. You can always just block from an async method. With proper documentation so when you are reusing/maintaining this code a year from now, you don't swear at your past self. :)

If you still spawn your threads, well, yes, it's bad. Because it will not give you any benefit as the thread is still allocated and consuming resources for the specific purpose of running your worker function. Running a few threads to be able to do work in parallel within a service has a minimal impact on your application.
If DoSomething() is synchronous, you could switch to the Timer class instead. It allows multiple timers to use a smaller amount of threads.
If it's important that the jobs can complete, you can modify your worker classes like this:
SemaphoreSlim _shutdownEvent = new SemaphoreSlim(0,1);
public async Task Stop()
{
return await _shutdownEvent.WaitAsync();
}
private void WorkerThreadFunc()
{
while (!shuttingDown)
{
DoSomething();
Thread.Sleep(10);
}
_shutdownEvent.Release();
}
.. which means that during shutdown you can do this:
var tasks = myServices.Select(x=> x.Stop());
Task.WaitAll(tasks);

A thread can only do one thing at a time. While it is working on your DoSomething it can't do anything else.
In an interview Eric Lippert described async-await in a restaurant metaphor. He suggests to use async-await only for functionality where your thread can do other things instead of waiting for a process to complete, like respond to operator input.
Alas, your thread is not waiting, it is doing hard work in DoSomething. And as long as DoSomething is not awaiting, your thread will not return from DoSomething to do the next thing.
So if your thread has something meaningful to do while procedure DoSomething is executing, it's wise to let another thread do the DoSomething, while your original thread is doing the meaningful stuff. Task.Run( () => DoSomething()) could do this for you. As long as the thread that called Task.Run doesn't await for this task, it is free to do other things.
You also want to cancel your process. DoSomething can't be cancelled. So even if cancellation is requested you'll have to wait until DoSomething is completed.
Below is your DoSomething in a form with a Start button and a Cancel button. While your thread is DoingSomething, one of the meaningful things your GUI thread may want to do is respond to pressing the cancel button:
void CancellableDoSomething(CancellationToken token)
{
while (!token.IsCancellationRequested)
{
DoSomething()
}
}
async Task DoSomethingAsync(CancellationToken token)
{
var task = Task.Run(CancellableDoSomething(token), token);
// if you have something meaningful to do, do it now, otherwise:
return Task;
}
CancellationTokenSource cancellationTokenSource = null;
private async void OnButtonStartSomething_Clicked(object sender, ...)
{
if (cancellationTokenSource != null)
// already doing something
return
// else: not doing something: start doing something
cancellationTokenSource = new CancellationtokenSource()
var task = AwaitDoSomethingAsync(cancellationTokenSource.Token);
// if you have something meaningful to do, do it now, otherwise:
await task;
cancellationTokenSource.Dispose();
cancellationTokenSource = null;
}
private void OnButtonCancelSomething(object sender, ...)
{
if (cancellationTokenSource == null)
// not doing something, nothing to cancel
return;
// else: cancel doing something
cancellationTokenSource.Cancel();
}

WCF Service Client with .NET 4.5 Task-Based Async Operations, await never returns

I've added a WCF service reference to my .NET 4.5 app, using the default "Generate task-based operations" option under "Allow Generation of asynchronous operations." I'm calling the service from an async method of my own, sort of like so:
public async Task<SomeData> GetDataAsync()
{
var client = new MyServiceClient();
var result = await client.GetSomeDataAsync();
return result;
}
The await client.GetSomeDataAsync() never completes (a breakpoint on the return statement never gets hit) and I don't get a time out or any other error, no exception is thrown, nothing. Fiddler shows that the client sent the request and the service responded almost instantly with the expected data, so the problem is on my side of the fence somehow.
If I switch to the synchronous version instead
var result = client.GetSomeData();
The call returns as expected.
What am I doing wrong?

My chest hairs are tingling, Mr. T. I strongly suspect that further up your (client-side) call stack, you have some code that is calling Task<T>.Result or Task.Wait, which significantly increases the possibility of a deadlock (as I explain on my blog).
By default, when you await a Task, the await will capture a "context" and use that to resume the async method. If this is a context like a UI thread context, and then the code blocks the UI thread (i.e., calling Result or Wait), then the async method cannot resume on that UI thread.
I pity the fool who would attempt to mix synchronous and asynchronous code. Just use async all the way (as I describe in an MSDN article).

What does CloudQueue.EndAddMessage(IAsyncResult) actually do?

Let's say I call
AsyncCallback callback = new AsyncCallback(QueueMessageAdded);
queue.BeginAddMessage(new CloudQueueMessage(message), callback, null);
where QueueMessageAdded is
private static void QueueMessageAdded(IAsyncResult result)
{
queue.EndAddMessage(result);
}
What does EndAddMessage do?
Including waiting for all callbacks to have been called, it is as slow as calling the synchronous version like this:
Parallel.ForEach(messages, message => queue.AddMessage(message));

First approach makes the request asynchronously and therefore your thread does not have to block while waiting for a response. Second approach, on the other hand, will use N threads, each of which will block until a response is received to its respective request.
Please refer to Asynchronous Programming Model (APM) for more information. All End* methods complete the asynchronous operation, meaning it will block until the operation finishes, return the operation's result if any, and do clean-up.

The first approach allow you to use concurrent requests! A single thread, can, with the first approach send hundreds of concurrent messages, even though the latency of a single POST request to get its reply is high. If you look at production code targeting ASB you can see some patterns in how APM/Async is used.