We Keep Coding

How to consume an awaitable in C++/Cli

async/await gained a lot of popularity in the C# world the past few years. Async functions also tend to spread rapidly within an application: Awaitables need to be awaited, therefore the calling function must be async and is therefore also an awaitable which needs to be awaited,... and so forth.
I'm using a C# library in a C++/Cli project. The library exposes async APIs. The Visual C++ compiler does not support async/await. Hence I have no way to await any of the APIs the library gives me.
I have the following options:
Call the async function and "letting it go": Not an option since I often need the return values to proceed.
Calling Wait() or accessing the Result property of the Task/Task<T> objects the async function returns: Causes the infamous dead-lock of the UI thread.
Is there any way to make this work? I wouldn't mind executing the async APIs synchronously if I have to.

To me, it sounds like your problem is more of a symptom of a bigger issue. I personally try to avoid as much as I can putting business logic in a C++-CLI module. Specifically, I try to limit a ref class functionality to 3 main areas:
Translating managed method calls to native method calls (that includes transforming parameters if necessary).
Translating native return values to managed return values. This can even be translating an asynchronous function receiving a callback to a method returning Task.
Translating native events (callbacks) to managed events.
I Might be wrong, but in your situation it sounds like your C++ code is not well decoupled and you end up wiring different parts of it inside a C++-CLI module.
As for your question, I would use Task.ContinueWith family of methods to perform the asynchronous continuation instead of async / await. Yet, If you want the continuation to be invoked on a specific SynchronizationContext (such as UI thread) then special care must be taken to supply the right TaskScheduler.

Must async methods be supported by OS or is async program level feature

Tutorials sometimes point implementation of own async methods as for example this code:
async public static Task GetHttpResponseAsync()
{
using (HttpClient httpClient = new HttpClient())
{
HttpResponseMessage response = await httpClient.GetAsync(...);
Console.WriteLine(response.Something);
}
}
what is clear to me is how async work generally, but none of tutorials explains how internally are implemented
httpClient.GetAsync(...);
which is really important to understand how asynchoronus code works in details. What make me curious is if internall operations of GetAsync (those method or other async method) are registered in some kind of container where this code is executed? Are async methods must be supported by operating system (f.e it uses windows api)? If I would like to implement my own asynchronous file downloader (from disk and without essential part from .NET framework), how would I implement it, should I register my method somewhere for further invocation?
It's pretty clear for me that internally, compiler makes state machine and after DoSomething() method do what it have to, it just invoke this state machine again to resume executing code after await.
Also what is unclear for me is that how async code can run on same thread. I think that maintaining state machine must be on the same thread but how the code from httpClient.GetAsync() can be run on the same thread and doesn't interrupt other operations (f.e gui). There must be something that make this code runs on separate thread (in all cases). Am I wrong? What I missed?
Additional explanation of my question: In JavaScript, as far as I know and understand, async methods works by registering them in some kind of container (which runs them one by one on separate thread) which executes this method. After execution of the method complete, result is returned to user context, It's clear for me, Is that work in the same way here?

In short, true asynchrony must be provided at the OS level. As you've noted, async/await is a language level feature that uses compiler generated state machines to "break up" your method into pieces that can run asynchronously, but it relies on OS primitives (interrupts, threads) to actually perform this work in an asynchronous manner.
However, it's important to note that there will often not be a thread created to handle your async operation. I'll defer to this expertly-written article to describe why this is the case: http://blog.stephencleary.com/2013/11/there-is-no-thread.html
On Windows, the primary mechanism for performing asynchronous work is with I/O Completion Ports. This is a Windows API that is used under the hood by many .NET types, including the HttpClient you're using.
Note that for non-I/O operations, you can also always use Threads, or better yet, the Thread Pool API, to perform background work that will complete asynchronously.
The ReadFile (or newer ReadFileEx) function in the Windows API is designed to work with async I/O. When you call ReadFile, you can use the FILE_FLAG_OVERLAPPED flag and pass an OVERLAPPED structure to the lpOverlapped argument, which enables async reads. I would encourage you to use this API to design your file downloader.
In summary:
There will not always be a thread created for async operations.
async/await is a language feature, but relies on various Windows APIs to achieve true asynchrony.
If the work you are doing is I/O bound, consider using Asynchronous I/O: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365683(v=vs.85).aspx
If the work you are doing is CPU bound, consider using Threading: https://msdn.microsoft.com/en-us/library/windows/desktop/ms684841(v=vs.85).aspx
Now that .NET Core is open source on GitHub, you can actually inspect the source code to see what's going on under the hood. Here is HttpClient.cs, which uses HttpMessageHandler.cs -> HttpClientHandler.Windows.cs -> WinHttpHandler.cs -> Interop.winhttp.cs -> PInvoke into the WinHTTP API native DLL -> Winsock sockets with an I/O completion port.

Best solution for async chicken and egg story

I have been applying async best practices to all my libraries. Basically it means:
Only use async when it's truly async (libraries shouldn't lie)
Define a synchronous method if and only if you have a faster synchronous method that won’t dead lock.
Postfix all async methods with Async
I worked on a library that is synchronous by nature. This means it has only sync methods. If the user wants to run the work on a separate thread than the UI thread, they can do that themselves by using Task.Factory (responsibility of the caller).
However, inside a handler / method / extensibility point, we want to show the user a message box. This is an async method (for example, WinRT ShowDialogAsync). Then this gives us the following options:
A. Move everything to async (so we have the option to use await in our handlers and don't block anything).
public async Task MyMethodAsync()
{
await _messageService.ShowAsync();
}
The advantage is that users can add async methods without having to use .Wait(). The downside is that we are lying as a library (it's not truly async).
I have considered making everything async, but I don't think that's a good idea either. It would make all libraries lie but prepare them in case we would need it. Remember that making everything async out of the box has a (small) performance impact as well.
B. Inside the handler that requires user input, call .Wait()
public void MyMethod()
{
_messageService.ShowAsync().Wait();
}
The advantage is that this will allow us to use async code inside sync methods. But... it will never be callable from the UI-thread because the _messageService dispatches to the UI thread (but it cannot do that because it's still waiting for the method, resulting in a deadlock). This method will work when used inside a Task.Factory.Run block (but the responsibility is up to the end-user):
await Task.Factory.Run(() => MyMethod());
The question
I feel that both have pros and cons, but what would you choose? Let the library lie (A) or only allow the method to be called from a background thread (B)? Or maybe there are other options I've overseen.
If I go for A, it means I have to bump the major version every time (because it's actually a breaking change) whenever a user requests to convert a method to an async signature method.

Define a synchronous method if and only if you have a faster synchronous method that won’t dead lock.
I'd say "define a synchronous method if you have synchronous work to do". It doesn't matter how fast it is. The burden is on the caller to determine if it's too slow and they need to use Task.Run.
However, inside a handler / method / extensibility point
If this is an Observer kind of extensibility, consider just using events or observables.
However, it sounds like you want more of a Strategy kind of extensibility, where your invoking code must wait for and/or change its behavior based on the result of the callback.
I have considered making everything async, but I don't think that's a good idea either.
Async all the way is a guideline, not a strict command. It definitely applies in the 99% case, but this could be one of the exceptions. I would try not to make a library async just for the sake of a possibly-async Strategy pattern; I'd investigate other extension possibilities first. There is a valid argument for making the library async, if you view the Strategy callback as a dependency (the library would be async because its dependency is (possibly) async).
As you've discovered, there's no clean way to do sync-over-async. There are a few different hacks (such as blocking from a background thread), but you'll first need to decide whether you need to call your library from the UI thread.
If you do, then there's just two options: make the library async, or use a nested message loop. I strongly avoid nested message loops, especially in libraries; I'm just mentioning it for sake of completeness.
If you can impose on the user a requirement to only call the library from a non-UI thread, then you can apply other hacks. E.g., blocking the background thread.
There's not an easy solution, sorry.
As far as me personally... if the library needs an async Strategy, then I would lean towards making the library async. But it does depend on what kind of library it is, whether there were backwards-compatibility issues, etc. And the first thing I'd look into is a different kind of extensibility point.

as you can read here :
https://msdn.microsoft.com/en-us/magazine/jj991977.aspx
Async All the Way
Asynchronous code reminds me of the story of a fellow who mentioned that the world was suspended in space and was immediately challenged by an elderly lady claiming that the world rested on the back of a giant turtle. When the man enquired what the turtle was standing on, the lady replied, “You’re very clever, young man, but it’s turtles all the way down!” As you convert synchronous code to asynchronous code, you’ll find that it works best if asynchronous code calls and is called by other asynchronous code—all the way down (or “up,” if you prefer). Others have also noticed the spreading behavior of asynchronous programming and have called it “contagious” or compared it to a zombie virus. Whether turtles or zombies, it’s definitely true that asynchronous code tends to drive surrounding code to also be asynchronous. This behavior is inherent in all types of asynchronous programming, not just the new async/await keywords.
“Async all the way” means that you shouldn’t mix synchronous and asynchronous code without carefully considering the consequences. In particular, it’s usually a bad idea to block on async code by calling Task.Wait or Task.Result. This is an especially common problem for programmers who are “dipping their toes” into asynchronous programming, converting just a small part of their application and wrapping it in a synchronous API so the rest of the application is isolated from the changes. Unfortunately, they run into problems with deadlocks. After answering many async-related questions on the MSDN forums, Stack Overflow and e-mail, I can say this is by far the most-asked question by async newcomers once they learn the basics: “Why does my partially async code deadlock?”

Correct way to rewrite my Utils library with Async/Await

I have a lot of Utils classes that I use for many different projects, most of them are static and made out of static methods that usually don't even call each others.
My intention is to take advantages from the new async/await features but without rewriting everything, so my question is: can I just add a new method for each existing method named MethodAsync with the Task.Run(() => MethodName)?
example:
//old code that will not be removed
static void DoSomething()
{ ... }
//new code that will be added
static async Task DoSomethingAsync()
{
//not really sure if Im supposed to use await/async here.
//isn't Task awaitable even without async??
return await Task.Run(() => DoSomething());
}
Basically in the old code I just had a normal sync method while in the new one I have an async method that could even run in another thread if the CLR see it as a CPU-bound method.
If I correctly understand, every asyncronous method contains by definition an await to an awaitable object which is a Task or another asyncronous method.
that means that whenever I can use an async .NET method, I should await it and mark my caller method as async.
But, every other method that do not call any async method but could take some time to complete should be called with a Task.Run call.
right?
EDIT
so I have read all the posted links, the best practices on msdn and a few blog posts but I still need a complete routine to follow when coding with the new async/await feature.
this is what I get so far:
1) every .NET method that has an async alternative should use the async alternative. (As far as I know the .NET async methods already exists ONLY for methods that can be async).
2) every method that use async methods should be made async too.
3) every method that do not use async methods (cause there aren't available) but still takes some cpu-time to execute should be made async by wrapping them using Task.Run (I understand that in this case it should be the client to use Task.Run if they want but since im only adding these wrappers for methods that takes more than 50ms to execute and there will be still available the non-async version of the method, I still don't see why I shouldn't place this wrapper in the library).
4) every method that takes non-cpu-time cause it's waiting for other sources (like internet, database, events, etc...) should use TaskFactory.FromAsync or TaskCompletionSource.
5) System.Threading.Tasks.Parallel.Invoke(method1, method2, etc...) is now deprecated. From what I read Task.Run already run concurrent threads if the CLR thinks that concurrency is required. So it seems that Task.Run already uses Parallel.Invoke when needed.

I was finally able to find good resources that cleared all my doubts:
the first one is "The Task-based Asynchronous Pattern" available at http://www.microsoft.com/en-us/download/details.aspx?id=19957
This document explains the async/await feature, how/when to use it, it contains many practical examples and a few of very usefull static methods that im now using on every project!
The second one is "The zen of async: Best practices for best performance" available at http://channel9.msdn.com/Events/BUILD/BUILD2011/TOOL-829T
which is a complete overview of the async feature, with some coverage of the Parallel features and it also explains why it should never be the library that use Task.Run() method but it should be a consumer choice instead.
So in the end I was really confusing multithreading with asynchronous code and I couldn't grasp the advantages of async code because I was just looking at 1 single method at the time while the real benefits of async code can only be seen if the entire project (or at least a consistent portion of it) is written following the async pattern.
For example in asp.net, if there isn't any blocking code (everything written in a async fashion way), then a thread can serve another client while yours is awaiting an async operation, improving scalability, while in a xaml application a thread that launch an async operation can immediately go back to supporting your UI instead of just waiting that operation to be over, improving responsiveness.

This has been discussed in Should I expose asynchronous wrappers for synchronous methods? The part of this article that I thought highlighted why this is bad design is below:
Consider, for example, a simple method like Dictionary<TKey,TValue>.Add(TKey,TValue). This is a really fast method, right? Typically, yes, but remember how dictionary works: it needs to hash the key in order to find the right bucket to put it into, and it needs to check for equality of the key with other entries already in the bucket. Those hashing and equality checks can result in calls to user code, and who knows what those operations do or how long they take. Should every method on dictionary have an asynchronous wrapper exposed? That’s obviously an extreme example, but there are simpler ones, like Regex. The complexity of the regular expression pattern provided to Regex as well as the nature and size of the input string can have significant impact on the running time of matching with Regex, so much so that Regex now supports optional timeouts… should every method on Regex have an asynchronous equivalent? I really hope not.
Obviously I advise you to read the entire article too, but I hope the above highlights a great reason why exposing *Async() methods should not wrap synchronous ones in a library.
I hope this helps.

Asynchronous methods and asynchronous delegates

C# 3.0 in a nutshell says asynchronous methods and asynchronous delegates looks similar but the behavior is very different.
Here is what the book says about both.
Asynchronous methods
Rarely or never blocks any thread.
Begin method may not immediately return to the caller.
An agreed protocol with no C# language support.
Asynchronous delegates
May block for any length of time
BeginInvoke return immediately to the caller.
Built-in compiler support.
The book also says, The purpose of asynchronous methods is to allow many tasks to run on few threads; the purpose of asynchronous delegates is to execute a task in parallel with the caller.
When I looked into the BeginRead() method in System.IO.Stream class through reflector, it is using a delegate and calling BeginInvoke on that. So an asynchronous method is using an asynchronous delegate internally.
In such case, how can one say their behaviors are different? Since it uses delegates internally, how a comparison like the above is possible?
Do you think working with a delegate's BeginXXX method is the way to execute a function in parallel to the caller?
What is the proper way to implement asynchronous methods by maintaining all the advantages like making good use of CPU?
Any thoughts?

At the core, there are two main behaviors you may see when you call BeginFoo() with a callback.
Work is started on a background thread, and that thread will be used the entire time up until the work is completed and the callback is invoked (e.g. because the work is synchronous).
Though some work happens on a background thread, the thread need not be in use the entire time (e.g. because the work involves System IO which can schedule callbacks on e.g. the IOCompletionPort).
When you use a delegate, behavior #1 above happens.
Some APIs (that have underlying support for non-blocking IO calls) support behavior #2.
In the specific case of 'Stream', I am not sure, but my guess is it is an abstract base class and so this is merely the default behavior for a subclass that implements only a synchronous version of Read. A 'good' subclass would override BeginRead/EndRead to have a non-blocking implementation.
The advantage of #2, as you said, is that you can have e.g. 100 pending IO calls without consuming 100 threads (threads are expensive).

The implementation can be different; for example, an async IO call may choose to make use of completion ports to minimise the cost to the system while not doing anything.
It is certainly a way; you could also use BackgroundWorker, ThreadPool.QueueUserWorkItem, or Parallel.For (etc) in .NET 4.0
Varies per implementation
I think what the book is trying to highlight is that delegates always include this pattern:
a synchronous call (Invoke) that can block
an async call (BeginInvoke) that shouldn't really block unless the thread-pool is saturated
but it isn't the only pattern. Also; more recently (for example, the async IO methods in Silverlight, or in WebClient): rather than an IAsyncResult, an event is used to signal completion.