I had a developer challenge the use of asynchronous server side code the other day. He asked why asynchronous code is superior to synchronous code on a server with no UI thread to block? I gave him the typical thread exhaustion answer but after thinking about it for a while I was no longer sure my answer was correct. After doing a little research I found that the upper limit to threads in an OS is governed by memory not an arbitrary number. And servers like Kestrel support unlimited threads. So in "theory" the number of requests (threads) a server can block on in parallel is governed by memory. Which is no different than async code in .NET; it lifts stack variables to the heap but it's still memory bound.
I've always assumed that smarter people than me had thought this through and async code was the right way to handle IO bound code. But what are the measurable advantages of async .NET code when running in a dedicated server farm with no UI thread? Does a move to the cloud (AWS) change the answer?
Server-side asynchronous code purpose is completely different from asynchronous UI code.
Asynchronous UI code makes UI more responsive (especially when multiple CPU cores are available), it allows multiple UI tasks to run in parallel which improves UI user experience.
The purpose of server-side asynchronous code on the other hand is to minimise the resources necessary to serve multiple clients simultaneously. In fact it is beneficial even if there is only one CPU core or a single-threaded event loop like in Node.js. And it all boils down to a simple concept of
Asynchronous IO.
The difference between synchronous and asynchronous IO is that in case of the former the thread which initialises an IO operation is paused until the IO operation is completed (e.g. until DB request is executed or a file on a disk is read). The same thread is then un-paused once the IO operation is completed to process the result of it. Note: even though while paused the thread is most likely not using any CPU resources (it is probably put to sleep by a thread scheduler) its resources are still tied to this particular IO operation and are pretty much wasted while IO is executed by the hardware. Effectively with synchronous IO you will need at least one thread per currently being processed client request even though most of those threads are probably asleep waiting for their IO operations to complete. In .NET each thread has at least 1MB of stack allocated so if the server is currently processing say 1000 requests it leads to almost 1GB of memory allocated simply for thread stacks plus an additional burden for a thread scheduler and more time CPU spends doing context switches: the more threads there are the slower overall performance of the system. More memory allocated means less efficient memory/CPU caches usage too.
Asynchronous IO is more efficient because a worker thread only initialises an IO operation and instead of waiting for it to complete it is immediately switched to another useful task (e.g. continuation of another client's request processing) and when the IO operation is completed by the hardware the processing of the result is resumed on any available worker thread. As a result, depending on the ratio between overall time spent waiting for hardware to complete IO and the time spent doing CPU tasks (e.g. serialisation of the result of IO operation into JSON) this approach can use less threads to serve the same number of simultaneous client requests: if, say, 90% of the time is spent in IO we can potentially use only 100 thread to serve the same 1000 simultaneous requests. The more your server-side code is IO-bound vs CPU-bound the more simultaneous clients requests it can process using a given amount of resources: CPU and memory.
What is the drawback of asynchronous code? Mainly it is generally harder to write than synchronous. Asynchronous code uses callbacks to resume operation so instead of a simple linear code a programmer needs to pass a delegate (a continuation) to IO method which is later called by the system when IO operation is completed (potentially on a different thread). However modern C# with its async/await facilities makes this task less complicated and even makes asynchronous code to almost look like synchronous. The only thing to remember: the asynchronous code only works when it is asynchronous "all the way down": even a single Task.Wait or Task.Result somewhere in the stack of calls from initial HTTP request processing to DB request call makes entire code synchronous thus forcing the current working thread to wait for that Wait call to finish defeating the purpose. Note: await in C# code does not actually awaits to the result of the call but is converted by the compiler to a ContinueWith i.e. to a continuation callback though in practice it is a bit more complicated than that but luckily the complexity is hidden from a programmer so nowadays writing efficient asynchronous code is relatively straightforward task.
My understanding is that there is a difference between asynchronous IO and truly non-blocking IO.
asynchronous blocking IO means that there is a thread waiting for the IO to finish. (It isn't the main thread, so the app stays responsive.) Because the thread is waiting, the CPU will put the thread to sleep and so it won't eat up processing power, but this still takes up a thread which assumedly uses some resources.
With truly non-blocking IO, when you wait for a File System or network request, the system actually frees up that thread to go do other things and somehow notifies your program once the request is completed. This saves you whatever resources a sleeping thread takes up.
Are C# functions like System.IO.File.ReadAllBytesAsync and HttpClient.PostAsync truly non-blocking?
Is there any way to tell which async functions in C# are non-blocking?
If the async functions are blocking (they cause a thread to wait) then it seems like there would be no advantage to using async/await even in a webserver that handles lots of requests because all the operations take up threads anyway.
The .Net Socket async API manages threads automatically when using the BeginXXX methods. For example, if I have 100 active connections sending and receiving TCP messages, will be used around 3 threads. And it makes me curious.
How the API makes this thread management?
How all flow of connections are divided among the threads to be processed?
How the manager prioritizes which connections/readings/writings must be processed first?
My questions may not have sense because I don't know how it works and what to ask specifically, so sorry. Basically I need to know how this whole process works in low level.
The .Net Socket async API manages threads automatically when using the
BeginXXX methods.
This is not quite correct. APM Begin/End-style socket API do not manage threads at all. Rather, the completion AsyncCallback is called on a random thread, which is the thread where the asynchronous socket I/O operation has completed. Most likely, this is going to be an IOCP pool thread (I/O completion port thread), different from the thread on which you called the BeginXXX method. For more details, check Stephen Cleary's "There Is No Thread".
How the manager prioritizes which connections/readings/writings must
be processed first?
The case when there's no IOCP threads available to handle the completion of the async I/O operation is called TheadPool starvation. It happens when all pool threads are busy executing some code (e.g., processing the received socket messages), or are blocked with a blocking call like WaitHandle.WaitOne(). In this case, the I/O completion routine is queued to ThreadPool to be executed when a thread becomes available, on FIFO basis.
You have an option to increase the size of ThreadPool with SetMinThreads/SetMaxThreads APIs, but doing so isn't always a good idea. The number of actual concurrent threads is anyway limited by the number of CPU/cores, so you'd rather want to finish any CPU-bound processing work as soon as possible and release the thread to go back to the pool.
I wonder whether existing I/O bound APM calls in .net API (BeginGetResponse, BeginRead, etc.) uses a thread from threadpool or uses the current thread until the callback. I know that it is "async" all the way down to the hardware/network card. I also know that the callback is executed on threadpool. My question is that: All contents of BeginGetResponse are executed on Threadpool or the contents until waiting for I/O are executed on current thread; then the rest is executed on threadpool.
I hope that the question is clear. I really wonder how BeginGetResponse is implemented underhood.
APM is more general mechanism. But the cases you are talking about use the operating system's support for I/O completion ports. The general idea is that your main thread calls the BeginXxx() method. Under the hood, it calls ThreadPool.BindHandle(), that sets up the plumbing to get the port to automatically start a TP thread when the I/O operation completes. That thread calls your callback method.
Core idea that no thread is waiting while the I/O operation takes place.
This is supported for MessageQueue, FileStream, PipeStream, Socket, FileSystemWatcher, IpcChannel and SerialPort.
BeginXxx execute on the current thread. You can easily verify this for yourself using e.g. Reflector. Moreover, sometimes the callback is executed on the current thread too. One case is if an error occurs early, and another is when the actual asynchronous I/O operation blocks — this happens sometimes, as asynchronous I/O is not guaranteed not to block.
The IAsyncResult approach using worker pool threads is available only for some tasks. Like FileIO (not directory enumeration), LDAP query (v2.0), ADO .net queries.
If you have it and can take the complexity, use the APM. They are usually built by .net folks as it takes some complexity.
Otherwise, use hand built if you think you will get speed.
Using explicit threads gives you more control. Specifically, you can choose to have foreground threads, which will keep your application "alive" after the main thread returns from Main. Explicit threads can also specify their COM threading apartment.
The general rule is to use the threadpool when you have a queue of work items to do, and use an explicit thread when you have an architectural need for them.
Many operations use IO completion ports.
This means that no thread is used while waiting for the operation. Once the operation is complete, the callback is called on a thread-pool thread or using some other synchronization context.
I've been reading some async articles here: http://www.asp.net/web-forms/tutorials/aspnet-45/using-asynchronous-methods-in-aspnet-45 and the author says :
When you’re doing asynchronous work, you’re not always using a thread.
For example, when you make an asynchronous web service request,
ASP.NET will not be using any threads between the async method call
and the await.
So what I am trying to understand is, how does it become async if we don't use any Threads for concurrent execution? What does it mean "you're not always using a thread."?
Let me first explain what I know regarding working with threads (A quick example, of course Threads can be used in different situations other than UI and Worker methodology here)
You have UI Thread to take input, give output.
You can handle things in UI Thread but it makes the UI unresponsive.
So lets say we have a stream-related operation and we need to download some sort of data.
And we also allow users to do other things while it is being downloaded.
We create a new worker thread which downloads the file and changes the progress bar.
Once it is done, there is nothing to do so thread is killed.
We continue from UI thread.
We can either wait for the worker thread in UI thread depending on the situation but before that while the file is being downloaded, we can do other things with UI thread and then wait for the worker thread.
Isn't the same for async programming? If not, what's the difference? I read that async programming uses ThreadPool to pull threads from though.
Threads are not necessary for asynchronous programming.
"Asynchronous" means that the API doesn't block the calling thread. It does not mean that there is another thread that is blocking.
First, consider your UI example, this time using actual asynchronous APIs:
You have UI Thread to take input, give output.
You can handle things in UI Thread but it makes the UI unresponsive.
So lets say we have a stream-related operation and we need to download some sort of data.
And we also allow users to do other things while it is being downloaded.
We use asynchronous APIs to download the file. No worker thread is necessary.
The asynchronous operation reports its progress back to the UI thread (which updates the progress bar), and it also reports its completion to the UI thread (which can respond to it like any other event).
This shows how there can be only one thread involved (the UI thread), yet also have asynchronous operations going on. You can start up multiple asynchronous operations and yet only have one thread involved in those operations - no threads are blocked on them.
async/await provides a very nice syntax for starting an asynchronous operation and then returning, and having the rest of the method continue when that operation completes.
ASP.NET is similar, except it doesn't have a main/UI thread. Instead, it has a "request context" for every incomplete request. ASP.NET threads come from a thread pool, and they enter the "request context" when they work on a request; when they're done, they exit their "request context" and return to the thread pool.
ASP.NET keeps track of incomplete asynchronous operations for each request, so when a thread returns to the thread pool, it checks to see if there are any asynchronous operations in progress for that request; if there are none, then the request is complete.
So, when you await an incomplete asynchronous operation in ASP.NET, the thread will increment that counter and return. ASP.NET knows the request isn't complete because the counter is non-zero, so it doesn't finish the response. The thread returns to the thread pool, and at that point: there are no threads working on that request.
When the asynchronous operation completes, it schedules the remainder of the async method to the request context. ASP.NET grabs one of its handler threads (which may or may not be the same thread that executed the earlier part of the async method), the counter is decremented, and the thread executes the async method.
ASP.NET vNext is slightly different; there's more support for asynchronous handlers throughout the framework. But the general concept is the same.
For more information:
My async/await intro post tries to be both an intro yet also reasonably complete picture of how async and await work.
The official async/await FAQ has lots of great links that go into a lot of detail.
The MSDN magazine article It's All About the SynchronizationContext exposes some of the plumbing underneath.
First time when I saw async and await, I thougth they were C# Syntactic sugar for Asynchronous Programming Model. I was wrong, async and await are more than that. It is a brand new asynchronous pattern Task-based Asynchronous Pattern, http://www.microsoft.com/en-us/download/details.aspx?id=19957 is a good article to get start. Most of the FCL classes which inplement TAP are call APM methods (BegingXXX() and EndXXX()). Here are two code snaps for TAP and AMP:
TAP sample:
static void Main(string[] args)
{
GetResponse();
Console.ReadLine();
}
private static async Task<WebResponse> GetResponse()
{
var webRequest = WebRequest.Create("http://www.google.com");
Task<WebResponse> response = webRequest.GetResponseAsync();
Console.WriteLine(new StreamReader(response.Result.GetResponseStream()).ReadToEnd());
return response.Result;
}
APM sample:
static void Main(string[] args)
{
var webRequest = WebRequest.Create("http://www.google.com");
webRequest.BeginGetResponse(EndResponse, webRequest);
Console.ReadLine();
}
static void EndResponse(IAsyncResult result)
{
var webRequest = (WebRequest) result.AsyncState;
var response = webRequest.EndGetResponse(result);
Console.WriteLine(new StreamReader(response.GetResponseStream()).ReadToEnd());
}
Finally these two will be the same, because GetResponseAsync() call BeginGetResponse() and EndGetResponse() inside. When we reflector the source code of GetResponseAsync(), we will get code like this:
task = Task<WebResponse>.Factory.FromAsync(
new Func<AsyncCallback, object, IAsyncResult>(this.BeginGetResponse),
new Func<IAsyncResult, WebResponse>(this.EndGetResponse), null);
For APM, in the BeginXXX(), there is an argument for a callback method which will invoked when the task (typically is an IO heavy operation) was completed. Creating a new thread and asynchronous, both of them will immediately return in main thread, both of them are unblocked. On performance side, creating new thread will cost more resource when process I/O-bound operations such us read file, database operation and network read. There are two disadvantages in creating new thread,
like in your mentioned article, there are memory cost and CLR are
limitation on thread pool.
Context switch will happen. On the other hander, asynchronous will
not create any thread manually and it will not have context switch
when the the IO-bound operations return.
Here is an picture which can help to understand the differences:
This diagram is from a MSDN article "Asynchronous Pages in ASP.NET 2.0", which explain very detail about how the old asynchronous working in ASP.NET 2.0.
About Asynchronous Programming Model, please get more detail from Jeffrey Richter's article "Implementing the CLR Asynchronous Programming Model", also there are more detail on his book "CLR via Csharp 3rd Edition" in chapter 27.
Let’s imagine that you are implementing a web application and as each client request comes in to
your server, you need to make a database request. When a client request comes in, a thread pool
thread will call into your code. If you now issue a database request synchronously, the thread will block
for an indefinite amount of time waiting for the database to respond with the result. If during this time
another client request comes in, the thread pool will have to create another thread and again this
thread will block when it makes another database request. As more and more client requests come in,
more and more threads are created, and all these threads block waiting for the database to respond.
The result is that your web server is allocating lots of system resources (threads and their memory) that
are barely even used!
And to make matters worse, when the database does reply with the various results, threads become
unblocked and they all start executing. But since you might have lots of threads running and relatively
few CPU cores, Windows has to perform frequent context switches, which hurts performance even
more. This is no way to implement a scalable application.
To read data from the file, I now call ReadAsync instead of Read. ReadAsync internally allocates a
Task object to represent the pending completion of the read operation. Then, ReadAsync
calls Win32’s ReadFile function (#1). ReadFile allocates its IRP, initializes it just like it did in the
synchronous scenario (#2), and then passes it down to the Windows kernel (#3). Windows adds the IRP
to the hard disk driver’s IRP queue (#4), but now, instead of blocking your thread, your thread is
allowed to return to your code; your thread immediately returns from its call to ReadAsync (#5, #6,
and #7). Now, of course, the IRP has not necessarily been processed yet, so you cannot have code after
ReadAsync that attempts to access the bytes in the passed-in Byte[].