We Keep Coding

.NET Task Performance with 1000s of blocked Tasks

I have some .NET4 code that needs to know if/when a network request times out.
Is the following code going to cause a new Thread to be added to the .NET ThreadPool each time a task runs, and then release it when it exits?
var wait = new Task(() =>
{
using (var pauseEvent = new ManualResetEvent(false))
pauseEvent.WaitOne(TimeSpan.FromMilliseconds(delay));
}).ContinueWith(action);
wait.Start()
https://stackoverflow.com/a/15096427/464603 suggests this approach would work, but have performance implications for the general system.
If so, how would you recommend handling a high number of request timeouts/s - probably 1000timeouts/s when bursting?
In Python I have previously used something like a tornado IOLoop to make sure this isn't heavy on the Kernel / ThreadPool.

I have some .NET4 code that needs to know if/when a network request times out.
The easiest way to do this is to use a timeout right at the API level, e.g., WebRequest.Timeout or CancellationTokenSource.CancelAfter. That way the operation itself will actually stop with an error when the timeout occurs. This is the proper way to do a timeout.
Doing a timed wait is quite different. (Your code does a timed wait). With a timed wait, it's only the wait that times out; the operation is still going, consuming system resources, and has no idea that it's supposed to stop.
If you must do a timed wait on a WaitHandle like ManualResetEvent, then you can use ThreadPool.RegisterWaitForSingleObject, which allows a thread pool thread to wait for 31 objects at a time instead of just one. However, I would consider this a last-ditch extreme solution, only acceptable if the code simply cannot be modified to use proper timeouts.
P.S. Microsoft.Bcl.Async adds async/await support for .NET 4.
P.P.S. Don't ever use StartNew or ContinueWith without explicitly specifying a scheduler. As I describe on my blog, it's dangerous.

First of all, adding Tasks to Thread Pool doesn't necessarily cause new Thread to be added to Thread Pool. When you add a new Task to Thread Pool it is added to internal queue. Existing Threads from Thread Pool take Tasks from this queue one by one and execute them. Thread Pool will start new Threads or stop them as it deems appropriate.
Adding Task with blocking logic inside will cause Threads from Thread Pool to block. It means that they won't be able to execute other Tasks from queue, which will lead to performance issues.
One way to add delay to some action is to use Task.Delay method which internally uses timers.
Task.Delay(delay).ContinueWith(action);
This will not block any Threads from Thread Pool. After specified delay, action will be added to Thread Pool and executed.
You may also directly use timers.
As someone suggested in comment, you may also use async methods. I believe the following code would be equivalent of your sample.
public async Task ExecuteActionAfterDelay()
{
await Task.Delay(3000);
action();
}
You might also want to look at this question Asynchronously wait for Task<T> to complete with timeout.

With C# and .NET, why doesn't the runtime automatically operate asynchronously?

This is such a basic (read noob) question regarding the .NET async/await library but I thought I'd ask it anyhow before rewriting our api's to be awaitable.
The Question
Why wouldn't the runtime simply evaluate any given thread that has a lot of idle time and automatically operate asynchronous whenever it gets to the blocking call.
Example of some routine
Web request to some app...
App starts database call...
Wait for response...(idle and long)
Receive recordset...
Return to client...
If I were the runtime environment, wouldn't it be wise to simply jot down that step 3 takes a while so I should use the current thread at that point, during its idle moments, to help out other routines that would normally be waiting for our current thread to be available?
Isn't it possible that at some point in the future we'll be able to toggle a flag in the app.config (or web.config) that says <system.runtime><asyncBehavior enableAsynchronousWhenIdle=true /></system.runtime>?

Sure it's possible but it completely breaks the current programming model. Before when you had a blocking call you were guaranteed that no other code would run on your thread. This change now allows re-entrant calls on the same thread.
For instance consider this case:
static int _processCount;
static object _lockObj = new object();
public Response ProcessRequest(Request request) {
lock (_lockObj) {
_processCount++;
var savedCount = _processCount;
// Make long running request
if (savedCount != _processCount)
throw new InvalidOperationException("Is my lock broken?");
}
}
Before we allow processing requests during the long running process this code is fine, but if we allow new requests to be processed on the thread while it is making a long running request we open up the possibility of this case.
Process Request A
Process Request A waits for the long running operation
The idle processing uses the thread to process Request B.
Request B enters the lock because locks have thread affinity
Request B waits for the long running operation
Request A returns from the long running operation and throws an exception because it's state has been corrupted.
So the code needs to be written in such a way that it is aware of the reentrancy potential. There is no way for the Framework to know if your code will break so that change will never happen.

.NET (nor any framework) isn't that smart. Unless you explicitly program code to run asynchronously, it has no way of knowing if any particularly code should run asynchronously. It can't look at your code and say, "Oh here's some code that runs awhile and blocking the UI thread, so I should run this in a separate thread so that the UI can update." As far as the framework is concerned--you intended it to operate that way--and it has no intelligence to override the way you coded into something more efficient.

Basically because both synchronous and asynchronous approaches are useful, and sometimes blocking is fine. Asynchrony isn't a silver bullet.
In the other hand, turning all synchronous code into asynchronous operations may break a lot of code base, both from the base class library (BCL) and third-party code, because asynchronous operations should synchronize access to shared resources and objects, and current synchronous code can be an actual bomb!

Deadlock when accessing StackExchange.Redis

I'm running into a deadlock situation when calling StackExchange.Redis.
I don't know exactly what is going on, which is very frustrating, and I would appreciate any input that could help resolve or workaround this problem.
In case you have this problem too and don't want to read all this;
I suggest that you'll try setting PreserveAsyncOrder to false.
ConnectionMultiplexer connection = ...;
connection.PreserveAsyncOrder = false;
Doing so will probably resolve the kind of deadlock that this Q&A is about and could also improve performance.
Our setup
The code is run as either a Console application or as an Azure Worker Role.
It exposes a REST api using HttpMessageHandler so the entry point is async.
Some parts of the code have thread affinity (is owned by, and must be run by, a single thread).
Some parts of the code is async-only.
We are doing the sync-over-async and async-over-sync anti-patterns. (mixing await and Wait()/Result).
We're only using async methods when accessing Redis.
We're using StackExchange.Redis 1.0.450 for .NET 4.5.
Deadlock
When the application/service is started it runs normally for a while then all of a sudden (almost) all incoming requests stop functioning, they never produce a response. All those requests are deadlocked waiting for a call to Redis to complete.
Interestingly, once the deadlock occur, any call to Redis will hang but only if those calls are made from an incoming API request, which are run on the thread pool.
We are also making calls to Redis from low priority background threads, and these calls continue to function even after the deadlock occurred.
It seems as if a deadlock will only occur when calling into Redis on a thread pool thread. I no longer think this is due to the fact that those calls are made on a thread pool thread. Rather, it seems like any async Redis call without continuation, or with a sync safe continuation, will continue to work even after the deadlock situation has occurred. (See What I think happens below)
Related
StackExchange.Redis Deadlocking
Deadlock caused by mixing await and Task.Result (sync-over-async, like we do). But our code is run without synchronization context so that doesn't apply here, right?
How to safely mix sync and async code?
Yes, we shouldn't be doing that. But we do, and we'll have to continue doing so for a while. Lots of code that needs to be migrated into the async world.
Again, we don't have a synchronization context, so this should not be causing deadlocks, right?
Setting ConfigureAwait(false) before any await has no effect on this.
Timeout exception after async commands and Task.WhenAny awaits in StackExchange.Redis
This is the thread hijacking problem. What's the current situation on this? Could this be the problem here?
StackExchange.Redis async call hangs
From Marc's answer:
...mixing Wait and await is not a good idea. In addition to deadlocks, this is "sync over async" - an anti-pattern.
But he also says:
SE.Redis bypasses sync-context internally (normal for library code), so it shouldn't have the deadlock
So, from my understanding StackExchange.Redis should be agnostic to whether we're using the sync-over-async anti-pattern. It's just not recommended as it could be the cause of deadlocks in other code.
In this case, however, as far as I can tell, the deadlock is really inside StackExchange.Redis. Please correct me if I'm wrong.
Debug findings
I've found that the deadlock seems to have its source in ProcessAsyncCompletionQueue on line 124 of CompletionManager.cs.
Snippet of that code:
while (Interlocked.CompareExchange(ref activeAsyncWorkerThread, currentThread, 0) != 0)
{
// if we don't win the lock, check whether there is still work; if there is we
// need to retry to prevent a nasty race condition
lock(asyncCompletionQueue)
{
if (asyncCompletionQueue.Count == 0) return; // another thread drained it; can exit
}
Thread.Sleep(1);
}
I've found that during the deadlock; activeAsyncWorkerThread is one of our threads that is waiting for a Redis call to complete. (our thread = a thread pool thread running our code). So the loop above is deemed to continue forever.
Without knowing the details, this sure feels wrong; StackExchange.Redis is waiting for a thread that it thinks is the active async worker thread while it is in fact a thread that is quite the opposite of that.
I wonder if this is due to the thread hijacking problem (which I don't fully understand)?
What to do?
The main two question I'm trying to figure out:
Could mixing await and Wait()/Result be the cause of deadlocks even when running without synchronization context?
Are we running into a bug/limitation in StackExchange.Redis?
A possible fix?
From my debug findings it seems as the problem is that:
next.TryComplete(true);
...on line 162 in CompletionManager.cs could under some circumstances let the current thread (which is the active async worker thread) wander off and start processing other code, possibly causing a deadlock.
Without knowing the details and just thinking about this "fact", then it would seem logical to temporarily release the active async worker thread during the TryComplete invocation.
I guess that something like this could work:
// release the "active thread lock" while invoking the completion action
Interlocked.CompareExchange(ref activeAsyncWorkerThread, 0, currentThread);
try
{
next.TryComplete(true);
Interlocked.Increment(ref completedAsync);
}
finally
{
// try to re-take the "active thread lock" again
if (Interlocked.CompareExchange(ref activeAsyncWorkerThread, currentThread, 0) != 0)
{
break; // someone else took over
}
}
I guess my best hope is that Marc Gravell would read this and provide some feedback :-)
No synchronization context = The default synchronization context
I've written above that our code does not use a synchronization context. This is only partially true: The code is run as either a Console application or as an Azure Worker Role. In these environments SynchronizationContext.Current is null, which is why I wrote that we're running without synchronization context.
However, after reading It's All About the SynchronizationContext I've learned that this is not really the case:
By convention, if a thread’s current SynchronizationContext is null, then it implicitly has a default SynchronizationContext.
The default synchronization context should not be the cause of deadlocks though, as UI-based (WinForms, WPF) synchronization context could - because it does not imply thread affinity.
What I think happens
When a message is completed its completion source is checked for whether it is considered sync safe. If it is, the completion action is executed inline and everything is fine.
If it is not, the idea is to execute the completion action on a newly allocated thread pool thread. This too works just fine when ConnectionMultiplexer.PreserveAsyncOrder is false.
However, when ConnectionMultiplexer.PreserveAsyncOrder is true (the default value), then those thread pool threads will serialize their work using a completion queue and by ensuring that at most one of them is the active async worker thread at any time.
When a thread becomes the active async worker thread it will continue to be that until it have drained the completion queue.
The problem is that the completion action is not sync safe (from above), still it is executed on a thread that must not be blocked as that will prevent other non sync safe messages from being completed.
Notice that other messages that are being completed with a completion action that is sync safe will continue to work just fine, even though the active async worker thread is blocked.
My suggested "fix" (above) would not cause a deadlock in this way, it would however mess with the notion of preserving async completion order.
So maybe the conclusion to make here is that it is not safe to mix await with Result/Wait() when PreserveAsyncOrder is true, no matter whether we are running without synchronization context?
(At least until we can use .NET 4.6 and the new TaskCreationOptions.RunContinuationsAsynchronously, I suppose)

These are the workarounds I've found to this deadlock problem:
Workaround #1
By default StackExchange.Redis will ensure that commands are completed in the same order that result messages are received. This could cause a deadlock as described in this question.
Disable that behavior by setting PreserveAsyncOrder to false.
ConnectionMultiplexer connection = ...;
connection.PreserveAsyncOrder = false;
This will avoid deadlocks and could also improve performance.
I encourage anyone that run into to deadlock problems to try this workaround, since it's so clean and simple.
You'll loose the guarantee that async continuations are invoked in the same order as the underlying Redis operations are completed. However, I don't really see why that is something you would rely on.
Workaround #2
The deadlock occur when the active async worker thread in StackExchange.Redis completes a command and when the completion task is executed inline.
One can prevent a task from being executed inline by using a custom TaskScheduler and ensure that TryExecuteTaskInline returns false.
public class MyScheduler : TaskScheduler
{
public override bool TryExecuteTaskInline(Task task, bool taskWasPreviouslyQueued)
{
return false; // Never allow inlining.
}
// TODO: Rest of TaskScheduler implementation goes here...
}
Implementing a good task scheduler may be a complex task. There are, however, existing implementations in the ParallelExtensionExtras library (NuGet package) that you can use or draw inspiration from.
If your task scheduler would use its own threads (not from the thread pool), then it might be a good idea to allow inlining unless the current thread is from the thread pool. This will work because the active async worker thread in StackExchange.Redis is always a thread pool thread.
public override bool TryExecuteTaskInline(Task task, bool taskWasPreviouslyQueued)
{
// Don't allow inlining on a thread pool thread.
return !Thread.CurrentThread.IsThreadPoolThread && this.TryExecuteTask(task);
}
Another idea would be to attach your scheduler to all of its threads, using thread-local storage.
private static ThreadLocal<TaskScheduler> __attachedScheduler
= new ThreadLocal<TaskScheduler>();
Ensure that this field is assigned when the thread starts running and cleared as it completes:
private void ThreadProc()
{
// Attach scheduler to thread
__attachedScheduler.Value = this;
try
{
// TODO: Actual thread proc goes here...
}
finally
{
// Detach scheduler from thread
__attachedScheduler.Value = null;
}
}
Then you can allow inlining of tasks as long as its done on a thread that is "owned" by the custom scheduler:
public override bool TryExecuteTaskInline(Task task, bool taskWasPreviouslyQueued)
{
// Allow inlining on our own threads.
return __attachedScheduler.Value == this && this.TryExecuteTask(task);
}

I am guessing a lot based on the detailed information above and not knowing the source code you have in place. It sounds like you may be hitting some internal, and configurable, limits in .Net. You shouldn't be hitting those, so my guess is that you are not disposing of objects since they are floating between threads which won't allow you to use a using statement to cleanly handle their object lifetimes.
This details the limitations on HTTP requests. Similar to the old WCF issue when you didn't dispose of the connection and then all WCF connections would fail.
Max number of concurrent HttpWebRequests
This is more of a debugging aid, since I doubt you really are using all the TCP ports, but good info on how to find how many open ports you have and to where.
https://msdn.microsoft.com/en-us/library/aa560610(v=bts.20).aspx

Why ThreadAbortException does not throw in catch block

Suppose I have this code:
static void Main(string[] args)
{
var thread = new Thread(() =>
{
try
{
throw new InvalidOperationException();
}
catch (Exception)
{
Thread.Sleep(Timeout.Infinite);
}
});
thread.Start();
Thread.Sleep(TimeSpan.FromSeconds(1));
thread.Abort();
thread.Join();
}
It starts thread, then thread is going into sleep in catch block and after that we are trying abort thread.
Abort method have to raise ThreadAbortException. But in catch block it does not happen.
It's documented:
The thread that calls Abort might block if the thread that is being
aborted is in a protected region of code, such as a catch block,
finally block, or constrained execution region. If the thread that
calls Abort holds a lock that the aborted thread requires, a deadlock
can occur.
My question is why. Why is it working that way? Because in catch block we can raise any exceptions and all works like it have to.
UPDATE:
From the link by Jordão. Accepted because it's the most understandable clarification.
Constrained Execution Regions The .NET Framework 2.0 introduces
Constrained Execution Regions (CER), which impose restrictions both on
the runtime and on the developer. In a region of code marked as a CER,
the runtime is constrained from throwing certain asynchronous
exceptions that would prevent the region from executing in its
entirety. The developer is also constrained in the actions that can be
performed in the region. This creates a framework and an enforcement
mechanism for authoring reliable managed code, making it a key player
in the reliability story for the .NET Framework 2.0. For the runtime
to meet its burden, it makes two accommodations for CERs. First, the
runtime will delay thread aborts for code that is executing in a CER.
In other words, if a thread calls Thread.Abort to abort another thread
that is currently executing within a CER, the runtime will not abort
the target thread until execution has left the CER. Second, the
runtime will prepare CERs as soon as is possible to avoid
out-of-memory conditions. This means that the runtime will do
everything up front that it would normally do during the code region's
JIT compilation. It will also probe for a certain amount of free stack
space to help eliminate stack overflow exceptions. By doing this work
up front, the runtime can better avoid exceptions that might occur
within the region and prevent resources from being cleaned up
appropriately. To use CERs effectively, developers should avoid
certain actions that might result in asynchronous exceptions. The code
is constrained from performing certain actions, including things like
explicit allocations, boxing, virtual method calls (unless the target
of the virtual method call has already been prepared), method calls
through reflection, use of Monitor.Enter (or the lock keyword in C#
and SyncLock in Visual Basic®), isinst and castclass instructions on
COM objects, field access through transparent proxies, serialization,
and multidimensional array accesses. In short, CERs are a way to move
any runtime-induced failure point from your code to a time either
before the code runs (in the case of JIT compiling), or after the code
completes (for thread aborts). However, CERs really do constrain the
code you can write. Restrictions such as not allowing most allocations
or virtual method calls to unprepared targets are significant,
implying a high development cost to authoring them. This means CERs
aren't suited for large bodies of general-purpose code, and they
should instead be thought of as a technique to guarantee execution of
small regions of code.

The problem is that the thread you're attempting to abort is running inside a catch clause.
This will abort the thread:
static void Main(string[] args) {
var thread = new Thread(() => {
Thread.Sleep(Timeout.Infinite);
});
thread.Start();
Thread.Sleep(TimeSpan.FromSeconds(1));
thread.Abort();
thread.Join();
}
From this article:
In the .NET Framework 2.0, the CLR delays graceful thread aborts by default over CERs, finally blocks, catch blocks, static constructors, and unmanaged code.
This feature exists to keep the .NET framework more reliable in the face of certain asynchronous exceptions. Read the article I linked for the full story.
Your code basically misbehaves and a host would probably escalate that thread to a rude thread abort:
Rude thread aborts and rude application domain unloads are used by CLR hosts to ensure that runaway code can be kept in check. Of course, failure to run finalizers or non-CER finally blocks due to these actions presents the CLR host with new reliability problems, since there's a good chance these actions will leak the resources the back-out code was supposed to clean up.

This is by design, this was introduced in Fx 3 or 4.
You can look up the different versions form your own link and find different descriptions.
Allowing an AbortException inside those protected regions (as in Fx 1.x) can lead to very unpredictable situations and an unstable Process.
Note that Thread.Abort() is (was) generally dis-advised. And so is long-running code in any catch or finally clause.
Disallowing Abort to interrupt a catch clause addresses some of the issues with Abort. But it's still not perfect.

I suspect the point is that while you're in a catch or finally block, you're probably trying to clean up after yourself already. If an asynchronous exception can be triggered at that point, it's going to be really hard to do any sort of reliable cleanup.
Joe Duffy's blog post about asynchronous exceptions is likely to clarify this more than I can...

Background consumer thread lifetime management best practices

I have a C# class library which starts up a background consumer thread (lazily) which listens for tasks to complete from a producer/consumer queue. This class library can be used from any type of .NET application and is currently in use under an ASP.NET MVC web site. The consumer thread is blocked most of the time until a request comes into the queue. There should be only one consumer thread for any given app domain.
I want to be able to gracefully shut down the consumer thread when the application exits, regardless of what type of application it is, e.g. Windows Forms, Console, or WPF application. The application code itself should be ignorant of the fact that this thread is lingering around waiting to be terminated.
How can I solve this thread lifetime issue from the class library's perspective? Is there some global application shutdown event I can tie into to tear down the thread gracefully then?
Right now, since it's in an ASP.NET MVC app domain, it really doesn't matter since those never get torn down gracefully anyway. But now that I'm starting to use it in scheduled console application tasks designed to terminate, I want to solve this problem once and for all. The console application does not terminate since the consumer thread is still active and blocked waiting for requests. I've exposed a public static Thread property on the class to issue an Abort() call when the console app is exiting but this is, quite frankly, disgusting.
Any pointers would be appreciated! Again, I don't want to have to write Windows Forms or WPF or console application specific code to solve the issue. A nice generic solution that every consumer of the class library can use would be best.

Set the thread's IsBackground property to true. Then it will not prevent the process from ending.
http://msdn.microsoft.com/en-us/library/system.threading.thread.isbackground.aspx
Alternately, instead of using Abort, use Interrupt. Much less disgusting.

This is a bit tricky. You are right on track that you want to avoid an abort as that might terminate the thread in the middle of an important operation (writing to a file, etc.). Setting the IsBackground property would have the same effect.
What you need to do is make your blocking queue cancellable. Too bad you are not using .NET 4.0 yet otherwise you could have used the BlockingCollection class. No worries though. You will just have to modify your custom queue so that it polls periodically for a stopping signal. Here is a quick and dirty implementation that I just whipped up using the canonical implementation of a blocking queue as a starting point.
public class CancellableBlockingCollection<T>
{
private Queue<T> m_Queue = new Queue<T>();
public T Take(WaitHandle cancel)
{
lock (m_Queue)
{
while (m_Queue.Count <= 0)
{
if (!Monitor.Wait(m_Queue, 1000))
{
if (cancel.WaitOne(0))
{
throw new ThreadInterruptedException();
}
}
}
return m_Queue.Dequeue();
}
}
public void Add(T data)
{
lock (m_Queue)
{
m_Queue.Enqueue(data);
Monitor.Pulse(m_Queue);
}
}
}
Notice how the Take method accepts a WaitHandle that can be tested for the cancel signal. This could be the same wait handle used in other parts of your code. The idea here is that the wait handle gets polled on a certain interval inside Take and if it is signaled then the whole method throws. The assumption is that throwing inside Take is okay since it is at a safe point because the consumer could not have been in the middle of an important operation. You can just allow the thread to disintegrate upon the exception at this point. Here is what your consumer thread may look like.
ManualResetEvent m_Cancel = new ManualResetEvent(false);
CancellableBlockingCollection<object> m_Queue = new CancellableBlockingCollection<object>();
private void ConsumerThread()
{
while (true)
{
object item = m_Queue.Take(m_Cancel); // This will throw if the event is signalled.
}
}
There are various other ways you could cancel the Take method. I chose to use a WaitHandle, but it could easily be done using a simple bool flag for example.
Update:
As Steven pointed out in the comments Thread.Interrupt does essentially this already. It will cause a thread to throw an exception on a blocking call...pretty much what I was after in this example, but with a lot more code. One caveat with Thread.Interrupt is that it only works during the canned blocking calls in the .NET BCL (like WaitOne, etc.). So you would not be able to cancel a long running computation in progress like you would if you used a more manual approach like the one in this answer. It is definitely great tool to keep in your back pocket and might just be useful in your specific scenario.

If you are using .net 4.0 a CancelationToken can be used to signal the thread that it is time to gracefully shut down. It is very useful as most Wait commands allow you to pass it a token and if the thread is in a wait when the thread is canceled it will automatically come out of the wait.