We Keep Coding

How to handle a deadlock in third-party code

We have a third-party method Foo which sometimes runs in a deadlock for unknown reasons.
We are executing an single-threaded tcp-server and call this method every 30 seconds to check that the external system is available.
To mitigate the problem with the deadlock in the third party code we put the ping-call in a Task.Run to so that the server does not deadlock.
Like
async Task<bool> WrappedFoo()
{
var timeout = 10000;
var task = Task.Run(() => ThirdPartyCode.Foo());
var delay = Task.Delay(timeout);
if (delay == await Task.WhenAny(delay, task ))
{
return false;
}
else
{
return await task ;
}
}
But this (in our opinion) has the potential to starve the application of free threads. Since if one call to ThirdPartyCode.Foo deadlock the thread will never recover from this deadlock and if this happens often enough we might run out of resources.
Is there a general approach how one should handle deadlocking third-party code?
A CancellationToken won't work because the third-party-api does not provide any cancellation options.
Update:
The method at hand is from the SAPNCO.dll provided by SAP to establish and test rfc-connections to a sap-system, therefore the method is not a simple network-ping. I renamed the method in the question to avoid further misunderstandings

Is there a general approach how one should handle deadlocking third-party code?
Yes, but it's not easy or simple.
The problem with misbehaving code is that it can not only leak resources (e.g., threads), but it can also indefinitely hold onto important resources (e.g., some internal "handle" or "lock").
The only way to forcefully reclaim threads and other resources is to end the process. The OS is used to cleaning up misbehaving processes and is very good at it. So, the solution here is to start a child process to do the API call. Your main application can communicate with its child process by redirected stdin/stdout, and if the child process ever times out, the main application can terminate it and restart it.
This is, unfortunately, the only reliable way to cancel uncancelable code.

Cancelling a task is a collaborative operation in that you pass a CancellationToken to the desired method and externally you use CancellationTokenSource.Cancel:
public void Caller()
{
try
{
CancellationTokenSource cts=new CancellationTokenSource();
Task longRunning= Task.Run(()=>CancellableThirdParty(cts.Token),cts.Token);
Thread.Sleep(3000); //or condition /signal
cts.Cancel();
}catch(OperationCancelledException ex)
{
//treat somehow
}
}
public void CancellableThirdParty(CancellationToken token)
{
while(true)
{
// token.ThrowIfCancellationRequested() -- if you don't treat the cancellation here
if(token.IsCancellationRequested)
{
// code to treat the cancellation signal
//throw new OperationCancelledException($"[Reason]");
}
}
}
As you can see in the code above , in order to cancel an ongoing task , the method running inside it must be structured around the CancellationToken.IsCancellationRequested flag or simply CancellationToken.ThrowIfCancellationRequested method ,
so that the caller just issues the CancellationTokenSource.Cancel.
Unfortunately if the third party code is not designed around CancellationToken ( it does not accept a CancellationToken parameter ), then there is not much you can do.

Your code isn't cancelling the blocked operation. Use a CancellationTokenSource and pass a cancellation token to Task.Run instead :
var cts=new CancellationTokenSource(timeout);
try
{
await Task.Run(() => ThirdPartyCode.Ping(),cts.Token);
return true;
}
catch(TaskCancelledException)
{
return false;
}
It's quite possible that blocking is caused due to networking or DNS issues, not actual deadlock.
That still wastes a thread waiting for a network operation to complete. You could use .NET's own Ping.SendPingAsync to ping asynchronously and specify a timeout:
var ping=new Ping();
var reply=await ping.SendPingAsync(ip,timeout);
return reply.Status==IPStatus.Success;
The PingReply class contains far more detailed information than a simple success/failure. The Status property alone differentiates between routing problems, unreachable destinations, time outs etc

Test run stops when doing several multi-threaded tests in a row

I've got a class with a static ConcurrentQueue. One class receives messages and puts them on the queue, whilst a different thread on this class reads them from that queue and processes them one at a time. That method is aborted with a cancellationtoken.
The method that empties the queue looks like this:
public async Task HandleEventsFromQueueAsync(CancellationToken ct, int pollDelay = 25)
{
while (true)
{
if (ct.IsCancellationRequested)
{
return;
}
if(messageQueue.TryDequeue(out ConsumeContext newMessage))
{
handler.Handle(newMessage);
}
try
{
await Task.Delay(pollDelay, ct).ConfigureAwait(true);
}
catch (TaskCanceledException)
{
return;
}
}
}
My testing methods look like this:
CancellationToken ct = source.Token;
Thread thread = new Thread(async () => await sut.HandleEventsFromQueueAsync(ct));
thread.Start();
EventListener.messageQueue.Enqueue(message1);
EventListener.messageQueue.Enqueue(message2);
await Task.Delay(1000);
source.Cancel(false);
mockedHandler.Verify(x => x.Handle(It.IsAny<ConsumeContext>()), Times.Exactly(2));
So I start my dequeueing method in its own thread, with a fresh cancellation token. Then I enqueue a couple of messages, give the process a second to handle them, and then use source.Cancel(false) to put an end to the thread and make the method return. Then I check that the handler was called the right number of times. Of course I'm testing this in a couple variations, with different message types and different times when I abort the dequeueing method.
The issue is that when I run any of my tests individually, they all succeed. But when I try to run them as a group, Visual Studio does not run every test. There's no error message, and the tests it does run succeed fine, but the run just stops after the second test.
I do not have an idea why this happens. My tests are all identical in structure. I'm aborting the dequeueing thread properly every time.
What could compel Visual Studio to stop a test run, without throwing any kind of error?

You are passing an async lambda to the Thread constructor. The Thread constructor doesn't understand async delegates (does not accept a Func<Task> argument), so you end up with an async void lambda. Async void methods should be avoided for anything that it's not an event handler. What happens in your case is that the explicitly created thread is terminated when the code hits the first await, and the rest of the body runs in ThreadPool threads. It seems that the code never fails with an exception, otherwise the process would crash (this is the default behavior of async void methods).
Suggestions:
Use a Task instead of a Thread. This way you'll have something to await before exiting the test.
CancellationToken ct = source.Token;
Task consumerTask = Task.Run(() => sut.HandleEventsFromQueueAsync(ct));
EventListener.messageQueue.Enqueue(message1);
EventListener.messageQueue.Enqueue(message2);
await Task.Delay(1000);
source.Cancel(false);
await consumerTask; // Wait the task to complete
mockedHandler.Verify(x => x.Handle(It.IsAny<ConsumeContext>()), Times.Exactly(2));
Consider using a BlockingCollection or an asynchronous queue like a Channel instead of a ConcurrentQueue. Polling is an awkward and inefficient technique. With a blocking or async queue you'll not be obliged to do loops waiting for new messages to arrive. You'll be able to enter a waiting state, and notified instantly when a new message arrives.
Configure the awaiting with ConfigureAwait(false). ConfigureAwait(true) is the default and does nothing.
Consider propagating cancellation by throwing an OperationCanceledException. This is the standard way of propagating cancellation in .NET. So instead of:
if (ct.IsCancellationRequested) return;
...it is preferable to do this:
ct.ThrowIfCancellationRequested();

I have solved my own issue. Turns out that the newly created thread threw an exception, and when threads throw exceptions those are ignored, but they still stop the unit test from happening. After fixing the issue causing the exception, the tests work fine.

Waiting on a continuous UI background polling task

I am somewhat new to parallel programming C# (When I started my project I worked through the MSDN examples for TPL) and would appreciate some input on the following example code.
It is one of several background worker tasks. This specific task pushes status messages to a log.
var uiCts = new CancellationTokenSource();
var globalMsgQueue = new ConcurrentQueue<string>();
var backgroundUiTask = new Task(
() =>
{
while (!uiCts.IsCancellationRequested)
{
while (globalMsgQueue.Count > 0)
ConsumeMsgQueue();
Thread.Sleep(backgroundUiTimeOut);
}
},
uiCts.Token);
// Somewhere else entirely
backgroundUiTask.Start();
Task.WaitAll(backgroundUiTask);
I'm asking for professional input after reading several topics like Alternatives to using Thread.Sleep for waiting, Is it always bad to use Thread.Sleep()?, When to use Task.Delay, when to use Thread.Sleep?, Continuous polling using Tasks
Which prompts me to use Task.Delay instead of Thread.Sleep as a first step and introduce TaskCreationOptions.LongRunning.
But I wonder what other caveats I might be missing? Is polling the MsgQueue.Count a code smell? Would a better version rely on an event instead?

First of all, there's no reason to use Task.Start or use the Task constructor. Tasks aren't threads, they don't run themselves. They are a promise that something will complete in the future and may or may not produce any results. Some of them will run on a threadpool thread. Use Task.Run to create and run the task in a single step when you need to.
I assume the actual problem is how to create a buffered background worker. .NET already offers classes that can do this.
ActionBlock< T >
The ActionBlock class already implements this and a lot more - it allows you to specify how big the input buffer is, how many tasks will process incoming messages concurrently, supports cancellation and asynchronous completion.
A logging block could be as simple as this :
_logBlock=new ActionBlock<string>(msg=>File.AppendAllText("myLog.txt",msg));
The ActionBlock class itself takes care of buffering the inputs, feeding new messages to the worker function when it arrives, potentially blocking senders if the buffer gets full etc. There's no need for polling.
Other code can use Post or SendAsync to send messages to the block :
_block.Post("some message");
When we are done, we can tell the block to Complete() and await for it to process any remaining messages :
_block.Complete();
await _block.Completion;
Channels
A newer, lower-level option is to use Channels. You can think of channels as a kind of asynchronous queue, although they can be used to implement complex processing pipelines. If ActionBlock was written today, it would use Channels internally.
With channels, you need to provide the "worker" task yourself. There's no need for polling though, as the ChannelReader class allows you to read messages asynchronously or even use await foreach.
The writer method could look like this :
public ChannelWriter<string> LogIt(string path,CancellationToken token=default)
{
var channel=Channel.CreateUnbounded<string>();
var writer=channel.Writer;
_=Task.Run(async ()=>{
await foreach(var msg in channel.Reader.ReadAllAsync(token))
{
File.AppendAllText(path,msg);
}
},token).ContinueWith(t=>writer.TryComplete(t.Exception);
return writer;
}
....
_logWriter=LogIt(somePath);
Other code can send messages by using WriteAsync or TryWrite, eg :
_logWriter.TryWrite(someMessage);
When we're done, we can call Complete() or TryComplete() on the writer :
_logWriter.TryComplete();
The line
.ContinueWith(t=>writer.TryComplete(t.Exception);
is needed to ensure the channel is closed even if an exception occurs or the cancellation token is signaled.
This may seem too cumbersome at first. Channels allow us to easily run initialization code or carry state from one message to the next. We could open a stream before the loop starts and use it instead of reopening the file each time we call File.AppendAllText, eg :
public ChannelWriter<string> LogIt(string path,CancellationToken token=default)
{
var channel=Channel.CreateUnbounded<string>();
var writer=channel.Writer;
_=Task.Run(async ()=>{
//***** Can't do this with an ActionBlock ****
using(var writer=File.AppendText(somePath))
{
await foreach(var msg in channel.Reader.ReadAllAsync(token))
{
writer.WriteLine(msg);
//Or
//await writer.WriteLineAsync(msg);
}
}
},token).ContinueWith(t=>writer.TryComplete(t.Exception);
return writer;
}

Definitely Task.Delay is better than Thread.Sleep, because you will not be blocking the thread on the pool, and during the wait the thread on the pool will be available to handle other tasks. Then, you don't need to make your task long-running. Long-running tasks are run in a dedicated thread, and then Task.Delay is meaningless.
Instead, I will recommend a different approach. Just use System.Threading.Timer and make your life simple. Timers are kernel objects that will run their callback on the thread pool, and you will not have to worry about delay or sleep.

The TPL Dataflow library is the preferred tool for this kind of job. It allows building efficient producer-consumer pairs quite easily, and more complex pipelines as well, while offering a complete set of configuration options. In your case using a single ActionBlock should be enough.
A simpler solution you might consider is to use a BlockingCollection. It has the advantage of not requiring the installation of any package (because it is built-in), and it's also much easier to learn. You don't have to learn more than the methods Add, CompleteAdding, and GetConsumingEnumerable. It also supports cancellation. The drawback is that it's a blocking collection, so it blocks the consumer thread while waiting for new messages to arrive, and the producer thread while waiting for available space in the internal buffer (only if you specify a boundedCapacity in the constructor).
var uiCts = new CancellationTokenSource();
var globalMsgQueue = new BlockingCollection<string>();
var backgroundUiTask = new Task(() =>
{
foreach (var item in globalMsgQueue.GetConsumingEnumerable(uiCts.Token))
{
ConsumeMsgQueueItem(item);
}
}, uiCts.Token);
The BlockingCollection uses a ConcurrentQueue internally as a buffer.

'await' does not return, when my Task is started from a custom TaskScheduler

Background:
I have a "Messenger" class. It sends messages. But due to limitations, let's say it can only send - at most - 5 messages at a time.
I have a WPF application which queues messages as needed, and waits for the queued message to be handled before continuing. Due to the asynchronous nature of the application, any number of messages could be awaited at any given time.
Current Implementation:
To accomplish this, I've implemented a Task<Result> SendMessage(Message message) API within my messaging class. Internal to the messaging class is a custom TaskScheduler (the LimitedConcurrencyTaskScheduler from MSDN), with its concurrency level set to 5. In this way, I would expect that no matter how many messages are queued, only 5 will be sent out at a time, and my client application will patiently wait until its respective message has been handled.
Problem:
When I await the SendMessage method, I can see via the debugger that the message was completed and the result returned, but my code never executes beyond the awaited method call!
Is there some special considerations that need to be made, when awaiting a Task which was scheduled using a different TaskScheduler?
Snipped Code:
From my client/consuming function:
public async Task Frobulate()
{
Message myMessage = new Message(x, y, z);
await messenger.SendMessage(myMessage);
//Code down here never executes!
}
From my messenger class:
private TaskScheduler _messengerTaskScheduler = new LimitedConcurrencyLevelTaskScheduler(5);
private TaskFactory _messengerTaskFactory = new TaskFactory(_messengerScheduler);
public Task<Result> SendMessage(Message message)
{
//My debugger has verified that "InternalSendMessage" has completed,
//but the caller's continuation appears to never execute
return _messengerTaskFactory.StartNew(() => InternalSendMessage(message));
}
Update:
The 'freeze' does not actually appear to be caused by my custom TaskScheduler; when I queue up the Task with the default TaskFactory, the same behavior occurs! There must be something else happening at a more fundamental level, likely due to my own stupidity.

Based on the comments, you probably have a deadlock because you're blocking on async code.
When using async, whenever there are thread restrictions on the SynchronizationContext or TaskScheduler and the code blocks using Task.Result or Task.Wait there's a possibility of deadlocking. The asynchronous operation needs a thread to finish execution, which it can't get because the SynchronizationContext (or TaskScheduler in your case) is waiting for that same exact operation to complete before allowing "new" ones to run.
Go deeper in Stephen Cleary's blog post: Don't Block on Async Code

What is the best way to wait on a network packet using C#'s new async feature

I've recently been playing around with the new Async CTP, and I've come across a situation where I'm not sure how to proceed.
In my current code base, I'm using a concept of "jobs" and a "job manager". Jobs exist solely for the purpose of handling an initial message, sending a response, and then waiting the response.
I already have existing code based around synchronous sockets, where a network thread is waiting on data to arrive, and then passing it along to an event handler, and eventually to the job manager.
The job manager looks for what job would handle the message, and passes it along.
So the scenario is this:
Job manager gets a new message and launches a job.
The job starts, processes the message, and sends a reply message.
At this point the job would wait for a response to the reply.
Here's a pseudocode example:
class MyJob : Job
{
public override void RunJob( IPacketMsg packet )
{
// handle packet
var myReply = new Packet();
SendReply( myReply );
await GetResponse();
}
}
But I'm not entirely sure how to proceed at step 3. The job manager will get the response and then hand it along to the running job. But I'm not sure how to make the job wait for the response.
I've considered creating an awaited Task that simply blocks on a WaitHandle, but is this the best solution?
Are there any other things I could do in this case?
Edit
On the subject of the Async CTP, what happens in a situation where the UI is not being used. I've read over Eric Lippert's Async blog, but I don't believe it ever touched on the subject of how everything works in the background without a UI thread (does it spin off a background worker or...?)

Job manager gets a new message and launches a job.
The job starts, processes the message, and sends a reply message.
At this point the job would wait for a response to the reply.
First off, I should mention that the Async CTP handles asynchronous operations very well, but asynchronous events not so much. You may want to consider an Rx-based approach. But let's proceed for the moment with the Async CTP.
You have two basic options to create Tasks:
With a delegate. e.g., Task.Factory.StartNew will run a delegate on the thread pool. Custom task factories and schedulers give you more options for task delegates (e.g., specifying the delegate must be run on an STA thread).
Without a delegate. e.g., TaskFactory.FromAsync wraps an existing Begin/End method pair, TaskEx.FromResult returns a "future constant", and TaskCompletionSource can be used to control a Task explicitly (both FromAsync and FromResult use TCS internally).
If the job processing is CPU-bound, it makes sense to pass it off to Task.Factory.StartNew. I'm going to assume the job processing is CPU-bound.
Job manager pseudo-code:
// Responds to a new message by starting a new job on the thread pool.
private void RespondToNewMessage(IPacketMsg message)
{
IJob job = ..;
Task.Factory.StartNew(job.RunJob(message));
}
// Holds tasks waiting for a response.
private ConcurrentDictionary<int, TaskCompletionSource<IResponse>> responseTasks = ..;
// Asynchronously gets a response for the specified reply.
public Task<IResponse> GetResponseForReplyAsync(int replyId)
{
var tcs = new TaskCompletionSource<IResponse>();
responseTasks.Add(replyId, tcs);
return tcs.Task;
}
// Responds to a new response by completing and removing its task.
private void RespondToResponse(IResponse response)
{
var tcs = responseTasks[response.ReplyId];
responseTasks.Remove(response.ReplyId);
tcs.TrySetComplete(response);
}
The idea is that the job manager also manages a list of oustanding responses. In order for this to happen, I introduced a simple int reply identifier that the job manager can use to determine which response goes with which reply.
Now jobs can work like this:
public override void RunJob(IPacketMsg packet)
{
// handle packet
var myReply = new Packet();
var response = jobManager.GetResponseForReplyAsync(myReply.ReplyId);
SendReply(myReply);
await response;
}
There's a few tricky things since we're placing the jobs on the thread pool thread:
GetResponseForReplyAsync must be invoked (registering the task) before the reply is sent, and is then awaited later. This is to avoid the situation where a reply may be sent and a response received before we have a chance to register for it.
RespondToResponse will remove the task registration before completing it, just in case completing the task causes another reply to be sent with the same id.
If the jobs are short enough that they don't need to be placed on the thread pool thread, then the solution can be simplified.

On the subject of the Async CTP, what happens in a situation where the UI is not being used. I've read over Eric Lippert's Async blog, but I don't believe it ever touched on the subject of how everything works in the background without a UI thread (does it spin off a background worker or...?)
await will return to its synchronization context. In a UI process, this is a UI message loop. In ASP.NET, this is the ASP.NET thread pool. In other situations (Console applications and Win32 services), there is no context, so continuations are queued to the ThreadPool. This is not usually desired behavior, so I wrote an AsyncContext class that can be used in those situations.
BackgroundWorker is not used. In a server-side scenario such as yours, it's not uncommon to not have a background thread at all.

You would simply wire up the rest of your event handler with the await pattern like so:
public async void RunJob(IPacketMsg msg)
{
// Do Stuff
var response = await GetResponse();
// response is "string", not "Task<string>"
// Do More Stuff
}
public Task<string> GetResponse()
{
return Task.Factory.StartNew(() =>
{
_networkThingy.WaitForDataAvailable();
return _networkThingy.ResponseString;
});
}
When your get response task finishes, the rest of the method picks up execution on your current synchronization context. Until then, however, your method execution is yielded (so any code after the wait is not run until the task started in GetResponse finishes)