We Keep Coding

How to handle a deadlock in third-party code

We have a third-party method Foo which sometimes runs in a deadlock for unknown reasons.
We are executing an single-threaded tcp-server and call this method every 30 seconds to check that the external system is available.
To mitigate the problem with the deadlock in the third party code we put the ping-call in a Task.Run to so that the server does not deadlock.
Like
async Task<bool> WrappedFoo()
{
var timeout = 10000;
var task = Task.Run(() => ThirdPartyCode.Foo());
var delay = Task.Delay(timeout);
if (delay == await Task.WhenAny(delay, task ))
{
return false;
}
else
{
return await task ;
}
}
But this (in our opinion) has the potential to starve the application of free threads. Since if one call to ThirdPartyCode.Foo deadlock the thread will never recover from this deadlock and if this happens often enough we might run out of resources.
Is there a general approach how one should handle deadlocking third-party code?
A CancellationToken won't work because the third-party-api does not provide any cancellation options.
Update:
The method at hand is from the SAPNCO.dll provided by SAP to establish and test rfc-connections to a sap-system, therefore the method is not a simple network-ping. I renamed the method in the question to avoid further misunderstandings

Is there a general approach how one should handle deadlocking third-party code?
Yes, but it's not easy or simple.
The problem with misbehaving code is that it can not only leak resources (e.g., threads), but it can also indefinitely hold onto important resources (e.g., some internal "handle" or "lock").
The only way to forcefully reclaim threads and other resources is to end the process. The OS is used to cleaning up misbehaving processes and is very good at it. So, the solution here is to start a child process to do the API call. Your main application can communicate with its child process by redirected stdin/stdout, and if the child process ever times out, the main application can terminate it and restart it.
This is, unfortunately, the only reliable way to cancel uncancelable code.

Cancelling a task is a collaborative operation in that you pass a CancellationToken to the desired method and externally you use CancellationTokenSource.Cancel:
public void Caller()
{
try
{
CancellationTokenSource cts=new CancellationTokenSource();
Task longRunning= Task.Run(()=>CancellableThirdParty(cts.Token),cts.Token);
Thread.Sleep(3000); //or condition /signal
cts.Cancel();
}catch(OperationCancelledException ex)
{
//treat somehow
}
}
public void CancellableThirdParty(CancellationToken token)
{
while(true)
{
// token.ThrowIfCancellationRequested() -- if you don't treat the cancellation here
if(token.IsCancellationRequested)
{
// code to treat the cancellation signal
//throw new OperationCancelledException($"[Reason]");
}
}
}
As you can see in the code above , in order to cancel an ongoing task , the method running inside it must be structured around the CancellationToken.IsCancellationRequested flag or simply CancellationToken.ThrowIfCancellationRequested method ,
so that the caller just issues the CancellationTokenSource.Cancel.
Unfortunately if the third party code is not designed around CancellationToken ( it does not accept a CancellationToken parameter ), then there is not much you can do.

Your code isn't cancelling the blocked operation. Use a CancellationTokenSource and pass a cancellation token to Task.Run instead :
var cts=new CancellationTokenSource(timeout);
try
{
await Task.Run(() => ThirdPartyCode.Ping(),cts.Token);
return true;
}
catch(TaskCancelledException)
{
return false;
}
It's quite possible that blocking is caused due to networking or DNS issues, not actual deadlock.
That still wastes a thread waiting for a network operation to complete. You could use .NET's own Ping.SendPingAsync to ping asynchronously and specify a timeout:
var ping=new Ping();
var reply=await ping.SendPingAsync(ip,timeout);
return reply.Status==IPStatus.Success;
The PingReply class contains far more detailed information than a simple success/failure. The Status property alone differentiates between routing problems, unreachable destinations, time outs etc

Cancel task in another class C#

I have a problem with the cancellationTokenSource in C#
public class Building {
public CancellationTokenSource BuildTokenSource;
public void StartBuilt()
{
BuildTokenSource = new CancellationTokenSource();
buildingService.buildTask = Task.Run(async () =>
{
await clock.Delay(BUILT_TIME);
}, BuildTokenSource.Token);
}
public void CancelBuilt()
{
if (BuildTokenSource != null)
{
BuildTokenSource.Cancel();
}
}
}
In another class in want to detect if the task is cancel like this but it doesn't work. The catch exception in never triggered
public async Task<Building> GetBuildingOfUserTask()
{
double remainingTime = unitService.GetRemainingTime();
if (remainingTime <= 2000 && remainingTime > 0.0)
{
Building building = GetBuilding();
CancellationToken cancellation = building.BuildTokenSource.Token;
try
{
await buildTask;
}
catch (OperationCanceledException) when (cancellation.IsCancellationRequested)
{
return GetBuildingOfUser();
}
}
return GetBuildingOfUser();
}
Anyone have any idea why this is not working and in this case a solution?

Does clock.Delay(BUILT_TIME) have an overload that accepts a CancellationToken? If so, use that.
The problem is if the code is already awaiting clock.Delay(BUILT_TIME) when you cancel, clock.Delay would not know that it needs throw an exception.

I don't see anywhere calling CancelBuilt() + you must call BuildTokenSource.Token.ThrowIfCancellationRequested() in order to raise OperationCanceledException exception
The token.Cancel() should be called outside the async method consuming the cancelationToken. Also consuming async method must call (usually in every step) Token.ThrowIfCancellationRequested();

Using a Mediator (Mediator design pattern) will be a better fit I believe. This is practically a pub-sub model that will publish an event on cancel and will inform all subscribers.
The first class will have the mediator instance as a reference (a read-only field, a property) to publish the cancel event and the other one should have the same instance as reference in order to be informed when the event actually takes place. Another point that you should take is the subscription that should be cancelled when the instance of the class that contains 'GetBuildingOfUserTask' method is destroyed.
What do you think?

I agree the answer of HadascokJ and I would like to bring more light.
You have a main task started at Task buildingService.buildTask and it subordinated one started at await clock.Delay(BUILT_TIME);
The first Task manages the CancellationToken but the subordinated one not. To figure it better replace your clock.Delay(BUILT_TIME) with a Task Task.Delay(int millisecondsDelay, CancellationToken cancellationToken); and provide, of course the CancelationToken. You will see that, in this case, the subordinated Task will be canceled. Also invoking void CancellationToken.CancelAfter(int millisecondsDelay)
As you do not provide the CancellationToken to the subordinated Task, ones the main Task has been started, neither the main nor its subordinated task will be canceled.
In the other hand, to cancel at execution of the subordinated Task, provide to the subordinated task some logic to manage the CancelationToken into its corresponding method and invoke the CancelationToken.ThrowIfCancellationRequested() whenever is necessary, which throws the OperationCanceledException.
At least, try to split long task into several small ones.
I use to manage async. Tasks, those have to be run sequentially into a Queue of task able to observe these TaskStatus. To deal with that, I have one implementation at github if you need. I call it FifoTaskQueue

Cancelling async uploading task

I've got an Uploaderclass with one method -Upload
public static int Upload(string endpoint,object objectToBeUploaded)
{
Source.Token.ThrowIfCancellationRequested();
var repos = new UploadRepository(endpoint);
return repos.Upload(objectToBeUploaded);
}
The Source is a static CancellationTokenSource available in the project.
I also have a list of endpoints I need to upload a certain object for.
The code in the Form (it's a very small project using WinForms) looks like this:
private async Task UploadObjectAsync(
string endpoint,
object objectToBeUploaded)
{
try
{
int elementId = await Task.Factory.StartNew(
() => Uploader.Upload(endpoint,objectToBeUploaded));
//do something with the returned value..
}
catch(OperationCanceledEception ex)
{
//handle the exception..
}
}
And then I set the btnUpload.Click handler like this so I can later use it:
this.btnUpload.Click += async (s, e) =>
{
foreach(var endpoint in endpoints)
{
await UploadObjectASsync(endpoint,someObject);
}
}
The problem is that whenever I start uploading to all the endpoints(how they are obtained is irrelevant) and I decide to cancel the uploading process using Source.Cancel(); the first UploadObjectAsyncwill always go through since
the Source.Token.ThrowIfCancellationRequested(); check in the Upload method has already been passed. The rest of tasks will be cancelled normally and handled gracefully.
How am I to restructure this code in order to make sure that the first UploadObjectAsync Task will also be cancelled?
It is worth mentioning that I also don't have access to the source code of the uploading process itself (service reference) - the repos.Upload(objectToBeUploaded) in my Upload method.

You need to make your UploadRepository.Upload take a CancellationToken.
Specially when that's the one doing the I/O operation.. That's when the async/await really pays-off.
That will also help you get rid of that: Task.Factory.StartNew since the Upload method will return Task already. There will be no need to spin off a task.
In your current setup, given enough time for the tasks to start (and go through your ThrowIfCancellationRequested) you won't be able to cancel any upload. Even if it takes 30 seconds.
Also, you might be interested in: Task.Run

There isn't anything practical you can do. The Upload method doesn't take a token. The first task has already passed the cancelation check by the time you hit the cancel button. You can prove to yourself the cancel is a timing issue by adding a 10 second sleep ahead of throw if cancelled call. All tasks would then cancel.

The problem is that you can't stop the process that happens inside the Upload function unless it checks for the state of the CancellationToken any terminates itself.
So what you could do is to abort the thread that is executing by doing something like this:
int elementId = await Task.Factory.StartNew(() =>
{
try
{
using (Source.Token.Register(Thread.CurrentThread.Interrupt))
{
return Uploader.Upload(endpoint, objectToBeUploaded));
}
}
catch (ThreadInterruptedException ex)
{
throw new OperationCanceledEception(ex)
}
}, Source.Token);
Using the Source.Token.Register(delegate) function you cause the token to call that function in case the token is cancelled. This way the thread that is currently executing the the uploaded and should throw a exception right away.
This method only works in case the thread enters the WaitSleepJoin-State from time to time, because the exception is only raised in case the thread is in that state. Have a look at the documentation of the Thread.Interrupt function.
The alternative is to use Thread.Abort and the ThreadAbortedException. This will kill your thread in any case, but it may corrupt the internal state of your service, because locks the thread holds won't be released properly. So be very careful using this method.

async function never executed why in c# with quartz task backgrounder in c# [duplicate]

I have a multi-tier .Net 4.5 application calling a method using C#'s new async and await keywords that just hangs and I can't see why.
At the bottom I have an async method that extents our database utility OurDBConn (basically a wrapper for the underlying DBConnection and DBCommand objects):
public static async Task<T> ExecuteAsync<T>(this OurDBConn dataSource, Func<OurDBConn, T> function)
{
string connectionString = dataSource.ConnectionString;
// Start the SQL and pass back to the caller until finished
T result = await Task.Run(
() =>
{
// Copy the SQL connection so that we don't get two commands running at the same time on the same open connection
using (var ds = new OurDBConn(connectionString))
{
return function(ds);
}
});
return result;
}
Then I have a mid level async method that calls this to get some slow running totals:
public static async Task<ResultClass> GetTotalAsync( ... )
{
var result = await this.DBConnection.ExecuteAsync<ResultClass>(
ds => ds.Execute("select slow running data into result"));
return result;
}
Finally I have a UI method (an MVC action) that runs synchronously:
Task<ResultClass> asyncTask = midLevelClass.GetTotalAsync(...);
// do other stuff that takes a few seconds
ResultClass slowTotal = asyncTask.Result;
The problem is that it hangs on that last line forever. It does the same thing if I call asyncTask.Wait(). If I run the slow SQL method directly it takes about 4 seconds.
The behaviour I'm expecting is that when it gets to asyncTask.Result, if it's not finished it should wait until it is, and once it is it should return the result.
If I step through with a debugger the SQL statement completes and the lambda function finishes, but the return result; line of GetTotalAsync is never reached.
Any idea what I'm doing wrong?
Any suggestions to where I need to investigate in order to fix this?
Could this be a deadlock somewhere, and if so is there any direct way to find it?

Yep, that's a deadlock all right. And a common mistake with the TPL, so don't feel bad.
When you write await foo, the runtime, by default, schedules the continuation of the function on the same SynchronizationContext that the method started on. In English, let's say you called your ExecuteAsync from the UI thread. Your query runs on the threadpool thread (because you called Task.Run), but you then await the result. This means that the runtime will schedule your "return result;" line to run back on the UI thread, rather than scheduling it back to the threadpool.
So how does this deadlock? Imagine you just have this code:
var task = dataSource.ExecuteAsync(_ => 42);
var result = task.Result;
So the first line kicks off the asynchronous work. The second line then blocks the UI thread. So when the runtime wants to run the "return result" line back on the UI thread, it can't do that until the Result completes. But of course, the Result can't be given until the return happens. Deadlock.
This illustrates a key rule of using the TPL: when you use .Result on a UI thread (or some other fancy sync context), you must be careful to ensure that nothing that Task is dependent upon is scheduled to the UI thread. Or else evilness happens.
So what do you do? Option #1 is use await everywhere, but as you said that's already not an option. Second option which is available for you is to simply stop using await. You can rewrite your two functions to:
public static Task<T> ExecuteAsync<T>(this OurDBConn dataSource, Func<OurDBConn, T> function)
{
string connectionString = dataSource.ConnectionString;
// Start the SQL and pass back to the caller until finished
return Task.Run(
() =>
{
// Copy the SQL connection so that we don't get two commands running at the same time on the same open connection
using (var ds = new OurDBConn(connectionString))
{
return function(ds);
}
});
}
public static Task<ResultClass> GetTotalAsync( ... )
{
return this.DBConnection.ExecuteAsync<ResultClass>(
ds => ds.Execute("select slow running data into result"));
}
What's the difference? There's now no awaiting anywhere, so nothing being implicitly scheduled to the UI thread. For simple methods like these that have a single return, there's no point in doing an "var result = await...; return result" pattern; just remove the async modifier and pass the task object around directly. It's less overhead, if nothing else.
Option #3 is to specify that you don't want your awaits to schedule back to the UI thread, but just schedule to the thread pool. You do this with the ConfigureAwait method, like so:
public static async Task<ResultClass> GetTotalAsync( ... )
{
var resultTask = this.DBConnection.ExecuteAsync<ResultClass>(
ds => return ds.Execute("select slow running data into result");
return await resultTask.ConfigureAwait(false);
}
Awaiting a task normally would schedule to the UI thread if you're on it; awaiting the result of ContinueAwait will ignore whatever context you are on, and always schedule to the threadpool. The downside of this is you have to sprinkle this everywhere in all functions your .Result depends on, because any missed .ConfigureAwait might be the cause of another deadlock.

This is the classic mixed-async deadlock scenario, as I describe on my blog. Jason described it well: by default, a "context" is saved at every await and used to continue the async method. This "context" is the current SynchronizationContext unless it it null, in which case it is the current TaskScheduler. When the async method attempts to continue, it first re-enters the captured "context" (in this case, an ASP.NET SynchronizationContext). The ASP.NET SynchronizationContext only permits one thread in the context at a time, and there is already a thread in the context - the thread blocked on Task.Result.
There are two guidelines that will avoid this deadlock:
Use async all the way down. You mention that you "can't" do this, but I'm not sure why not. ASP.NET MVC on .NET 4.5 can certainly support async actions, and it's not a difficult change to make.
Use ConfigureAwait(continueOnCapturedContext: false) as much as possible. This overrides the default behavior of resuming on the captured context.

I was in the same deadlock situation but in my case calling an async method from a sync method, what works for me was:
private static SiteMetadataCacheItem GetCachedItem()
{
TenantService TS = new TenantService(); // my service datacontext
var CachedItem = Task.Run(async ()=>
await TS.GetTenantDataAsync(TenantIdValue)
).Result; // dont deadlock anymore
}
is this a good approach, any idea?

Just to add to the accepted answer (not enough rep to comment), I had this issue arise when blocking using task.Result, event though every await below it had ConfigureAwait(false), as in this example:
public Foo GetFooSynchronous()
{
var foo = new Foo();
foo.Info = GetInfoAsync.Result; // often deadlocks in ASP.NET
return foo;
}
private async Task<string> GetInfoAsync()
{
return await ExternalLibraryStringAsync().ConfigureAwait(false);
}
The issue actually lay with the external library code. The async library method tried to continue in the calling sync context, no matter how I configured the await, leading to deadlock.
Thus, the answer was to roll my own version of the external library code ExternalLibraryStringAsync, so that it would have the desired continuation properties.
wrong answer for historical purposes
After much pain and anguish, I found the solution buried in this blog post (Ctrl-f for 'deadlock'). It revolves around using task.ContinueWith, instead of the bare task.Result.
Previously deadlocking example:
public Foo GetFooSynchronous()
{
var foo = new Foo();
foo.Info = GetInfoAsync.Result; // often deadlocks in ASP.NET
return foo;
}
private async Task<string> GetInfoAsync()
{
return await ExternalLibraryStringAsync().ConfigureAwait(false);
}
Avoid the deadlock like this:
public Foo GetFooSynchronous
{
var foo = new Foo();
GetInfoAsync() // ContinueWith doesn't run until the task is complete
.ContinueWith(task => foo.Info = task.Result);
return foo;
}
private async Task<string> GetInfoAsync
{
return await ExternalLibraryStringAsync().ConfigureAwait(false);
}

quick answer :
change this line
ResultClass slowTotal = asyncTask.Result;
to
ResultClass slowTotal = await asyncTask;
why? you should not use .result to get the result of tasks inside most applications except console applications if you do so your program will hang when it gets there
you can also try the below code if you want to use .Result
ResultClass slowTotal = Task.Run(async ()=>await asyncTask).Result;

Executing async function synchronously

I have done a lot of search on this topic, and I read most of the posts here on this site regarding this topic, however I'm still confused and I need a straight forward answer. Here is my situation:
I have an established Winform application that I can't make it all 'async'. I'm forced now to use an external library that is all written as async functions.
In my application I have
/// <summary>
/// This function I can't change it to an 'async'
/// </summary>
public void MySyncFunction()
{
//This function is my point in my application where I have to call the
//other 'async' functions but I can't change the function itself to 'async'
try
{
//I need to call the MyAsyncDriverFunction() as if it is a synchronous function
//I need the driver function to finish execution and return before processing the code that follows it
//I also need to be able to catch any exceptions
MyAsyncDriverFunction();
//Rest of the code have to wait for the above function to return
}
catch (Exception exp)
{
//Need to be able to handle the exception thrown
//from the MyAsyncDriverFunction here.
}
}
public static async Task<IEnumerable<string>> MyAsyncDriverFunction()
{
try
{
var strCollection = await AsyncExternalLibraryFunction1();
var strCollection2 = await AsyncExternalLibraryFunction2();
return strCollection;
}
catch (Exception exp)
{
//Need to be able to catch an exception and re-throw it to the caller function
}
}
As outlined in the code, I need to be able to:
I can't change my MySyncFunction to an async
Call the "MyAsyncDriverFunction" in a sync way, where it have to wait for it to finish all its work before I process the code that follows
Be able to handle exceptions in both functions (from what I read so far this is tricky?)
I need a simple way using the standard API, I can't use any third party library (even if I wanted to)

however I'm still confused and I need a straight forward answer.
That's because there isn't a "straight-forward" answer.
The only proper solution is to make MySyncFunction asynchronous. Period. All other solutions are hacks, and there is no hack that works perfectly in all scenarios.
I go into full details in my recent MSDN article on brownfield async development, but here's the gist:
You can block with Wait() or Result. As others have noted, you can easily cause a deadlock, but this can work if the asynchronous code never resumes on its captured context.
You can push the work to a thread pool thread and then block. However, this assumes that the asynchronous work is capable of being pushed to some other arbitrary thread and that it can resume on other threads, thus possibly introducing multithreading.
You can push the work to a thread pool thread that executes a "main loop" - e.g., a dispatcher or my own AsyncContext type. This assumes the asynchronous work is capable of being pushed to another thread but removes any concerns about multithreading.
You can install a nested message loop on the main thread. This will execute the asynchronous code on the calling thread, but also introduces reentrancy, which is extremely difficult to reason about correctly.
In short, there is no one answer. Every single approach is a hack that works for different kinds of asynchronous code.

Simply calling .Result or .Wait against your async method will deadlock because you're in the context of a GUI application. See https://msdn.microsoft.com/en-us/magazine/jj991977.aspx (chapter 'Async All the Way') for a nice explanation.
The solution to your problem is not easy, but it has been described in details by Stephen Cleary: here.
So you should use the Nito.AsyncEx library (available on Nuget).
If you really can't add the library he wrote to your project, you could check the source code and use portions of it, the MIT license allows it.

Just add a .Result call at the end of the method call.
var strCollection = MyAsyncDriverFunction().Result;

I'm not sure what the experts would say, but based on the Stephen Cleary advices I end up with the following idea. Having the following class
public sealed class AsyncTask
{
public static void Run(Func<Task> asyncFunc)
{
var originalContext = SynchronizationContext.Current;
bool restoreContext = false;
try
{
if (originalContext != null && originalContext.GetType() != typeof(SynchronizationContext))
{
restoreContext = true;
SynchronizationContext.SetSynchronizationContext(new SynchronizationContext());
}
var task = asyncFunc();
task.GetAwaiter().GetResult();
}
finally
{
if (restoreContext) SynchronizationContext.SetSynchronizationContext(originalContext);
}
}
public static TResult Run<TResult>(Func<Task<TResult>> asyncFunc)
{
var originalContext = SynchronizationContext.Current;
bool restoreContext = false;
try
{
if (originalContext != null && originalContext.GetType() != typeof(SynchronizationContext))
{
restoreContext = true;
SynchronizationContext.SetSynchronizationContext(new SynchronizationContext());
}
var task = asyncFunc();
return task.GetAwaiter().GetResult();
}
finally
{
if (restoreContext) SynchronizationContext.SetSynchronizationContext(originalContext);
}
}
}
and use it as follows
public void MySyncFunction()
{
try
{
AsyncTask.Run(() => MyAsyncDriverFunction());
}
catch (Exception exp)
{
}
}
would do what you are asking for without deadlock. The key point is to "hide" the current synchronization context during the asynchronous task execution and force using of the default synchronization context which is known to use thread pool for Post method. Again, I'm not sure if this is good or bad idea and what side effects it could introduce, but once you asked, I'm just sharing it.

Try change "await AsyncExternalLibraryFunction1()" to "AsyncExternalLibraryFunction1().Wait()" and next to it, and remove
async for function "MyAsyncDriverFunction"