We Keep Coding

How to handle a deadlock in third-party code

We have a third-party method Foo which sometimes runs in a deadlock for unknown reasons.
We are executing an single-threaded tcp-server and call this method every 30 seconds to check that the external system is available.
To mitigate the problem with the deadlock in the third party code we put the ping-call in a Task.Run to so that the server does not deadlock.
Like
async Task<bool> WrappedFoo()
{
var timeout = 10000;
var task = Task.Run(() => ThirdPartyCode.Foo());
var delay = Task.Delay(timeout);
if (delay == await Task.WhenAny(delay, task ))
{
return false;
}
else
{
return await task ;
}
}
But this (in our opinion) has the potential to starve the application of free threads. Since if one call to ThirdPartyCode.Foo deadlock the thread will never recover from this deadlock and if this happens often enough we might run out of resources.
Is there a general approach how one should handle deadlocking third-party code?
A CancellationToken won't work because the third-party-api does not provide any cancellation options.
Update:
The method at hand is from the SAPNCO.dll provided by SAP to establish and test rfc-connections to a sap-system, therefore the method is not a simple network-ping. I renamed the method in the question to avoid further misunderstandings

Is there a general approach how one should handle deadlocking third-party code?
Yes, but it's not easy or simple.
The problem with misbehaving code is that it can not only leak resources (e.g., threads), but it can also indefinitely hold onto important resources (e.g., some internal "handle" or "lock").
The only way to forcefully reclaim threads and other resources is to end the process. The OS is used to cleaning up misbehaving processes and is very good at it. So, the solution here is to start a child process to do the API call. Your main application can communicate with its child process by redirected stdin/stdout, and if the child process ever times out, the main application can terminate it and restart it.
This is, unfortunately, the only reliable way to cancel uncancelable code.

Cancelling a task is a collaborative operation in that you pass a CancellationToken to the desired method and externally you use CancellationTokenSource.Cancel:
public void Caller()
{
try
{
CancellationTokenSource cts=new CancellationTokenSource();
Task longRunning= Task.Run(()=>CancellableThirdParty(cts.Token),cts.Token);
Thread.Sleep(3000); //or condition /signal
cts.Cancel();
}catch(OperationCancelledException ex)
{
//treat somehow
}
}
public void CancellableThirdParty(CancellationToken token)
{
while(true)
{
// token.ThrowIfCancellationRequested() -- if you don't treat the cancellation here
if(token.IsCancellationRequested)
{
// code to treat the cancellation signal
//throw new OperationCancelledException($"[Reason]");
}
}
}
As you can see in the code above , in order to cancel an ongoing task , the method running inside it must be structured around the CancellationToken.IsCancellationRequested flag or simply CancellationToken.ThrowIfCancellationRequested method ,
so that the caller just issues the CancellationTokenSource.Cancel.
Unfortunately if the third party code is not designed around CancellationToken ( it does not accept a CancellationToken parameter ), then there is not much you can do.

Your code isn't cancelling the blocked operation. Use a CancellationTokenSource and pass a cancellation token to Task.Run instead :
var cts=new CancellationTokenSource(timeout);
try
{
await Task.Run(() => ThirdPartyCode.Ping(),cts.Token);
return true;
}
catch(TaskCancelledException)
{
return false;
}
It's quite possible that blocking is caused due to networking or DNS issues, not actual deadlock.
That still wastes a thread waiting for a network operation to complete. You could use .NET's own Ping.SendPingAsync to ping asynchronously and specify a timeout:
var ping=new Ping();
var reply=await ping.SendPingAsync(ip,timeout);
return reply.Status==IPStatus.Success;
The PingReply class contains far more detailed information than a simple success/failure. The Status property alone differentiates between routing problems, unreachable destinations, time outs etc

Tasks, priorities, scheduling and async/await

I'm trying to monitor a connection by performing regular 'pings' with a parallel 'timeout' task in case the ping doesn't come back in time.
var pingTask = await Task.WhenAny(
Task.Run(() =>
{
try
{
PingThing();
return true;
}
catch
{
return false;
}
}),
Task.Run(() =>
{
Task.Delay(5000).Wait();
return false;
})
);
if (!pingTask.Result) //etc...
I seem to be getting cases where 'other stuff' is taking priority, and this code gets put to one side. So where the pings normally take milliseconds, I'm getting five second timeouts.
The thing is, when I initially put a stopwatch inside the code to check, it didn't say anything like 5 seconds. Which has me wondering whether the CPU isn't shutting down my thread(s) to do other work. But that in turn makes me wonder why it would shut down the first Task, but keep the second one up (otherwise the 'timeout' Task would also take longer).
So I have several questions:
is this a plausible hypothesis?
is there a way to set the priority inside a task?
if so is that a good or bad idea?
is using await here a bad idea (does it somehow signal low priority)?
is there a way to monitor what all the threads are doing to try and work out what the 'other stuff' is that's stealing my processor time?
Leaky abstractions FTW...

How to call third party method and then if failed remove all resources its using

I am trying this code to convert a document using aspose word, I am trying to terminate the process of conversion of document when it takes too long (as it leads to memory leak and eats up all system resources), I never looked at threading or async before, but now as our production server is broken I am looking for a quick fix before digging any deeper,
This is what I tried but it kills the thread but keeps all the resources, I read some posts from What's wrong with using Thread.Abort() but I am not sure what is the best way to move forward, should I use Task and async but I don't know how to use them in this context,
RunWithTimeout(() =>
{
status = AsposeConversion.ConvertToPDF(licensePath, fileName);
}, TimeSpan.FromMilliseconds(1000 * 60 * 4));
public static bool RunWithTimeout(ThreadStart threadStart, TimeSpan timeout)
{
Thread workerThread = new Thread(threadStart);
workerThread.Start();
bool finished = workerThread.Join(timeout);
if (!finished)
workerThread.Abort();
return finished;
}
Should I create a new process ? but then how can I timeout it or run conversion code line in it.
Edit
Sorry for confusion, I said it takes too long, but in real it never returns even for 4 hours I tested so far..

Since AsposeConversion.ConvertToPDF is a blocking method, you cannot send a signal to tell the thread to end gracefully, as the thread will not check the signal until the blocking method has completed.
All I can suggest is to let the thread run until AsposeConversion.ConvertToPDF returns, then check within thread if the maximum allowed time has passed, and if so clean up before terminating.
Edit - After knowing the method call does not return at all
It looks like this is an issue with Aspose, so you should find out why that method call is never returning and see if there is something you are doing wrong, or any workaround if it is a bug.
If you don't manage to make the method work as expected, as an extreme solution you could try to run the code inside a Job object. It will allow you to execute your code as a separate process and limit the memory the process can use. You can then handle the OutOfMemoryException and clean up as necessary.
It is horrible, I know, but will prevent your server from crashing while you wait for a workaround / bug fix.

Creating a c# windows service to poll a database

I am wanting to write a service that polls a database and performs an operation depending on the data being brought back.
I am not sure what is the best way of doing this, I can find a few blogs about it and this stack overflow question Polling Service - C#. However I am wary that they are all quite old and possibly out of date.
Can anyone advise me on the current advice or best practices (if there are any) on doing something like this or point me in the direction of a more recent blog post about this. From what I can gather either using a timer or tpl tasks are two potential ways of doing this.
If timers are still suggested then how will they work when the service is stopped because the operations I intend for these services to do could potentially take 30+ minutes, this is why I say use tasks because I can use a task cancellation token but these throw exceptions when cancelled (correct me if I am wrong) and I don't think I really want that behaviour (although correct me if you think there is a reason I will want that).
Sorry that I may be asking quite a lot in a single question but I'm not entirely sure myself what I am asking.

Go with a Windows service for this. Using a scheduled task is not a bad idea per se, but since you said the polls can occur every 2 minutes then you are probably better off going with the service. The service will allow you to maintain state between polls and you would have more control over the timing of the polls as well. You said the operation might take 30+ minutes once it is kicked off so maybe you would want to defer polls until the operation complete. That is a bit easier to do when the logic is ran as a service.
In the end it does not really matter what mechanism you use to generate the polls. You could use a timer or a dedicated thread/task that sleeps or whatever. Personally, I find the dedicated thread/task easier to work with than a timer for these kinds of things because it is easier to control the polling interval. Also, you should definitely use the cooperative cancellation mechanism provided with the TPL. It does not necessary throw exceptions. It only does so if you call ThrowIfCancellationRequested. You can use IsCancellationRequested instead to just check the cancellation token's state.
Here is a very generic template you might use to get started.
public class YourService : ServiceBase
{
private CancellationTokenSource cts = new CancellationTokenSource();
private Task mainTask = null;
protected override void OnStart(string[] args)
{
mainTask = new Task(Poll, cts.Token, TaskCreationOptions.LongRunning);
mainTask.Start();
}
protected override void OnStop()
{
cts.Cancel();
mainTask.Wait();
}
private void Poll()
{
CancellationToken cancellation = cts.Token;
TimeSpan interval = TimeSpan.Zero;
while (!cancellation.WaitHandle.WaitOne(interval))
{
try
{
// Put your code to poll here.
// Occasionally check the cancellation state.
if (cancellation.IsCancellationRequested)
{
break;
}
interval = WaitAfterSuccessInterval;
}
catch (Exception caught)
{
// Log the exception.
interval = WaitAfterErrorInterval;
}
}
}
}
Like I said, I normally use a dedicated thread/task instead of a timer. I do this because my polling interval is almost never constant. I usually start slowing the polls down if a transient error is detected (like network or server availability issues) that way my log file does not fill up with the same error message over and over again in rapid succession.

You have a few options. To start with what could be essentially the easiest option, you could decide to create your app as a console application and run the executable as a task in the Windows Task Scheduler. All you would need to do is assign your executable as the program to start in the task and have the task scheduler handle the timing interval for you. This is probably the preferred way if you don't care about state and will prevent you from having to worry about creating and managing a windows service if you don't really need to. See the following link for how to use the scheduler.
Windows Task Scheduler
The next way you could do this would be to create a windows service and in that service use a timer, specifically System.Timers.Timer. Essentially you would set the timer interval to the amount of time you would like to have pass before you run your process. Then you would sign up for the timers tick event which would fire every time that interval occurred. In this event you would essentially have the process you would like to run; this could kick off addition threads if you would like. Then after that initial setup you would just call the timers Start() function or set the Enabled property to True to start the timer. A good example of what this would look like can be found in the example on MSDN page describing the object. There are plenty of tutorials out there that show how to set up a windows service so I won't bother with going into that specifically.
MSDN: System.Timers.Timer
Finally and more complex would be to set up a windows service that listens for a SqlDependency. This technique is useful if things can occur in the database outside your application yet you need to be made aware of it in your application or some other service. The following link has a good tutorial on how to set up a SqlDependency in an application.
Using SqlDependency To Monitor SQL Database Changes
Two things I would like to point out from your original post that are not specific to the question you had.
If you are writing a true windows service you don't want the service to stop. The service should be running constantly and if an exception does occur it should be handled appropriately and not stop the service.
A cancellation token doesn't have to throw an exception; simply not calling ThrowIfCancellationRequested() will cause the exception not to be thrown or if this is a CancellationTokenSource set the argument to false on the Cancel method then subsequently check the token to see if cancellation is requested in your threads and return out of the thread gracefully if so.
For example:
CancellationTokenSource cts = new CancellationTokenSource();
ParallelOptions options = new ParallelOptions
{
CancellationToken = cts.Token
};
Parallel.ForEach(data, options, i =>
{
try
{
if (cts.IsCancellationRequested) return;
//do stuff
}
catch (Exception ex)
{
cts.Cancel(false);
}
});

How can I return a response from a WCF call and then do more work?

I have a synchronous web service call that returns a message. I need to quickly return a message that basically says that order was received. I then need to spend a couple of minutes processing the order, but cannot block the service call for that long. So how can I return from the web service, and then do some more stuff? I'm guessing I need to fork some other thread or something before I return, but I'm not sure of the best approach.
string ProcessOrder(Order order)
{
if(order.IsValid)
{
return "Great!";
//Then I need to process the order
}
}

You can open a new thread and have it do what you need, while you're main thread returns great.
string ProcessOrder(Order order)
{
if(order.IsValid)
{
//Starts a new thread
ThreadPool.QueueUserWorkItem(th =>
{
//Process Order here
});
return "Great!";
}
}

You could start your big amount of work in a seperate thread
public string ProcessOrder(Order order)
{
if(order.IsValid)
{
System.Threading.ParameterizedThreadStart pts = new System.Threading.ParameterizedThreadStart(DoHardWork);
System.Threading.Thread t = new System.Threading.Thread(pts);
t.Start(order);
return "Great!!!";
}
}
public void DoHardWork(object order)
{
//Stuff Goes Here
}

Is the work you're doing "important?" I assume it is. You could use a thread, but you'll have to be ok with the possibility that your work might get interrupted if the machine restarts or possibly if the asp.net worker process recycles. This would likely lead to the work not getting done even though you already told the client you had accepted it. This might be or not depending on your use case.
I would consider taking the work item you receive from the synchronous service request and putting it in a persistent queue. An easy way to do this is to use a transational MSMQ queue. Your synchronous service puts the work request in the queue and you have a few worker threads pulling work requests out of the queue. Wrap your queue read and the work in a transaction and don't commit the transaction until the work is completed. If you machine or process shuts down in the middle of a request, it will be restarted automatically the next time it starts up.

You could also look at utilizing the PIAB (Policy Injection Application Block) to accomplish work after a method call.