We Keep Coding

Polly weird behavior (infinite loop?)

I'm using CircuitBreaker with Polly the first time and I'm facing a problem:
My program executes everything without throwing an Error.
As soon as I get to this codeline, the program Executes it and nothing happens. I cant debug any further. The program is executing this method for ever without a timeout:
var cB = Policy.Handle<Exception>()
.CircuitBreakerAsync(3, TimeSpan.FromMinutes(1),
(ex, t) =>
{
Console.WriteLine("Circuit broken!");
},
() =>
{
Console.WriteLine("Circuit Reset!");
});
HttpClient client = new HttpClient();
var response = await cB.ExecuteAsync((ct) => // <--- Executing this line of code lasts forever
{
var request = new HttpRequestMessage(HttpMethod.Get, new Uri(
endPoint
));
return client.SendAsync(request, ct);
}, System.Threading.CancellationToken.None);
This is a .NET Framework 4.8 Project.
I created a new Project. Also a .NET Framework 4.8 Project and copy-pasted the code in the picture
into the new project.
And: It runs fine!!
So I suggest, there is a configuration? or an adjustment? in the Project I'm working in, which isn't allowing Polly to work correctly.
But what can it be? I get no feedback by e.g. an error...
Does somebody know what can interact with Polly like that?

The observed behaviour was that the application did not make any progress and get stuck at the await ExecuteAsync line. Without knowing what the ExecuteAsync does people can suspect:
either we have an infinite loop
or we have a deadlock
Since the Circuit Breaker implementation (so called Engine) does not perform any kind of things that can cause infinite loop that's why we can exclude that as a possible root cause.
cancellationToken.ThrowIfCancellationRequested();
breakerController.OnActionPreExecute();
try
{
TResult result = action(context, cancellationToken);
if (shouldHandleResultPredicates.AnyMatch(result))
{
breakerController.OnActionFailure(new DelegateResult<TResult>(result), context);
}
else
{
breakerController.OnActionSuccess(context);
}
return result;
}
catch (Exception ex)
{
Exception handledException = shouldHandleExceptionPredicates.FirstMatchOrDefault(ex);
if (handledException == null)
{
throw;
}
breakerController.OnActionFailure(new DelegateResult<TResult>(handledException), context);
if (handledException != ex)
{
ExceptionDispatchInfo.Capture(handledException).Throw();
}
throw;
}
So the deadlock has remained as our primary suspect. Back in the old days the ASP.NET MVC was using SynchronizationContext. As Stephen Clearly describe it in detail it was used to continue the operation on the same thread as the invoker of the async method.
In other words the awaiter and the awaitee are waiting for each other. The caller thread waits for the ExecuteAsync to finish and the ExecuteAsync waits for the caller thread to continue.
With the ConfigureAwait(false) you are basically stating that please do not capture the SynchronizationContext, so the ExecuteAsync can continued on any thread it should not be the caller thread.
In case of ASP.NET Core we don't have to worry about this because the SynchronizationContext.Current is null :)

How to handle a deadlock in third-party code

We have a third-party method Foo which sometimes runs in a deadlock for unknown reasons.
We are executing an single-threaded tcp-server and call this method every 30 seconds to check that the external system is available.
To mitigate the problem with the deadlock in the third party code we put the ping-call in a Task.Run to so that the server does not deadlock.
Like
async Task<bool> WrappedFoo()
{
var timeout = 10000;
var task = Task.Run(() => ThirdPartyCode.Foo());
var delay = Task.Delay(timeout);
if (delay == await Task.WhenAny(delay, task ))
{
return false;
}
else
{
return await task ;
}
}
But this (in our opinion) has the potential to starve the application of free threads. Since if one call to ThirdPartyCode.Foo deadlock the thread will never recover from this deadlock and if this happens often enough we might run out of resources.
Is there a general approach how one should handle deadlocking third-party code?
A CancellationToken won't work because the third-party-api does not provide any cancellation options.
Update:
The method at hand is from the SAPNCO.dll provided by SAP to establish and test rfc-connections to a sap-system, therefore the method is not a simple network-ping. I renamed the method in the question to avoid further misunderstandings

Is there a general approach how one should handle deadlocking third-party code?
Yes, but it's not easy or simple.
The problem with misbehaving code is that it can not only leak resources (e.g., threads), but it can also indefinitely hold onto important resources (e.g., some internal "handle" or "lock").
The only way to forcefully reclaim threads and other resources is to end the process. The OS is used to cleaning up misbehaving processes and is very good at it. So, the solution here is to start a child process to do the API call. Your main application can communicate with its child process by redirected stdin/stdout, and if the child process ever times out, the main application can terminate it and restart it.
This is, unfortunately, the only reliable way to cancel uncancelable code.

Cancelling a task is a collaborative operation in that you pass a CancellationToken to the desired method and externally you use CancellationTokenSource.Cancel:
public void Caller()
{
try
{
CancellationTokenSource cts=new CancellationTokenSource();
Task longRunning= Task.Run(()=>CancellableThirdParty(cts.Token),cts.Token);
Thread.Sleep(3000); //or condition /signal
cts.Cancel();
}catch(OperationCancelledException ex)
{
//treat somehow
}
}
public void CancellableThirdParty(CancellationToken token)
{
while(true)
{
// token.ThrowIfCancellationRequested() -- if you don't treat the cancellation here
if(token.IsCancellationRequested)
{
// code to treat the cancellation signal
//throw new OperationCancelledException($"[Reason]");
}
}
}
As you can see in the code above , in order to cancel an ongoing task , the method running inside it must be structured around the CancellationToken.IsCancellationRequested flag or simply CancellationToken.ThrowIfCancellationRequested method ,
so that the caller just issues the CancellationTokenSource.Cancel.
Unfortunately if the third party code is not designed around CancellationToken ( it does not accept a CancellationToken parameter ), then there is not much you can do.

Your code isn't cancelling the blocked operation. Use a CancellationTokenSource and pass a cancellation token to Task.Run instead :
var cts=new CancellationTokenSource(timeout);
try
{
await Task.Run(() => ThirdPartyCode.Ping(),cts.Token);
return true;
}
catch(TaskCancelledException)
{
return false;
}
It's quite possible that blocking is caused due to networking or DNS issues, not actual deadlock.
That still wastes a thread waiting for a network operation to complete. You could use .NET's own Ping.SendPingAsync to ping asynchronously and specify a timeout:
var ping=new Ping();
var reply=await ping.SendPingAsync(ip,timeout);
return reply.Status==IPStatus.Success;
The PingReply class contains far more detailed information than a simple success/failure. The Status property alone differentiates between routing problems, unreachable destinations, time outs etc

Task gets stuck in "[Scheduled and Waiting to Run]"

I've run into an issue with tasks I can't seem to figure out. This application makes repeated HTTP calls via WebClient to several servers. It maintains a dictionary of tasks that are running the HTTP calls, and every five seconds it checks for results, then once the results are in it makes an HTTP call again. This goes on for the lifetime of the application.
Recently, it has started having a problem where the tasks are randomly getting stuck in WaitingForActivation. In the debugger the task shows as "[Scheduled and waiting to run]", but it never runs.
This is the function that it's running, when I click on the "Scheduled" task in the debugger, it points to the DownloadStringTaskAsync() line:
private static async Task<string> DownloadString(string url)
{
using (var client = new WebClient()) {
try {
var result = await client.DownloadStringTaskAsync(url).ConfigureAwait(false);
return result;
} catch (WebException) {
return null;
}
}
}
The code that is actually creating the task that runs the above function is this. It only hits this line once the existing task is completed, Task.IsCompleted never returns true since it's stuck in scheduled status. Task.Status gets stuck in WaitingForActivation.
tasks[resource] = Task.Run(() => DownloadString("http://" + resources[resource] + ":8181/busy"));
The odd thing about this is that, as far as I can tell, this code ran perfectly fine for two years, until we recently did a server migration which included an upgraded OS and spawned a network packet loss issue. That's when we started noticing this particular problem, though I don't see how either of those would be related.
Also, this tends to only happen after the application has been running for several thousand seconds. It runs perfectly fine for a while until tasks, one-by-one, start getting stuck. After about a day, there's usually four or five tasks stuck in scheduled. Since it usually takes time for the first task to get stuck, that seems to me like there would be a race condition of some sort, but I don't see how that could be the case.
Is there a reason a task would get stuck in scheduled and never actually run?

I'm not familiar with ancient WebClient (maybe it contains bugs) but can suggest the recommended by Microsoft way to get a response from a server using System.Net.Http.HttpClient. Also HttpClient is rather faster works with multiple requests per endpoint, especially in .NET Core/.NET 5.
// HttpClient is intended to be instantiated once per application, rather than per-use
private static readonly HttpClient client = new HttpClient();
private static async Task<string> DownloadString(string url)
{
try
{
return await client.GetStringAsync(url).ConfigureAwait(false);
}
catch (HttpRequestException ex)
{
Debug.WriteLine(ex.Message);
return null;
}
}
Also remove Task.Run, it's a kind of redundancy.
tasks[resource] = DownloadString($"http://{resources[resource]}:8181/busy");
Asynchronous programming - read the article. You have to get a difference between I/O-bound and CPU-bound work, and don't spawn Threads without special need for concurrency. You need no Thread here.

Process ends without exception during asynchronous operation

Ok, so I got this little call to a public API using HttpClient:
using (var httpClient = new HttpClient()) {
var response = await httpClient.GetAsync(uri);
}
As soon as the operation reaches the GetAsync() line, the application closes without any exception, neither in the call stack nor in the event callback I've set up to catch unhandled app domain exceptions. I've already figured out it might be because await observes the executing task and supresses any exceptions, however if I remove the await keyword, the operation runs fine! What could be the problem here?

You probably fire that operation without waiting/awaiting for that operation to complete. So the asynchronous operation is fired, but the application goes on and ends before the operation itself has had a chance to complete.
Simply await the root call (or use Wait on the task if you can't use await in that context).

HttpClient.PostAsJsonAsync crashing without throwing exception

I have an ASP.NET MVC app with WebAPI controllers, and a console app that uses these controllers. The console app runs from a scheduled task and fetches data from remote sources, parses it, and posts it to the MVC app before exiting.
This works well for several of the controllers, but one of the calls is crashing the console app without throwing an exception. The caller code which is used for all the controllers:
public async Task<string> Post<T>(string urlRoot, string url, T data)
{
var result = "";
try
{
var httpClient = GetHttpClient(urlRoot);
var response = await httpClient.PostAsJsonAsync(url, data); // Exits here
if (response.IsSuccessStatusCode)
{
result = await response.Content.ReadAsAsync<string>();
}
}
catch (Exception e)
{
Debug.WriteLine(e.ToString());
}
return result;
}
The program exits when calling await httpClient.PostAsJsonAsync(url, data). Using breakpoints, neither the catch block nor the if statement are reached. The call is being made however, as the web API controller is being called with the correct data.
The programs share the same code for the T being passed across the API call.
From the output window:
The program '[9640] ProgramName.vshost.exe: Managed (v4.0.30319)' has exited with code 0 (0x0).
I was wondering whether the size of the posted data could be a problem, but I've not found any documentation stating size limits.
So my questions are:
What could be causing the console app to prematurely exit?
How can I detect this and prevent the program exiting?

One of possible problems, that you just don't await execution of Post method. Here is simplified version of what i am talking about:
static void Main(string[] args)
{
Action testAction = async () =>
{
Console.WriteLine("In");
await Task.Delay(100);
Console.WriteLine("First delay");
await Task.Delay(100);
Console.WriteLine("Second delay");
};
testAction.Invoke();
Thread.Sleep(150);
}
testAction will be aborted on second await Task, and console exits with 0 code. And output will be:
In
First delay
Press any key to continue . . .
In my case, i would just add Console.ReadKey() call at the end of Main method. In your case something different might be required

Please take a look at this question witch enumerates some Exceptions you need to handle:
.NET Global exception handler in console application
Application.ThreadException += MYThreadHandler;
AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(CurrentDomain_UnhandledException);
Also consider this aproach for Task Exceptions:
static void Main(string[] args)
{
Task<int> task = new Task<int>(Test);
task.ContinueWith(ExceptionHandler, TaskContinuationOptions.OnlyOnFaulted);
task.Start();
Console.ReadLine();
}
static int Test()
{
throw new Exception();
}
static void ExceptionHandler(Task<int> task)
{
var exception = task.Exception;
Console.WriteLine(exception);
}
You can find more detail in one of the answers of this question:
catch exception that is thrown in different thread

Once you hit await control will be returned to the caller of Post.
Rather than have await in the post method, use ContinueWith from the returned task, something like: Task continuation on UI thread or Wait on the returned task: http://msdn.microsoft.com/en-gb/library/dd537610.aspx

I would suggest testing to see if the JSON.Net serializer is capable of serializing your type.

"exited with code 0" means the program exited gracefully. Thus, it might be the caller of Post<T> or even above callers determine that it was time to end the program. If you are familiar with debugging tools such as WinDbg, you can set native breakpoints at ntdll functions to further diagnose the case, but normally you just need to review your code base.