I'm using CircuitBreaker with Polly the first time and I'm facing a problem:
My program executes everything without throwing an Error.
As soon as I get to this codeline, the program Executes it and nothing happens. I cant debug any further. The program is executing this method for ever without a timeout:
var cB = Policy.Handle<Exception>()
.CircuitBreakerAsync(3, TimeSpan.FromMinutes(1),
(ex, t) =>
{
Console.WriteLine("Circuit broken!");
},
() =>
{
Console.WriteLine("Circuit Reset!");
});
HttpClient client = new HttpClient();
var response = await cB.ExecuteAsync((ct) => // <--- Executing this line of code lasts forever
{
var request = new HttpRequestMessage(HttpMethod.Get, new Uri(
endPoint
));
return client.SendAsync(request, ct);
}, System.Threading.CancellationToken.None);
This is a .NET Framework 4.8 Project.
I created a new Project. Also a .NET Framework 4.8 Project and copy-pasted the code in the picture
into the new project.
And: It runs fine!!
So I suggest, there is a configuration? or an adjustment? in the Project I'm working in, which isn't allowing Polly to work correctly.
But what can it be? I get no feedback by e.g. an error...
Does somebody know what can interact with Polly like that?
The observed behaviour was that the application did not make any progress and get stuck at the await ExecuteAsync line. Without knowing what the ExecuteAsync does people can suspect:
either we have an infinite loop
or we have a deadlock
Since the Circuit Breaker implementation (so called Engine) does not perform any kind of things that can cause infinite loop that's why we can exclude that as a possible root cause.
cancellationToken.ThrowIfCancellationRequested();
breakerController.OnActionPreExecute();
try
{
TResult result = action(context, cancellationToken);
if (shouldHandleResultPredicates.AnyMatch(result))
{
breakerController.OnActionFailure(new DelegateResult<TResult>(result), context);
}
else
{
breakerController.OnActionSuccess(context);
}
return result;
}
catch (Exception ex)
{
Exception handledException = shouldHandleExceptionPredicates.FirstMatchOrDefault(ex);
if (handledException == null)
{
throw;
}
breakerController.OnActionFailure(new DelegateResult<TResult>(handledException), context);
if (handledException != ex)
{
ExceptionDispatchInfo.Capture(handledException).Throw();
}
throw;
}
So the deadlock has remained as our primary suspect. Back in the old days the ASP.NET MVC was using SynchronizationContext. As Stephen Clearly describe it in detail it was used to continue the operation on the same thread as the invoker of the async method.
In other words the awaiter and the awaitee are waiting for each other. The caller thread waits for the ExecuteAsync to finish and the ExecuteAsync waits for the caller thread to continue.
With the ConfigureAwait(false) you are basically stating that please do not capture the SynchronizationContext, so the ExecuteAsync can continued on any thread it should not be the caller thread.
In case of ASP.NET Core we don't have to worry about this because the SynchronizationContext.Current is null :)
Related
Background (ignorable)
I am trying to re-create a bug situation on my development machine, described in this github issue: https://github.com/dotnet/aspnetcore/issues/28568. The actual bug I am seeing is when my application runs in Kubernetes. The details of that bug are documented here: Handling Errors in Health Checks
Problem
I am trying to get an HttpClient to abort a connection, to hopefully cause a System.Threading.Tasks.TaskCanceledException in my ASP.Net Core WebAPI Service. The text I am working off (found in the GitHub issue linked above) says that my issue happens:
when the client disconnect[s] before the request has been completely processed and is caused by the HttpContext.RequestAborted cancellation token.
I need to recreate this issue on my development machine. To do that I have created a console application with an HttpClient that calls my service endpoint. I have tried two different ways to recreate this so far. First I call:
var sendTask = httpClient.SendAsync(message, cancellationTokenSource.Token);
Then, in the first attempt I called:
cancellationTokenSource.Cancel();
In the second attempt, I took out the call to Cancel and put in this:
httpClient.Dispose();
I put the logic in a loop and ran it a bunch of times, but I was unable to reproduce the exception I am looking for (TaskCanceledException) in the service (confusingly it is thrown in the client when Cancel is called).
Is there a way to case the HttpClient to disconnect before the call is done?
I realized that I had not been resetting my CancellationTokenSource after I called Cancel. Once I did that, about 1 in 20ish calls started throwing the TaskCanceledException in my service.
Just incase others want it (and for later reference), here is the code that I used to generate the error:
var url = "https://localhost:7249/health/liveness";
var httpClient = new HttpClient();
for (int loopIndex = 0; loopIndex < 1000; loopIndex++)
{
try
{
CancellationTokenSource cancellationTokenSource = new CancellationTokenSource();
var message = new HttpRequestMessage(HttpMethod.Get, url);
// Important: don't await this or it will cause the call to finish before continuing.
httpClient.SendAsync(message, cancellationTokenSource.Token);
cancellationTokenSource.Cancel();
}
catch (Exception e) { }
Console.WriteLine($"Next {loopIndex}");
}
Console.WriteLine("All Done");
I am using a BackgroundService object in an aspnet core application.
Regarding the way the operations that run in the ExecuteAsync method are implemented, the Aspnet core fails to initialize or stop correctly. Here is what I tried:
I implemented the abstract ExecuteAsync method the way it is explained in the documentation.
the pipeline variable is an IEnumerable<IPipeline> that is injected in the constructor.
public interface IPipeline {
Task Start();
Task Cycle();
Task Stop();
}
...
protected override async Task ExecuteAsync(CancellationToken stoppingToken) {
log.LogInformation($"Starting subsystems");
foreach(var engine in pipeLine) {
try {
await engine.Start();
}
catch(Exception ex) {
log.LogError(ex, $"{nameof(engine)} failed to start");
}
}
log.LogInformation($"Runnning main loop");
while(!stoppingToken.IsCancellationRequested) {
foreach(var engine in pipeLine) {
try {
await engine.Cycle();
}
catch(Exception ex) {
log.LogError(ex, $"{engine.GetType().Name} error in Cycle");
}
}
}
log.LogInformation($"Stopping subsystems");
foreach(var engine in pipeLine) {
try {
await engine.Stop();
}
catch(Exception ex) {
log.LogError(ex, $"{nameof(engine)} failed to stop");
}
}
}
Because of the current development state of the project, there are many "nop" Pipeline that contains an empty Cycle() operation that is implemented this way:
public async Task Cycle() {
await Task.CompletedTask;
}
What I noticed is:
If at least one IPipeline object contains an actual asynchronous method (await Task.Delay(1)), then everything runs smoothly and I can stop the service gracefully using CTRL+C.
If all IPipeline objects contains await Task.CompletedTask;,
Then on one hand, aspnetcore fails to initialize correctly. I mean, there is no "Now listening on: http://[::]:10001 Application started. Press Ctrl+C to shut down." on the console.
On the other, when I hit CTRL+C, the console shows "Application is shutting down..." but the cycle loop continues to run as if the CancellationToken was never requested to stop.
So basically, if I change a single Pipeline object to this:
public async Task Cycle() {
await Task.Delay(1);
}
Then everything is fine, and I dont understand why. Can someone explain me what I did not understood regarding Task processing ?
The simplest workaround is to add await Task.Yield(); as line one of ExecuteAsync.
I am not an expert... but the "problem" is that all the code inside this ExecuteAsync actually running synchronously.
If all the "cycles" return a Task that has completed synchronously (as Task.CompletedTask will be) then the while and therefore the ExecuteAsync method never "yield"s.
The framework essentially does a foreach over the registered IHostedServices and calls StartAsync. If your service does not yield then the foreach gets blocked. So any other services (including the AspNetCore host) will not be started. As bootstrapping cannot finish, things like ctrl-C handling etc also never get setup.
Putting await Task.Delay(1) in one of the cycles "releases" or "yields" the Task. This allows the host to "capture" the task and continue. Which allows the wiring up of Cancellation etc to happen.
Putting Task.Yield() at the top of ExecuteAsync is just the more direct way of achieving this and means the cycles do not need to be aware of this "issue" at all.
note: this is usually only really an issue in testing... 'cos why would you have a no-op cycle?
note2: If you are likely to have "compute" cycles (ie they don't do any IO, database, queues etc) then switching to ValueTask will help with perf/allocations.
I call the below code once every second to poll a camera but after running for a day or two, it stops working.
public List<CameraEvent> GetEventsSince(CaptureTime afterDateTime)
{
string uri = string.Format(
"http://{0}/ISAPI/channels/1/events/detect", _ipAddress);
using (var client = new HttpClient())
{
client.Timeout = TimeSpan.FromSeconds(5);
AddBasicAuth(client);
try
{
HttpResponseMessage response =
client.PostAsync(
uri, new StringContent(GetPicTimeXml(afterDateTime))).Result;
logger.Debug(
string.Format("Status code response={0}", response.StatusCode));
if (response.StatusCode == HttpStatusCode.Unauthorized ||
response.StatusCode == HttpStatusCode.Forbidden)
{
// 401
currentState = 2;
return new List<CameraEvent>();
}
if (response.StatusCode == HttpStatusCode.OK)
{
// OK
currentState = 0;
}
List<CameraEvent> events = new CameraHttpResponseHandler()
.HandleHttpResponse(response);
AppendPlateImages(events);
return events;
}
catch (AggregateException ex)
{
//if (ex.InnerException is TaskCanceledException)
//{
// // Timeout
// currentState = 1;
//}
logger.Error("AggregateException", ex);
}
catch (Exception ex)
{
logger.Error("Generic exception", ex);
}
return new List<CameraEvent>();
}
}
The error I get is:
2015-08-17 07:59:57,310 [16] ERROR CameraHttpClient AggregateException
System.AggregateException: One or more errors occurred. --->
System.InvalidOperationException: There were not enough free threads
in the ThreadPool to complete the operation.
The parent thread calling GetEventsSince is a background worker thread running in a loop if that makes any difference.
Has anyone seen this issue or have any suggestions on what might be causing threads to be used up?
Hard to say for certain, but if the root cause of the threadpool starvation is this method, then it's a great example of why asynchronous code is beneficial on the server.
HttpClient is an asynchronous API, meaning that if you properly await calls, you free up a thread and send it back to the threadpool until the call returns. By calling .Result, you are blocking the thread for the entire duration of the call. Say this method takes several seconds beginning to end, and that 99.9% of that time is waiting on I/O (not an unreasonable guess). The way it is, you are consuming a thread for 100% of that time. If you refactor it to run asynchronously, your thread consumption drops to 0.1% of the time, and the threadpool is suddenly much fuller on average.
So I would begin by marking the method async (use Task<List<CameraEvent>> as the return type) and using await instead of .Result where asynchronous APIs are used. I don't know what CameraHttpResponseHandler.HandleHttpResponse does exactly, but I'm guessing there's blocking on I/O there too and it should be converted as well as well and called using await.
This has implications on how the root application calls this method. I'd need to see that code to suggest the best approach. TPL Dataflow might be a good fit here - not only is it helpful in calling async methods at regular intervals, but it also supports throttling concurrency as sort of a safeguard against problems like this.
I have an ASP.NET MVC app with WebAPI controllers, and a console app that uses these controllers. The console app runs from a scheduled task and fetches data from remote sources, parses it, and posts it to the MVC app before exiting.
This works well for several of the controllers, but one of the calls is crashing the console app without throwing an exception. The caller code which is used for all the controllers:
public async Task<string> Post<T>(string urlRoot, string url, T data)
{
var result = "";
try
{
var httpClient = GetHttpClient(urlRoot);
var response = await httpClient.PostAsJsonAsync(url, data); // Exits here
if (response.IsSuccessStatusCode)
{
result = await response.Content.ReadAsAsync<string>();
}
}
catch (Exception e)
{
Debug.WriteLine(e.ToString());
}
return result;
}
The program exits when calling await httpClient.PostAsJsonAsync(url, data). Using breakpoints, neither the catch block nor the if statement are reached. The call is being made however, as the web API controller is being called with the correct data.
The programs share the same code for the T being passed across the API call.
From the output window:
The program '[9640] ProgramName.vshost.exe: Managed (v4.0.30319)' has exited with code 0 (0x0).
I was wondering whether the size of the posted data could be a problem, but I've not found any documentation stating size limits.
So my questions are:
What could be causing the console app to prematurely exit?
How can I detect this and prevent the program exiting?
One of possible problems, that you just don't await execution of Post method. Here is simplified version of what i am talking about:
static void Main(string[] args)
{
Action testAction = async () =>
{
Console.WriteLine("In");
await Task.Delay(100);
Console.WriteLine("First delay");
await Task.Delay(100);
Console.WriteLine("Second delay");
};
testAction.Invoke();
Thread.Sleep(150);
}
testAction will be aborted on second await Task, and console exits with 0 code. And output will be:
In
First delay
Press any key to continue . . .
In my case, i would just add Console.ReadKey() call at the end of Main method. In your case something different might be required
Please take a look at this question witch enumerates some Exceptions you need to handle:
.NET Global exception handler in console application
Application.ThreadException += MYThreadHandler;
AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(CurrentDomain_UnhandledException);
Also consider this aproach for Task Exceptions:
static void Main(string[] args)
{
Task<int> task = new Task<int>(Test);
task.ContinueWith(ExceptionHandler, TaskContinuationOptions.OnlyOnFaulted);
task.Start();
Console.ReadLine();
}
static int Test()
{
throw new Exception();
}
static void ExceptionHandler(Task<int> task)
{
var exception = task.Exception;
Console.WriteLine(exception);
}
You can find more detail in one of the answers of this question:
catch exception that is thrown in different thread
Once you hit await control will be returned to the caller of Post.
Rather than have await in the post method, use ContinueWith from the returned task, something like: Task continuation on UI thread or Wait on the returned task: http://msdn.microsoft.com/en-gb/library/dd537610.aspx
I would suggest testing to see if the JSON.Net serializer is capable of serializing your type.
"exited with code 0" means the program exited gracefully. Thus, it might be the caller of Post<T> or even above callers determine that it was time to end the program. If you are familiar with debugging tools such as WinDbg, you can set native breakpoints at ntdll functions to further diagnose the case, but normally you just need to review your code base.
I am testing for using Task in my application.
Now, I have done the fallowing:
var task = Task.Factory.StartNew(() => {
// long running process
throw new Exception("test"); // throwing TestException
});
task.ContinueWith(x => MyErrorHandler(task.Exception), TaskContinuationOptions.OnlyOnFaulted);
void MyErrorHandler(Exception error) {
MessageBox.Show(error.Message);
}
The idea is, that the long running process will executed and the user can working on without any blocking of the UI. If there was a problem (exception) it will be show, after the long running process has finished (in the normal case, there would be no exception)
Is this the correct way I have used it or does I have to do it on another way? Are there any problems, which I can get on this way and I don't see now?
This will work, as you're explicitly checking Task.Exception, which will prevent the exception from being unobserved.
I would make a couple of recommendations here.
First, if this is truly a long running task, you might want to specify that:
var task = Task.Factory.StartNew(() => {
// long running process
throw new Exception("test"); // throwing TestException
}, TaskCreationOptions.LongRunning);
Second, you don't need the closure over task:
// use x.Exception, since x is the task
task.ContinueWith(x => MyErrorHandler(x.Exception), TaskContinuationOptions.OnlyOnFaulted);
You may also want to have this run on the main thread, especially if you decide you want to use something more elaborate (in your UI) for reporting:
// This will work if MyErrorHandler uses a UI control, since it'll marshal back to the current synchronization context
task.ContinueWith(x => MyErrorHandler(x.Exception),
CancellationToken.None,
TaskContinuationOptions.OnlyOnFaulted,
TaskScheduler.FromCurrentSynchronizationContext());
(This is only required if you're going to use UI controls, etc, in your error handler.)
Also - If you're using .NET 4.5 or the async targetting pack for .NET 4, you can simplify this by using the new async/await support. If you flag your method as async, you can do:
try
{
await Task.Factory.StartNew(() => {
// long running process
throw new Exception("test"); // throwing TestException
}, TaskCreationOptions.LongRunning);
}
catch(Exception error)
{
MyErrorHandler(error);
}