I'm making calls to an external API and want to deal with the event that a call returns an Unauthorized HttpResponseMessage. When this happens I want to refresh the access token and make the call again.
I'm trying to use Polly with the following code:
public async Task<HttpResponseMessage> MakeGetRequestAsync()
{
var retryPolicy = Policy
.HandleResult<HttpResponseMessage>(r => r.StatusCode == HttpStatusCode.Unauthorized)
.Retry(1, (exception, retryCount) =>
{
RefreshAccessToken();
});
var result = await retryPolicy.ExecuteAsync(() => CallApiAsync());
return result;
}
private async Task<HttpResponseMessage> CallApiAsync()
{
var url = Options.ResourceSandboxUrl;
var httpClient = new HttpClient();
SetRequestHeaders(httpClient);
var response = await httpClient.GetAsync(url);
response.StatusCode = HttpStatusCode.Unauthorized;
return response;
}
I put breakpoints on the ExecuteAsync statement and in DoSomethingAsync - when I step over ExecuteAsync DoSomethingAsync is not called and control is returned to the function that called MakeGetRequestAsync
I don't understand why DoSomethingAsync is not called - can anyone help me with what I'm trying to achieve?
I've looked at the Polly documentation & Polly questions on Stack Overflow but I can't figure out what's going on..
To use ExecuteAsync() you must declare the policy as .RetryAsync(...), not .Retry(...).
If your actual code reads exactly as the code sample above, the .ExecuteAsync(...) will be throwing for the mismatch between .Retry(...) [a sync policy] and .ExecuteAsync(...) [an async execution]. Since this exception is thrown, CallApiAsync() is indeed never invoked. You should be able to see the thrown exception, when calling MakeGetRequestAsync()
Overall code approach looks good tho: this retry-refreshing-authentication is a proven pattern with Polly!
I'm replying to this old question just to point out the Polly wiki page where this pattern was official documented:
retry-to-refresh-authorization
In particular this is the code snippet suggested:
var authorisationEnsuringPolicy = Policy
.HandleResult<HttpResponseMessage>(r => r.StatusCode == HttpStatusCode.Unauthorized)
.RetryAsync(
retryCount: 1, // Consider how many retries. If auth lapses and you have valid credentials, one should be enough; too many tries can cause some auth systems to blacklist.
onRetryAsync: async (outcome, retryNumber, context) => FooRefreshAuthorizationAsync(context),
/* more configuration */);
var response = authorisationEnsuringPolicy.ExecuteAsync(context => DoSomethingThatRequiresAuthorization(context), cancellationToken);
The FooRefreshAuthorizationAsync(...) method can obtain a new authorization token and pass it to the delegate executed through the policy using Polly.Context.
I might be late to the game but I leave here a post for future readers. I have created two slightly different solutions for this refresh token with retry problem.
Retry policy, Delegating Handler and Custom exception
Here is sequence diagram which depicts the communication flow
Here is the full source code
Retry policy, Delegating Handler and Polly.Context
This version separates the responsibilities in a different way:
Here is sequence diagram which depicts the communication flow
Here is the full source code
this is how async await work in .net, when the execution of your code reaches an await, two things will happened
the current thread of your code should release to make your code async, that means, your method should return
when the task you await is complete, your method should continue from where it used to be
Related
I'm using flurl and polly as part of my code to post asynchronously - I'm having an issue with the retry policy and the file content getting disposed, on a retry the FileContent option has been disposed.
I will get the following error:
System.ObjectDisposedException: Cannot access a disposed object. Object name: 'Flurl.Http.Content.FileContent'.
If someone could advise where best to place the logic to ensure FileContent is available for the retry
var endpoint = $"api endpoint here";
if (!File.Exists(filePath))
{
return Task.CompletedTask;
}
var fileBytesLength = _fileSystem.FileInfo.FromFileName(filePath).Length;
var httpContent = new Flurl.Http.Content.FileContent(filePath, (int)fileBytesLength);
return _statusRetryPolicy
.StatusAsyncRetryPolicy()
.ExecuteAsync(() => endpoint
.WithOAuthBearerToken(accessToken)
.WithTimeout(TimeSpan.FromMinutes(timeOutMinutes))
.WithHeader("Content-Type", "application/zip")
.OnError((flurlCall) =>
{
_logger.LogError(flurlCall.Exception, flurlCall.Exception.Message);
})
.PostAsync(httpContent));
and the Retry Policy:
public AsyncRetryPolicy StatusAsyncRetryPolicy() => Policy
.Handle<FlurlHttpException>(RetryPolicyHelpers.IsTransientError)
.WaitAndRetryAsync(5, retryAttempt =>
{
var nextAttemptIn = TimeSpan.FromSeconds(Math.Pow(2, retryAttempt));
_logger.LogWarning($"StatusRetryPolicy: Retry attempt {retryAttempt} to make request. Next try in {nextAttemptIn.TotalSeconds} seconds.");
return nextAttemptIn;
});
I have several suggestions for your code.
!File.Exists
I assume the filePath is provided by the user. If that file does not exist then short-cutting the execution and saying everything went fine << seems to me lazy error handling.
Early exiting / failing fast is a good pattern but please let the caller of your method know that some pre-condition(s) failed prior any real work has been done.
"application/zip"
Since you haven't shared with us all relevant code that's why it is hard to tell, but please make sure that the filePath is pointing to a real zip compressed file (not just checking that the file's extension is .zip).
return ....ExecuteAsync
As it was stated by dropoutcoder you should await the ExecuteAsync to be sure that method-scoped httpContent is not disposed.
.PostAsync(httpContent))
Most of the time those endpoints that can be called with POST verb are not implemented in an idempotent way. In other words it might happen that without request de-duplication you create duplicates or even worse with retries.
Please make sure that retry logic is applicable in your use case.
WithOAuthBearerToken(accessToken)
Creating a retry logic to refresh the accessToken if it is expired seems to me a more reasonable choice for a retry logic. Here you can find two alternative implementations how to achieve that.
WithTimeout(TimeSpan.FromMinutes(timeOutMinutes))
Because you are mixing Flurl resilience features with Polly resilience features it is pretty hard to tell without reading the documentation (or experimenting) that this timeout is for all retry attempts (global) or for each attempt (local).
I would suggest to use Polly's Timeout with Policy.Wrap to define an overarching timeout constraint or a per request timeout limit.
_logger.LogWarning
Last but not least I would suggest to use WaitAndRetryAsync in the way that you perform the logging inside the onRetry delegate
.WaitAndRetryAsync(5,
retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)),
onRetry:(ex, nextAttemptIn, retryAttempt) =>
_logger.LogWarning($"StatusRetryPolicy: Retry attempt {retryAttempt} to make request. Next try in {nextAttemptIn.TotalSeconds} seconds."));
Here you can access the exception which triggers a new retry attempt.
I just read about Polly library and I need to handle 3 retries when communicating from a desktop agent to a server.
Currently I like the exponential backoff.
However, I struggle to understand how to implement this in my code. This is what I have done so far:
using (HttpClient client = new HttpClient())
{
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", bearerToken);
client.DefaultRequestHeaders.Add("TenantId", tenantId);
#if DEBUG
var url = "http://localhost:5001/api/v1/RetrievePaymentDetails?orderpaymentid={orderPaymentId";
#else
//TODO: add URL once the application is in production!
var url = "intent2.api";
#endif
Policy
.Handle<HttpRequestException>()
.WaitAndRetry(new[]
{
TimeSpan.FromSeconds(5),
TimeSpan.FromSeconds(15),
TimeSpan.FromSeconds(30)
});
using (HttpResponseMessage response = await client.GetAsync($"{url}"))
{
using (HttpContent content = response.Content)
{
HttpContentHeaders headers = content.Headers;
var result = JsonConvert.DeserializeObject<PaymentDetailsDto>(response.Content.ReadAsStringAsync().Result);
Currency currencyFromDb = (Currency)Enum.Parse(typeof(Currency), result.Currency);
var result2 = await this.posManager.PayPos(result.Amount, currencyFromDb, result.SumUpUsername, result.SumUpPassword);
if (result2.TransactionCode == null)
{
return this.Request.CreateResponse(HttpStatusCode.BadRequest, result2);
}
return this.Request.CreateResponse(HttpStatusCode.OK, result2);
//return this.Request.CreateResponse<object>(result2);
}
}
}
I turned off my web server so I can simulate this exercise.
Now, every time it hits that URL it throws a HttpRequestException and does not retry at all. How do I go about using this?
Your are not using the policy. Defining one does not apply it to anything. Here is an example of a specialized transient error policy from the HttpPolicyExtensions. But this general pattern applies to any policy:
static AsyncRetryPolicy<HttpResponseMessage> transientRetry3Policy = HttpPolicyExtensions
.HandleTransientHttpError()
.RetryAsync(3);
And here is how you apply it to a request:
var someresult = await transientRetry3Policy
.ExecuteAndCaptureAsync(async () => await client.SendAsync(msg));
Your code could (and should) be improved in so many different ways.
Because we are not on code review that's why I will only focus on the Polly related stuff.
Retry in general
There is really important precondition for retry: the action (which might be called several times) should be idempotent and side-effect free. As an educated guess based on the provided code you are trying to retrieve information about a purchase which has already occurred. I suppose such query is written in an idempotent way.
I quite not sure that your posManager is idempotent and side-effect free (I suppose not). So my point is that you should choose wisely the scope of your resilient strategy, because it can cause damage if it is carried out without care.
Retry and its friends
Retry itself is just a policy not a resilient strategy. We can talk about strategy when we combine multiple policies in order to gain resilient behavior.
For example, imagine a situation when your payment service is highly overloaded and your response times goes insane. Let's say 99 seconds. Are you willing to wait for that? The default timeout of HttpClient is 100 seconds, which is a lot. If you don't control that your system will hang for a while. That's where Timeout comes into the picture.
It can be used for each individual query or as a global timeout for the whole process (including retries). So for example:
local timeout: 5 seconds
global timeout: 90 seconds
Another related policy is called CircuitBreaker which can help you to prevent flooding the downstream system (which is the payment service in your case). If your payment service already has hard time to serve requests then we should not send new ones to it.
CircuitBreaker monitors the successive failures and if a given threshold is exceeded then it will prevent all outgoing requests for a predefined amount of time to help the downstream system to get on foot again.
If you put all of this together then you will have a fairly resilient solution:
Global timeout
Retry
CircuitBreaker
Local timeout
Transient Http Errors
The HandleTransientHttpError method mentioned by Crowcoder handles either HttpRequestException or status code of 408 or 5xx. If your payment service returns another response status, for example 429 (Too Many Request) then it will not handle that. It's worth to bear in mind.
Crowcoder's answer with exponential backoff policy
// Exponential backoff policy
AsyncRetryPolicy<HttpResponseMessage> exponentialBackoffPolicy = HttpPolicyExtensions
.HandleTransientHttpError()
// Use sleepDurationProvider parameter
.WaitAndRetryAsync(retryCount, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));
// Using policy
var someresult = await exponentialBackoffPolicy
.ExecuteAndCaptureAsync(async () => await client.SendAsync(msg));
I have this very simple WebApi method:
[HttpGet]
public IHttpActionResult Foo()
{
Thread.Sleep(3000);
return Ok("Bar");
}
And I have these two methods in a console application that call it:
async Task UsingWebClient()
{
Task<string> task = new WebClient().DownloadStringTaskAsync (new Uri ("http://localhost.fiddler:63710/api/producttype/Foo"));
Console.WriteLine("WebClient - Before calling wait");
string result = await task;
Console.WriteLine("WebClient - After calling wait");
}
async Task UsingHttpClient()
{
Task<string> task = new HttpClient().GetStringAsync (new Uri ("http://localhost.fiddler:63710/api/producttype/Foo"));
Console.WriteLine("HttpClient - Before calling wait");
string result = await task;
Console.WriteLine("HttpClient - After calling wait");
}
And I am calling these methods from LinqPad like this:
async Task Main()
{
await UsingWebClient();
await UsingHttpClient();
}
I was monitoring the traffic using Fiddler and I noticed that:
when using WebClient the request to the web api is made immediately
and then execution continues to Console.WriteLine("WebClient - Before
calling wait");
when using HttpClient the request to the web api is not made until
the call to await task;
I'm trying to understand why the request is not made immediately when using HttpClient. Can anyone point me in the right direction?
This is not a duplicate question. I'm not looking for reasons to choose one option over the other - I'll use HttpClient. I would like to know specifically why the request is created at a later stage when using HttpClient.
Thanks,
David
Since both of the requests are async, none of them should delay execution of your current thread (significantly).
It is possible, though, that one of them can send the request before the current thread reaches the next line, while the other cannot.
These kinds of timing issues can happen in asynchronous/parallel environments and they're nothing to worry about as long as you don't separate logically successive operations.
When I try to handle a route asynchronously and within that route I do the authentication, Nancy throws a RouteExecutionEarlyExitException and then the status code 500 (Internal Server Error) is returned instead of 401 (Unauthorized).
My route handler looks roughly like this:
Get["route/to/private/stuff", true] = async (args, ct) =>
{
this.RequiresAuthentication()
// process request
}
I followed the guidelines here to configure stateless authentication.
It works as expected (throws exception internally, but returns 401) when my route handler looks like this:
Get["route/to/private/stuff"] = args =>
{
this.RequiresAuthentication()
// process request
}
How do I get Nancy to return 401 in an async route handler with per-route authentication?
I'm using Nancy 1.0.0.
Ok, now I got it. Unfortunately I provided too little information.
I wasn't quite honest of how I process the request. Nancy fails with a 500 instead of a 401 when the RouteExecutionEarlyExitException is thrown synchronously, like here:
Get["/", true] = (args, ct) =>
{
this.RequiresAuthentication();
return Task.FromResult((object) "Hello World!");
};
AFAIK that wouldn't happen when Nancy used await when invoking the route handler (i.e. await handler(args, ct)) but it can happen when the handler is invoked synchronously (i.e. handler(args, ct).ContinueWith(...)).
And it seems as Nancy uses the latter approach.
So the solution is to either stick with the async modifier or ensure that exceptions are only thrown asynchronously (e.g. within Task.Run).
I'm writing an application that proxies some HTTP requests using the ASP.NET Web API and I am struggling to identify the source of an intermittent error.
It seems like a race condition... but I'm not entirely sure.
Before I go into detail here is the general communication flow of the application:
Client makes a HTTP request to Proxy 1.
Proxy 1 relays the contents of the HTTP request to Proxy 2
Proxy 2 relays the contents of the HTTP request to the Target Web Application
Target Web App responds to the HTTP request and the response is streamed (chunked transfer) to Proxy 2
Proxy 2 returns the response to Proxy 1 which in turn responds to the original calling Client.
The Proxy applications are written in ASP.NET Web API RTM using .NET 4.5.
The code to perform the relay looks like so:
//Controller entry point.
public HttpResponseMessage Post()
{
using (var client = new HttpClient())
{
var request = BuildRelayHttpRequest(this.Request);
//HttpCompletionOption.ResponseHeadersRead - so that I can start streaming the response as soon
//As it begins to filter in.
var relayResult = client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead).Result;
var returnMessage = BuildResponse(relayResult);
return returnMessage;
}
}
private static HttpRequestMessage BuildRelayHttpRequest(HttpRequestMessage incomingRequest)
{
var requestUri = BuildRequestUri();
var relayRequest = new HttpRequestMessage(incomingRequest.Method, requestUri);
if (incomingRequest.Method != HttpMethod.Get && incomingRequest.Content != null)
{
relayRequest.Content = incomingRequest.Content;
}
//Copies all safe HTTP headers (mainly content) to the relay request
CopyHeaders(relayRequest, incomingRequest);
return relayRequest;
}
private static HttpRequestMessage BuildResponse(HttpResponseMessage responseMessage)
{
var returnMessage = Request.CreateResponse(responseMessage.StatusCode);
returnMessage.ReasonPhrase = responseMessage.ReasonPhrase;
returnMessage.Content = CopyContentStream(responseMessage);
//Copies all safe HTTP headers (mainly content) to the response
CopyHeaders(returnMessage, responseMessage);
}
private static PushStreamContent CopyContentStream(HttpResponseMessage sourceContent)
{
var content = new PushStreamContent(async (stream, context, transport) =>
await sourceContent.Content.ReadAsStreamAsync()
.ContinueWith(t1 => t1.Result.CopyToAsync(stream)
.ContinueWith(t2 => stream.Dispose())));
return content;
}
The error that occurs intermittently is:
An asynchronous module or handler completed while an asynchronous operation was still pending.
This error usually occurs on the first few requests to the proxy applications after which the error is not seen again.
Visual Studio never catches the Exception when thrown.
But the error can be caught in the Global.asax Application_Error event.
Unfortunately the Exception has no Stack Trace.
The proxy applications are hosted in Azure Web Roles.
Any help identifying the culprit would be appreciated.
Your problem is a subtle one: the async lambda you're passing to PushStreamContent is being interpreted as an async void (because the PushStreamContent constructor only takes Actions as parameters). So there's a race condition between your module/handler completing and the completion of that async void lambda.
PostStreamContent detects the stream closing and treats that as the end of its Task (completing the module/handler), so you just need to be sure there's no async void methods that could still run after the stream is closed. async Task methods are OK, so this should fix it:
private static PushStreamContent CopyContentStream(HttpResponseMessage sourceContent)
{
Func<Stream, Task> copyStreamAsync = async stream =>
{
using (stream)
using (var sourceStream = await sourceContent.Content.ReadAsStreamAsync())
{
await sourceStream.CopyToAsync(stream);
}
};
var content = new PushStreamContent(stream => { var _ = copyStreamAsync(stream); });
return content;
}
If you want your proxies to scale a bit better, I also recommend getting rid of all the Result calls:
//Controller entry point.
public async Task<HttpResponseMessage> PostAsync()
{
using (var client = new HttpClient())
{
var request = BuildRelayHttpRequest(this.Request);
//HttpCompletionOption.ResponseHeadersRead - so that I can start streaming the response as soon
//As it begins to filter in.
var relayResult = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
var returnMessage = BuildResponse(relayResult);
return returnMessage;
}
}
Your former code would block one thread for each request (until the headers are received); by using async all the way up to your controller level, you won't block a thread during that time.
I would like to add some wisdom for anyone else who landed here with the same error, but all of your code seems fine. Look for any lambda expressions passed into functions across the call-tree from where this occurs.
I was getting this error on a JavaScript JSON call to an MVC 5.x controller action. Everything I was doing up and down the stack was defined async Task and called using await.
However, using Visual Studio's "Set next statement" feature I systematically skipped over lines to determine which one caused it. I kept drilling down into local methods until I got to a call into an external NuGet package. The called method took an Action as a parameter and the lambda expression passed in for this Action was preceded by the async keyword. As Stephen Cleary points out above in his answer, this is treated as an async void, which MVC does not like. Luckily said package had *Async versions of the same methods. Switching to using those, along with some downstream calls to the same package fixed the problem.
I realize this is not a novel solution to the problem, but I passed over this thread a few times in my searches trying to resolve the issue because I thought I didn't have any async void or async <Action> calls, and I wanted to help someone else avoid that.
A slightly simpler model is that you can actually just use the HttpContents directly and pass them around inside the relay. I just uploaded a sample illustrating how you can rely both requests and responses asynchronously and without buffering the content in a relatively simple manner:
http://aspnet.codeplex.com/SourceControl/changeset/view/7ce67a547fd0#Samples/WebApi/RelaySample/ReadMe.txt
It is also beneficial to reuse the same HttpClient instance as this allows you to reuse connections where appropriate.