I'm using Polly to handle some scenarios like request throttled and timeouts.
The policies were added directly in the Startup.cs which would be like this :
var retries = //applying the retries, let say I set to 25 times with 10s delay. Total 250s.
serviceCollection
.AddHttpClient<IApplicationApi, ApplicationApi>()
.AddPolicyHandler((services, request) => GetRetryPolicy<ApplicationApi>(retries, services));
The Policy:
static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy<T>(List<TimeSpan> retries, IServiceProvider services)
{
return HttpPolicyExtensions
.HandleTransientHttpError()
.OrResult(msg => msg.StatusCode == System.Net.HttpStatusCode.TooManyRequests)
.WaitAndRetryAsync(retries,
onRetry: (outcome, timespan, retryAttempt, context) =>
{
//do some logging
}
}
In ApplicationApi.cs do something like this:
private readonly HttpClient _httpClient;
public ApplicationApi(HttpClient httpClient)
{
_httpClient = httpClient;
}
public void CallApi()
{
var url = "https://whateverurl.com/someapi"
using (var request = new HttpRequestMessage(HttpMethod.Get, url))
{
var response = await _httpClient.SendAsync(request);
var respMessage = await
response.Content.ReadAsStringAsync();
}
}
Now let say I don't specify the HttpClient.Timeout, which then will use default timeout : 100s.
Now I have a problem with heavy throttling. Polly will retry until the throttling resolved, or it reach the max retry.
But, the program will thrown an exception on the 10th retry since it already more than 100s elapsed on httpclient since the first request it got throttled.
Seems like the first http request that got throttled still on and not closed, or I may be wrong.
What causing this? Is it a normal behavior of Polly retries?
How can I make it close the connection on each retries so I don't have to set a very high HttpClient.Timeout value.
I also implemented the Polly timeout policy to cut request if more than some specified second then retry until it succeed. But the Polly behavior still like this. So I need to set httpclient timeout > total elapsed time on retries
**UPDATE
Code updated. So I just realized there's using statement for the request.
***UPDATE
I've created a repo that reproduce the behavior here : https://github.com/purnadika/PollyTestWebApi
The short answer is that your observed behaviour is due to fact how AddPolicyHandler and PolicyHttpMessageHandler work.
Whenever you register a new Typed HttpClient without any policy (.AddHttpClient) then you basically create a new HttpClient like this:
var handler = new HttpClientHandler();
var client = new HttpClient(handler);
Of course it is much more complicated, but from our topic perspective it works like that.
If you register a new Typed HttpClient with a policy (.AddHttpClient().AddPolicyHandler()) then you create a new HttpClient like this
var handler = new PolicyHttpMessageHandler(yourPolicy);
handler.InnerHandler = new HttpClientHandler();
var client = new HttpClient(handler);
So the outer handler will be the Polly's MessageHandler and the inner is the default ClientHandler.
Polly's MessageHandler has the following documentation comment:
/// <para>
/// Take care when using policies such as Retry or Timeout together as HttpClient provides its own timeout via
/// <see cref="HttpClient.Timeout"/>. When combining Retry and Timeout, <see cref="HttpClient.Timeout"/> will act as a
/// timeout across all tries; a Polly Timeout policy can be configured after a Retry policy in the configuration sequence,
/// to provide a timeout-per-try.
/// </para>
By using the AddPolicyHandler the HttpClient's Timeout will act as a global timeout.
The solution
There is workaround, namely avoiding the usage of AddPolicyHandler.
So, rather than decorating your Typed Client at the registration time you can decorate only the specific HttpClient method call inside your typed client.
Here is a simplified example based on your dummy project:
ConfigureServices
services.AddHttpClient<IApplicationApi, ApplicationApi>(client => client.Timeout = TimeSpan.FromSeconds(whateverLowValue));
_MainRequest
var response = await GetRetryPolicy().ExecuteAsync(async () => await _httpClient.GetAsync(url));
Here I would like to emphasize that you should prefer GetAsync over SendAsync since the HttpRequestMessage can not be reused.
So, if you would write the above code like this
using (var request = new HttpRequestMessage(HttpMethod.Get, url))
{
var response = await GetRetryPolicy().ExecuteAsync(async () => await _httpClient.SendAsync(request));
}
then you would receive the following exception:
InvalidOperationException: The request message was already sent. Cannot send the same request message multiple times.
So, with this workaround the HttpClient's Timeout will not act as a global / overarching timeout over the retry attempts.
Polly works to retry the top-level HttpClient request, so HttpClient's timeout applies to all retries. That's the whole point of using Polly, to retry requests in a way that's transparent to the top-level code.
If retrying for over a minute failed to work, the retry policy isn't good enough. Retrying over and over with a fixed delay will only result in more 429 responses, as all the requests that failed will be retried at the same time. This will result in wave after wave of identical requests hitting the server, resulting in 429s once again.
To avoid this, exponential backoff and jitter are used to introduce an increasing random delay to retries.
From the linked sample:
var delay = Backoff.DecorrelatedJitterBackoffV2(
medianFirstRetryDelay: TimeSpan.FromSeconds(1),
retryCount: 5);
var retryPolicy = Policy
.Handle<FooException>()
.WaitAndRetryAsync(delay);
The Polly page for the Jitter strategy explain how this works. The distribution graph of delays shows that even with 5 retries, the retry intervals don't clamp together.
This means there's less chance of multiple HttpClient calls retrying at the same time, resulting in renewed throttling
Related
I'm using flurl and polly as part of my code to post asynchronously - I'm having an issue with the retry policy and the file content getting disposed, on a retry the FileContent option has been disposed.
I will get the following error:
System.ObjectDisposedException: Cannot access a disposed object. Object name: 'Flurl.Http.Content.FileContent'.
If someone could advise where best to place the logic to ensure FileContent is available for the retry
var endpoint = $"api endpoint here";
if (!File.Exists(filePath))
{
return Task.CompletedTask;
}
var fileBytesLength = _fileSystem.FileInfo.FromFileName(filePath).Length;
var httpContent = new Flurl.Http.Content.FileContent(filePath, (int)fileBytesLength);
return _statusRetryPolicy
.StatusAsyncRetryPolicy()
.ExecuteAsync(() => endpoint
.WithOAuthBearerToken(accessToken)
.WithTimeout(TimeSpan.FromMinutes(timeOutMinutes))
.WithHeader("Content-Type", "application/zip")
.OnError((flurlCall) =>
{
_logger.LogError(flurlCall.Exception, flurlCall.Exception.Message);
})
.PostAsync(httpContent));
and the Retry Policy:
public AsyncRetryPolicy StatusAsyncRetryPolicy() => Policy
.Handle<FlurlHttpException>(RetryPolicyHelpers.IsTransientError)
.WaitAndRetryAsync(5, retryAttempt =>
{
var nextAttemptIn = TimeSpan.FromSeconds(Math.Pow(2, retryAttempt));
_logger.LogWarning($"StatusRetryPolicy: Retry attempt {retryAttempt} to make request. Next try in {nextAttemptIn.TotalSeconds} seconds.");
return nextAttemptIn;
});
I have several suggestions for your code.
!File.Exists
I assume the filePath is provided by the user. If that file does not exist then short-cutting the execution and saying everything went fine << seems to me lazy error handling.
Early exiting / failing fast is a good pattern but please let the caller of your method know that some pre-condition(s) failed prior any real work has been done.
"application/zip"
Since you haven't shared with us all relevant code that's why it is hard to tell, but please make sure that the filePath is pointing to a real zip compressed file (not just checking that the file's extension is .zip).
return ....ExecuteAsync
As it was stated by dropoutcoder you should await the ExecuteAsync to be sure that method-scoped httpContent is not disposed.
.PostAsync(httpContent))
Most of the time those endpoints that can be called with POST verb are not implemented in an idempotent way. In other words it might happen that without request de-duplication you create duplicates or even worse with retries.
Please make sure that retry logic is applicable in your use case.
WithOAuthBearerToken(accessToken)
Creating a retry logic to refresh the accessToken if it is expired seems to me a more reasonable choice for a retry logic. Here you can find two alternative implementations how to achieve that.
WithTimeout(TimeSpan.FromMinutes(timeOutMinutes))
Because you are mixing Flurl resilience features with Polly resilience features it is pretty hard to tell without reading the documentation (or experimenting) that this timeout is for all retry attempts (global) or for each attempt (local).
I would suggest to use Polly's Timeout with Policy.Wrap to define an overarching timeout constraint or a per request timeout limit.
_logger.LogWarning
Last but not least I would suggest to use WaitAndRetryAsync in the way that you perform the logging inside the onRetry delegate
.WaitAndRetryAsync(5,
retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)),
onRetry:(ex, nextAttemptIn, retryAttempt) =>
_logger.LogWarning($"StatusRetryPolicy: Retry attempt {retryAttempt} to make request. Next try in {nextAttemptIn.TotalSeconds} seconds."));
Here you can access the exception which triggers a new retry attempt.
I just read about Polly library and I need to handle 3 retries when communicating from a desktop agent to a server.
Currently I like the exponential backoff.
However, I struggle to understand how to implement this in my code. This is what I have done so far:
using (HttpClient client = new HttpClient())
{
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", bearerToken);
client.DefaultRequestHeaders.Add("TenantId", tenantId);
#if DEBUG
var url = "http://localhost:5001/api/v1/RetrievePaymentDetails?orderpaymentid={orderPaymentId";
#else
//TODO: add URL once the application is in production!
var url = "intent2.api";
#endif
Policy
.Handle<HttpRequestException>()
.WaitAndRetry(new[]
{
TimeSpan.FromSeconds(5),
TimeSpan.FromSeconds(15),
TimeSpan.FromSeconds(30)
});
using (HttpResponseMessage response = await client.GetAsync($"{url}"))
{
using (HttpContent content = response.Content)
{
HttpContentHeaders headers = content.Headers;
var result = JsonConvert.DeserializeObject<PaymentDetailsDto>(response.Content.ReadAsStringAsync().Result);
Currency currencyFromDb = (Currency)Enum.Parse(typeof(Currency), result.Currency);
var result2 = await this.posManager.PayPos(result.Amount, currencyFromDb, result.SumUpUsername, result.SumUpPassword);
if (result2.TransactionCode == null)
{
return this.Request.CreateResponse(HttpStatusCode.BadRequest, result2);
}
return this.Request.CreateResponse(HttpStatusCode.OK, result2);
//return this.Request.CreateResponse<object>(result2);
}
}
}
I turned off my web server so I can simulate this exercise.
Now, every time it hits that URL it throws a HttpRequestException and does not retry at all. How do I go about using this?
Your are not using the policy. Defining one does not apply it to anything. Here is an example of a specialized transient error policy from the HttpPolicyExtensions. But this general pattern applies to any policy:
static AsyncRetryPolicy<HttpResponseMessage> transientRetry3Policy = HttpPolicyExtensions
.HandleTransientHttpError()
.RetryAsync(3);
And here is how you apply it to a request:
var someresult = await transientRetry3Policy
.ExecuteAndCaptureAsync(async () => await client.SendAsync(msg));
Your code could (and should) be improved in so many different ways.
Because we are not on code review that's why I will only focus on the Polly related stuff.
Retry in general
There is really important precondition for retry: the action (which might be called several times) should be idempotent and side-effect free. As an educated guess based on the provided code you are trying to retrieve information about a purchase which has already occurred. I suppose such query is written in an idempotent way.
I quite not sure that your posManager is idempotent and side-effect free (I suppose not). So my point is that you should choose wisely the scope of your resilient strategy, because it can cause damage if it is carried out without care.
Retry and its friends
Retry itself is just a policy not a resilient strategy. We can talk about strategy when we combine multiple policies in order to gain resilient behavior.
For example, imagine a situation when your payment service is highly overloaded and your response times goes insane. Let's say 99 seconds. Are you willing to wait for that? The default timeout of HttpClient is 100 seconds, which is a lot. If you don't control that your system will hang for a while. That's where Timeout comes into the picture.
It can be used for each individual query or as a global timeout for the whole process (including retries). So for example:
local timeout: 5 seconds
global timeout: 90 seconds
Another related policy is called CircuitBreaker which can help you to prevent flooding the downstream system (which is the payment service in your case). If your payment service already has hard time to serve requests then we should not send new ones to it.
CircuitBreaker monitors the successive failures and if a given threshold is exceeded then it will prevent all outgoing requests for a predefined amount of time to help the downstream system to get on foot again.
If you put all of this together then you will have a fairly resilient solution:
Global timeout
Retry
CircuitBreaker
Local timeout
Transient Http Errors
The HandleTransientHttpError method mentioned by Crowcoder handles either HttpRequestException or status code of 408 or 5xx. If your payment service returns another response status, for example 429 (Too Many Request) then it will not handle that. It's worth to bear in mind.
Crowcoder's answer with exponential backoff policy
// Exponential backoff policy
AsyncRetryPolicy<HttpResponseMessage> exponentialBackoffPolicy = HttpPolicyExtensions
.HandleTransientHttpError()
// Use sleepDurationProvider parameter
.WaitAndRetryAsync(retryCount, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));
// Using policy
var someresult = await exponentialBackoffPolicy
.ExecuteAndCaptureAsync(async () => await client.SendAsync(msg));
I'm trying to set a default timeout for my HttpClient calls to 5 seconds.
I've done this via CancellationTokenSource.
Here's the pertinent bit of code:
var cancellationToken = new CancellationTokenSource();
cancellationToken.CancelAfter(TimeSpan.FromSeconds(5));
var result = _httpClient.SendAsync(request, cancellationToken.Token);
Works as i expected in terms of the calling code getting a "Task was cancelled" error (i tested in a .NET 4.7 console app), but i noticed in Fiddler the request was still running for 1 minute, until it finally gave up:
Can someone explain this behaviour?
I would expect the underlying request to also get cancelled when the cancellation is triggered.
_httpClient is instantiated like: new HttpClient { BaseAddress = baseAddress }
I know there's the the Timeout setting, but not sure if I should be using that or cancellation tokens? My guess is Timeout is for the non-async/await cases?
As Damien said in the comments, HttpClient re-uses connections as much as possible, hence the reason why the connection is not closed on cancel.
When canceling a request like that, the HttpClient will just stop sending/receiving data to/from the other end. It will not send anything to inform the other end that it was cancelled. So the timeout you see of 1 minute depends on the behavior of the other end of your connection.
Also, if you want to cancel each request after 5 seconds, you can as well set the Timeout property of _httpClient to TimeSpan.FromSeconds(5). The behavior will be exactly the same (a TaskCanceledException will be thrown if the other end doesn't respond within 5 seconds).
If anyone is interested, you can try the following approach to applying your own timeout per HttpClient request. It seems to work for me, restricting the SendAsync() to 2 seconds and returning immediately when the timeout occurs:
private async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, TimeSpan? timeout = null)
{
if (timeout is null)
{
return await _httpClient.SendAsync(request);
}
else
{
using (var cts = new CancellationTokenSource(timeout.Value))
{
var sendTask = _httpClient.SendAsync(request);
while (!sendTask.IsCompleted)
{
cts.Token.ThrowIfCancellationRequested();
await Task.Delay(10).ConfigureAwait(false);
}
return await sendTask.ConfigureAwait(false);
}
}
}
I'm making calls to an external API and want to deal with the event that a call returns an Unauthorized HttpResponseMessage. When this happens I want to refresh the access token and make the call again.
I'm trying to use Polly with the following code:
public async Task<HttpResponseMessage> MakeGetRequestAsync()
{
var retryPolicy = Policy
.HandleResult<HttpResponseMessage>(r => r.StatusCode == HttpStatusCode.Unauthorized)
.Retry(1, (exception, retryCount) =>
{
RefreshAccessToken();
});
var result = await retryPolicy.ExecuteAsync(() => CallApiAsync());
return result;
}
private async Task<HttpResponseMessage> CallApiAsync()
{
var url = Options.ResourceSandboxUrl;
var httpClient = new HttpClient();
SetRequestHeaders(httpClient);
var response = await httpClient.GetAsync(url);
response.StatusCode = HttpStatusCode.Unauthorized;
return response;
}
I put breakpoints on the ExecuteAsync statement and in DoSomethingAsync - when I step over ExecuteAsync DoSomethingAsync is not called and control is returned to the function that called MakeGetRequestAsync
I don't understand why DoSomethingAsync is not called - can anyone help me with what I'm trying to achieve?
I've looked at the Polly documentation & Polly questions on Stack Overflow but I can't figure out what's going on..
To use ExecuteAsync() you must declare the policy as .RetryAsync(...), not .Retry(...).
If your actual code reads exactly as the code sample above, the .ExecuteAsync(...) will be throwing for the mismatch between .Retry(...) [a sync policy] and .ExecuteAsync(...) [an async execution]. Since this exception is thrown, CallApiAsync() is indeed never invoked. You should be able to see the thrown exception, when calling MakeGetRequestAsync()
Overall code approach looks good tho: this retry-refreshing-authentication is a proven pattern with Polly!
I'm replying to this old question just to point out the Polly wiki page where this pattern was official documented:
retry-to-refresh-authorization
In particular this is the code snippet suggested:
var authorisationEnsuringPolicy = Policy
.HandleResult<HttpResponseMessage>(r => r.StatusCode == HttpStatusCode.Unauthorized)
.RetryAsync(
retryCount: 1, // Consider how many retries. If auth lapses and you have valid credentials, one should be enough; too many tries can cause some auth systems to blacklist.
onRetryAsync: async (outcome, retryNumber, context) => FooRefreshAuthorizationAsync(context),
/* more configuration */);
var response = authorisationEnsuringPolicy.ExecuteAsync(context => DoSomethingThatRequiresAuthorization(context), cancellationToken);
The FooRefreshAuthorizationAsync(...) method can obtain a new authorization token and pass it to the delegate executed through the policy using Polly.Context.
I might be late to the game but I leave here a post for future readers. I have created two slightly different solutions for this refresh token with retry problem.
Retry policy, Delegating Handler and Custom exception
Here is sequence diagram which depicts the communication flow
Here is the full source code
Retry policy, Delegating Handler and Polly.Context
This version separates the responsibilities in a different way:
Here is sequence diagram which depicts the communication flow
Here is the full source code
this is how async await work in .net, when the execution of your code reaches an await, two things will happened
the current thread of your code should release to make your code async, that means, your method should return
when the task you await is complete, your method should continue from where it used to be
I need to change a HttpClient.Timeout property after it made a request(s). When I try, I get an exception:
This instance has already started one or more requests. Properties can only be modified before sending the first request.
Is there any way to avoid this?
There isn't much you can do to change this. This is just default behavior in the HttpClient implementation.
The Timeout property must be set before the GetRequestStream or GetResponse method is called.
From HttpClient.Timeout Remark Section
In order to change the timeout, it would be best to create a new instance of an HttpClient.
client = new HttpClient();
client.Timeout = 20; //set new timeout
Internally the Timeout property is used to set up a CancellationTokenSource which will abort the async operation when that timeout is reached. Since some overloads of the HttpClient methods accept CancellationTokens, we can create helper methods to have a custom timeouts for specific operations:
public async Task<string> GetStringAsync(string requestUri, TimeSpan timeout)
{
using (var cts = new CancellationTokenSource(timeout))
{
HttpResponseMessage response = await _httpClient.GetAsync(requestUri, cts.Token)
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
Lack of support for custom request-level timeouts has always been a shortcoming of HttpClient in my mind. If you don't mind a small library dependency, Flurl.Http [disclaimer: I'm the author] supports this directly:
"http://api.com/endpoint".WithTimeout(30).GetJsonAsync<T>();
This is a true request-level setting; all calls to the same host use a shared HttpClient instance under the hood, and concurrent calls with different timeouts will not conflict. There's a configurable global default (100 seconds initially, same as HttpClient).