I'm using retry policy in .net core application and am getting timeouts after exceeding 100 seconds period.
Might I use Poly in some incorrect way or it's by design and only timeout period increase might help?
Here is the way I use Poly:
Startup:
// populate timeouts array from appsettings
var resilencyOptions = services.BuildServiceProvider().GetRequiredService<IOptions<ResiliencyOptions>>().Value;
var attempts = resilencyOptions.TimeOutsInSeconds.Count;
TimeSpan[] timeouts = new TimeSpan[attempts];
int i = 0;
foreach (var timeout in resilencyOptions.TimeOutsInSeconds)
{
timeouts[i++] = TimeSpan.FromSeconds(timeout);
}
// register
services.AddTransient<LoggingDelegatingHandler>();
services.AddHttpClient<IMyClient, MyClient>()
.AddHttpMessageHandler<LoggingDelegatingHandler>()
.AddPolicyHandler(ResiliencyPolicy.GetRetryPolicy(attempts, timeouts))
.AddPolicyHandler(ResiliencyPolicy.GetCircuitBreakerPolicy());
Library:
/// <summary>
/// Resiliency policy.
/// </summary>
public class ResiliencyPolicy
{
/// <summary>
/// Get a retry policy.
/// </summary>
/// <param name="numberofAttempts"> Количество попыток.</param>
/// <param name="timeOfAttempts"> Массив с таймаутами между попытками, если передается неполный или пустой, попытки делаются в секундах 2^.</param>
/// <returns></returns>
public static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy(int numberofAttempts = 5, TimeSpan[] timeOfAttempts = null)
{
// In case timeOfAttempts is null or its elements count doesnt correspond to number of attempts provided,
// we will wait for:
// 2 ^ 1 = 2 seconds then
// 2 ^ 2 = 4 seconds then
// 2 ^ 3 = 8 seconds then
// 2 ^ 4 = 16 seconds then
// 2 ^ 5 = 32 seconds
return HttpPolicyExtensions
.HandleTransientHttpError()
.WaitAndRetryAsync(
retryCount: numberofAttempts,
sleepDurationProvider: retryAttempt => ((timeOfAttempts == null) || (timeOfAttempts.Length != numberofAttempts)) ?
TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)) :
timeOfAttempts[retryAttempt],
onRetry: (exception, retryCount, context) =>
{
Logging.Global.LogError($"Retry {retryCount} of {context.PolicyKey} at {context.OperationKey}, due to: {exception}.");
});
}
/// <summary>
/// Get circuit breaker policy.
/// </summary>
/// <param name="numberofAttempts">количество попыток</param>
/// <param name="durationOfBreaksInSeconds">количество секунд (таймаут) между попытками</param>
/// <returns></returns>
public static IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicy(int numberofAttempts = 5, int durationOfBreaksInSeconds = 30)
{
return HttpPolicyExtensions
.HandleTransientHttpError()
.CircuitBreakerAsync(
handledEventsAllowedBeforeBreaking: numberofAttempts,
durationOfBreak: TimeSpan.FromSeconds(durationOfBreaksInSeconds)
);
}
}
Calling from custom http client:
public class MyClient : IMyClient
{
private readonly HttpClient _httpClient;
private readonly ILogger<MyClient> _logger;
public MyClient(HttpClient httpClient, ILogger<MyClient> logger)
{
_httpClient = httpClient;
_logger = logger;
}
public async Task<bool> Notify(string url, Guid id, string orderId, int state, int category, DateTime date, CancellationToken cancellationToken)
{
// prepare request
var request = new
{
Id = id,
OrderId = orderId,
State = state,
Category = category,
Date = date
};
var data = new StringContent(JsonSerializer.Serialize(request), Encoding.UTF8, "application/json");
// send request
_logger.LogInformation("sending request to {url}", url);
var response = await _httpClient.PostAsync(url, data, cancellationToken);
// process response
if (response.IsSuccessStatusCode)
return true;
var content = await response.Content.ReadAsStringAsync(cancellationToken);
response.Content?.Dispose();
throw new HttpRequestException($"{response.ReasonPhrase}. {content.Replace("\"", "").TrimEnd()}", null, response.StatusCode);
}
}
Controller simulating endpoint availability:
[ApiController]
[Route("[controller]")]
public class RabbitController : ControllerBase
{
private static int _numAttempts;
public RabbitController(IBus client)
{
_client = client;
}
[HttpPost("ProcessTestREST")]
public IActionResult ProcessTestREST(Object data)
{
_numAttempts++;
if (_numAttempts%4==3)
{
return Ok();
}
else
{
return StatusCode((int)HttpStatusCode.InternalServerError, "Something went wrong");
}
}
}
I am getting this error:
"The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing."
The important thing to note here, and it's definitely not intuitive, is that the HttpClient.Timeout applies to the ENTIRE collection of calls, which includes all retries and waits: https://github.com/App-vNext/Polly/wiki/Polly-and-HttpClientFactory#use-case-applying-timeouts
The default for HttpClient is 100 seconds, so if your retries and waits exceed that then Polly will throw the TimeoutException.
There's a couple ways to address this:
Set HttpClient.Timeout to the max length of time you'd expect it to take for all your retries.
Put the timeout policy BEFORE the retry policy, which makes it act like a global timeout policy.
In my case I did #1, because I want my timeout policy to apply independently to each request, so I kept my timeout policy AFTER my retry policy. The docs further explain how this works.
Check https://learn.microsoft.com/en-us/aspnet/core/fundamentals/http-requests?view=aspnetcore-6.0#dynamically-select-policies
var timeoutPolicy = Policy.TimeoutAsync<HttpResponseMessage>(
TimeSpan.FromSeconds(10));
var longTimeoutPolicy = Policy.TimeoutAsync<HttpResponseMessage>(
TimeSpan.FromSeconds(30));
builder.Services.AddHttpClient("PollyDynamic")
.AddPolicyHandler(httpRequestMessage =>
httpRequestMessage.Method == HttpMethod.Get ? timeoutPolicy : longTimeoutPolicy);
The timeout policy should be set during the AddHttpClient phase to override the 100 seconds default value as defined in the official documentation.
Your timeout for polly related requests, should cover the biggest value of your retry policy.
Beware to use custom clients in case you want to ignore the retries, so that the timeout is the default one.
You need to make sure that the timeout for the HttpClient is greater than any of the timeouts for your Polly policies. You need to use the AddHttpClient overload, changing the default timeout for the client from 100 seconds.
var notFoundTimeout = TimeSpan.FromMinutes(5);
var transientTimeout = TimeSpan.FromSeconds(5);
var clientTimeout = notFoundTimeout.Add(new TimeSpan(0, 1, 0));
var notFoundRetryPolicy = Policy.Handle<HttpRequestException>() // 404 not found errors
.OrResult<HttpResponseMessage>(response => response.StatusCode == System.Net.HttpStatusCode.NotFound)
.WaitAndRetryAsync(3, (int tryIndex) => notFoundTimeout);
services.AddHttpClient(CLIENT_NAME, config => config.Timeout = clientTimeout)
.AddPolicyHandler(notFoundRetryPolicy)
.AddTransientHttpErrorPolicy(
builder => builder.WaitAndRetryAsync(3, (int tryIndex) => transientTimeout));
I might be late to the game but allow me to put my 2 cents.
All the other answers are focusing on the 100 seconds default value of HttpClient's Timeout property and try to solve that issue. The real problem is how the AddPolicyHandler works under the hood.
I have detailed here how the PolicyHttpMessageHandler ruins the party. In case of typed HttpClient the solution is to move the policy inside the typed client to avoid the usage of AddPolicyHandler.
You have already separated the policies into a dedicated class ResiliencyPolicy. (BTW you can declare the class as static). I would recommend to expose a combined policy instead of exposing two policies.
public static IAsyncPolicy<HttpResponseMessage> GetCombinedPolicy(int attempts = 5, TimeSpan[] timeouts = null)
=> Policy.WrapAsync<HttpResponseMessage>(GetRetryPolicy(attempts, timeouts), GetCircuitBreakerPolicy())
You may try this during construct your HttpClient:
HttpClient client = new();
client.Timeout = TimeSpan.FromMinutes(5); // or your desire
Related
Using:
Frontend: Angular 14,
API: .NET Core 5, c#, MVC
IDS: .NET Core 5, c#, Razor as per ID standard
For my web app I have an instance of IdentityServer 4 running. This worked perfectly fine and without hick ups for about a year. Since recently when the app starts the login still works flawlessly and provides the token as per usual.
However, any API request thereafter return a 500 error, for about 1 minute or so, after which it works fine and without issue. Until the app is in 'rest' position (i.e. no active users) it starts of with the same error for the same amount of time.
I tried installing serilog to see if I can catch the error on the API side, to no avail.
There are no errors in the logged serilog file.
The only errors I can find are in the ASP.NET logs, which generally llok like the below;
fail: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[3]
Exception occurred while processing message.
System.InvalidOperationException: IDX20803: Unable to obtain configuration from: 'https://<ids_server_url>/.well-known/openid-configuration'.
---> System.IO.IOException: IDX20804: Unable to retrieve document from: 'https://<ids_server_url>/.well-known/openid-configuration'.
---> System.Net.Http.HttpRequestException: Only one usage of each socket address (protocol/network address/port) is normally permitted. (<ids_server_url>:443)
---> System.Net.Sockets.SocketException (10048): Only one usage of each socket address (protocol/network address/port) is normally permitted.
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
Nor can I catch the error on the IDS side, as that also seems to be working fine.
Accessing the .well-known/openid-configuration directly (i.e. from browser) gives a direct and correct response.
Several posts on SO indicated to add the below;
IdentityModelEventSource.ShowPII = true;
ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol =
SecurityProtocolType.Tls |
SecurityProtocolType.Tls11 |
SecurityProtocolType.Tls12;
// | SecurityProtocolType.Tls13;
This didn't seem to do anything at all to improve the error.
Would anybody be able to point me in the directions of any other possibilities?
Especially the fact that it is only about a minute at the startup of the app seems to be weird?
I thought it might be the startup of IDS instance, but given that the actual login window repsonds directly and without delay, it implies that the IDS instance is active and running?
Any ideas would be helpfull?
update: 19/02/2023
With the help of #Tore Nestenius I have been able to add some logging to the initial process but the behaviour remains erratic and only on the deployed instance. (Likely because of app_pool shutting down)
Last night according to logger, after 6 failed attempts there was a succesfull query of the openid-configuration
JwtBearerBackChannelListener
#### SendASync: https://<ids_server_url>/.well-known/openid-configuration
#### success: True
#### completed: True
#### loadtime: 132
#### url: https://<ids_server_url>/.well-known/openid-configuration
But...
The subsequent process fails (again)
fail: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[3]
What's more is that the initial call that the frontend makes is to a non-authorized endpoint (i.e. a public endpoint) there should not be a need for any token verification on that call?
If I query the backend on that endpoint directly from the browser it responds immediately, hence the backend appears to be working correctly? (i.e. api & database respond as expected when queried from the browser) yet in the API ASP logs it indicates a failed jwtHandler call? Weird...
Could it be a timing issue that, when you deploy your application that the client starts to request the discovery document before IdentityServer is up and running?
In AddOpenIDConnect and JwtBearer, you can define your own BackchannelHttpHandler, like this:
.AddJwtBearer(opt =>
{
opt.BackchannelHttpHandler = new JwtBearerBackChannelListener();
opt.BackchannelTimeout = TimeSpan.FromSeconds(60); //default 60s
...
}
This handler is used when it needs to load and reload the discovery document.
A sample handler can look like this:
public class JwtBearerBackChannelListener : DelegatingHandler
{
public JwtBearerBackChannelListener() : base(new HttpClientHandler())
{
Console.WriteLine();
}
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
Console.WriteLine("JwtBearerBackChannelListener");
Console.WriteLine("#### SendASync: " + request.RequestUri);
var sw = new Stopwatch();
sw.Start();
var result = base.SendAsync(request, cancellationToken);
result.ContinueWith(t =>
{
sw.Stop();
Console.WriteLine("#### success: " + result.IsFaulted);
Console.WriteLine("#### loadtime: " + sw.ElapsedMilliseconds.ToString());
Console.WriteLine("#### url: " + request.RequestUri);
Serilog.Log.Logger.ForContext("SourceContext", "JwtBearerBackChannelListener")
.ForContext("url", request.RequestUri)
.ForContext("loadtime", sw.ElapsedMilliseconds.ToString() + " ms")
.ForContext("success", result.IsCompletedSuccessfully)
.Information("Loading IdentityServer configuration");
});
return result;
}
}
This allows you to add more extensive logging and also even custom retry logic.
It is important that IdentityServer is up-and-running before the client/api starts.
One approach to solve this is to add a middleware that blocks incoming requests from being processed until IdentityServer is online, like this:
Sample code for a waiting middleware
namespace PaymentAPI.Middleware
{
/// <summary>
/// Extension method to register the middleware
/// </summary>
public static class WaitForIdentityServerMiddlewareExtensions
{
public static IApplicationBuilder UseWaitForIdentityServer(this IApplicationBuilder builder, WaitForIdentityServerOptions options)
{
return builder.UseMiddleware<WaitForIdentityServerMiddleware>(options);
}
}
public class WaitForIdentityServerOptions
{
public string Authority { get; set; }
}
/// <summary>
/// ASP.NET Core middleware that will wait for IdentityServer to respond
///
/// It will return a 503 SERVICE UNAVAILABLE if IdentityServer is not responding
///
/// This middleware is only in use until the first successfull response from IdentityServer.
/// After that this module will not do anything.
///
/// It will add the following response headers to the resonse when we return a 503 error:
///
/// - x-reason: Waiting for IdentityServer
/// - Cache-Control: no-store,no-cache,max-age=0
/// - Retry-After: 5
///
/// The authority URL will be taken from the
///
/// Written by Tore Nestenius to be used in the IdentityServer in production training class.
/// https://www.tn-data.se
///
/// </summary>
public class WaitForIdentityServerMiddleware
{
/// <summary>
/// number of seconds between each attempt to contact IdentityServer
/// </summary>
private int secondsBetweenRetries = 2;
/// <summary>
/// How many seconds should we wait before we give up waiting?
/// </summary>
private int httpRequestTimeout = 3;
/// <summary>
/// True when we have been able to reach IdentityServer
/// </summary>
private bool _identityServerReady = false;
private readonly RequestDelegate _next;
private readonly string _discoveryUrl;
private readonly SemaphoreSlim _refreshLock;
private DateTimeOffset _syncAfter = DateTimeOffset.MinValue;
private readonly DateTime _startTime;
public WaitForIdentityServerMiddleware(RequestDelegate next, IConfiguration configuration, WaitForIdentityServerOptions options)
{
_next = next;
_startTime = DateTime.UtcNow;
_discoveryUrl = buildDiscoveryUrl(options.Authority);
_refreshLock = new SemaphoreSlim(1);
}
public async Task InvokeAsync(HttpContext context)
{
//Has IdentityServer has succesfully responsed yet?
if (_identityServerReady == false)
{
//Fail fast if we should wait a bit or if there is already a request is in progress
if (_syncAfter > DateTimeOffset.UtcNow ||
_refreshLock.CurrentCount == 0)
{
//We are waiting to not overload IdentitytServer with to many requests
//Just terminate the request with a 503 Service Unavailable response
CreateServiceUnavailableResponse(context);
return;
}
//Make sure we only do one request at the time
await _refreshLock.WaitAsync().ConfigureAwait(false);
try
{
//Still not answering?
if (_identityServerReady == false)
{
_identityServerReady = await TryToReachIdentityServer(context);
}
}
catch (Exception exc)
{
Log.Logger.ForContext("SourceContext", "WaitForIdentityServerMiddleware")
.ForContext("DiscoveryUrl", _discoveryUrl)
.ForContext("Exception", exc.Message)
.ForContext("Path", context.Request.Path)
.Fatal("Exception while trying to reach IdentityServer");
}
finally
{
_refreshLock.Release();
_syncAfter = DateTimeOffset.UtcNow.AddSeconds(secondsBetweenRetries);
}
}
if (_identityServerReady)
{
// Call the next delegate/middleware in the pipeline
await _next(context);
}
else
{
//As we did not succeeed, let's terminate return a 503 SERVICE UNAVAILABLE error back to the client
CreateServiceUnavailableResponse(context);
return;
}
}
/// <summary>
/// Create a service unavailable 503 error response
/// </summary>
/// <param name="context"></param>
private void CreateServiceUnavailableResponse(HttpContext context)
{
context.Response.Headers.Add("x-reason", "Waiting for IdentityServer");
context.Response.Headers.Add("Retry-After", "5"); //Add a retry again header, with 5 seconds
context.Response.Headers.Add("Cache-Control", "no-store,no-cache,max-age=0"); //Don't cache this response
context.Response.StatusCode = (int)HttpStatusCode.ServiceUnavailable; //503 status code
}
/// <summary>
/// Try to reach the IdentityServer discovery endpoint
/// </summary>
/// <returns>True if successfull</returns>
private async Task<bool> TryToReachIdentityServer(HttpContext context)
{
var client = new HttpClient();
client.Timeout = TimeSpan.FromSeconds(httpRequestTimeout);
var response = await client.GetAsync(_discoveryUrl);
//Should we log?
if (response.IsSuccessStatusCode == false)
{
var secondsSinceStart = (int)DateTime.UtcNow.Subtract(_startTime).TotalSeconds;
Log.Logger.ForContext("SourceContext", "WaitForIdentityServerMiddleware")
.ForContext("DiscoveryUrl", _discoveryUrl)
.ForContext("Path", context.Request.Path)
.ForContext("Tried for over", secondsSinceStart.ToString() + " seconds")
.Information("Failed to reach IdentityServer at startup");
}
return response.IsSuccessStatusCode;
}
/// <summary>
/// Construct the discovery endpoint URL
/// </summary>
/// <param name="authority"></param>
/// <returns></returns>
private string buildDiscoveryUrl(string authority)
{
string Url = authority;
if (!Url.EndsWith("/", StringComparison.Ordinal))
{
Url = Url + "/";
}
Url = Url + ".well-known/openid-configuration";
return Url;
}
}
}
Then to use the handler:
//Make sure its placed before app.UseAuthentication();
//Wait for IdentityServer to startup
app.UseWaitForIdentityServer(new WaitForIdentityServerOptions()
{ Authority = _configuration["openid:authority"] });
I'm working with a very flaky API. Sometimes I get 500 Server Error with Timeout, some other time I also get 500 Server Error because I gave it input that it can't handle
SqlDateTime overflow. Must be between 1/1/1753 12:00:00 AM and 12/31/9999 11:59:59 PM.
Both of these cases give me HttpRequestException but I can look into the reply message from the server and determine the cause of the exception. If it is a timeout error, I should try again. If it is a bad input I should re-throw the exception, because no amount of retries will fix the problem of bad data.
What I'd like to do with Polly is to check on response message before attempting to retry. But all the samples I've seen so far only included type of exception.
I've come up with this so far:
HttpResponseMessage response = null;
String stringContent = null;
Policy.Handle<FlakyApiException>()
.WaitAndRetry(5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)),
async (exception, timeSpan, context) =>
{
response = await client.PostAsync(requestUri, new StringContent(serialisedParameters, Encoding.UTF8, "application/json"));
stringContent = await response.Content.ReadAsStringAsync();
if (response.StatusCode == HttpStatusCode.InternalServerError && stringContent.Contains("Timeout"))
{
throw new FlakyApiException(stringContent);
}
});
Is there a better way to do this kind of checking?
In general, you can configure Polly policies to respond to the results of an execution (not just an exception), for example check an HttpResponseMessage.StatusCode with a predicate. Examples here in the Polly readme.
There is not however an in-built way to configure a single Polly policy to respond additionally to the content of the response message. This is because (as your example shows) obtaining that content requires a second async call, which may itself raise network errors.
This tl;dr engenders complications about how to express (in a simple syntax) a single policy which manages two different async steps with potentially different error handling for each step. Prior related discussion on Polly github: comment welcome.
As such, where a sequence requires two separate async calls, the Polly team currently recommends expressing this as two separate policies, similar to the example in the end of this answer.
The particular example in your question may not work because the onRetryAsync delegate (throwing FlakyApiException) is not itself guarded by the policy. A policy only guards the execution of delegates executed through .Execute/ExecuteAsync(...).
One approach could be to use two policies, a retry policy which retries all typical http exceptions and status codes including 500s; then inside that a Polly FallbackPolicy which traps the status code 500 representing SqlDateTime overflow, and excludes that from being retried by rethrowing as some distinguishing exception (CustomSqlDateOverflowException).
IAsyncPolicy<HttpResponseMessage> rejectSqlError = Policy<HttpResponseMessage>
.HandleResult(r => r.StatusCode == HttpStatusCode.InternalServerError)
.FallbackAsync(async (delegateOutcome, context, token) =>
{
String stringContent = await delegateOutcome.Result.Content.ReadAsStringAsync(); // Could wrap this line in an additional policy as desired.
if (delegateOutcome.Result.StatusCode == HttpStatusCode.InternalServerError && stringContent.Contains("SqlDateTime overflow"))
{
throw new CustomSqlDateOverflowException(); // Replace 500 SqlDateTime overflow with something else.
}
else
{
return delegateOutcome.Result; // render all other 500s as they were
}
}, async (delegateOutcome, context) => { /* log (if desired) that InternalServerError was checked for what kind */ });
IAsyncPolicy<HttpResponseMessage> retryPolicy = Policy<HttpResponseMessage>
.Handle<HttpRequestException>()
.OrResult(r => r.StatusCode == HttpStatusCode.InternalServerError)
.OrResult(r => /* condition for any other errors you want to handle */)
.WaitAndRetry(5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)),
async (exception, timeSpan, context) =>
{
/* log (if desired) retry being invoked */
});
HttpResponseMessage response = await retryPolicy.WrapAsync(rejectSqlError)
.ExecuteAsync(() => client.PostAsync(requestUri, new StringContent(serialisedParameters, Encoding.UTF8, "application/json"), cancellationToken));
For Http, I chose to solve this problem using DelegatingHandler (DH) pattern, and polly. There is no HandleResultAsync(), so the issue still exists for a generalized question.
With polly, I avoid a solution that has "coupling".
I've had great success with using a retry policy in a DelegatingHandler as it follows SRP, and provides a nice SoC (see this SO post). Here is the retry DH I use typically for reference.
For your question at hand, there are 2 things: retry, and conditions to retry on. Building on my retry DH, I exploded it into two DelegatingHandlers: a retry DH that retries on a "signal", and a latter retry signaling DH that signals a retry. HttpRequestMessage's .Properties (or .Options) bag is used to signal.
I find it easily maintainable, and is not complex by avoiding nested polly policies or blocking call. I have few APIs using the async request/reply pattern, so the retry DH (used for polling) is reusable (nugetized), and the retry signaling DH is different as per the API. You can obviously combine them into one by inlining the signaling code into the action arg.
HttpClient CoR (chain of responsibility):
... -> retry on signal DH -> retry signaling DH -> ...
Here is the retry signaling DH for your conditions to retry.
public class RetrySignalingOnConditionHandler : DelegatingHandler
{
protected override async Task<HttpResponseMessage> SendAsync(
HttpRequestMessage request,
CancellationToken cancellationToken)
{
var response = await base.SendAsync(request, cancellationToken);
// tweak conditions accordingly
if (response.StatusCode == (HttpStatusCode)500)
{
request.Properties[RequestProperties.RetrySignal] = true;
return response;
}
var content = await response.Content.ReadAsStringAsync(cancellationToken);
if (content.Contains("Timeout"))
{
request.Properties[RequestProperties.RetrySignal] = true;
return response;
}
return response;
}
}
internal static class RequestProperties
{
internal static string RetrySignal = nameof(RetrySignal);
}
Here is the retry DH that retries on the signal. It resets the signal before the attempt.
public class ExponentialBackoffRetryOnSignalHandler : DelegatingHandler
{
private readonly IAsyncPolicy<(HttpRequestMessage request, HttpResponseMessage response)> retryPolicy;
public ExponentialBackoffRetryOnSignalHandler(
IRetrySettings retrySettings)
{
_ = retrySettings
?? throw new ArgumentNullException(nameof(retrySettings));
var sleepDurations = Backoff.ExponentialBackoff(
initialDelay: TimeSpan.FromMilliseconds(retrySettings.RetryDelayInMilliseconds),
retryCount: retrySettings.RetryCount);
retryPolicy = Policy
.HandleResult<(HttpRequestMessage request, HttpResponseMessage response)>(tuple =>
tuple.request.Properties.TryGetValue(RequestProperties.RetrySignal, out var retrySignaledObj) && (bool)retrySignaledObj)
.WaitAndRetryAsync(
sleepDurations: sleepDurations,
onRetry: (responseResult, delay, retryAttempt, context) =>
{
// note: response can be null in case of handled exception
responseResult.Result.response?.Dispose();
});
}
protected override async Task<HttpResponseMessage> SendAsync(
HttpRequestMessage request,
CancellationToken cancellationToken)
{
var tuple = await retryPolicy.ExecuteAsync(
action: async (ct) =>
{
request.Properties.Remove(RequestProperties.RetrySignal);
var response = await base.SendAsync(request, ct)
.ConfigureAwait(false);
return (request, response);
},
cancellationToken: cancellationToken)
.ConfigureAwait(false);
return tuple.response;
}
}
public interface IRetrySettings
{
int RetryCount { get; }
int RetryDelayInMilliseconds { get; }
}
Here is the full code that I use along with tests.
If I understand your question correctly then you want to retry only if the status code is 500 and the body contains Timeout. If that's the case then you can define your policy just like this
Policy<HttpResponseMessage>
.HandleResult(response =>
response.StatusCode == System.Net.HttpStatusCode.InternalServerError
&& response.Content.ReadAsStringAsync().GetAwaiter().GetResult().Contains("Timeout"))
.WaitAndRetry(5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt);
UPDATE #1
Just to clarify. Even tough .GetAwaiter().GetResult() should be avoided whenever possible, here I consider it as a valid use case to utilize it:
There is no HandleResultAsync builder method, so we have to use HandleResult sync method here
First we filter for 500 status code and then we lazily evaluate the response body
I assumed the response body is fairly small due to the fact we should not expose too much information in case of Internal Server Error
An HttpRequestMessage object can only be used one time; future attempts to use the same object throw an exception. I'm using Polly to retry some requests and I'm hitting this issue. I know how I can clone a request, there are plenty of examples on SO, but I can't figure out how to clone a request and send that new request whenever Polly retries. How can I accomplish this?
These are my policies, for reference. This is a Xamarin app. I want to retry a few times in case of network failures, and if the response is unauthorized I want to re-auth with saved credentials and try the original request again.
public static PolicyWrap<HttpResponseMessage> RetryPolicy
{
get => WaitAndRetryPolicy.WrapAsync(ReAuthPolicy);
}
private static IAsyncPolicy WaitAndRetryPolicy
{
get => Policy.Handle<WebException>().WaitAndRetryAsync(4, _ => TimeSpan.FromSeconds(2));
}
private static IAsyncPolicy<HttpResponseMessage> ReAuthPolicy
{
get => Policy.HandleResult<HttpResponseMessage>(x => x.StatusCode == HttpStatusCode.Unauthorized)
.RetryAsync((_, __) => CoreService.LogInWithSavedCredsAsync(true));
}
This doesn't work because of the HttpRequestMessage reuse, but it's what I'm trying to accomplish:
var request = new HttpRequestMessage(HttpMethod.Post, "some_endpoint")
{
Content = new StringContent("some content")
};
request.Content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
var policyResponse = await ConnectivityHelper.RetryPolicy
.ExecuteAndCaptureAsync(() => _client.SendAsync(request)).ConfigureAwait(false);
// handle outcome
The code to throw InvalidOperationException if an HttpRequestMessage is reused is a validation step within HttpClient itself.
Source code link
private static void CheckRequestMessage(HttpRequestMessage request)
{
if (!request.MarkAsSent())
{
throw new InvalidOperationException(SR.net_http_client_request_already_sent);
}
}
Source code link
internal bool MarkAsSent()
{
return Interlocked.Exchange(ref sendStatus, messageAlreadySent) == messageNotYetSent;
}
You can put the polly retry policy in a DelegatingHandler and that works. It also provides a nice SoC (separation of concerns). If, in future, you want to not retry or change retry behavior, you simply remove the DelegatingHandler or change it. Note to dispose off the HttpRequestMessage and intermediate HttpResponseMessages objects. Here is one that I use with good results (retry policy).
Your question is an open-ended, and generally SO is not good for those (see). But here goes. I call this a "reactive" approach as it uses the token right up until its ttl, and fetches the new one. Note that this doesn't incur 401s by using the token ttl.
# gets token with its ttl
tokenService: iTokenService
# use retry policy in DH here
httpClient
string getTokenAsync():
# calls out for token
# note: tokens typically have a ttl
# returns cached token till its tll, or gets a new token which is then cached
cachedTokenService: iCachedTokenService
tokenCached
tokenTtl
iTokenService
string getTokenAsync():
# returns tokenCached or gets a new token based on ttl
# note: fetches with some buffer before ttl to avoid failures on edge
# note: buffer as 2x http timeout is good enough
# DH that adds the cached token to the outgoing "work" request
tokenHandler: delegatingHandler
iCachedTokenService
task<response> sendAsync(request, ct):
# gets token, and adds token to request header
# worker service
workService: iWorkService
# uses tokenHandler DH
httpClient
workAsync():
# ...
Well, the simplest solution is to move the creation of the HttpRequestMessage inside the ExecuteAndCaptureAsync delegate. In other words do not reuse rather recreate it:
var policyResponse = await ConnectivityHelper.RetryPolicy
.ExecuteAndCaptureAsync(async () => {
var request = new HttpRequestMessage(HttpMethod.Post, "some_endpoint")
{
Content = new StringContent("some content", Encoding.UT8, "application/json")
};
return await _client.SendAsync(request)).ConfigureAwait(false);
});
Or simply prefer PostAsync over SendAsync
var policyResponse = await ConnectivityHelper.RetryPolicy.ExecuteAndCaptureAsync(
async () =>
await _client.PostAsync("some_endpoint",
new StringContent("some content", Encoding.UT8, "application/json"))
.ConfigureAwait(false)
});
I am trying to set the timeout value of anglesharp.io using HttpClientHandler.
This issue suggests it is possible: https://github.com/AngleSharp/AngleSharp/issues/266
I am getting page like this:
NetworkCredential proxyCreds = new NetworkCredential(proxy.User, proxy.Pass);
WebProxy wProxy = new WebProxy(proxy.Ip + ":" + proxy.Port, false)
{
UseDefaultCredentials = false,
Credentials = proxyCreds,
};
HttpClientHandler httpClientHandler = new HttpClientHandler()
{
Proxy = wProxy,
PreAuthenticate = true,
UseDefaultCredentials = false
};
var config = Configuration.Default.WithRequesters(httpClientHandler);
var document = await BrowsingContext.New(config).OpenAsync(address);
I cannot see any properties available to set the timeout. How do I set the timeout?
The comments above are right. AngleSharp is abstracting requesters away - to allow multiple types of requesters and provide flexibility if needed. The essential interface is IRequester (note: there is no HTTP on purpose - in AngleSharp.Io we also find, e.g., a FileRequester that accesses the local file system for file:// URIs).
We could now either implement our own requester or just use the HttpClientRequester from AngleSharp.Io with the constructor overload accepting an HttpClient instance.
var client = new HttpClient();
client.Timeout = MyCustomTimeout; //Whatever value you want it to be
var requester = new HttpClientRequester(client);
Now the question is how can you use this requester? We just create a configuration (as usual) and use the default loader extension method (as usual), however, this time with our custom requester.
For pre 0.10 this looks as follows:
// Assumes we do not want to provide custom options for the loaders
var requesters = new [] { requester };
var configuration = Configuration.Default.WithDefaultLoader(requesters: requesters);
For 0.10 and later this looks a bit different:
var configuration = Configuration.Default.WithRequester(requester).WithDefaultLoader();
Hope this helps!
Sometimes it's needed to have different timeouts on the level of HttpClient and HttpClientHandler, for example kind of retry logic, seamless for HttpClient - wait for 5 minutes, but retry every minute. In this case one can use a delegating handler like following:
public class RetryHandler : DelegatingHandler
{
private readonly TimeSpan timeout;
public RetryHandler(TimeSpan timeout)
{
this.timeout = timeout;
}
private async Task<HttpResponseMessage> Delay(
CancellationToken cancellationToken)
{
await Task.Delay(timeout, cancellationToken);
return null;
}
protected override async Task<HttpResponseMessage> SendAsync(
HttpRequestMessage request,
CancellationToken cancellationToken)
{
for (; ; )
{
cancellationToken.ThrowIfCancellationRequested();
var delayTask = Delay(cancellationToken);
var firstCompleted = await Task.WhenAny(
base.SendAsync(request, cancellationToken), delayTask);
if (firstCompleted != delayTask)
return await firstCompleted;
}
}
}
Use case:
var client = new HttpClient(
new RetryHandler(TimeSpan.FromMinutes(1)))
{
Timeout = TimeSpan.FromMinutes(5)
};
client.PostAsync(...);
I am working on an ASP.Net Core API that calls a 3rd party API using an HttpClient GetAsync method. If the 3rd party API is off line, the call times out but in the response that is returned, I do not see any info related to not being able to connect. The response object's StatusCode and ResponsePhrase properties say "Not Found", which is a bit misleading. Is there a way for me to know if the 3rd party API is actually running?
For example, listed below is my code for making a cal to the 3rd party API;
public async Task<SAPIAddAlertSubscriberResponse> AddAlertSubscriberAsync(SAPIAddAlertSubscriberRequest a_request, CancellationToken a_cancellationToken)
{
try
{
SAPIAddAlertSubscriberResponse sapiResponse = new SAPIAddAlertSubscriberResponse();
using (HttpClient client = new HttpClient())
{
var policy = SetRetryPolicy();
HttpResponseMessage response = null;
var uri = $#"{m_appSettings.Value.SAPIUrl88}SAPIAddAlertSubscriber";
string jsonInString = JsonConvert.SerializeObject(a_request);
SetHttpClientHeader(client);
await policy.ExecuteAsync(async token =>
{
response = await client.PostAsync(uri, new StringContent(jsonInString, Encoding.UTF8, "application/json"));
}, a_cancellationToken);
if (response.IsSuccessStatusCode)
{
var result = await response.Content.ReadAsStringAsync();
result = TrimWBResponse(result, "SAPIAddAlertSubscriber");
sapiResponse = JsonConvert.DeserializeObject<SAPIAddAlertSubscriberResponse>(result);
}
else
{
sapiResponse.queue.Add(SetSapiError(response, uri, $"add an AlertSubscriber for CarrierLoginId {a_request.CarrierLoginPrimaryID}"));
}
}
return sapiResponse;
}
catch (Exception ex)
{
m_logger.LogError(1, ex, "An exception has occurred in the AddAlertSubscriberAsync method call.");
return null;
}
}
When I make the call at a time that the 3rd party API is down, the call to the 3rd party API holds until the timeout is reached and then returns response.IsSuccessStatusCode = false. But when I examine the values in the response object's StatusCode and ResponsePhrase properties, they both just say "Not Found".
Is there a way for me to return more information that could indicate that the reason was that the API was not available?
EDIT: Added code for SetRetryPolicy
/// <summary>
/// Creates a retry policy for a Polly instance.
/// </summary>
/// <returns>The Polly.Retry.RetryPolicy object</returns>
/// <remarks>
/// Supports implementation of a Polly (https://github.com/App-vNext/Polly) transient exception handler to apply exponential backoff solution for timeout errors.
/// If a timeout transient error occurs, the call will retry after an exponentially increasing time period (starting at 1/10 second) , up to the
/// RetryAttempts value in the settings database
/// <remarks>
private Polly.Retry.RetryPolicy SetRetryPolicy()
{
int retries = 0;
var policy = Policy.Handle<Exception>()
.WaitAndRetryAsync((int)m_appSettings.Value.SAPIRetryAttempts, attempts => TimeSpan.FromSeconds(0.1 * Math.Pow(2, attempts)),
(exception, calculateWaitDuration) =>
{
m_logger.LogError(1, exception, "A transient exception has occurred{NewLine} ... automatically delaying for [{WaitDuration}] ms. This is retry [{Retries}].", Environment.NewLine, calculateWaitDuration, retries);
retries++;
});
return policy;
}
Thanks in advance for any help you can provide.