I'd like to use WaitAndRetryAsync to help retry http 429 (throttling) errors. The retry delay is returned as a property on the exception itself.
But I need to add the accumulated time and abandon the retry loop if the overall duration exceeds a certain amount.
policy = Policy.Handle<DocumentClientException>(ex => ex.StatusCode == (HttpStatusCode)429)
.WaitAndRetryAsync(
retryCount: retries,
sleepDurationProvider: (retryCount, exception, context) => {
DocumentClientException dce = exception as DocumentClientException;
// Here I would like to check the total time and NOT return a RetryAfter value if my overall time is exceeded. Instead re-throw the 'exception'.
return dce.RetryAfter;
},
onRetryAsync: async (res, timespan, retryCount, context) => {
});
When the overall time is exceeded I'd like to re-throw the 'exception' handled in the sleepDurationProvider.
Is there a better way to handle this?
This first example below limits the total waits between retries to a total timespan myWaitLimit, but takes no account of how long the calls to CosmosDB spend before returning DocumentClientException. Because Polly Context is execution-scoped, this is thread-safe. Something like:
policy = Policy.Handle<DocumentClientException>(ex => ex.StatusCode == (HttpStatusCode)429)
.WaitAndRetryAsync(
retryCount: retries,
sleepDurationProvider: (retryCount, exception, context) => {
DocumentClientException dce = exception as DocumentClientException;
TimeSpan toWait = dce.RetryAfter;
TimeSpan waitedSoFar;
if (!Context.TryGetValue("WaitedSoFar", out waitedSoFar)) waitedSoFar = TimeSpan.Zero; // (probably some extra casting actually needed between object and TimeSpan, but this kind of idea ...)
waitedSoFar = waitedSoFar + toWait;
if (waitedSoFar > myWaitLimit)
throw dce; // or use ExceptionDispatchInfo to preserve stack trace
Context["WaitedSoFar"] = waitedSoFar; // (magic string "WaitedSoFar" only for readability; of course you can factor this out)
return toWait;
},
onRetryAsync: async (res, timespan, retryCount, context) => {
});
An alternative approach could limit the overall execution time (when 429s occur) using a timing-out CancellationToken. The below approach will not retry further after the CancellationToken has been signalled. This approach is modelled to be close to the functionality requested in the question, but the timeout clearly only takes effect if a 429 response is returned and the sleepDurationProvider delegate is invoked.
CancellationTokenSource cts = new CancellationTokenSource();
cts.CancelAfter(/* my timeout */);
var policy = Policy.Handle<DocumentClientException>(ex => ex.StatusCode == (HttpStatusCode)429)
.WaitAndRetryAsync(
retryCount: retries,
sleepDurationProvider: (retryCount, exception, context) => {
if (cts.IsCancellationRequested) throw exception; // or use ExceptionDispatchInfo to preserve stack trace
DocumentClientException dce = exception as DocumentClientException;
return dce.RetryAfter;
},
onRetryAsync: async (res, timespan, retryCount, context) => {
});
If you don't wish to define policy in the same scope as using it and close over the variable cts (as the above example does), you can pass the CancellationTokenSource around using Polly Context as described in this blog post.
Alternatively, Polly provides a TimeoutPolicy. Using PolicyWrap you can wrap this outside the retry policy. A timeout can then be imposed on the overall execution whether a 429 occurs or not.
If the strategy is intended to manage Cosmos DB async calls which do not inherently take a CancellationToken, you would need to use TimeoutStrategy.Pessimistic if you wanted to enforce timeout at that time interval. However, note from the wiki how TimeoutStrategy.Pessimistic operates: it allows the calling thread to walk away from the uncancellable call, but doesn't unilaterally cancel the uncancellable call. That call might either later fault, or continue to completion.
Obviously, consider what is best from among the above options, according to your context.
Related
Please let me know is there any chance in the below code the "response" turns to be null, that results an error if I try to get response.result and response.statuscode value in onRetry block.
Sample code below
AsyncRetryPolicy<HttpResponseMessage> retryPolicyPost = Policy
.Handle<HttpRequestException>()
.OrResult<HttpResponseMessage>(r => r != null && (!r.IsSuccessStatusCode))
.WaitAndRetryAsync(3, times => TimeSpan.FromMilliseconds(times * 30000),
onRetry: (response, delay, retryCount, context) =>
{
Console.WriteLine(retryCount);
Console.WriteLine(response.Result);
Console.WriteLine(response.Result.StatusCode);
});
I appreciate your response.
is there any chance in the below code the "response" turns to be null
Your response parameter's type is DelegateResult<HttpResponseMessage>, which can't be null inside the onRetry delegate.
BUT the response.Result can be null! The DelegateResult defines two properties which are mutually exclusive: Exception, Result.
So, if the retry logic is triggered because an HttpRequestException was thrown then response.Result will be null, but the result.Exception will be an HttpRequestException.
If the retry logic is triggered because the HttpResponseMessage's StatusCode is different than 200 then response.Result will be an HttpResponseMessage, but the result.Exception will be null.
This this piece of code is error-prone, it might throw NRE:
Console.WriteLine(response.Result.StatusCode);
To fix this you can do the following:
var statusCode = response.Result?.StatusCode ?? ((HttpRequestException)response.Exception).StatusCode;
Console.WriteLine(statusCode);
Please note that HttpRequestException's StatusCode returns with an HttpStatusCode?.
I'm working with a very flaky API. Sometimes I get 500 Server Error with Timeout, some other time I also get 500 Server Error because I gave it input that it can't handle
SqlDateTime overflow. Must be between 1/1/1753 12:00:00 AM and 12/31/9999 11:59:59 PM.
Both of these cases give me HttpRequestException but I can look into the reply message from the server and determine the cause of the exception. If it is a timeout error, I should try again. If it is a bad input I should re-throw the exception, because no amount of retries will fix the problem of bad data.
What I'd like to do with Polly is to check on response message before attempting to retry. But all the samples I've seen so far only included type of exception.
I've come up with this so far:
HttpResponseMessage response = null;
String stringContent = null;
Policy.Handle<FlakyApiException>()
.WaitAndRetry(5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)),
async (exception, timeSpan, context) =>
{
response = await client.PostAsync(requestUri, new StringContent(serialisedParameters, Encoding.UTF8, "application/json"));
stringContent = await response.Content.ReadAsStringAsync();
if (response.StatusCode == HttpStatusCode.InternalServerError && stringContent.Contains("Timeout"))
{
throw new FlakyApiException(stringContent);
}
});
Is there a better way to do this kind of checking?
In general, you can configure Polly policies to respond to the results of an execution (not just an exception), for example check an HttpResponseMessage.StatusCode with a predicate. Examples here in the Polly readme.
There is not however an in-built way to configure a single Polly policy to respond additionally to the content of the response message. This is because (as your example shows) obtaining that content requires a second async call, which may itself raise network errors.
This tl;dr engenders complications about how to express (in a simple syntax) a single policy which manages two different async steps with potentially different error handling for each step. Prior related discussion on Polly github: comment welcome.
As such, where a sequence requires two separate async calls, the Polly team currently recommends expressing this as two separate policies, similar to the example in the end of this answer.
The particular example in your question may not work because the onRetryAsync delegate (throwing FlakyApiException) is not itself guarded by the policy. A policy only guards the execution of delegates executed through .Execute/ExecuteAsync(...).
One approach could be to use two policies, a retry policy which retries all typical http exceptions and status codes including 500s; then inside that a Polly FallbackPolicy which traps the status code 500 representing SqlDateTime overflow, and excludes that from being retried by rethrowing as some distinguishing exception (CustomSqlDateOverflowException).
IAsyncPolicy<HttpResponseMessage> rejectSqlError = Policy<HttpResponseMessage>
.HandleResult(r => r.StatusCode == HttpStatusCode.InternalServerError)
.FallbackAsync(async (delegateOutcome, context, token) =>
{
String stringContent = await delegateOutcome.Result.Content.ReadAsStringAsync(); // Could wrap this line in an additional policy as desired.
if (delegateOutcome.Result.StatusCode == HttpStatusCode.InternalServerError && stringContent.Contains("SqlDateTime overflow"))
{
throw new CustomSqlDateOverflowException(); // Replace 500 SqlDateTime overflow with something else.
}
else
{
return delegateOutcome.Result; // render all other 500s as they were
}
}, async (delegateOutcome, context) => { /* log (if desired) that InternalServerError was checked for what kind */ });
IAsyncPolicy<HttpResponseMessage> retryPolicy = Policy<HttpResponseMessage>
.Handle<HttpRequestException>()
.OrResult(r => r.StatusCode == HttpStatusCode.InternalServerError)
.OrResult(r => /* condition for any other errors you want to handle */)
.WaitAndRetry(5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)),
async (exception, timeSpan, context) =>
{
/* log (if desired) retry being invoked */
});
HttpResponseMessage response = await retryPolicy.WrapAsync(rejectSqlError)
.ExecuteAsync(() => client.PostAsync(requestUri, new StringContent(serialisedParameters, Encoding.UTF8, "application/json"), cancellationToken));
For Http, I chose to solve this problem using DelegatingHandler (DH) pattern, and polly. There is no HandleResultAsync(), so the issue still exists for a generalized question.
With polly, I avoid a solution that has "coupling".
I've had great success with using a retry policy in a DelegatingHandler as it follows SRP, and provides a nice SoC (see this SO post). Here is the retry DH I use typically for reference.
For your question at hand, there are 2 things: retry, and conditions to retry on. Building on my retry DH, I exploded it into two DelegatingHandlers: a retry DH that retries on a "signal", and a latter retry signaling DH that signals a retry. HttpRequestMessage's .Properties (or .Options) bag is used to signal.
I find it easily maintainable, and is not complex by avoiding nested polly policies or blocking call. I have few APIs using the async request/reply pattern, so the retry DH (used for polling) is reusable (nugetized), and the retry signaling DH is different as per the API. You can obviously combine them into one by inlining the signaling code into the action arg.
HttpClient CoR (chain of responsibility):
... -> retry on signal DH -> retry signaling DH -> ...
Here is the retry signaling DH for your conditions to retry.
public class RetrySignalingOnConditionHandler : DelegatingHandler
{
protected override async Task<HttpResponseMessage> SendAsync(
HttpRequestMessage request,
CancellationToken cancellationToken)
{
var response = await base.SendAsync(request, cancellationToken);
// tweak conditions accordingly
if (response.StatusCode == (HttpStatusCode)500)
{
request.Properties[RequestProperties.RetrySignal] = true;
return response;
}
var content = await response.Content.ReadAsStringAsync(cancellationToken);
if (content.Contains("Timeout"))
{
request.Properties[RequestProperties.RetrySignal] = true;
return response;
}
return response;
}
}
internal static class RequestProperties
{
internal static string RetrySignal = nameof(RetrySignal);
}
Here is the retry DH that retries on the signal. It resets the signal before the attempt.
public class ExponentialBackoffRetryOnSignalHandler : DelegatingHandler
{
private readonly IAsyncPolicy<(HttpRequestMessage request, HttpResponseMessage response)> retryPolicy;
public ExponentialBackoffRetryOnSignalHandler(
IRetrySettings retrySettings)
{
_ = retrySettings
?? throw new ArgumentNullException(nameof(retrySettings));
var sleepDurations = Backoff.ExponentialBackoff(
initialDelay: TimeSpan.FromMilliseconds(retrySettings.RetryDelayInMilliseconds),
retryCount: retrySettings.RetryCount);
retryPolicy = Policy
.HandleResult<(HttpRequestMessage request, HttpResponseMessage response)>(tuple =>
tuple.request.Properties.TryGetValue(RequestProperties.RetrySignal, out var retrySignaledObj) && (bool)retrySignaledObj)
.WaitAndRetryAsync(
sleepDurations: sleepDurations,
onRetry: (responseResult, delay, retryAttempt, context) =>
{
// note: response can be null in case of handled exception
responseResult.Result.response?.Dispose();
});
}
protected override async Task<HttpResponseMessage> SendAsync(
HttpRequestMessage request,
CancellationToken cancellationToken)
{
var tuple = await retryPolicy.ExecuteAsync(
action: async (ct) =>
{
request.Properties.Remove(RequestProperties.RetrySignal);
var response = await base.SendAsync(request, ct)
.ConfigureAwait(false);
return (request, response);
},
cancellationToken: cancellationToken)
.ConfigureAwait(false);
return tuple.response;
}
}
public interface IRetrySettings
{
int RetryCount { get; }
int RetryDelayInMilliseconds { get; }
}
Here is the full code that I use along with tests.
If I understand your question correctly then you want to retry only if the status code is 500 and the body contains Timeout. If that's the case then you can define your policy just like this
Policy<HttpResponseMessage>
.HandleResult(response =>
response.StatusCode == System.Net.HttpStatusCode.InternalServerError
&& response.Content.ReadAsStringAsync().GetAwaiter().GetResult().Contains("Timeout"))
.WaitAndRetry(5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt);
UPDATE #1
Just to clarify. Even tough .GetAwaiter().GetResult() should be avoided whenever possible, here I consider it as a valid use case to utilize it:
There is no HandleResultAsync builder method, so we have to use HandleResult sync method here
First we filter for 500 status code and then we lazily evaluate the response body
I assumed the response body is fairly small due to the fact we should not expose too much information in case of Internal Server Error
An HttpRequestMessage object can only be used one time; future attempts to use the same object throw an exception. I'm using Polly to retry some requests and I'm hitting this issue. I know how I can clone a request, there are plenty of examples on SO, but I can't figure out how to clone a request and send that new request whenever Polly retries. How can I accomplish this?
These are my policies, for reference. This is a Xamarin app. I want to retry a few times in case of network failures, and if the response is unauthorized I want to re-auth with saved credentials and try the original request again.
public static PolicyWrap<HttpResponseMessage> RetryPolicy
{
get => WaitAndRetryPolicy.WrapAsync(ReAuthPolicy);
}
private static IAsyncPolicy WaitAndRetryPolicy
{
get => Policy.Handle<WebException>().WaitAndRetryAsync(4, _ => TimeSpan.FromSeconds(2));
}
private static IAsyncPolicy<HttpResponseMessage> ReAuthPolicy
{
get => Policy.HandleResult<HttpResponseMessage>(x => x.StatusCode == HttpStatusCode.Unauthorized)
.RetryAsync((_, __) => CoreService.LogInWithSavedCredsAsync(true));
}
This doesn't work because of the HttpRequestMessage reuse, but it's what I'm trying to accomplish:
var request = new HttpRequestMessage(HttpMethod.Post, "some_endpoint")
{
Content = new StringContent("some content")
};
request.Content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
var policyResponse = await ConnectivityHelper.RetryPolicy
.ExecuteAndCaptureAsync(() => _client.SendAsync(request)).ConfigureAwait(false);
// handle outcome
The code to throw InvalidOperationException if an HttpRequestMessage is reused is a validation step within HttpClient itself.
Source code link
private static void CheckRequestMessage(HttpRequestMessage request)
{
if (!request.MarkAsSent())
{
throw new InvalidOperationException(SR.net_http_client_request_already_sent);
}
}
Source code link
internal bool MarkAsSent()
{
return Interlocked.Exchange(ref sendStatus, messageAlreadySent) == messageNotYetSent;
}
You can put the polly retry policy in a DelegatingHandler and that works. It also provides a nice SoC (separation of concerns). If, in future, you want to not retry or change retry behavior, you simply remove the DelegatingHandler or change it. Note to dispose off the HttpRequestMessage and intermediate HttpResponseMessages objects. Here is one that I use with good results (retry policy).
Your question is an open-ended, and generally SO is not good for those (see). But here goes. I call this a "reactive" approach as it uses the token right up until its ttl, and fetches the new one. Note that this doesn't incur 401s by using the token ttl.
# gets token with its ttl
tokenService: iTokenService
# use retry policy in DH here
httpClient
string getTokenAsync():
# calls out for token
# note: tokens typically have a ttl
# returns cached token till its tll, or gets a new token which is then cached
cachedTokenService: iCachedTokenService
tokenCached
tokenTtl
iTokenService
string getTokenAsync():
# returns tokenCached or gets a new token based on ttl
# note: fetches with some buffer before ttl to avoid failures on edge
# note: buffer as 2x http timeout is good enough
# DH that adds the cached token to the outgoing "work" request
tokenHandler: delegatingHandler
iCachedTokenService
task<response> sendAsync(request, ct):
# gets token, and adds token to request header
# worker service
workService: iWorkService
# uses tokenHandler DH
httpClient
workAsync():
# ...
Well, the simplest solution is to move the creation of the HttpRequestMessage inside the ExecuteAndCaptureAsync delegate. In other words do not reuse rather recreate it:
var policyResponse = await ConnectivityHelper.RetryPolicy
.ExecuteAndCaptureAsync(async () => {
var request = new HttpRequestMessage(HttpMethod.Post, "some_endpoint")
{
Content = new StringContent("some content", Encoding.UT8, "application/json")
};
return await _client.SendAsync(request)).ConfigureAwait(false);
});
Or simply prefer PostAsync over SendAsync
var policyResponse = await ConnectivityHelper.RetryPolicy.ExecuteAndCaptureAsync(
async () =>
await _client.PostAsync("some_endpoint",
new StringContent("some content", Encoding.UT8, "application/json"))
.ConfigureAwait(false)
});
I´m using both Refit and Polly to call to restful API´s and I´m wondering what the retry (if any) policy for Refits ApiException should be?
public static PolicyWrap MyRetryPolicy()
{
// Try few times with little more time between... maybe the
// connection issue gets resolved
var wireServerNetworkIssue = Policy.Handle<WebException>()
.WaitAndRetryAsync(new[] {
TimeSpan.FromSeconds(1),
TimeSpan.FromSeconds(2),
TimeSpan.FromSeconds(4)});
var policyList = new List<Policy>();
// But if there is something wrong with the api
// I should do what (if general)?
var api = Policy.Handle<ApiException>()
.RetryAsync(1, onRetry: async (exception, i) =>
{
await Task.Run(() =>
{
// What would be normal to do here?
// Try again or do some circuit braking?
});
});
policyList.Add(wireServerNetworkIssue);
policyList.Add(api);
return Policy.Wrap(policyList.ToArray());
}
And then I use it like this
try
{
myApi = RestService.For<MyApi>("api base url");
var policyWrapper = Policies.Policies.MyRetryPolicyWrapper();
var response = await policy.ExecuteAsync(() => myApi.SendReceiptAsync(receipt));
}
catch (ApiException apiEx)
{
//Do something if the retry policy did´t fix it.
}
catch (WebException webEx)
{
//Do something if the retry policy did´t fix it.
}
The Question
What would be a normal retry policy for ApiExceptions? Would you just circuit brake or under what general circumstances would you do something to recover?
The answer is probably "it depends on what your service returns" but I just have to ask.
If the ApiExceptions returned contain meaningful HttpStatusCode StatusCode properties, you could certainly choose which of those StatusCodes merit a retry; the Polly readme suggests:
int[] httpStatusCodesWorthRetrying = {408, 500, 502, 503, 504};
For ApiExceptions specific to the API called, only knowledge of what those API-specific errors represent, can guide whether to retry them.
If you choose to circuit-break on too many exceptions of some kind, that should be achieved by wrapping a circuit-breaker into your PolicyWrap, rather than within the onRetry delegate of the retry policy. Polly discusses 'Why circuit-break?' here, and links to a number of other circuit-breaker blog posts at the foot of the readme circuit-breaker section.
Could someone point me to a resource that would help explain how web api (specifically using Owin Self Host) handles request cancellation?
Here's the sequence of events i'm observing:
someone makes a GET from chrome
Api controller (through some layers) fires off an async SQL query
someone hits the X button in chrome (i don't know exactly what happens on socket for this)
What happens next in Web Api??
There was some code running in a controller, does the thread running it get aborted? If it was an async controller awaiting another Task, does that task still have an awaiter in case it returns with an exception?
For context: I do have an async controller awaiting a Task (this is the only call site) which looks to be throwing an unobserved exception in some edge cases. I haven't been able to isolate or re-produce yet :)
I did find something called HttpResponse.ClientDisconnectedToken, but don't know well that is supported in Owin Selfhost + is it even the good thing to use for all user cancels.
I've dealt with this by handing the System.OperationCanceledException in a custom middleware I've registered before WebApi.
public class ExceptionHanldingMiddleware : OwinMiddleware
{
public override async Task Invoke(IOwinContext context)
{
try
{
await Next.Invoke(context);
}
catch (OperationCanceledException) when (context.Request.CallCancelled.IsCancellationRequested)
{
//swallow user-agent cancelling request.
_log.Trace($"client disconnected on request for: {context.Request.Path}.");
}
catch (Exception ex)
{
_log.Error(ex);
context.Response.StatusCode = (int) HttpStatusCode.InternalServerError;
context.Response.ReasonPhrase = "Internal Server Error";
}
}
}
As you stated that your async controller is awaiting for a Task, which sometimes got some exception, I suggest you ContinueWith extension method for a task, which can be run only then your task is faulted, like this:
task.ContinueWith(
t =>
logger.Error(t.Exception.Message, t.Exception);
, TaskContinuationOptions.OnlyOnFaulted);
This is a default mechanism to handle the exceptions, and this will work in OWIN application.
Second, as for the cancellation: task can be started with a CancellationToken structure, which can be used for a cancelling the task during the execution. You can read more in the MSDN article.
HttpResponse.ClientDisconnectedToken is used for a situation when the client has been disconnected and the request should not be proceed in execution.
You can use this token, or create your own with CancellationTokenSource, like this:
var source = new CancellationTokenSource();
var token = source.Token;
var task = Task.Factory.StartNew(() =>
{
// Were we already canceled?
ct.ThrowIfCancellationRequested();
var moreToDo = true;
while (moreToDo)
{
// Poll on this property if you have to do
// other cleanup before throwing.
if (ct.IsCancellationRequested)
{
// Clean up here, then...
ct.ThrowIfCancellationRequested();
}
}
}, token);