I'm attempting to log something before retrying a web api call using Polly in a .net core web api.
I know the web api is failing and returning a 503 response code however there's nothing in my console log as part of the retry call. Any ideas why and how to resolve this?
var retryPolicy = Policy
.Handle<HttpRequestException>()
.Or<SocketException>()
.WaitAndRetryAsync(new[]
{
TimeSpan.FromSeconds(2),
TimeSpan.FromSeconds(5),
TimeSpan.FromSeconds(10)
}, (exception, timeSpan, retryCount, context) =>
{
Console.Write("RETRYING - " + DateTime.Now.Second);
});
await retryPolicy.ExecuteAsync(async () =>
{
var serviceReturnLabel = await this.stockTransfersServiceClient.GetPRReturnLabel(ItemSourceType.ReturnLocker);
if (serviceReturnLabel != null && serviceReturnLabel.Accepted)
{
returnLabel = serviceReturnLabel.PRLabel;
}
});
The retry policy exposes a hook where you can wire up a custom code which will be called before the retry penalty. In other words this delegate will be called whenever the policy should be triggered but before the wait between the two attempts.
This hook is called onRetry or onRetryAsync depending whether your policy is sync or async respectively.
Here you can see when will these user defined custom delegates be called:
Sync Retry Policy
Async Retry Policy
So, you have wired up to the right hook.
Now you have to make sure that policy is triggered. You can use Console.Write or some logger to push information from your delegate to the standard output.
Or you can simply set a breakpoint in your anonymous lambda delegate to make sure that it is called during debugging.
If it is not called then you have to check the following:
Are the thrown exception handled?
Is there any exception at all?
From a policy perspective there can be two kinds of exceptions: handled and unhandled. The former can trigger a new attempt if the threshold is not reached yet. The latter won't trigger another attempt rather it will re-throw the original exception. (Reference)
In your case the policy has been setup to trigger either when a HttpRequestException is thrown or when a SocketException. If the thrown exception is none of these then it is considered unhandled from the policy perspective.
Your policy won't be triggered if there was no exception. There is one typical mistake that we have made several times. Let's suppose we expect that the http response should be 200. Whenever is not success then we want to issue a retry. We might utilize the HandleTransientHttpError (Ref) extension. But that extension watches only the 408 and 5xx status codes. So if we receive for example 429 (too many requests) then no retry will happen. We have to explicitly call the EnsureSuccessStatusCode (Ref) method to throw error if the response was not successful.
Related
I have a .net 6 console app that I configured with polly policies depending on what each service does.
Program.cs
try
{
//other setup code
services
.AddHttpClient<ISubjectData, SubjectData>()
.AddTransientHttpErrorPolicy(ConfigurePolicy);
//other setup code
IAsyncPolicy<HttpResponseMessage> ConfigurePolicy(PolicyBuilder<HttpResponseMessage> policy)
{
try
{
return policy.Or<TaskCanceledException>()
.WaitAndRetryAsync(Backoff.DecorrelatedJitterBackoffV2(TimeSpan.FromSeconds(10), 5));
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
}
}
catch(Exception e)
{
Console.WriteLine(e);
}
The policy is working, however, the program is throwing an unhandled exception which is TaskCanceledException because the HttpClient timed out, which isn't being caught by either catch statements or the policy in ConfigurePolicy.
How can I catch this error, since it is crashing the app?
Also, is there a way to allow Polly to override HTTP client timeout depending on how long it takes to complete all retries?
TL;DR:
You can't catch the exception because you wrapped the policy definition with try- catches.
In case of Polly the policy definition and its execution are separated.
You define a policy (or a combination of policies)
You are decorating some method(s) with the above policy
You are executing the decorated method explicitly or implicitly through the policy.
Whenever you register a typed/named HttpClient and decorate its methods with an IAsyncPolicy<HttpResponseMessage> then you are doing only the first two steps.
Whenever you call a method on the HttpClient, like PostAsync or SendAsync that's where the policy is being executed on your behalf / implicitly (via a special DelegatingHandler).
That's where you can catch the exceptions
ones which is thrown either by the HttpClient (like TaskCancelledException, HttpRequestException, etc.)
or those that are thrown by one of the policies (like TimeOutRejectedException, BrokenCircuitException, etc.)
UPDATE #1
What is confusing to me, is why does a try-catch block that covers my entire Program.cs not catch the exception, but a try-catch block in the scope of the request does? Aren't exceptions propagated back to the original caller? (The original caller would be a method inside my Program.cs file)
Try catch inside the ConfigurePolicy
As I stated above, here you are building the policy, not executing it. So, the try-catch here can be used to detect and resolve misconfiguration.
Try catch inside Program
Yet again the AddHttpClient and AddTransientHttpErrorPolicy are just builder methods. They are not executing any http request on your behalf. So, wrapping this code into try-catch can help you to spot misconfiguration.
Your try-catch is covering those exceptions which are thrown by the Main but not those which are thrown by different threads which are not in the Main.
I am using Polly's retry policy for my unsuccessful call. But it is not catching the exception and retrying.
Using:
Polly 7.2.3
.NET6.0
Nsubstitute 4.2.2
Setup:
var delay = Backoff.DecorrelatedJitterBackoffV2(TimeSpan.FromMilliseconds(RetryDelay), RetryCount);
_retryPolicy = Policy.Handle<HttpRequestException>()
.Or<CustomException>()
.OrResult<string>(response => !string.IsNullOrEmpty(response))
.WaitAndRetryAsync(delay);
Usage:
public async Task ProcessRequest()
{
var errors = await _retryPolicy.ExecuteAsync(async () => await this.RetryProcessRequest());
}
private async Task<string> RetryProcessRequest()
{
var token = await _tokenInfrastructure.GetTokenAsync();
return await _timezoneInfrastructure.ProcessRequest(token);
}
Unit test:
[Fact]
public async Task ProcessRequest_Throws()
{
string errors = _fixture.Create<string>();
var token = _fixture.Create<string>();
// Retry policy configured to retry 3 times on failed call
var expectedReceivedCalls = 4;
// this is throwing but Polly is not catching it and not retrying
_tokenInfrastructure.GetTokenAsync().Returns(Task.FromException<string>(new HttpRequestException()));
// this errors can be caught by Polly as configured and retrying
_timezoneInfrastructure.ProcessRequest(token).Returns(errors);
await _timezoneOrchestration.Awaiting(o => o.ProcessRequest()).Should()
.ThrowAsync<HttpRequestException>();
await _tokenInfrastructure.Received(expectedReceivedCalls).GetTokenAsync();
await _timezoneInfrastructure.Received(expectedReceivedCalls).ProcessRequest(Arg.Any<string>());
}
After doing Rubber duck debugging found my mistake. Actually, Polly was configured well and retrying.
this line of code was never calling because above we were getting exceptions.
return await _timezoneInfrastructure.ProcessRequest(token);
In Unit tests, it was expecting some retry calls:
_timezoneInfrastructure.Received(expectedReceivedCalls).ProcessRequest(Arg.Any<string>());
This post is not answer for the OP's question (the problem has already been addressed here). It is more like a set of suggestions (you can call it code review if you wish).
Exponential backoff
I'm glad to see that you are using the V2 of the backoff logic which utilizes the jitter in a proper way.
My only concern here is that depending on the actual values of RetryDelay and RetryCount the sleepDuration might explode: It can easily reach several minutes. I would suggest two solutions in that case:
Change factor parameter of the DecorrelatedJitterBackoffV2 from 2 (which is the default) to a lower number
Or try to top the max sleepDuration, here I have detailed one way to do that
Combined Retry logic
Without knowing what does RetryProcessRequest do, it seems like this _retryPolicy smashes two different policies into one. I might be wrong, so this section could suggest something which is not applicable for your code.
I assume this part decorates the _tokenInfrastructure.GetTokenAsync call
_retryPolicy = Policy.Handle<HttpRequestException>()
.WaitAndRetryAsync(delay);
whereas this part decorates the _timezoneInfrastructure.ProcessRequest call
_retryPolicy = Policy.Handle<CustomException>()
.OrResult<string>(response => !string.IsNullOrEmpty(response))
.WaitAndRetryAsync(delay);
Based on your naming I assume that these are different downstream systems: tokenInfrastructure, timezoneInfrastructure. I would suggest to create separate policies for them. You might want to apply different Timeout for them or use separate Circuit Breakers.
Naming
I know naming is hard and I assume your method names (ProcessRequest, RetryProcessRequest or ProcessRequest_Throws) are dumyfied for StackOverflow. If not then please try to spend some time to come up with more expressive names.
Component testing
Your ProcessRequest_Throws test is not really a unit test. It is more likely a component test. You are testing there the integration between the Polly's policy and the decorated code.
If you would test only the correctness of the policy setup or test only the decorated code (with NoOpPolicy) then they were unit tests.
I have implemented Polly in it's own "retry" HttpClient DelegateHandler in a dll written to .NET Standard 2.0. I have the Polly v7.2.3 package. My HttpClient is running separate from an HttpClientFactory since only one instance will ever exist during the short lifetime of the dll.
My problem is this: The code executes great when my internet is working. However, when I disconnect my internet it throws a TaskCanceledException on the first retry and does not retry any more. Here are the relevant parts of my code...
inside the ctor of my typed HttpClient:
this.Client = new System.Net.Http.HttpClient(
new ATCacheDelegatingHandler(
new RetryPolicyDelegatingHandler(
new HttpClientHandler()))));
inside my Retry delegating handler:
this.RetryPolicy =
Policy.Handle<HttpRequestException>()
.Or<TaskCanceledException>()
.WaitAndRetryAsync(numRetries,
retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt-1) * 15));
So I did my research here on SO and found this very promising explanation and solution that suggested I call Dispose on the result. HttpClient Polly WaitAndRetry policy
Here is my updated code using that solution. The call to WaitAndRetryAsync complains it is unable to resolve the OnRetry method because it is looking for an 'Action<Exception, TimeSpan>'
private void WaitAndRetry(int numRetries)
{
this.RetryPolicy =
Policy.Handle<HttpRequestException>()
.Or<TaskCanceledException>()
.WaitAndRetryAsync(numRetries,
retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt-1) * 15)
, OnRetry); // reference to the method below
}
// unable to match to these parameters from the "WaitAndRetryAsync" call above
private Task OnRetry(DelegateResult<HttpResponseMessage> response, TimeSpan span, int retryCount, Context context)
{
if (response == null)
return Task.CompletedTask;
// this is the "Dispose" call from that SO solution I referenced above
response.Result?.Dispose();
return Task.CompletedTask;
}
Unfortunately there is NO support for a DelegateResult<HttpResponseMessage> parameter in the version of Polly I am using. All onRetry support expects the first parameter to be an "Exception". I am dead in the water using the Dispose solution if I can't access the disposable object.
Update: I want to be able to call Dispose() to affect the fix from the other StackOverflow feedback. But I can't because the onRetry method does not support the same set of parameters (i.e., the "response" object). It looks like the Polly API has changed. If so, what is the new way to gain access to the response so I can Dispose of it? Either that or is there another way to resolve the error I am getting?
So I am stuck trying to get this solution working or finding another way to resolve this exception. I welcome any feedback on how to specify the object to Dispose. Alternative approaches are also welcome.
All you need to do is to specify the HttpResponseMessage as a return type when you declare your policy.
IAsyncPolicy<HttpResponseMessage> retryPolicy = Policy<HttpResponseMessage>
.Handle<HttpRequestException>()
.Or<TaskCanceledException>()
.WaitAndRetryAsync(numRetries,
retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt - 1) * 15),
OnRetry);
So, instead of Policy.Handle... use Policy<HttpResponseMessage>.Handle...
Asp.NET core logs each request that enters based on configuration. Now i'd like to have the same functionality for Flurl requests i sent. Most notably, I of course would like to know when a requests fails or does not complete. For debugging I found logging all requests in a verbose matter was extremely helpful.
Sure can. For cross-cutting concerns like logging, you want to use Flurl's event handlers, specifically BeforeCall, AfterCall, OnError, and their async equivalents (BeforeCallAsync, AfterCallAsync, OnErrorAsync). Here's an error logging example:
private async Task HandleFlurlErrorAsync(HttpCall call)
{
await LogErrorAsync(call.Exception.Message);
call.ExceptionHandled = true; // prevents exception from bubbling up, if desired
}
// Configure once at startup:
FlurlHttp.Configure(settings => settings.OnErrorAsync = HandleFlurlErrorAsync);
I am trying to implement linear retry policy for a cloud queue. Previously I was managing the retry logic programatically on each dequeue but I saw the RetryPolicy member on QueueRequestOptions and thought I could set up the retry policy on the first add and have the cloud queue manage retries automatically. Unfortunately the code below doesn't seem to do anything. It still retries almost immediately and it retries 5 times. I have tried setting it on the on creation and it also doesn't work.
What am I missing?
Thanks!
await cloudQueue.CreateIfNotExistsAsync();
var linearRetryPolicy = new LinearRetry(TimeSpan.FromMinutes(5), 1);
var options = new QueueRequestOptions { RetryPolicy = linearRetryPolicy };
await cloudQueue.AddMessageAsync(new CloudQueueMessage(JsonConvert.SerializeObject(queueItem)), null, null, options, null);
I asked a colleague about this and he suggested that the retry policy probably relates to attempts to add the message to the queue and not to attempts to process the message from the queue.
I tested this by implementing IRetryPolicy and then disabling the storage emulator just before the call to AddMessageSync. Sure enough ShouldRetry is called after each failed attempt to add the message to the queue.
Hope this helps anyone who was similarly confused.
The RetryPolicy is actually a delegate that when evaluated returns a Microsoft.WindowsAzure.StorageClient.ShouldRetry delegate.It provides a lightweight mechanism to construct state-full retry instances in controlled manner. When each operation begins it will evaluate the RetryPolicy which will cause the CLR to create a state object behind the scenes containing the parameters used to configure the policy.
Example if simple linear retry policy
public static RetryPolicy LinearRetry(int retryCount, TimeSpan intervalBetweenRetries)
{
return () =>
{
return (int currentRetryCount, Exception lastException, out TimeSpan retryInterval) =>
{
// Do custom work here
// Set backoff
retryInterval = intervalBetweenRetries;
// Decide if we should retry, return bool
return currentRetryCount < retryCount;
};
};
}
code inside
return () => {
}
conforms to the signature for the Microsoft.WindowsAzure.StorageClient.ShouldRetry delegate and will contain the specifics of your implementation.
Once you have constructed a retry policy as above you can configure your client to use it via
Cloud[Table/Blob/Queue].Client.RetryPolicy = LinearRetry(<retryCount, intervalBetweenRetries>).
It worked for me. Hope it helps.