We Keep Coding

Pass ILogger<T> to Polly retry policy without HttpRequestMessage

When making HTTP calls using an instance of HttpClient wrapped in Polly's retry policy and injected into a controller using dependency injection, I want to send ILogger<T> from the call site to a delegate of the retry policy (e.g., onRetry), so logs are registered more appropriately.
Polly docs explain how to achieve this by sending ILogger<T> from the calls site to the retry delegates leveraging Context encapsulated in an HttpRequestMessage request.
However, this solution works when you leverage a method of the HttpClient that takes HttpRequestMessage in one of its overloads. For instance, client.SendAsync.
However, not every method of HttpClient take HttpRequestMessage. For instance, I'm using client.GetStreamAsync, which none of its overloads take HttpRequestMessage.
In this case, I wonder how you would pass the Ilogger<T> to Polly's retry delegates.

Options that does not work for your use case
Using the Context object with HttpRequestMessage
As you have stated in your question this is not applicable, since you don't have a HttpRequestMessage instance on which you could access the Context via the request.GetPolicyExecutionContext call.
Using AddPolicyHandler + IServiceProvider
The AddPolicyHandler has an overload which provides access to the IServiceProvider and to the HttpRequestMessage. You could obtain an ILoggerFactory via provider.GetRequiredService<ILoggerFactory>() and then you could call factory.CreateLogger<T>.
The problem with this approach is that you don't know T at policy registration time, since you want to use the Controller as T.
Options that could work for your use case
Defining the policy inside your Controller
If you would define the policy inside the same class where you have the intention to use it then you could access the ILogger<YourController>.
There are two drawbacks of this approach:
You have to define (more or less) the same policy in every place where you want to use it
You have to explicitly call the ExecuteAsync
The first issue can be addressed via the PolicyRegistry
Registering the policy into PolicyRegistry and using Context
You can register your policy/ies into a PolicyRegistry and then you can obtain them (via IReadOnlyPolicyRegistry) inside your controller. This approach lets you define your policy in the way that you can retrieve an ILogger from the Context inside the onRetry. And you can specify the Context when you call the ExecuteAsync
var context = new Polly.Context().WithLogger(yourControllerLogger);
await policy.ExecuteAsync(async (ct) => ..., context);
Registering the policy into PolicyRegistry and using try-catch
The previous approach used the Context to transfer an object between the policy definition and its usage. One can say that this separation is a bit fragile since the coupling between these two is not explicit rather via a magic Context object.
An alternative solution could be to perform logging only inside your the ExecuteAsync to avoid the usage of the Context
await policy.ExecuteAsync(async () =>
try
{
...
}
catch(Exception ex) //filter for the retry's trigger exception(s)
{
yourControllerLogger.LogError(...);
});
As you can see none of the above solutions is perfect since you want to couple the policy and its usage via logging.
UPDATE #1
I'm not a big fan of defining policy inside a controller, because I generally reuse a policy (and accordingly the HttpClientFactory) in different controllers.
As I said above, this is one option out of three. The other two options do not require you to define your policy inside the controller class. You can define them inside the startup
var registry = new PolicyRegistry()
{
{ "YourPolicyName", resilientStrategy }
};
services.AddPolicyRegistry(registry);
and then retrieve the given policy inside the controller
private readonly IAsyncPolicy policy;
public YourController(IReadOnlyPolicyRegistry<string> registry)
{
policy = registry.Get<IAsyncPolicy>("YourPolicyName"):
}
I suppose there is no other cleaner solution
If you want to / need to use the controller's logger inside the onRetry delegate then I'm unaware of any cleaner solution.
If you want to use that logger to be able to correlate the controller's log with the policy's log then I would rather suggest to use a correlation id per request and include that into your logs. Steve Gordon has a nuget package called correlationId which can help you to achieve that.

Polly not catching Nsubstitute's mocked exception

I am using Polly's retry policy for my unsuccessful call. But it is not catching the exception and retrying.
Using:
Polly 7.2.3
.NET6.0
Nsubstitute 4.2.2
Setup:
var delay = Backoff.DecorrelatedJitterBackoffV2(TimeSpan.FromMilliseconds(RetryDelay), RetryCount);
_retryPolicy = Policy.Handle<HttpRequestException>()
.Or<CustomException>()
.OrResult<string>(response => !string.IsNullOrEmpty(response))
.WaitAndRetryAsync(delay);
Usage:
public async Task ProcessRequest()
{
var errors = await _retryPolicy.ExecuteAsync(async () => await this.RetryProcessRequest());
}
private async Task<string> RetryProcessRequest()
{
var token = await _tokenInfrastructure.GetTokenAsync();
return await _timezoneInfrastructure.ProcessRequest(token);
}
Unit test:
[Fact]
public async Task ProcessRequest_Throws()
{
string errors = _fixture.Create<string>();
var token = _fixture.Create<string>();
// Retry policy configured to retry 3 times on failed call
var expectedReceivedCalls = 4;
// this is throwing but Polly is not catching it and not retrying
_tokenInfrastructure.GetTokenAsync().Returns(Task.FromException<string>(new HttpRequestException()));
// this errors can be caught by Polly as configured and retrying
_timezoneInfrastructure.ProcessRequest(token).Returns(errors);
await _timezoneOrchestration.Awaiting(o => o.ProcessRequest()).Should()
.ThrowAsync<HttpRequestException>();
await _tokenInfrastructure.Received(expectedReceivedCalls).GetTokenAsync();
await _timezoneInfrastructure.Received(expectedReceivedCalls).ProcessRequest(Arg.Any<string>());
}

After doing Rubber duck debugging found my mistake. Actually, Polly was configured well and retrying.
this line of code was never calling because above we were getting exceptions.
return await _timezoneInfrastructure.ProcessRequest(token);
In Unit tests, it was expecting some retry calls:
_timezoneInfrastructure.Received(expectedReceivedCalls).ProcessRequest(Arg.Any<string>());

This post is not answer for the OP's question (the problem has already been addressed here). It is more like a set of suggestions (you can call it code review if you wish).
Exponential backoff
I'm glad to see that you are using the V2 of the backoff logic which utilizes the jitter in a proper way.
My only concern here is that depending on the actual values of RetryDelay and RetryCount the sleepDuration might explode: It can easily reach several minutes. I would suggest two solutions in that case:
Change factor parameter of the DecorrelatedJitterBackoffV2 from 2 (which is the default) to a lower number
Or try to top the max sleepDuration, here I have detailed one way to do that
Combined Retry logic
Without knowing what does RetryProcessRequest do, it seems like this _retryPolicy smashes two different policies into one. I might be wrong, so this section could suggest something which is not applicable for your code.
I assume this part decorates the _tokenInfrastructure.GetTokenAsync call
_retryPolicy = Policy.Handle<HttpRequestException>()
.WaitAndRetryAsync(delay);
whereas this part decorates the _timezoneInfrastructure.ProcessRequest call
_retryPolicy = Policy.Handle<CustomException>()
.OrResult<string>(response => !string.IsNullOrEmpty(response))
.WaitAndRetryAsync(delay);
Based on your naming I assume that these are different downstream systems: tokenInfrastructure, timezoneInfrastructure. I would suggest to create separate policies for them. You might want to apply different Timeout for them or use separate Circuit Breakers.
Naming
I know naming is hard and I assume your method names (ProcessRequest, RetryProcessRequest or ProcessRequest_Throws) are dumyfied for StackOverflow. If not then please try to spend some time to come up with more expressive names.
Component testing
Your ProcessRequest_Throws test is not really a unit test. It is more likely a component test. You are testing there the integration between the Polly's policy and the decorated code.
If you would test only the correctness of the policy setup or test only the decorated code (with NoOpPolicy) then they were unit tests.

How to log something before retry using Polly?

I'm attempting to log something before retrying a web api call using Polly in a .net core web api.
I know the web api is failing and returning a 503 response code however there's nothing in my console log as part of the retry call. Any ideas why and how to resolve this?
var retryPolicy = Policy
.Handle<HttpRequestException>()
.Or<SocketException>()
.WaitAndRetryAsync(new[]
{
TimeSpan.FromSeconds(2),
TimeSpan.FromSeconds(5),
TimeSpan.FromSeconds(10)
}, (exception, timeSpan, retryCount, context) =>
{
Console.Write("RETRYING - " + DateTime.Now.Second);
});
await retryPolicy.ExecuteAsync(async () =>
{
var serviceReturnLabel = await this.stockTransfersServiceClient.GetPRReturnLabel(ItemSourceType.ReturnLocker);
if (serviceReturnLabel != null && serviceReturnLabel.Accepted)
{
returnLabel = serviceReturnLabel.PRLabel;
}
});

The retry policy exposes a hook where you can wire up a custom code which will be called before the retry penalty. In other words this delegate will be called whenever the policy should be triggered but before the wait between the two attempts.
This hook is called onRetry or onRetryAsync depending whether your policy is sync or async respectively.
Here you can see when will these user defined custom delegates be called:
Sync Retry Policy
Async Retry Policy
So, you have wired up to the right hook.
Now you have to make sure that policy is triggered. You can use Console.Write or some logger to push information from your delegate to the standard output.
Or you can simply set a breakpoint in your anonymous lambda delegate to make sure that it is called during debugging.
If it is not called then you have to check the following:
Are the thrown exception handled?
Is there any exception at all?
From a policy perspective there can be two kinds of exceptions: handled and unhandled. The former can trigger a new attempt if the threshold is not reached yet. The latter won't trigger another attempt rather it will re-throw the original exception. (Reference)
In your case the policy has been setup to trigger either when a HttpRequestException is thrown or when a SocketException. If the thrown exception is none of these then it is considered unhandled from the policy perspective.
Your policy won't be triggered if there was no exception. There is one typical mistake that we have made several times. Let's suppose we expect that the http response should be 200. Whenever is not success then we want to issue a retry. We might utilize the HandleTransientHttpError (Ref) extension. But that extension watches only the 408 and 5xx status codes. So if we receive for example 429 (too many requests) then no retry will happen. We have to explicitly call the EnsureSuccessStatusCode (Ref) method to throw error if the response was not successful.

Azure Functions CloudQueue AddMessageAsync QueueRequestOptions LinearRetryPolicy Doesn't Do Anything

I am trying to implement linear retry policy for a cloud queue. Previously I was managing the retry logic programatically on each dequeue but I saw the RetryPolicy member on QueueRequestOptions and thought I could set up the retry policy on the first add and have the cloud queue manage retries automatically. Unfortunately the code below doesn't seem to do anything. It still retries almost immediately and it retries 5 times. I have tried setting it on the on creation and it also doesn't work.
What am I missing?
Thanks!
await cloudQueue.CreateIfNotExistsAsync();
var linearRetryPolicy = new LinearRetry(TimeSpan.FromMinutes(5), 1);
var options = new QueueRequestOptions { RetryPolicy = linearRetryPolicy };
await cloudQueue.AddMessageAsync(new CloudQueueMessage(JsonConvert.SerializeObject(queueItem)), null, null, options, null);

I asked a colleague about this and he suggested that the retry policy probably relates to attempts to add the message to the queue and not to attempts to process the message from the queue.
I tested this by implementing IRetryPolicy and then disabling the storage emulator just before the call to AddMessageSync. Sure enough ShouldRetry is called after each failed attempt to add the message to the queue.
Hope this helps anyone who was similarly confused.

The RetryPolicy is actually a delegate that when evaluated returns a Microsoft.WindowsAzure.StorageClient.ShouldRetry delegate.It provides a lightweight mechanism to construct state-full retry instances in controlled manner. When each operation begins it will evaluate the RetryPolicy which will cause the CLR to create a state object behind the scenes containing the parameters used to configure the policy.
Example if simple linear retry policy
public static RetryPolicy LinearRetry(int retryCount, TimeSpan intervalBetweenRetries)
{
return () =>
{
return (int currentRetryCount, Exception lastException, out TimeSpan retryInterval) =>
{
// Do custom work here
// Set backoff
retryInterval = intervalBetweenRetries;
// Decide if we should retry, return bool
return currentRetryCount < retryCount;
};
};
}
code inside
return () => {
}
conforms to the signature for the Microsoft.WindowsAzure.StorageClient.ShouldRetry delegate and will contain the specifics of your implementation.
Once you have constructed a retry policy as above you can configure your client to use it via
Cloud[Table/Blob/Queue].Client.RetryPolicy = LinearRetry(<retryCount, intervalBetweenRetries>).
It worked for me. Hope it helps.

Retry after rehydrating token with polly

We're using the latest Polly to handle our retry and circuit breaker policies for interacting with three APIs.
Basic flow is:
A) Read data from Product Catalogue (API)
B) Get Unique Merchant Token (API)
C) Update Merchant Catalogue (with new item) (API)
Due to the load on Merchant Catalogue API (third party, can't work around this yet!) we get bounced sometimes. Polly is configured to nicely retry this if it fails and circuit breaker pattern style, back off.
We realised that it was being tripped consistently because our Merchant Token was marked as invalid even though the server spat the dummy - the third party marks a token used even on error.
Reading this article which is what we based our solution on initially, we are thinking using the context to reload/refresh the auth token. However I'm a bit confused how I can have a policy that refreshes that token when that logic is not in the wiring up (startup) and instead in the handler that runs the policy.
var authMerchTokenPolicy = Policy<HttpResponseMessage>
.HandleResult(r => r.StatusCode == 500)
.RetryAsync(1, onRetryAsync: async (ex, i, context) => await RefreshMerchantAuthorization(context["httpClient"]));
Is the above example stating that I implement RefreshMerchantAuthorization in the startup class?
I haven't seen a concrete example which is where the confusion lies - and the original developer has since left who wrote this (ironically named Paulie!)

Is the above example stating that I implement RefreshMerchantAuthorization in the startup class?
Polly's Context class allows you to carry any custom data, with Dictionary<string, object>-like semantics. So you can also pass the handler class in to the policy via the Context.
For RefreshMerchantAuthorization(...) an instance method on a class FooHandler, then you can configure the policy in StartUp:
var authMerchTokenPolicy = Policy<HttpResponseMessage>
.HandleResult(r => r.StatusCode == 500)
.RetryAsync(1, onRetryAsync: async (ex, i, context) =>
await ((FooHandler)context["handler"]).RefreshMerchantAuthorization(context["httpClient"]));
With, at the policy usage site within FooHandler:
var httpResponseMessage =
await authMerchTokenPolicy.ExecuteAsync(context => context["httpClient"].GetAsync(uri),
contextData: new Dictionary<string, object> {
{"httpClient", httpClient},
{"handler", this}
});
This is all assuming the RefreshMerchantAuthorization(...) isn't/can't be made static (if static it can be referenced directly from the StartUp class as a static method).

If I understand your question correctly then you are basically interested about how could you call the RefreshMerchantAuthorization method if it is not defined inside the Startup class rather somewhere else?
Let's suppose you have an interface (for example: IRefresher) and an implementation (for example Refresher) which contains the RefreshMerchantAuthorization method.
You have registered these into the DI container (for example: .AddScoped<IRefresher, Refresher>()).
You are using this authMerchTokenPolicy to decorate a pre-configured HttpClient (for example: .AddHttpClient(...).AddPolicyHandler(authMerchTokenPolicy))
Then you can use an overload of AddPolicyHandler which allows you access the IServiceProvider and the HttpRequestMessage:
.AddPolicyHandler((provider, request) => Policy<HttpResponseMessage>
.HandleResult(r => r.StatusCode == 500)
.RetryAsync(1, async (_, __, context) =>
{
var refresher = sp.GetRequiredService<IRefresher>();
await refresher.RefreshMerchantAuthorization(context["httpClient"]);
}));