I am using Polly's retry policy for my unsuccessful call. But it is not catching the exception and retrying.
Using:
Polly 7.2.3
.NET6.0
Nsubstitute 4.2.2
Setup:
var delay = Backoff.DecorrelatedJitterBackoffV2(TimeSpan.FromMilliseconds(RetryDelay), RetryCount);
_retryPolicy = Policy.Handle<HttpRequestException>()
.Or<CustomException>()
.OrResult<string>(response => !string.IsNullOrEmpty(response))
.WaitAndRetryAsync(delay);
Usage:
public async Task ProcessRequest()
{
var errors = await _retryPolicy.ExecuteAsync(async () => await this.RetryProcessRequest());
}
private async Task<string> RetryProcessRequest()
{
var token = await _tokenInfrastructure.GetTokenAsync();
return await _timezoneInfrastructure.ProcessRequest(token);
}
Unit test:
[Fact]
public async Task ProcessRequest_Throws()
{
string errors = _fixture.Create<string>();
var token = _fixture.Create<string>();
// Retry policy configured to retry 3 times on failed call
var expectedReceivedCalls = 4;
// this is throwing but Polly is not catching it and not retrying
_tokenInfrastructure.GetTokenAsync().Returns(Task.FromException<string>(new HttpRequestException()));
// this errors can be caught by Polly as configured and retrying
_timezoneInfrastructure.ProcessRequest(token).Returns(errors);
await _timezoneOrchestration.Awaiting(o => o.ProcessRequest()).Should()
.ThrowAsync<HttpRequestException>();
await _tokenInfrastructure.Received(expectedReceivedCalls).GetTokenAsync();
await _timezoneInfrastructure.Received(expectedReceivedCalls).ProcessRequest(Arg.Any<string>());
}
After doing Rubber duck debugging found my mistake. Actually, Polly was configured well and retrying.
this line of code was never calling because above we were getting exceptions.
return await _timezoneInfrastructure.ProcessRequest(token);
In Unit tests, it was expecting some retry calls:
_timezoneInfrastructure.Received(expectedReceivedCalls).ProcessRequest(Arg.Any<string>());
This post is not answer for the OP's question (the problem has already been addressed here). It is more like a set of suggestions (you can call it code review if you wish).
Exponential backoff
I'm glad to see that you are using the V2 of the backoff logic which utilizes the jitter in a proper way.
My only concern here is that depending on the actual values of RetryDelay and RetryCount the sleepDuration might explode: It can easily reach several minutes. I would suggest two solutions in that case:
Change factor parameter of the DecorrelatedJitterBackoffV2 from 2 (which is the default) to a lower number
Or try to top the max sleepDuration, here I have detailed one way to do that
Combined Retry logic
Without knowing what does RetryProcessRequest do, it seems like this _retryPolicy smashes two different policies into one. I might be wrong, so this section could suggest something which is not applicable for your code.
I assume this part decorates the _tokenInfrastructure.GetTokenAsync call
_retryPolicy = Policy.Handle<HttpRequestException>()
.WaitAndRetryAsync(delay);
whereas this part decorates the _timezoneInfrastructure.ProcessRequest call
_retryPolicy = Policy.Handle<CustomException>()
.OrResult<string>(response => !string.IsNullOrEmpty(response))
.WaitAndRetryAsync(delay);
Based on your naming I assume that these are different downstream systems: tokenInfrastructure, timezoneInfrastructure. I would suggest to create separate policies for them. You might want to apply different Timeout for them or use separate Circuit Breakers.
Naming
I know naming is hard and I assume your method names (ProcessRequest, RetryProcessRequest or ProcessRequest_Throws) are dumyfied for StackOverflow. If not then please try to spend some time to come up with more expressive names.
Component testing
Your ProcessRequest_Throws test is not really a unit test. It is more likely a component test. You are testing there the integration between the Polly's policy and the decorated code.
If you would test only the correctness of the policy setup or test only the decorated code (with NoOpPolicy) then they were unit tests.
Related
I have .net core weabpi (see code below). I am using polly retry policy (see policy below). I would like to unit test endpoint (getProducts) and test polly retry
I have found these examples but it is not clear how to unit test endpoint and retry policy?
services
.AddHttpClient<IProductService, ProductService>()
.AddPolicyHandler(GetRetryPolicy(3, 2));
static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy(int retryCount, int breakDuration)
{
return HttpPolicyExtensions
.HandleTransientHttpError()
.OrResult(msg => msg.StatusCode == System.Net.HttpStatusCode.NotFound)
.WaitAndRetryAsync(retryCount, retryAttempt => TimeSpan.FromSeconds(Math.Pow(breakDuration,
retryAttempt)));
}
.Net core api:
public interface IProductService
{
Task<IEnumerable<ProductResponse>> GetProducts(string productType);
}
public class ProductService: IProductService
{
private readonly HttpClient _httpClient;
public ProductService(HttpClient httpClient)
{
_httpClient = httpClient;
}
public async Task<IEnumerable<ProductResponse>> GetProducts(string productType)
{
var response = await _httpClient.GetAsync("uri");
...
}
}
The sad truth is you can't really unit test your retry+endpoint logic and here are my reasonings why:
The retry is registered on the top of the HttpClient via the DI (AddPolicyHandler). When you are unit testing then you are not relying on the DI rather on individual components.
1.1 So, an integration test might be more suitable for this. I've already detailed how can you do that via WireMock.Net: 1, 2. The basic idea is to create a local http server (to mock the downstream system) with a predefined response sequence.
After you have defined your retry policy with the max retry count and time penalties, you can not retrieve them easily. So, from a unit testing perspective it is really hard to make sure that the policy has been defined correctly (like the delay is specified in seconds, not in minutes). I've already created a github issue for this, but unfortunately the development of the V8 got stuck.
Back to your test case. The correct way to articulate your test scenario with the given-when-then structure should be written like this
Given a faulty downstream service which returns 5XX responses
When I call GetProducts
Then it is performed 4 times (1 initial + 3 retry attempts)
This is not a unit test. It is more like a component/integration test. Why? Because even though you could create an HttpClient mock but in that case there will be no retry policy there.
There is a workaround: you could manually decorate the underlying handler with the policy via the PolicyHttpMessageHandler. But that's a bad idea in a unit test, because you basically re-implemented the DI part inside your test. And with that you would test your test arrangement code, not your production code.
I'm attempting to log something before retrying a web api call using Polly in a .net core web api.
I know the web api is failing and returning a 503 response code however there's nothing in my console log as part of the retry call. Any ideas why and how to resolve this?
var retryPolicy = Policy
.Handle<HttpRequestException>()
.Or<SocketException>()
.WaitAndRetryAsync(new[]
{
TimeSpan.FromSeconds(2),
TimeSpan.FromSeconds(5),
TimeSpan.FromSeconds(10)
}, (exception, timeSpan, retryCount, context) =>
{
Console.Write("RETRYING - " + DateTime.Now.Second);
});
await retryPolicy.ExecuteAsync(async () =>
{
var serviceReturnLabel = await this.stockTransfersServiceClient.GetPRReturnLabel(ItemSourceType.ReturnLocker);
if (serviceReturnLabel != null && serviceReturnLabel.Accepted)
{
returnLabel = serviceReturnLabel.PRLabel;
}
});
The retry policy exposes a hook where you can wire up a custom code which will be called before the retry penalty. In other words this delegate will be called whenever the policy should be triggered but before the wait between the two attempts.
This hook is called onRetry or onRetryAsync depending whether your policy is sync or async respectively.
Here you can see when will these user defined custom delegates be called:
Sync Retry Policy
Async Retry Policy
So, you have wired up to the right hook.
Now you have to make sure that policy is triggered. You can use Console.Write or some logger to push information from your delegate to the standard output.
Or you can simply set a breakpoint in your anonymous lambda delegate to make sure that it is called during debugging.
If it is not called then you have to check the following:
Are the thrown exception handled?
Is there any exception at all?
From a policy perspective there can be two kinds of exceptions: handled and unhandled. The former can trigger a new attempt if the threshold is not reached yet. The latter won't trigger another attempt rather it will re-throw the original exception. (Reference)
In your case the policy has been setup to trigger either when a HttpRequestException is thrown or when a SocketException. If the thrown exception is none of these then it is considered unhandled from the policy perspective.
Your policy won't be triggered if there was no exception. There is one typical mistake that we have made several times. Let's suppose we expect that the http response should be 200. Whenever is not success then we want to issue a retry. We might utilize the HandleTransientHttpError (Ref) extension. But that extension watches only the 408 and 5xx status codes. So if we receive for example 429 (too many requests) then no retry will happen. We have to explicitly call the EnsureSuccessStatusCode (Ref) method to throw error if the response was not successful.
Asp.NET core logs each request that enters based on configuration. Now i'd like to have the same functionality for Flurl requests i sent. Most notably, I of course would like to know when a requests fails or does not complete. For debugging I found logging all requests in a verbose matter was extremely helpful.
Sure can. For cross-cutting concerns like logging, you want to use Flurl's event handlers, specifically BeforeCall, AfterCall, OnError, and their async equivalents (BeforeCallAsync, AfterCallAsync, OnErrorAsync). Here's an error logging example:
private async Task HandleFlurlErrorAsync(HttpCall call)
{
await LogErrorAsync(call.Exception.Message);
call.ExceptionHandled = true; // prevents exception from bubbling up, if desired
}
// Configure once at startup:
FlurlHttp.Configure(settings => settings.OnErrorAsync = HandleFlurlErrorAsync);
I am trying to implement linear retry policy for a cloud queue. Previously I was managing the retry logic programatically on each dequeue but I saw the RetryPolicy member on QueueRequestOptions and thought I could set up the retry policy on the first add and have the cloud queue manage retries automatically. Unfortunately the code below doesn't seem to do anything. It still retries almost immediately and it retries 5 times. I have tried setting it on the on creation and it also doesn't work.
What am I missing?
Thanks!
await cloudQueue.CreateIfNotExistsAsync();
var linearRetryPolicy = new LinearRetry(TimeSpan.FromMinutes(5), 1);
var options = new QueueRequestOptions { RetryPolicy = linearRetryPolicy };
await cloudQueue.AddMessageAsync(new CloudQueueMessage(JsonConvert.SerializeObject(queueItem)), null, null, options, null);
I asked a colleague about this and he suggested that the retry policy probably relates to attempts to add the message to the queue and not to attempts to process the message from the queue.
I tested this by implementing IRetryPolicy and then disabling the storage emulator just before the call to AddMessageSync. Sure enough ShouldRetry is called after each failed attempt to add the message to the queue.
Hope this helps anyone who was similarly confused.
The RetryPolicy is actually a delegate that when evaluated returns a Microsoft.WindowsAzure.StorageClient.ShouldRetry delegate.It provides a lightweight mechanism to construct state-full retry instances in controlled manner. When each operation begins it will evaluate the RetryPolicy which will cause the CLR to create a state object behind the scenes containing the parameters used to configure the policy.
Example if simple linear retry policy
public static RetryPolicy LinearRetry(int retryCount, TimeSpan intervalBetweenRetries)
{
return () =>
{
return (int currentRetryCount, Exception lastException, out TimeSpan retryInterval) =>
{
// Do custom work here
// Set backoff
retryInterval = intervalBetweenRetries;
// Decide if we should retry, return bool
return currentRetryCount < retryCount;
};
};
}
code inside
return () => {
}
conforms to the signature for the Microsoft.WindowsAzure.StorageClient.ShouldRetry delegate and will contain the specifics of your implementation.
Once you have constructed a retry policy as above you can configure your client to use it via
Cloud[Table/Blob/Queue].Client.RetryPolicy = LinearRetry(<retryCount, intervalBetweenRetries>).
It worked for me. Hope it helps.
I have a query, IGetHamburgers, that calls an external API.
I've registered the implementation of IGetHamburgers in my DI container as a Singleton. Im using Polly as a Circuitbreaker, if two requests fails the circuit will open.
My goal is that all calls to the Hamburger api should go through the same circuitbreaker, if GetHamburgers fails, then all other calls should fail as well.
How should I use my Policy? Should I register my Policy as a field like this:
private Policy _policy;
private Policy Policy
{
get
{
if(this_policy != null)
{
return this_policy;
}
this._policy = Policy
.Handle<Exception>()
.CircuitBreaker(2, TimeSpan.FromMinutes(1));
return this._policy;
}
}
public object Execute(.......)
{
return Policy.Execute(() => this.hamburgerQuery.GetHamburgers());
}
OR
public object Execute(.......)
{
var breaker = Policy
.Handle<Exception>()
.CircuitBreaker(2, TimeSpan.FromMinutes(1));
return breaker.Execute(() => this.hamburgerQuery.GetHamburgers());
}
I guess that the first option is the correct way since then the Policy object will always be the same and can keep track of the exception count and stuff like that.
My question is, will option number two work as well? I've found a lot of samples/examples on Pollys Github but I can't find any "real world" examples where Polly is used together with DI and stuff like that?
I guess that the first option is the correct way since then the Policy object will always be the same and can keep track of the exception count and stuff like that.
Correct. This is described in the Polly wiki here. In brief:
Share the same breaker policy instance across call sites when you want those call sites to break in common - for instance they have a common downstream dependency.
Don't share a breaker instance across call sites when you want those call sites to have independent circuit state and break independently.
See this stackoverflow answer for a more extensive discussion of configuring policies separately from their usage, injecting them to usage sites by DI, and the effects of re-using the same instance (for example a singleton) versus using separate instances, across the full range (at June 2017) of Polly policies.
will option number two work as well?
No (for the converse reason: each call creates a separate instance, so won't share circuit statistics/states with other calls).