I am trying to implement linear retry policy for a cloud queue. Previously I was managing the retry logic programatically on each dequeue but I saw the RetryPolicy member on QueueRequestOptions and thought I could set up the retry policy on the first add and have the cloud queue manage retries automatically. Unfortunately the code below doesn't seem to do anything. It still retries almost immediately and it retries 5 times. I have tried setting it on the on creation and it also doesn't work.
What am I missing?
Thanks!
await cloudQueue.CreateIfNotExistsAsync();
var linearRetryPolicy = new LinearRetry(TimeSpan.FromMinutes(5), 1);
var options = new QueueRequestOptions { RetryPolicy = linearRetryPolicy };
await cloudQueue.AddMessageAsync(new CloudQueueMessage(JsonConvert.SerializeObject(queueItem)), null, null, options, null);
I asked a colleague about this and he suggested that the retry policy probably relates to attempts to add the message to the queue and not to attempts to process the message from the queue.
I tested this by implementing IRetryPolicy and then disabling the storage emulator just before the call to AddMessageSync. Sure enough ShouldRetry is called after each failed attempt to add the message to the queue.
Hope this helps anyone who was similarly confused.
The RetryPolicy is actually a delegate that when evaluated returns a Microsoft.WindowsAzure.StorageClient.ShouldRetry delegate.It provides a lightweight mechanism to construct state-full retry instances in controlled manner. When each operation begins it will evaluate the RetryPolicy which will cause the CLR to create a state object behind the scenes containing the parameters used to configure the policy.
Example if simple linear retry policy
public static RetryPolicy LinearRetry(int retryCount, TimeSpan intervalBetweenRetries)
{
return () =>
{
return (int currentRetryCount, Exception lastException, out TimeSpan retryInterval) =>
{
// Do custom work here
// Set backoff
retryInterval = intervalBetweenRetries;
// Decide if we should retry, return bool
return currentRetryCount < retryCount;
};
};
}
code inside
return () => {
}
conforms to the signature for the Microsoft.WindowsAzure.StorageClient.ShouldRetry delegate and will contain the specifics of your implementation.
Once you have constructed a retry policy as above you can configure your client to use it via
Cloud[Table/Blob/Queue].Client.RetryPolicy = LinearRetry(<retryCount, intervalBetweenRetries>).
It worked for me. Hope it helps.
Related
I am developing an integration solution that accesses a rate limited API. I am performing a variety of CRUD operations on the API using multiple HTTP verbs on different endpoints (on the same server though). I have been pointed towards Polly multiple times, but I haven't managed to come up with a solution that actually works.
This is what I have in my startup:
builder.Services
.AddHttpClient("APIClient", client =>
{
client.BaseAddress = new Uri(C.Configuration.GetValue<string>("APIBaseAddress"));
})
.AddTransientHttpErrorPolicy(builder =>
builder.WaitAndRetryAsync(new []
{
TimeSpan.FromSeconds(1),
TimeSpan.FromSeconds(5),
TimeSpan.FromSeconds(15),
}));
This is just resilience to retry in case of failure. I have a RateLimit policy in a singleton ApiWrapper class:
public sealed class ApiWrapper
{
private static readonly Lazy<ApiWrapper> lazy = new Lazy<ApiWrapper>(() => new ApiWrapper());
public static ApiWrapper Instance { get { return lazy.Value; } }
private IHttpClientFactory _httpClientFactory;
public readonly AsyncRateLimitPolicy RateLimit = Policy.RateLimitAsync(150, TimeSpan.FromSeconds(10), 50); // 150 actions within 10 sec, 50 burst
private ApiWrapper()
{
}
public void SetFactory(IHttpClientFactory httpClientFactory)
{
_httpClientFactory = httpClientFactory;
}
public HttpClient GetApiClient()
{
return _httpClientFactory.CreateClient("APIClient");
}
}
That policy is used in multiple other classes like this:
public class ApiConsumer
{
private HttpClient _httpClient = ApiWrapper.Instance.GetApiClient();
public async Task<bool> DoSomethingWithA(List<int> customerIDs)
{
foreach (int id in customerIDs)
{
HttpResponseMessage httpResponse = await ApiWrapper.Instance.RateLimit.ExecuteAsync(() => _httpClient.GetAsync($"http://some.endpoint"));
}
}
}
My expectation was that the rate limiter would not fire more requests than configured, but that does not seem to be true. From my understanding the way it works is that the rate limiter just throws an exception if there are more calls than the limit that has been configured. That's where I thought the Retry policy would come into play, so just try again after 5 or 15 seconds if it did not go through the limiter.
Then I played around a bit with Polly's Bulkhead policy, but as far as I can see that is meant to limit the amount of parallel executions.
I have multiple threads that may use different HttpClients (all created by the Factory like in the example above) with different methods and endpoints, but all use the same policies. Some threads run in parallel, some sequentially as I have to wait for their response before sending the next requests.
Any suggestions on how this can or should be achieved with Polly? (Or any other extension if there is good reason to)
Thanks again to #Neil and #Panagiotis for pointing me in the right direction. I wrongly assumed that the Polly rate limiter would actually delay API calls. I found a workaround that probably is not particularly nice, but for my purpose it does the trick.
I installed David Desmaisons RateLimiter package which is super simple to use. In my singleton I have now this:
public TimeLimiter RateLimiter = TimeLimiter.GetFromMaxCountByInterval(150, TimeSpan.FromSeconds(10));
I use this RateLimiter everywhere I make calls to an API endpoint like this:
HttpResponseMessage httpResponse = await ApiWrapper.Instance.RateLimiter.Enqueue(() => _httpClient.GetAsync($"http://some.endpoint"), _cancellationToken);
Does exactly what I originally expected from Polly.
In this post I would like to clarify things around rate limiter and rate gate
Similarity
Both concepts can be used to throttle requests.
They sit between the clients and the server and they know about the server's capacity.
Difference
The limiter as its name implies limits the transient traffic. It short-cuts the requests if there are too many.
The gate on the other hand holds/delays the requests until there is enough capacity.
Algorithms
The rate limiter usually implements the leaky-bucket or token-bucket algorithm
Polly's ratelimiter implements token bucket
The rate gate usually utilizies some queuing mechanism and timers
Sample implementation
I am using Polly's retry policy for my unsuccessful call. But it is not catching the exception and retrying.
Using:
Polly 7.2.3
.NET6.0
Nsubstitute 4.2.2
Setup:
var delay = Backoff.DecorrelatedJitterBackoffV2(TimeSpan.FromMilliseconds(RetryDelay), RetryCount);
_retryPolicy = Policy.Handle<HttpRequestException>()
.Or<CustomException>()
.OrResult<string>(response => !string.IsNullOrEmpty(response))
.WaitAndRetryAsync(delay);
Usage:
public async Task ProcessRequest()
{
var errors = await _retryPolicy.ExecuteAsync(async () => await this.RetryProcessRequest());
}
private async Task<string> RetryProcessRequest()
{
var token = await _tokenInfrastructure.GetTokenAsync();
return await _timezoneInfrastructure.ProcessRequest(token);
}
Unit test:
[Fact]
public async Task ProcessRequest_Throws()
{
string errors = _fixture.Create<string>();
var token = _fixture.Create<string>();
// Retry policy configured to retry 3 times on failed call
var expectedReceivedCalls = 4;
// this is throwing but Polly is not catching it and not retrying
_tokenInfrastructure.GetTokenAsync().Returns(Task.FromException<string>(new HttpRequestException()));
// this errors can be caught by Polly as configured and retrying
_timezoneInfrastructure.ProcessRequest(token).Returns(errors);
await _timezoneOrchestration.Awaiting(o => o.ProcessRequest()).Should()
.ThrowAsync<HttpRequestException>();
await _tokenInfrastructure.Received(expectedReceivedCalls).GetTokenAsync();
await _timezoneInfrastructure.Received(expectedReceivedCalls).ProcessRequest(Arg.Any<string>());
}
After doing Rubber duck debugging found my mistake. Actually, Polly was configured well and retrying.
this line of code was never calling because above we were getting exceptions.
return await _timezoneInfrastructure.ProcessRequest(token);
In Unit tests, it was expecting some retry calls:
_timezoneInfrastructure.Received(expectedReceivedCalls).ProcessRequest(Arg.Any<string>());
This post is not answer for the OP's question (the problem has already been addressed here). It is more like a set of suggestions (you can call it code review if you wish).
Exponential backoff
I'm glad to see that you are using the V2 of the backoff logic which utilizes the jitter in a proper way.
My only concern here is that depending on the actual values of RetryDelay and RetryCount the sleepDuration might explode: It can easily reach several minutes. I would suggest two solutions in that case:
Change factor parameter of the DecorrelatedJitterBackoffV2 from 2 (which is the default) to a lower number
Or try to top the max sleepDuration, here I have detailed one way to do that
Combined Retry logic
Without knowing what does RetryProcessRequest do, it seems like this _retryPolicy smashes two different policies into one. I might be wrong, so this section could suggest something which is not applicable for your code.
I assume this part decorates the _tokenInfrastructure.GetTokenAsync call
_retryPolicy = Policy.Handle<HttpRequestException>()
.WaitAndRetryAsync(delay);
whereas this part decorates the _timezoneInfrastructure.ProcessRequest call
_retryPolicy = Policy.Handle<CustomException>()
.OrResult<string>(response => !string.IsNullOrEmpty(response))
.WaitAndRetryAsync(delay);
Based on your naming I assume that these are different downstream systems: tokenInfrastructure, timezoneInfrastructure. I would suggest to create separate policies for them. You might want to apply different Timeout for them or use separate Circuit Breakers.
Naming
I know naming is hard and I assume your method names (ProcessRequest, RetryProcessRequest or ProcessRequest_Throws) are dumyfied for StackOverflow. If not then please try to spend some time to come up with more expressive names.
Component testing
Your ProcessRequest_Throws test is not really a unit test. It is more likely a component test. You are testing there the integration between the Polly's policy and the decorated code.
If you would test only the correctness of the policy setup or test only the decorated code (with NoOpPolicy) then they were unit tests.
I have implemented Polly in it's own "retry" HttpClient DelegateHandler in a dll written to .NET Standard 2.0. I have the Polly v7.2.3 package. My HttpClient is running separate from an HttpClientFactory since only one instance will ever exist during the short lifetime of the dll.
My problem is this: The code executes great when my internet is working. However, when I disconnect my internet it throws a TaskCanceledException on the first retry and does not retry any more. Here are the relevant parts of my code...
inside the ctor of my typed HttpClient:
this.Client = new System.Net.Http.HttpClient(
new ATCacheDelegatingHandler(
new RetryPolicyDelegatingHandler(
new HttpClientHandler()))));
inside my Retry delegating handler:
this.RetryPolicy =
Policy.Handle<HttpRequestException>()
.Or<TaskCanceledException>()
.WaitAndRetryAsync(numRetries,
retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt-1) * 15));
So I did my research here on SO and found this very promising explanation and solution that suggested I call Dispose on the result. HttpClient Polly WaitAndRetry policy
Here is my updated code using that solution. The call to WaitAndRetryAsync complains it is unable to resolve the OnRetry method because it is looking for an 'Action<Exception, TimeSpan>'
private void WaitAndRetry(int numRetries)
{
this.RetryPolicy =
Policy.Handle<HttpRequestException>()
.Or<TaskCanceledException>()
.WaitAndRetryAsync(numRetries,
retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt-1) * 15)
, OnRetry); // reference to the method below
}
// unable to match to these parameters from the "WaitAndRetryAsync" call above
private Task OnRetry(DelegateResult<HttpResponseMessage> response, TimeSpan span, int retryCount, Context context)
{
if (response == null)
return Task.CompletedTask;
// this is the "Dispose" call from that SO solution I referenced above
response.Result?.Dispose();
return Task.CompletedTask;
}
Unfortunately there is NO support for a DelegateResult<HttpResponseMessage> parameter in the version of Polly I am using. All onRetry support expects the first parameter to be an "Exception". I am dead in the water using the Dispose solution if I can't access the disposable object.
Update: I want to be able to call Dispose() to affect the fix from the other StackOverflow feedback. But I can't because the onRetry method does not support the same set of parameters (i.e., the "response" object). It looks like the Polly API has changed. If so, what is the new way to gain access to the response so I can Dispose of it? Either that or is there another way to resolve the error I am getting?
So I am stuck trying to get this solution working or finding another way to resolve this exception. I welcome any feedback on how to specify the object to Dispose. Alternative approaches are also welcome.
All you need to do is to specify the HttpResponseMessage as a return type when you declare your policy.
IAsyncPolicy<HttpResponseMessage> retryPolicy = Policy<HttpResponseMessage>
.Handle<HttpRequestException>()
.Or<TaskCanceledException>()
.WaitAndRetryAsync(numRetries,
retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt - 1) * 15),
OnRetry);
So, instead of Policy.Handle... use Policy<HttpResponseMessage>.Handle...
I'm attempting to log something before retrying a web api call using Polly in a .net core web api.
I know the web api is failing and returning a 503 response code however there's nothing in my console log as part of the retry call. Any ideas why and how to resolve this?
var retryPolicy = Policy
.Handle<HttpRequestException>()
.Or<SocketException>()
.WaitAndRetryAsync(new[]
{
TimeSpan.FromSeconds(2),
TimeSpan.FromSeconds(5),
TimeSpan.FromSeconds(10)
}, (exception, timeSpan, retryCount, context) =>
{
Console.Write("RETRYING - " + DateTime.Now.Second);
});
await retryPolicy.ExecuteAsync(async () =>
{
var serviceReturnLabel = await this.stockTransfersServiceClient.GetPRReturnLabel(ItemSourceType.ReturnLocker);
if (serviceReturnLabel != null && serviceReturnLabel.Accepted)
{
returnLabel = serviceReturnLabel.PRLabel;
}
});
The retry policy exposes a hook where you can wire up a custom code which will be called before the retry penalty. In other words this delegate will be called whenever the policy should be triggered but before the wait between the two attempts.
This hook is called onRetry or onRetryAsync depending whether your policy is sync or async respectively.
Here you can see when will these user defined custom delegates be called:
Sync Retry Policy
Async Retry Policy
So, you have wired up to the right hook.
Now you have to make sure that policy is triggered. You can use Console.Write or some logger to push information from your delegate to the standard output.
Or you can simply set a breakpoint in your anonymous lambda delegate to make sure that it is called during debugging.
If it is not called then you have to check the following:
Are the thrown exception handled?
Is there any exception at all?
From a policy perspective there can be two kinds of exceptions: handled and unhandled. The former can trigger a new attempt if the threshold is not reached yet. The latter won't trigger another attempt rather it will re-throw the original exception. (Reference)
In your case the policy has been setup to trigger either when a HttpRequestException is thrown or when a SocketException. If the thrown exception is none of these then it is considered unhandled from the policy perspective.
Your policy won't be triggered if there was no exception. There is one typical mistake that we have made several times. Let's suppose we expect that the http response should be 200. Whenever is not success then we want to issue a retry. We might utilize the HandleTransientHttpError (Ref) extension. But that extension watches only the 408 and 5xx status codes. So if we receive for example 429 (too many requests) then no retry will happen. We have to explicitly call the EnsureSuccessStatusCode (Ref) method to throw error if the response was not successful.
I am using WSO2 as my Identity Provider (IDP). It is putting the JWT in an header called "X-JWT-Assertion".
To feed this into the ASP.NET Core system, I added an OnMessageReceived event. This allows me to set the token to the value supplied in the header.
Here is the code that I have to do that (the key part is the last 3 lines of non-bracket code):
services.AddAuthentication(options =>
{
options.DefaultAuthenticateScheme = JwtBearerDefaults.AuthenticationScheme;
options.DefaultSignInScheme = CookieAuthenticationDefaults.AuthenticationScheme;
})
.AddCookie()
.AddJwtBearer(async options =>
{
options.TokenValidationParameters =
await wso2Actions.JwtOperations.GetTokenValidationParameters();
options.Events = new JwtBearerEvents()
{
// WSO2 sends the JWT in a different field than what is expected.
// This allows us to feed it in.
OnMessageReceived = context =>
{
context.Token = context.HttpContext.Request.Headers["X-JWT-Assertion"];
return Task.CompletedTask;
}
}
};
This all works perfectly except for the very first call after the service starts up. To be clear, every call, except for the first one works exactly as I want it to. (It puts the token in and updates the User object like I need.)
But for the first call, the OnMessageReceived is not hit. And the User object in my controller is not setup.
I checked HttpContext for that first call, and the "X-JWT-Assertion" header is in the Request.Headers list (with the JWT in it). But, for some reason, the OnMessageReceived event is not called for it.
How can I get OnMessageReceived to be called for the first invocation of a service operation for my service?
IMPORTANT NOTE: I figured out that the issue was async await in AddJwtBearer. (See my answer below.) That is what I really wanted out of this question.
However, since a bounty cannot be cancled, I will still award the bounty to anyone who can show a way to use AddJwtBearer with async await where it is awaiting an actual HttpClient call. Or show documentation of why async await is not supposed to be used with AddJwtBearer.
UPDATE:
The lambda is an Action method. It does not return anything. So trying to do asynchrony in it is not possible without it being fire and forget.
Also, this method is invoked on the first call. So the answer is to call anything you need in this method in advance and cache it. (However, I have not figured out a non-hack way to use dependency injected items to make this call.) Then during the first call, this lambda will be called. At that time you should pull the values you need from cache (thus not slowing down the first call much).
This is what I finally figured out.
The lambda for AddJwtBearer does not work with async await. My call to await wso2Actions.JwtOperations.GetTokenValidationParameters(); awaits just fine, but the call pipeline goes on without waiting for AddJwtBearer to finish.
With async await the call order goes like this:
The service starts up (and you wait a while for it all to be happy.)
A call is made to the service.
AddJwtBearer is called.
await wso2Actions.JwtOperations.GetTokenValidationParameters(); is called.
GetTokenValidationParameters() invokes an HttpClient with await.
The HttpClient does an awaited call to get the public signing key of the issuer.
While the HttpClient is awaiting, the rest of the original call goes through. No events had been setup yet, so it just goes on with the call pipeline as normal.
This is where it "appears to skip" the OnMessageReceived event.
The HttpClient gets the response with the public key.
Execution of AddJwtBearer continues.
The OnMessageReceived event is setup.
A second call is made to the service
Because the event was eventually setup, the event is called. (AddJwtBearer is only called on the first call.)
So, when the await happens (in this case it eventually hits an HttpClient call to get the Issuer Signing Key), the rest of the first call goes through. Because there was no event setup yet, it does not know to call the handler.
I changed the lambda of AddJwtBearer to not be async and it worked just fine.
Notes:
Two things seem odd here:
I would have thought that AddJwtBearer would be called at startup, not on the first call of the service.
I would have thought that AddJwtBearer would not support an async lambda signature if it could not correctly apply the await.
I am not sure if this is a bug or not, but I posted it as one just in case: https://github.com/dotnet/aspnetcore/issues/20799
The reason your first couple requests cannot trigger OnMessageReceived is not because of the async void delegate you are using, but the order of how the parameters being loaded and the events being attached.
You attach handlers to events after await, meaning you created a race condition here, that, if say some request arrives before the await is completed, there is no event handler attached to OnMessageReceived at all.
To fix this, you should attach event handlers before the first await. This will guarantee that you always have event handlers attached to OnMessageReceived.
Try this code:
services.AddAuthentication(opt =>
{
// ...
})
.AddJwtBearer(async opt =>
{
var tcs = new TaskCompletionSource<object>();
// Any code before the first await in this delegate can run
// synchronously, so if you have events to attach for all requests
// attach handlers before await.
opt.Events = new JwtBearerEvents
{
// This method is first event in authentication pipeline
// we have chance to wait until TokenValidationParameters
// is loaded.
OnMessageReceived = async context =>
{
// Wait until token validation parameters loaded.
await tcs.Task;
}
};
// This delegate returns if GetTokenValidationParametersAsync
// does not complete synchronously
try
{
opt.TokenValidationParameters = await GetTokenValidationParametersAsync();
}
finally
{
tcs.TrySetResult(true);
}
// Any code here will be executed as continuation of
// GetTokenValidationParametersAsync and may not
// be seen by first couple requests
});
You can use GetAwaiter().GetResult() to execute async code in startup. It will block the thread, but it's ok because it only run once and it's in the startup of the application.
However if you don't like to block the thread and insist on using await to get the options, you can use async await in Program.cs to get your options and store it in a static class and use it in startup.
public class Program
{
public static async Task Main(string[] args)
{
JwtParameter.TokenValidationParameters = await wso2Actions.JwtOperations.GetTokenValidationParameters();
CreateHostBuilder(args).Build().Run();
}
public static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.ConfigureWebHostDefaults(webBuilder =>
{
webBuilder.UseStartup<Startup>();
});
}
public static class JwtParameter
{
public static TokenValidationParameters TokenValidationParameters { get; set; }
}