I am trying to get Polly to try again on timeout after 3 seconds and also when certain http codes are returned. However, it doesn't time out until after 100 seconds when the HttpClient times out.
Here is my code:
private static Polly.Wrap.AsyncPolicyWrap<HttpResponseMessage> GetPolicy()
{
var timeoutPolicy = Policy.TimeoutAsync(3, Polly.Timeout.TimeoutStrategy.Optimistic);
var retryPolicy = Policy
.Handle<HttpRequestException>()
.OrResult<HttpResponseMessage>(r =>
r.StatusCode == HttpStatusCode.TooManyRequests ||
r.StatusCode == HttpStatusCode.ServiceUnavailable ||
r.StatusCode == HttpStatusCode.Forbidden)
.WaitAndRetryAsync(3, i => TimeSpan.FromSeconds(3));
var policy = retryPolicy.WrapAsync(timeoutPolicy);
return policy;
}
Update
As requested, here is code where I am using the policy.
var pollyResponse = await GetPolicy().ExecuteAndCaptureAsync(() =>
httpClient.SendAsync(GetMessage(HttpMethod.Delete, endpoint))
);
And the helper method that makes the HttpRequestMessage:
private HttpRequestMessage GetMessage<T>(HttpMethod method, string endpoint, T content)
{
var message = new HttpRequestMessage
{
Method = method,
RequestUri = new Uri(endpoint),
Headers = {
{ "MyCustomHeader", _value },
{ HttpRequestHeader.Accept.ToString(), "application/json" }
}
};
if (content != null)
{
var contentAsString = JsonSerializer.Serialize(content);
message.Content = new StringContent(contentAsString);
}
return message;
}
First, let me share with you the revised version of your GetPolicy:
private static IAsyncPolicy<HttpResponseMessage> GetStrategy()
{
var timeoutPolicy = Policy
.TimeoutAsync<HttpResponseMessage>(3, TimeoutStrategy.Optimistic,
onTimeoutAsync: (_, __, ___, ____) =>
{
Console.WriteLine("Timeout has occurred");
return Task.CompletedTask;
});
var retryPolicy = Policy
.Handle<HttpRequestException>()
.Or<TimeoutRejectedException>()
.OrResult<HttpResponseMessage>(r =>
r.StatusCode == (HttpStatusCode)429 ||
r.StatusCode == HttpStatusCode.ServiceUnavailable ||
r.StatusCode == HttpStatusCode.Forbidden)
.WaitAndRetryAsync(3, i => TimeSpan.FromSeconds(3),
onRetryAsync: (_, __, ___) =>
{
Console.WriteLine("Retry will fire soon");
return Task.CompletedTask;
});
return Policy.WrapAsync(retryPolicy, timeoutPolicy);
}
I've changed the return type because from the consumer perspective the PolicyWrap is just an implementation detail
You could also use the AsyncPolicy<T> abstract class as return type if you don't want to use an interface (IAsyncPolicy<T>)
I've added some debug logging (onTimeoutAsync, onRetryAsync) to be able to watch which policy triggers when
I've added an Or<TimeoutRejectedException>() builder function call on the retryPolicy to make sure that retry will be triggered in case of timeout
Please note that the Timeout policy will throw its own exception
I've also changed your retryPolicy.WrapAsync to a PolicyWrap because with that the escalation chain is more explicit
The left most policy is the most outer
The right most policy is the most inner
I've also changed the timeoutPolicy (.TimeoutAsync < HttpResponseMessage > ) to align with the retry policy (both of them are wrapping a delegate which might return a Task<HttpResponseMessage>)
In order to be able to test our resilience strategy (note the naming) I've created the following helper method:
private static HttpClient client = new HttpClient();
public static async Task<HttpResponseMessage> CallOverloadedAPI(int responseDelay = 5000, int responseCode = 200)
{
return await client.GetAsync($"http://httpstat.us/{responseCode}?sleep={responseDelay}");
}
It will issue a request against a website which will return with a specified status code after a predefined amount of time
If you haven't used this website before please visit: 1, 2
Now, let's call the website:
public static async Task Main()
{
HttpResponseMessage response;
try
{
response = await GetStrategy().ExecuteAsync(async () => await CallOverloadedAPI());
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Environment.Exit(-1);
}
Console.WriteLine("Finished");
}
The output:
Finished
Wait, what???
The thing is none of the policies have been triggered.
Why?
Because after 5 seconds we have received a response with 200.
But, we have set up a timeout, right?
Yes and no. :) Even though we have defined a timeout policy we haven't really connected that to the HttpClient
So, how can I connect?
Well, via CancellationToken
So, in case of timeout policy if a CancellationToken is in use then it can call its Cancel method to indicate the timeout fact to the HttpClient. And HttpClient will cancel the pending request.
Please note that, because we are using TimeoutPolicy the exception will be TimeoutRejectedException, not an OperationCanceledException.
So, let's modify our code to accept a CancellationToken
public static async Task<HttpResponseMessage> CallOverloadedAPI(int responseDelay = 5000, int responseCode = 200, CancellationToken token = default)
{
return await client.GetAsync($"http://httpstat.us/{responseCode}?sleep={responseDelay}", token);
}
We have to adjust the usage side as well:
public static async Task Main()
{
HttpResponseMessage response;
try
{
response = await GetStrategy().ExecuteAsync(async (ct) => await CallOverloadedAPI(token: ct), CancellationToken.None);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Environment.Exit(-1);
}
Console.WriteLine("Finished");
}
Now the output now will look like this:
Timeout has occurred
Retry will fire soon
Timeout has occurred
Retry will fire soon
Timeout has occurred
Retry will fire soon
Timeout has occurred
The delegate executed asynchronously through TimeoutPolicy did not complete within the timeout.
The last line is the Message of the TimeoutRejectedException.
Please note that if we remove the Or<TimeoutRejectedException>() call from the retryPolicy builder then the output will be the following:
Timeout has occurred
The delegate executed asynchronously through TimeoutPolicy did not complete within the timeout.
So, now retry will be triggered. There will be no escalation.
For the sake of completeness, here is the whole source code:
public static async Task Main()
{
HttpResponseMessage response;
try
{
response = await GetStrategy().ExecuteAsync(async (ct) => await CallOverloadedAPI(token: ct), CancellationToken.None);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Environment.Exit(-1);
}
Console.WriteLine("Finished");
}
private static AsyncPolicy<HttpResponseMessage> GetStrategy()
{
var timeoutPolicy = Policy
.TimeoutAsync<HttpResponseMessage>(3, TimeoutStrategy.Optimistic,
onTimeoutAsync: (_, __, ___, ____) =>
{
Console.WriteLine("Timeout has occurred");
return Task.CompletedTask;
});
var retryPolicy = Policy
.Handle<HttpRequestException>()
.Or<TimeoutRejectedException>()
.OrResult<HttpResponseMessage>(r =>
r.StatusCode == (HttpStatusCode)429 ||
r.StatusCode == HttpStatusCode.ServiceUnavailable ||
r.StatusCode == HttpStatusCode.Forbidden)
.WaitAndRetryAsync(3, i => TimeSpan.FromSeconds(3),
onRetryAsync: (_, __, ___) =>
{
Console.WriteLine("Retry will fire soon");
return Task.CompletedTask;
});
return Policy.WrapAsync(retryPolicy, timeoutPolicy);
}
private static HttpClient client = new HttpClient();
public static async Task<HttpResponseMessage> CallOverloadedAPI(int responseDelay = 5000, int responseCode = 200, CancellationToken token = default)
{
return await client.GetAsync($"http://httpstat.us/{responseCode}?sleep={responseDelay}", token);
}
Related
In the below code for each request we are calling rest client in an Async manner. Rest client is basically a http client which is calling some webapis in a batch. If suppose AsyncTaskCount value is 5 then 5 request will be called asynchronously and then in the while block we are getting the result for each call. If any response out of those 5 request has an exception then the response will be faulted and IsFaulted becomes true for that particular request and in the response we can get the inner exception.
private async Task<List<RequestResponse>> ProcessInvestment(List<Request> Requests, List<Result> Results, ILogger log)
{
var requestResponses = new List<RequestResponse>();
var asyncTaskCount = Convert.ToInt32(Environment.GetEnvironmentVariable("AsyncTaskCount"));
log.LogInformation($"Start processing {Requests.Count} in batches of {asyncTaskCount}");
for (int index = 0; index < Requests.Count; index = index + asyncTaskCount)
{
var requestBatch = Requests.Skip(index).Take(asyncTaskCount).ToList();
var requests = requestBatch.Select(x => _restClient.RequestResponse(x)).ToList();
while (requests.Count > 0)
{
// Identify the first task that completes.
Task<RequestResponse> requestResponseTask = await Task.WhenAny(requests);
var requestResponse = new RequestResponse();
// ***Remove the selected task from the list so that you don't process it more than once
requests.Remove(requestResponseTask);
if (!requestResponseTask.IsFaulted)
{
// Await the completed task.
requestResponse = await requestResponseTask;
requestResponses.Add(requestResponse);
}
else
{
if (requestResponseTask.Exception.InnerException != null && requestResponseTask.Exception.InnerException is Exception)
{
var result = new Result();
result = ResponseTransformComponent.ResponseToResult(((Exception)requestResponseTask.Exception.InnerException).Request, null);
result.SetBadRequestErrorDetails(((Exception)RequestResponseTask.Exception.InnerException).BadRequestResponse);
results.Add(Result);
}
else
{
throw requestResponseTask.Exception;
}
}
}
log.LogInformation($"Number of records processed = {requestResponses.Count}");
}
log.LogInformation($"Total invalid and Bad requests count = {results.Count}");
return RequestResponses;
}
Below is the code for restclient which is called from the above method.
public async Task<Response> RequestResponse(Request request)
{
var response = await GetDataFromService("calculation", "CalculateCapital", request);
return JsonConvert.DeserializeObject<Response>(response);
}
public async Task<string> GetDataFromService(string controller, string method, object request)
{
var client = _httpClientFactory.CreateClient(ServiceEnum.DCS);
string baseAddress = client.BaseAddress.ToString();
var requestUrl = $"api/{controller}/{method}";
var content = new StringContent(JsonConvert.SerializeObject(request), Encoding.UTF8, "application/json");
var response = await client.PostAsync(requestUrl, content).ConfigureAwait(false);
if (!response.IsSuccessStatusCode)
{
var responseResult = response.Content.ReadAsStringAsync().Result;
if (response.StatusCode == HttpStatusCode.BadRequest)
{
throw new CalculatorServiceException("Bad Request", JsonConvert.DeserializeObject<BadRequestResponse>(responseResult), (Request)request);
}
throw new Exception($"Status code: {response.StatusCode}. {responseResult}");
}
return response.Content.ReadAsStringAsync().Result;
}
GetDataFromService method is called from Calculate method. And GetDataFromService method will return a custom exception if the request is a Bad Request.
I am trying to write the unit test case for the above method and try to mock a request so that it will return a faulted task and then IsFaulted should become true. Below is the part of my unit test case.
_restClient
.When(a => a.RequestResponse(Arg.Any<Request>()))
.Do(a => {Task.FromException(new CalculatorServiceException(string.Empty, new BadRequestResponse { Message = string.Empty, ModelState = new Dictionary<string, string[]> { { "CalculationDates.StartDate", new string[] { "0002: Duration too short to execute calculations (CalculationDates.StartDate)" } } } }, Arg.Any<Request>())); });
If i mock my restclient method like above then it is throwing the exception instead of giving the response with IsFaulted to true. So how should i mock the restclient method so that it will return a faulted task which has an exception instead of throwing it. Please suggest.
Thanks in advance.
When..Do is for wiring up callbacks for when a member is called.
Try using Returns instead:
_restClient.RequestResponse(Arg.Any<Request>())
.Returns(x => Task.FromException<RequestResponse>(...));
// (assuming RequestResponse(..) returns a Task<RequestResponse>. Tweak as required)
Currently I'm using the following utility extension to read from a PipeReader with a specified timeout. The timeout is required to properly implement Keep-Alive within a HTTP server.
internal static async Task<ReadResult?> ReadWithTimeoutAsync(this PipeReader reader, TimeSpan timeout)
{
ReadResult? result = null;
var readTask = Task.Run(async () =>
{
result = await reader.ReadAsync();
});
var success = await Task.WhenAny(readTask, Task.Delay(timeout)) == readTask;
if (!success || (result == null))
{
return null;
}
return result;
}
This code is problematic in a couple of ways, as it introduces locking (within Task.Delay), a lot of allocations and a thread to be handled by the CPU.
Is there a more efficient way to use a PipeReader with read timeouts?
We can use a CancellationToken to implement the timeout in a more efficient manner:
using var cancellation = new CancellationTokenSource(timout);
try
{
Data = (await Reader.ReadAsync(cancellation.Token)).Buffer;
}
catch (OperationCanceledException)
{
return null;
}
So I have to use a library that essentially does a POST to a remote system that may choose to throttle the traffic. If it does, it returns 429 and a specific # of seconds to back off in the Retry-After header... at which point the framework reads and parses this value, and essentially does this
throw new ThrottledException(retryAfterSeconds);
How do I set up a Polly policy that will catch this custom exception, and then retry after exception.RetryAfter seconds?
OK, this was a bit more tricky than it needed to be, but only because I was sent on several wild goose chases by inscrutable compiler messages.
In this scenario the retry is communicated by a custom exception of type SigsThrottledException, which has a field that contains the requested backoff time in seconds.
var policy = Policy
.Handle<SigsThrottledException>(e => e.RetryAfterInSeconds > 0)
.WaitAndRetryAsync(
retryCount: retries,
sleepDurationProvider: (i, e, ctx) =>
{
var ste = (SigsThrottledException)e;
return TimeSpan.FromSeconds((double)ste.RetryAfterInSeconds);
},
onRetryAsync: async (e, ts, i, ctx) =>
{
// Do something here
};);
This is an example of how to use the policy. You can't just add it to an existing HttpClient or HttpClientFactory. You have to use it explicitly.
[TestMethod]
public async Task SigsPollyRetriesOnThrottle()
{
var retryResponse = new HttpResponseMessage
{
StatusCode = (HttpStatusCode)429,
Content = new StringContent("{}"),
};
retryResponse.Headers.Add("Retry-After", "1");
var mockMessageHandler = new Mock<HttpMessageHandler>();
mockMessageHandler.Protected()
.SetupSequence<Task<HttpResponseMessage>>("SendAsync", ItExpr.IsAny<HttpRequestMessage>(), ItExpr.IsAny<CancellationToken>())
.ReturnsAsync(retryResponse)
.ReturnsAsync(new HttpResponseMessage
{
StatusCode = HttpStatusCode.OK
});
var client = new HttpClient(mockMessageHandler.Object);
// Retry once after waiting 1 second
var retryPolicy = Policy
.Handle<SigsThrottledException>(e => e.RetryAfterInSeconds > 0)
.WaitAndRetryAsync(
retryCount: 1,
sleepDurationProvider: (i, e, ctx) =>
{
var ste = (SigsThrottledException)e;
return TimeSpan.FromSeconds((double)ste.RetryAfterInSeconds);
},
onRetryAsync: async (e, ts, i, ctx) =>
{
// Do something here
};);
Stopwatch stopWatch = new Stopwatch();
stopWatch.Start();
var response = await retryPolicy.ExecuteAsync(async () =>
{
Uri substrateurl = new Uri("https://substrate.office.com/");
return await SIGSClient.Instance.PostAsync(client, substrateurl, new UserInfo(), "faketoken", new Signal(), Guid.NewGuid()).ConfigureAwait(false);
}
);
Assert.AreEqual(response.StatusCode, HttpStatusCode.OK);
stopWatch.Stop();
Assert.IsTrue(stopWatch.ElapsedMilliseconds > 1000); // Make sure we actually waited at least a second
}
Let's say I have a data service class that fetches the data batch by batch and its chunks to the subscribers.
public class DataService {
public IObservable<IList<T>> QuerySegmentedObservable<T>(string tableName) where T : TableEntity, new(){
return Observable.Create<IList<T>>(async (observer, token) =>{
TableContinuationToken continuationToken = null;
do{
var currentSegment = CallData();
observer.OnNext(currentSegment.Results);
continuationToken = currentSegment.ContinuationToken;
} while (continuationToken != null);
observer.OnCompleted();
}
}
}
I am subscribing this observable as below.
public async Task<bool> MyMethod()
{
var tcs = new TaskCompletionSource<bool>();
var observable = _dataService.QuerySegmentedObservable<TSource>(_sourceTableName);
var dataCount = 0;
_databaseService.OpenConnection();
observable.Subscribe(async data =>
{
await _databaseService.DoSomething(data);
dataCount += data.Count;
Console.WriteLine($"Processing - {dataCount}");
},
err =>
{
Console.WriteLine($"Error - {err.Message}");
tcs.SetResult(false);
},
() =>
{
_databaseService.CloseConnection();
Console.WriteLine($"Finished");
tcs.SetResult(true);
}
);
return await tcs.Task;
}
The problem is that OnComplete() is called before the last OnNext() is finished. So, I ended up closing the connection before finishing the task that I am doing in Subscribe();
Is there any way to fix it? Thanks.
Rx does support async/await within operators. You're using it though within a subscription. So (hopefully) you can change your code to something like this:
public async Task<bool> MyMethod()
{
var tcs = new TaskCompletionSource<bool>();
_databaseService.OpenConnection();
var dataCount = 0;
_dataService.QuerySegmentedObservable<TSource>(_sourceTableName)
.SelectMany(async data =>
{
await _databaseService.DoSomething(data);
return data;
})
//.Finally(() => _databaseService.CloseConnection()) //This would be called on OnComplete and OnError, just like try-finally
.Subscribe(data =>
{
dataCount += data.Count;
Console.WriteLine($"Processing - {dataCount}");
},
err =>
{
Console.WriteLine($"Error - {err.Message}");
tcs.SetResult(false);
},
() =>
{
_databaseService.CloseConnection(); //Maybe move this to a Finally call?
Console.WriteLine($"Finished");
tcs.SetResult(true);
}
);
return await tcs.Task;
}
I can't really test it, so I hope that sets you on the right path. If you need more help, then please post a better MCVE.
This is the error I get:
One or more errors occurred.
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at System.Threading.Tasks.Task`1.get_Result()
at myProject.mymethod(Int32[] myIds) in path\MyClass.cs:line 758
And here's that method:
private void mymethod(int[] myIds)
{
var uri = new Uri(string.Format(UriPath, string.Format(MyPath)));
var client = GetHttpClient(uri);
var postModel = new PostModel
{
Ids = myIds,
LastUpdate = NewItem ? null : _lastUpdated
};
if (client != null)
{
var response = client.PostAsJsonAsync(uri, postModel).Result;//this line crashes
if (response.IsSuccessStatusCode)
{
//doSomething
}
}
}
I call a lot of methods like this and all of them work except this one. When it's hit, it takes a lot of time and then this exception is throws. With all of the other methods the error doesn't happen.
This is the inner exception:
Inner.System.Threading.Tasks.TaskCanceledException: A task was canceled.
Here's my GetCLient() method:
private HttpClient GetHttpClient(Uri uri)
{
var handler = new HttpClientHandler
{
CookieContainer = CoockieContainer
};
return new HttpClient(handler)
{
BaseAddress = uri
};
}
Here's the API method:
[HttpPost]
public IList<MyModel> MyAPIMethod(PostModel model)
{
List<MyModel> myTranslations;
using (var db = new myEntities(GetDbConnStrByUser(new GetCookies().GetUserName())))
{
myTranslations = db.tblTranslations.Where(it => model.Ids.Contains(it.id)
&& (!model.Update.HasValue || it.update > model.LastUpdate.Value))
.Select(it => new MyModel
{
Id = it.id,
Name = it.name,
Description = it.desc,
LanguageId = it.language_id
}).ToList();
}
return myTranslations.GroupBy(x => new { x.Id, x.LanguageId }).Select(x => x.First()).ToList();
}
Maybe a timeout occurs.
Fiddler returns this error: The wait operation timed out.
.Result tries to turn Task into T. It synchronously blocks waiting for the task to complete and will only return T if no exceptions occur. The two types of exceptions you should anticipate are OperationCanceledExceptions/TaskCanceledExceptions (when the HTTP request times out or a CancellationToken was used and the token was cancelled--I don't think the latter is applicable here) and general/HTTP-related exceptions.
You're probably running into the HTTP timeout scenario. Is it taking a long time for the exception to get thrown? If not, what does your GetHttpClient method look like? Is it explicitly setting timeouts or cancellation tokens?
If you want to stick with the synchronous approach and not returns Tasks, you should do something like this instead:
try
{
var response = client.PostAsJsonAsync(uri, postModel).Result;//this line crashes
}
catch (OperationCanceledException oce)
{
// Tell the user that the request timed our or you cancelled a CancellationToken
}
catch (Exception exc)
{
// Look at the HttpClient docs and try to catch something more specific. You don't want
// to swallow StackOverflowExceptions or OutOfMemoryExceptions.
}
If you're willing to take the async plunge, this is more what you're looking for:
private async Task mymethodAsync(int[] myIds)
{
var uri = new Uri(string.Format(UriPath, string.Format(MyPath)));
var client = GetHttpClient(uri);
var postModel = new PostModel { ... };
if (client != null)
{
try
{
var response = await client.PostAsJsonAsync(uri, postModel);
if (response.IsSuccessStatusCode)
{
//doSomething
}
}
catch (OperationCanceledException oce)
{
// because timeouts are still possible
}
catch (Exception exc) {
// Because other things can still go wrong too (HTTP 500, parsing errors)
}
}
}
See here for more info on how .Result call works w/HttpClient: What happens while waiting on a Task's Result?