We Keep Coding

grpc and polly - .net core 6

I'm trying to use Polly as retry policy handler for grpc in my .net core 6 project. I noticed that the retryFunc is never invoked. I started from this project gRPC & ASP.NET Core 3.1: Resiliency with Polly
class Program
{
static async Task Main(string[] args)
{
// DI
var services = new ServiceCollection();
var loggerFactory = LoggerFactory.Create(logging =>
{
logging.AddConsole();
logging.SetMinimumLevel(LogLevel.Debug);
});
var serverErrors = new HttpStatusCode[] {
HttpStatusCode.BadGateway,
HttpStatusCode.GatewayTimeout,
HttpStatusCode.ServiceUnavailable,
HttpStatusCode.InternalServerError,
HttpStatusCode.TooManyRequests,
HttpStatusCode.RequestTimeout
};
var gRpcErrors = new StatusCode[] {
StatusCode.DeadlineExceeded,
StatusCode.Internal,
StatusCode.NotFound,
StatusCode.ResourceExhausted,
StatusCode.Unavailable,
StatusCode.Unknown
};
Func<HttpRequestMessage, IAsyncPolicy<HttpResponseMessage>> retryFunc = (request) =>
{
return Policy.HandleResult<HttpResponseMessage>(r => {
var grpcStatus = StatusManager.GetStatusCode(r);
var httpStatusCode = r.StatusCode;
return (grpcStatus == null && serverErrors.Contains(httpStatusCode)) || // if the server send an error before gRPC pipeline
(httpStatusCode == HttpStatusCode.OK && gRpcErrors.Contains(grpcStatus.Value)); // if gRPC pipeline handled the request (gRPC always answers OK)
})
.WaitAndRetryAsync(3, (input) => TimeSpan.FromSeconds(3 + input), (result, timeSpan, retryCount, context) =>
{
var grpcStatus = StatusManager.GetStatusCode(result.Result);
Console.WriteLine($"Request failed with {grpcStatus}. Retry");
});
};
services.AddGrpcClient<CountryServiceClient>(o =>
{
o.Address = new Uri("https://localhost:5001");
}).AddPolicyHandler(retryFunc);
var provider = services.BuildServiceProvider();
var client = provider.GetRequiredService<CountryServiceClient>();
try
{
var countries = (await client.GetAllAsync(new EmptyRequest())).Countries.Select(x => new Country
{
CountryId = x.Id,
Description = x.Description,
CountryName = x.Name
}).ToList();
Console.WriteLine("Found countries");
countries.ForEach(x => Console.WriteLine($"Found country {x.CountryName} ({x.CountryId}) {x.Description}"));
}
catch (RpcException e)
{
Console.WriteLine(e.Message);
}
}
}
but at the end WaitAndRetryAsync is never called.
I created a small project available on github in order to reproduce it.
My test is fairly simple. I start the client without a listening back-end, expecting to read 3 times the output from Console.WriteLine($"Request failed with {grpcStatus}. Retry"); on the console. But the policy handler in never fired. I have the following exception instead
Status(StatusCode="Unavailable", Detail="Error connecting to
subchannel.", DebugException="System.Net.Sockets.SocketException
(10061): No connection could be made because the target machine
actively refused it.
without any retry.

This is not working for you because Retry is now built into Grpc. In order to make this work, register your service as follows:
var defaultMethodConfig = new MethodConfig
{
Names = { MethodName.Default },
RetryPolicy = new RetryPolicy
{
MaxAttempts = 3,
InitialBackoff = TimeSpan.FromSeconds(3),
MaxBackoff = TimeSpan.FromSeconds(3),
BackoffMultiplier = 1,
RetryableStatusCodes =
{
// Whatever status codes you want to look for
StatusCode.Unauthenticated, StatusCode.NotFound, StatusCode.Unavailable,
}
}
};
var services = new ServiceCollection();
services.AddGrpcClient<TestServiceClient>(o => {
o.Address = new Uri("https://localhost:5001");
o.ChannelOptionsActions.Add(options =>
{
options.ServiceConfig = new ServiceConfig {MethodConfigs = {defaultMethodConfig}};
});
});
That will add the retry policy to your client. One other thing that you might run into. I didn't realize this at the time, but in my service implementation, I was setting up errors something like this:
var response = new MyServiceResponse()
// something bad happens
context.Status = new Status(StatusCode.Internal, "Something went wrong");
return response;
The retry logic will not kick in if you implement your service like that, you actually have to do something more like this:
// something bad happens
throw new RpcException(new Status(StatusCode.Internal, "Something went wrong"));
The retry logic you configured when registering your client will then work. Hope that helps.

With the help of #PeterCsala I tried some fix.
As a first attempt I tried without DependencyInjection, registering the policy as follows
var policy = Policy
.Handle<Exception>()
.RetryAsync(3, (exception, count) =>
{
Console.WriteLine($"Request {count}, {exception.Message}. Retry");
});
var channel = GrpcChannel.ForAddress("https://localhost:5001");
TestServiceClient client = new TestServiceClient(channel);
await policy.ExecuteAsync(async () => await client.TestAsync(new Empty()));
This way it's working.
Then I came back to DI and used to register the policy as follows
IAsyncPolicy<HttpResponseMessage> policy =
Policy<HttpResponseMessage>.Handle<Exception>().RetryAsync(3, (exception, count) =>
{
Console.WriteLine($"Request {count}, {exception.Exception.Message}. Retry");
});
var services = new ServiceCollection();
services.AddGrpcClient<TestServiceClient>(o => {
o.Address = new Uri("https://localhost:5001");
}).AddPolicyHandler(policy);
var provider = services.BuildServiceProvider();
var client = provider.GetRequiredService<TestServiceClient>();
var testClient = (await client.TestAsync(new Empty()));
And still not working.
At the end it seems AddPolicyHandler is not suitable for grpc clients?

Polly does not timeout

I am trying to get Polly to try again on timeout after 3 seconds and also when certain http codes are returned. However, it doesn't time out until after 100 seconds when the HttpClient times out.
Here is my code:
private static Polly.Wrap.AsyncPolicyWrap<HttpResponseMessage> GetPolicy()
{
var timeoutPolicy = Policy.TimeoutAsync(3, Polly.Timeout.TimeoutStrategy.Optimistic);
var retryPolicy = Policy
.Handle<HttpRequestException>()
.OrResult<HttpResponseMessage>(r =>
r.StatusCode == HttpStatusCode.TooManyRequests ||
r.StatusCode == HttpStatusCode.ServiceUnavailable ||
r.StatusCode == HttpStatusCode.Forbidden)
.WaitAndRetryAsync(3, i => TimeSpan.FromSeconds(3));
var policy = retryPolicy.WrapAsync(timeoutPolicy);
return policy;
}
Update
As requested, here is code where I am using the policy.
var pollyResponse = await GetPolicy().ExecuteAndCaptureAsync(() =>
httpClient.SendAsync(GetMessage(HttpMethod.Delete, endpoint))
);
And the helper method that makes the HttpRequestMessage:
private HttpRequestMessage GetMessage<T>(HttpMethod method, string endpoint, T content)
{
var message = new HttpRequestMessage
{
Method = method,
RequestUri = new Uri(endpoint),
Headers = {
{ "MyCustomHeader", _value },
{ HttpRequestHeader.Accept.ToString(), "application/json" }
}
};
if (content != null)
{
var contentAsString = JsonSerializer.Serialize(content);
message.Content = new StringContent(contentAsString);
}
return message;
}

First, let me share with you the revised version of your GetPolicy:
private static IAsyncPolicy<HttpResponseMessage> GetStrategy()
{
var timeoutPolicy = Policy
.TimeoutAsync<HttpResponseMessage>(3, TimeoutStrategy.Optimistic,
onTimeoutAsync: (_, __, ___, ____) =>
{
Console.WriteLine("Timeout has occurred");
return Task.CompletedTask;
});
var retryPolicy = Policy
.Handle<HttpRequestException>()
.Or<TimeoutRejectedException>()
.OrResult<HttpResponseMessage>(r =>
r.StatusCode == (HttpStatusCode)429 ||
r.StatusCode == HttpStatusCode.ServiceUnavailable ||
r.StatusCode == HttpStatusCode.Forbidden)
.WaitAndRetryAsync(3, i => TimeSpan.FromSeconds(3),
onRetryAsync: (_, __, ___) =>
{
Console.WriteLine("Retry will fire soon");
return Task.CompletedTask;
});
return Policy.WrapAsync(retryPolicy, timeoutPolicy);
}
I've changed the return type because from the consumer perspective the PolicyWrap is just an implementation detail
You could also use the AsyncPolicy<T> abstract class as return type if you don't want to use an interface (IAsyncPolicy<T>)
I've added some debug logging (onTimeoutAsync, onRetryAsync) to be able to watch which policy triggers when
I've added an Or<TimeoutRejectedException>() builder function call on the retryPolicy to make sure that retry will be triggered in case of timeout
Please note that the Timeout policy will throw its own exception
I've also changed your retryPolicy.WrapAsync to a PolicyWrap because with that the escalation chain is more explicit
The left most policy is the most outer
The right most policy is the most inner
I've also changed the timeoutPolicy (.TimeoutAsync < HttpResponseMessage > ) to align with the retry policy (both of them are wrapping a delegate which might return a Task<HttpResponseMessage>)
In order to be able to test our resilience strategy (note the naming) I've created the following helper method:
private static HttpClient client = new HttpClient();
public static async Task<HttpResponseMessage> CallOverloadedAPI(int responseDelay = 5000, int responseCode = 200)
{
return await client.GetAsync($"http://httpstat.us/{responseCode}?sleep={responseDelay}");
}
It will issue a request against a website which will return with a specified status code after a predefined amount of time
If you haven't used this website before please visit: 1, 2
Now, let's call the website:
public static async Task Main()
{
HttpResponseMessage response;
try
{
response = await GetStrategy().ExecuteAsync(async () => await CallOverloadedAPI());
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Environment.Exit(-1);
}
Console.WriteLine("Finished");
}
The output:
Finished
Wait, what???
The thing is none of the policies have been triggered.
Why?
Because after 5 seconds we have received a response with 200.
But, we have set up a timeout, right?
Yes and no. :) Even though we have defined a timeout policy we haven't really connected that to the HttpClient
So, how can I connect?
Well, via CancellationToken
So, in case of timeout policy if a CancellationToken is in use then it can call its Cancel method to indicate the timeout fact to the HttpClient. And HttpClient will cancel the pending request.
Please note that, because we are using TimeoutPolicy the exception will be TimeoutRejectedException, not an OperationCanceledException.
So, let's modify our code to accept a CancellationToken
public static async Task<HttpResponseMessage> CallOverloadedAPI(int responseDelay = 5000, int responseCode = 200, CancellationToken token = default)
{
return await client.GetAsync($"http://httpstat.us/{responseCode}?sleep={responseDelay}", token);
}
We have to adjust the usage side as well:
public static async Task Main()
{
HttpResponseMessage response;
try
{
response = await GetStrategy().ExecuteAsync(async (ct) => await CallOverloadedAPI(token: ct), CancellationToken.None);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Environment.Exit(-1);
}
Console.WriteLine("Finished");
}
Now the output now will look like this:
Timeout has occurred
Retry will fire soon
Timeout has occurred
Retry will fire soon
Timeout has occurred
Retry will fire soon
Timeout has occurred
The delegate executed asynchronously through TimeoutPolicy did not complete within the timeout.
The last line is the Message of the TimeoutRejectedException.
Please note that if we remove the Or<TimeoutRejectedException>() call from the retryPolicy builder then the output will be the following:
Timeout has occurred
The delegate executed asynchronously through TimeoutPolicy did not complete within the timeout.
So, now retry will be triggered. There will be no escalation.
For the sake of completeness, here is the whole source code:
public static async Task Main()
{
HttpResponseMessage response;
try
{
response = await GetStrategy().ExecuteAsync(async (ct) => await CallOverloadedAPI(token: ct), CancellationToken.None);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Environment.Exit(-1);
}
Console.WriteLine("Finished");
}
private static AsyncPolicy<HttpResponseMessage> GetStrategy()
{
var timeoutPolicy = Policy
.TimeoutAsync<HttpResponseMessage>(3, TimeoutStrategy.Optimistic,
onTimeoutAsync: (_, __, ___, ____) =>
{
Console.WriteLine("Timeout has occurred");
return Task.CompletedTask;
});
var retryPolicy = Policy
.Handle<HttpRequestException>()
.Or<TimeoutRejectedException>()
.OrResult<HttpResponseMessage>(r =>
r.StatusCode == (HttpStatusCode)429 ||
r.StatusCode == HttpStatusCode.ServiceUnavailable ||
r.StatusCode == HttpStatusCode.Forbidden)
.WaitAndRetryAsync(3, i => TimeSpan.FromSeconds(3),
onRetryAsync: (_, __, ___) =>
{
Console.WriteLine("Retry will fire soon");
return Task.CompletedTask;
});
return Policy.WrapAsync(retryPolicy, timeoutPolicy);
}
private static HttpClient client = new HttpClient();
public static async Task<HttpResponseMessage> CallOverloadedAPI(int responseDelay = 5000, int responseCode = 200, CancellationToken token = default)
{
return await client.GetAsync($"http://httpstat.us/{responseCode}?sleep={responseDelay}", token);
}

Nsubstitute or moq an Async method so it will return a faulted task instead of exception

In the below code for each request we are calling rest client in an Async manner. Rest client is basically a http client which is calling some webapis in a batch. If suppose AsyncTaskCount value is 5 then 5 request will be called asynchronously and then in the while block we are getting the result for each call. If any response out of those 5 request has an exception then the response will be faulted and IsFaulted becomes true for that particular request and in the response we can get the inner exception.
private async Task<List<RequestResponse>> ProcessInvestment(List<Request> Requests, List<Result> Results, ILogger log)
{
var requestResponses = new List<RequestResponse>();
var asyncTaskCount = Convert.ToInt32(Environment.GetEnvironmentVariable("AsyncTaskCount"));
log.LogInformation($"Start processing {Requests.Count} in batches of {asyncTaskCount}");
for (int index = 0; index < Requests.Count; index = index + asyncTaskCount)
{
var requestBatch = Requests.Skip(index).Take(asyncTaskCount).ToList();
var requests = requestBatch.Select(x => _restClient.RequestResponse(x)).ToList();
while (requests.Count > 0)
{
// Identify the first task that completes.
Task<RequestResponse> requestResponseTask = await Task.WhenAny(requests);
var requestResponse = new RequestResponse();
// ***Remove the selected task from the list so that you don't process it more than once
requests.Remove(requestResponseTask);
if (!requestResponseTask.IsFaulted)
{
// Await the completed task.
requestResponse = await requestResponseTask;
requestResponses.Add(requestResponse);
}
else
{
if (requestResponseTask.Exception.InnerException != null && requestResponseTask.Exception.InnerException is Exception)
{
var result = new Result();
result = ResponseTransformComponent.ResponseToResult(((Exception)requestResponseTask.Exception.InnerException).Request, null);
result.SetBadRequestErrorDetails(((Exception)RequestResponseTask.Exception.InnerException).BadRequestResponse);
results.Add(Result);
}
else
{
throw requestResponseTask.Exception;
}
}
}
log.LogInformation($"Number of records processed = {requestResponses.Count}");
}
log.LogInformation($"Total invalid and Bad requests count = {results.Count}");
return RequestResponses;
}
Below is the code for restclient which is called from the above method.
public async Task<Response> RequestResponse(Request request)
{
var response = await GetDataFromService("calculation", "CalculateCapital", request);
return JsonConvert.DeserializeObject<Response>(response);
}
public async Task<string> GetDataFromService(string controller, string method, object request)
{
var client = _httpClientFactory.CreateClient(ServiceEnum.DCS);
string baseAddress = client.BaseAddress.ToString();
var requestUrl = $"api/{controller}/{method}";
var content = new StringContent(JsonConvert.SerializeObject(request), Encoding.UTF8, "application/json");
var response = await client.PostAsync(requestUrl, content).ConfigureAwait(false);
if (!response.IsSuccessStatusCode)
{
var responseResult = response.Content.ReadAsStringAsync().Result;
if (response.StatusCode == HttpStatusCode.BadRequest)
{
throw new CalculatorServiceException("Bad Request", JsonConvert.DeserializeObject<BadRequestResponse>(responseResult), (Request)request);
}
throw new Exception($"Status code: {response.StatusCode}. {responseResult}");
}
return response.Content.ReadAsStringAsync().Result;
}
GetDataFromService method is called from Calculate method. And GetDataFromService method will return a custom exception if the request is a Bad Request.
I am trying to write the unit test case for the above method and try to mock a request so that it will return a faulted task and then IsFaulted should become true. Below is the part of my unit test case.
_restClient
.When(a => a.RequestResponse(Arg.Any<Request>()))
.Do(a => {Task.FromException(new CalculatorServiceException(string.Empty, new BadRequestResponse { Message = string.Empty, ModelState = new Dictionary<string, string[]> { { "CalculationDates.StartDate", new string[] { "0002: Duration too short to execute calculations (CalculationDates.StartDate)" } } } }, Arg.Any<Request>())); });
If i mock my restclient method like above then it is throwing the exception instead of giving the response with IsFaulted to true. So how should i mock the restclient method so that it will return a faulted task which has an exception instead of throwing it. Please suggest.
Thanks in advance.

When..Do is for wiring up callbacks for when a member is called.
Try using Returns instead:
_restClient.RequestResponse(Arg.Any<Request>())
.Returns(x => Task.FromException<RequestResponse>(...));
// (assuming RequestResponse(..) returns a Task<RequestResponse>. Tweak as required)

How to fix MassTransit.EndpointNotFoundException when using Request/Response initialized with localhost?

We are using masstransit with Request/Response pattern as described here: http://masstransit-project.com/MassTransit/usage/request-response.html
For some reason when the bus is initialized with localhost I receive a MassTransit.EndpointNotFoundException on the respond side, using context.RespondAsync method.
When the bus is initialized with the actual IP it works good without any exception.
We are using the bus also with other patterns of masstransit initialized with localhost without any problems.
We tried using both, Request and GetResponse patterns on the request side thinking maybe the problem source is there, but with no success.
Following is the consumer code the Response side, which generate the exception on the RespondAsync:
public override async Task Consume(ConsumeContext<IImageRequest> context)
{
var msg = context.Message;
try
{
var image = GetImage(msg.Id);
await context.RespondAsync<IImageResult>(new
{
msg.Id,
AImage = image
});
}
catch (Exception e)
{
Logger.Instance.Error($"{e}");
}
}
Following is the request side:
Task.Run(async () =>
{
var reqResClient =
BusControl.GetBusControl.CreateRequestClient<IImageRequest, IImageResult>("requestimage");
var req = new ImageRequest(msg.Id);
var response = await reqResClient.Request(req);
}).Wait();
Bus initialization code:
_bus = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
cfg.Durable = true;
_host = cfg.Host(HostName, port, VirtualHost, h =>
{
h.Username(UserName);
h.Password(Password);
});
cfg.ReceiveEndpoint(host, subscriber.QueueName, ep =>
{
ep.Durable = !subscriber.AutoDelete;
ep.AutoDelete = subscriber.AutoDelete;
ep.Instance(subscriber.Consumer);
if (subscriber.PrefetchCount.HasValue)
ep.PrefetchCount = subscriber.PrefetchCount.Value;
if (subscriber.ConcurrencyLimit.HasValue)
ep.UseConcurrencyLimit(subscriber.ConcurrencyLimit.Value);
});
cfg.Message<T>(x =>
{
x.SetEntityName(entityName);
});
});
public void Start()
{
if (_state == BusState.NotInitialized)
throw new InvalidOperationException("Cannot start bus. Bus is not initialized");
_bus?.Start();
_state = BusState.Started;
}
I wanted to know if there's a solution for it? or is it a known problem?
Thanks,

Task cancellation with condition in case of positive response from server side

I have three tasks like this
public List<CheckResponse> Check(CheckRequest request)
{
var Responses = new List<CheckResponse>();
List<Task> TaskList = new List<Task>();
Task task1 = new Task(() =>
{
Service gatewayObject = new Service();
var response = gatewayObject.Check();
Responses.Add(response);
});
task1.Start();
TaskList.Add(task1);
Task task2 = new Task(() =>
{
Service gatewayObject = new Service();
var response = gatewayObject.Check();
Responses.Add(response);
});
task2.Start();
TaskList.Add(task2);
Task task3 = new Task(() =>
{
Service gatewayObject = new Service();
var response = gatewayObject.Check();
Responses.Add(response);
});
task3.Start();
TaskList.Add(task3);
Task.WaitAll(TaskList.ToArray());
return Responses;
}
Each of them make request to service and can return positive result
so I want to cancel other two task if any of them return positive result.
Service side code is not accessible for me so I can not make any changes on service side.