I'm trying to use Polly as retry policy handler for grpc in my .net core 6 project. I noticed that the retryFunc is never invoked. I started from this project gRPC & ASP.NET Core 3.1: Resiliency with Polly
class Program
{
static async Task Main(string[] args)
{
// DI
var services = new ServiceCollection();
var loggerFactory = LoggerFactory.Create(logging =>
{
logging.AddConsole();
logging.SetMinimumLevel(LogLevel.Debug);
});
var serverErrors = new HttpStatusCode[] {
HttpStatusCode.BadGateway,
HttpStatusCode.GatewayTimeout,
HttpStatusCode.ServiceUnavailable,
HttpStatusCode.InternalServerError,
HttpStatusCode.TooManyRequests,
HttpStatusCode.RequestTimeout
};
var gRpcErrors = new StatusCode[] {
StatusCode.DeadlineExceeded,
StatusCode.Internal,
StatusCode.NotFound,
StatusCode.ResourceExhausted,
StatusCode.Unavailable,
StatusCode.Unknown
};
Func<HttpRequestMessage, IAsyncPolicy<HttpResponseMessage>> retryFunc = (request) =>
{
return Policy.HandleResult<HttpResponseMessage>(r => {
var grpcStatus = StatusManager.GetStatusCode(r);
var httpStatusCode = r.StatusCode;
return (grpcStatus == null && serverErrors.Contains(httpStatusCode)) || // if the server send an error before gRPC pipeline
(httpStatusCode == HttpStatusCode.OK && gRpcErrors.Contains(grpcStatus.Value)); // if gRPC pipeline handled the request (gRPC always answers OK)
})
.WaitAndRetryAsync(3, (input) => TimeSpan.FromSeconds(3 + input), (result, timeSpan, retryCount, context) =>
{
var grpcStatus = StatusManager.GetStatusCode(result.Result);
Console.WriteLine($"Request failed with {grpcStatus}. Retry");
});
};
services.AddGrpcClient<CountryServiceClient>(o =>
{
o.Address = new Uri("https://localhost:5001");
}).AddPolicyHandler(retryFunc);
var provider = services.BuildServiceProvider();
var client = provider.GetRequiredService<CountryServiceClient>();
try
{
var countries = (await client.GetAllAsync(new EmptyRequest())).Countries.Select(x => new Country
{
CountryId = x.Id,
Description = x.Description,
CountryName = x.Name
}).ToList();
Console.WriteLine("Found countries");
countries.ForEach(x => Console.WriteLine($"Found country {x.CountryName} ({x.CountryId}) {x.Description}"));
}
catch (RpcException e)
{
Console.WriteLine(e.Message);
}
}
}
but at the end WaitAndRetryAsync is never called.
I created a small project available on github in order to reproduce it.
My test is fairly simple. I start the client without a listening back-end, expecting to read 3 times the output from Console.WriteLine($"Request failed with {grpcStatus}. Retry"); on the console. But the policy handler in never fired. I have the following exception instead
Status(StatusCode="Unavailable", Detail="Error connecting to
subchannel.", DebugException="System.Net.Sockets.SocketException
(10061): No connection could be made because the target machine
actively refused it.
without any retry.
This is not working for you because Retry is now built into Grpc. In order to make this work, register your service as follows:
var defaultMethodConfig = new MethodConfig
{
Names = { MethodName.Default },
RetryPolicy = new RetryPolicy
{
MaxAttempts = 3,
InitialBackoff = TimeSpan.FromSeconds(3),
MaxBackoff = TimeSpan.FromSeconds(3),
BackoffMultiplier = 1,
RetryableStatusCodes =
{
// Whatever status codes you want to look for
StatusCode.Unauthenticated, StatusCode.NotFound, StatusCode.Unavailable,
}
}
};
var services = new ServiceCollection();
services.AddGrpcClient<TestServiceClient>(o => {
o.Address = new Uri("https://localhost:5001");
o.ChannelOptionsActions.Add(options =>
{
options.ServiceConfig = new ServiceConfig {MethodConfigs = {defaultMethodConfig}};
});
});
That will add the retry policy to your client. One other thing that you might run into. I didn't realize this at the time, but in my service implementation, I was setting up errors something like this:
var response = new MyServiceResponse()
// something bad happens
context.Status = new Status(StatusCode.Internal, "Something went wrong");
return response;
The retry logic will not kick in if you implement your service like that, you actually have to do something more like this:
// something bad happens
throw new RpcException(new Status(StatusCode.Internal, "Something went wrong"));
The retry logic you configured when registering your client will then work. Hope that helps.
With the help of #PeterCsala I tried some fix.
As a first attempt I tried without DependencyInjection, registering the policy as follows
var policy = Policy
.Handle<Exception>()
.RetryAsync(3, (exception, count) =>
{
Console.WriteLine($"Request {count}, {exception.Message}. Retry");
});
var channel = GrpcChannel.ForAddress("https://localhost:5001");
TestServiceClient client = new TestServiceClient(channel);
await policy.ExecuteAsync(async () => await client.TestAsync(new Empty()));
This way it's working.
Then I came back to DI and used to register the policy as follows
IAsyncPolicy<HttpResponseMessage> policy =
Policy<HttpResponseMessage>.Handle<Exception>().RetryAsync(3, (exception, count) =>
{
Console.WriteLine($"Request {count}, {exception.Exception.Message}. Retry");
});
var services = new ServiceCollection();
services.AddGrpcClient<TestServiceClient>(o => {
o.Address = new Uri("https://localhost:5001");
}).AddPolicyHandler(policy);
var provider = services.BuildServiceProvider();
var client = provider.GetRequiredService<TestServiceClient>();
var testClient = (await client.TestAsync(new Empty()));
And still not working.
At the end it seems AddPolicyHandler is not suitable for grpc clients?
Related
I Have one Consumer-A, and i want to create multimple endpoints witj this consumer. For companies that can be added at runtime. Each company should have its own queue. Is it possible to do with MassTransit InMemmory?
Must be something like this
Consumer-A(with SomeID-a)
Consumer-A(with SomeID-b)
and many other..
And when I sent a message to the queue it was processed by the exact consumer (only 1 concurrent)
I`ve tried this
await using var provider = new ServiceCollection()
.AddMassTransit(x =>
{
x.AddConsumer<ConsumerServiceA>()
.Endpoint(e =>
{
e.Name = "endpint-service";
e.Temporary = false;
e.ConcurrentMessageLimit = 1;
e.InstanceId = SomeId-a;
});
})
.BuildServiceProvider();
I run it when new company created
I solved it by this
var consumer = new ConsumerA();
_bus = Bus.Factory.CreateUsingInMemory(cfg =>
{
cfg.ConcurrentMessageLimit = 1;
});
_bus.ConnectReceiveEndpoint("queue-a", x =>
{
x.Consumer(() => consumer);
});
I am writing sample applications in .Net core to interact with Kafka.
I have downloaded Kafka and Zookeeper official docker images to my machine.
I am using Confluent.Kafka nuget package for both producer and consumer. I am able to produce the message to Kafka. But my consumer part is not working.
Below is my producer and consumer code snippet. I am not sure what mistake I'm doing here.
Do we need to Explicitly create a consumer group?
Consumer Code (This code not working. Thread waiting at consumer.Consume(cToken);)
var config = new ConsumerConfig
{
BootstrapServers = "localhost:9092",
GroupId = "myroupidd",
AutoOffsetReset = AutoOffsetReset.Earliest,
};
var cToken = ctokenSource.Token;
var consumerBuilder = new ConsumerBuilder<Null, string>(config);
consumerBuilder.SetPartitionsAssignedHandler((consumer, partitionlist) =>
{
consumer.Assign(new TopicPartition("myactual-toppics", 0) { });
Console.WriteLine("inside SetPartitionsAssignedHandler action");
});
using var consumer = consumerBuilder.Build();
consumer.Subscribe("myactual-toppics");
while (!cToken.IsCancellationRequested)
{
var consumeResult = consumer.Consume(cToken);
if (consumeResult.Message != null)
Console.WriteLine(consumeResult.Message.Value);
}
Producer Code (This is working fine. I am able to see the messages using Conduckto Tool)
var config = new ProducerConfig
{
BootstrapServers = "localhost:9092",
ClientId = Dns.GetHostName(),
};
using (var producer = new ProducerBuilder<Null, string>(config).Build())
{
while (!stoppingToken.IsCancellationRequested)
{
var top = new TopicPartition("myactual-toppics", 0);
var result = await producer.ProduceAsync(top, new Message<Null, string> { Value = "My First Message" });
Console.WriteLine($"Publishedss1234" );
await Task.Delay(5000, stoppingToken);
Say, I have the following minimal API:
var builder = WebApplication.CreateBuilder(args);
// Routing options
builder.Services
.Configure<RouteOptions>(options =>
{
options.LowercaseUrls = true;
options.LowercaseQueryStrings = true;
});
await using var app = builder.Build();
// API
app.MapGet("/api/customers/{id:int}", async (VRContext db, int id) =>
await db.Customers.FindAsync(id) switch
{
{ } customer => Results.Ok(customer),
null => Results.NotFound(new { Requested_Id = id, Message = $"Customer not found." })
});
//app.MapControllers();
await app.RunAsync();
When I pass non-existing id, I get the following JSON:
{
"requested_Id": 15,
"message": "Customer not found."
}
The problem is that letter I in requested_Id is not lowercase, although I configured it in Configure<RouteOptions>. But when I start using fully-fledged controller, then I correctly get requested_id. How do I achieve the same with MapGet?
Configure the default Json options (Note the using for the correct namespace)
using Microsoft.AspNetCore.Http.Json;
var builder = WebApplication.CreateBuilder(args);
builder.Services.Configure<JsonOptions>(
options =>
{
options.SerializerOptions.PropertyNamingPolicy = new LowerCaseNamingPolicy();
});
await using var app = builder.Build();
// API
app.MapGet("/api/customers/{id:int}", (int id) =>
Results.NotFound(new { Requested_Id = id, Message = $"Customer not found." })
);
await app.RunAsync();
internal class LowerCaseNamingPolicy : JsonNamingPolicy
{
public override string ConvertName(string name) => name.ToLowerInvariant();
}
You can use below line fo code to convert all words in URL as lowercase
services.AddRouting(options => options.LowercaseUrls = true);
So I have to use a library that essentially does a POST to a remote system that may choose to throttle the traffic. If it does, it returns 429 and a specific # of seconds to back off in the Retry-After header... at which point the framework reads and parses this value, and essentially does this
throw new ThrottledException(retryAfterSeconds);
How do I set up a Polly policy that will catch this custom exception, and then retry after exception.RetryAfter seconds?
OK, this was a bit more tricky than it needed to be, but only because I was sent on several wild goose chases by inscrutable compiler messages.
In this scenario the retry is communicated by a custom exception of type SigsThrottledException, which has a field that contains the requested backoff time in seconds.
var policy = Policy
.Handle<SigsThrottledException>(e => e.RetryAfterInSeconds > 0)
.WaitAndRetryAsync(
retryCount: retries,
sleepDurationProvider: (i, e, ctx) =>
{
var ste = (SigsThrottledException)e;
return TimeSpan.FromSeconds((double)ste.RetryAfterInSeconds);
},
onRetryAsync: async (e, ts, i, ctx) =>
{
// Do something here
};);
This is an example of how to use the policy. You can't just add it to an existing HttpClient or HttpClientFactory. You have to use it explicitly.
[TestMethod]
public async Task SigsPollyRetriesOnThrottle()
{
var retryResponse = new HttpResponseMessage
{
StatusCode = (HttpStatusCode)429,
Content = new StringContent("{}"),
};
retryResponse.Headers.Add("Retry-After", "1");
var mockMessageHandler = new Mock<HttpMessageHandler>();
mockMessageHandler.Protected()
.SetupSequence<Task<HttpResponseMessage>>("SendAsync", ItExpr.IsAny<HttpRequestMessage>(), ItExpr.IsAny<CancellationToken>())
.ReturnsAsync(retryResponse)
.ReturnsAsync(new HttpResponseMessage
{
StatusCode = HttpStatusCode.OK
});
var client = new HttpClient(mockMessageHandler.Object);
// Retry once after waiting 1 second
var retryPolicy = Policy
.Handle<SigsThrottledException>(e => e.RetryAfterInSeconds > 0)
.WaitAndRetryAsync(
retryCount: 1,
sleepDurationProvider: (i, e, ctx) =>
{
var ste = (SigsThrottledException)e;
return TimeSpan.FromSeconds((double)ste.RetryAfterInSeconds);
},
onRetryAsync: async (e, ts, i, ctx) =>
{
// Do something here
};);
Stopwatch stopWatch = new Stopwatch();
stopWatch.Start();
var response = await retryPolicy.ExecuteAsync(async () =>
{
Uri substrateurl = new Uri("https://substrate.office.com/");
return await SIGSClient.Instance.PostAsync(client, substrateurl, new UserInfo(), "faketoken", new Signal(), Guid.NewGuid()).ConfigureAwait(false);
}
);
Assert.AreEqual(response.StatusCode, HttpStatusCode.OK);
stopWatch.Stop();
Assert.IsTrue(stopWatch.ElapsedMilliseconds > 1000); // Make sure we actually waited at least a second
}
We are using masstransit with Request/Response pattern as described here: http://masstransit-project.com/MassTransit/usage/request-response.html
For some reason when the bus is initialized with localhost I receive a MassTransit.EndpointNotFoundException on the respond side, using context.RespondAsync method.
When the bus is initialized with the actual IP it works good without any exception.
We are using the bus also with other patterns of masstransit initialized with localhost without any problems.
We tried using both, Request and GetResponse patterns on the request side thinking maybe the problem source is there, but with no success.
Following is the consumer code the Response side, which generate the exception on the RespondAsync:
public override async Task Consume(ConsumeContext<IImageRequest> context)
{
var msg = context.Message;
try
{
var image = GetImage(msg.Id);
await context.RespondAsync<IImageResult>(new
{
msg.Id,
AImage = image
});
}
catch (Exception e)
{
Logger.Instance.Error($"{e}");
}
}
Following is the request side:
Task.Run(async () =>
{
var reqResClient =
BusControl.GetBusControl.CreateRequestClient<IImageRequest, IImageResult>("requestimage");
var req = new ImageRequest(msg.Id);
var response = await reqResClient.Request(req);
}).Wait();
Bus initialization code:
_bus = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
cfg.Durable = true;
_host = cfg.Host(HostName, port, VirtualHost, h =>
{
h.Username(UserName);
h.Password(Password);
});
cfg.ReceiveEndpoint(host, subscriber.QueueName, ep =>
{
ep.Durable = !subscriber.AutoDelete;
ep.AutoDelete = subscriber.AutoDelete;
ep.Instance(subscriber.Consumer);
if (subscriber.PrefetchCount.HasValue)
ep.PrefetchCount = subscriber.PrefetchCount.Value;
if (subscriber.ConcurrencyLimit.HasValue)
ep.UseConcurrencyLimit(subscriber.ConcurrencyLimit.Value);
});
cfg.Message<T>(x =>
{
x.SetEntityName(entityName);
});
});
public void Start()
{
if (_state == BusState.NotInitialized)
throw new InvalidOperationException("Cannot start bus. Bus is not initialized");
_bus?.Start();
_state = BusState.Started;
}
I wanted to know if there's a solution for it? or is it a known problem?
Thanks,