I Have one Consumer-A, and i want to create multimple endpoints witj this consumer. For companies that can be added at runtime. Each company should have its own queue. Is it possible to do with MassTransit InMemmory?
Must be something like this
Consumer-A(with SomeID-a)
Consumer-A(with SomeID-b)
and many other..
And when I sent a message to the queue it was processed by the exact consumer (only 1 concurrent)
I`ve tried this
await using var provider = new ServiceCollection()
.AddMassTransit(x =>
{
x.AddConsumer<ConsumerServiceA>()
.Endpoint(e =>
{
e.Name = "endpint-service";
e.Temporary = false;
e.ConcurrentMessageLimit = 1;
e.InstanceId = SomeId-a;
});
})
.BuildServiceProvider();
I run it when new company created
I solved it by this
var consumer = new ConsumerA();
_bus = Bus.Factory.CreateUsingInMemory(cfg =>
{
cfg.ConcurrentMessageLimit = 1;
});
_bus.ConnectReceiveEndpoint("queue-a", x =>
{
x.Consumer(() => consumer);
});
Related
I'm trying to use Polly as retry policy handler for grpc in my .net core 6 project. I noticed that the retryFunc is never invoked. I started from this project gRPC & ASP.NET Core 3.1: Resiliency with Polly
class Program
{
static async Task Main(string[] args)
{
// DI
var services = new ServiceCollection();
var loggerFactory = LoggerFactory.Create(logging =>
{
logging.AddConsole();
logging.SetMinimumLevel(LogLevel.Debug);
});
var serverErrors = new HttpStatusCode[] {
HttpStatusCode.BadGateway,
HttpStatusCode.GatewayTimeout,
HttpStatusCode.ServiceUnavailable,
HttpStatusCode.InternalServerError,
HttpStatusCode.TooManyRequests,
HttpStatusCode.RequestTimeout
};
var gRpcErrors = new StatusCode[] {
StatusCode.DeadlineExceeded,
StatusCode.Internal,
StatusCode.NotFound,
StatusCode.ResourceExhausted,
StatusCode.Unavailable,
StatusCode.Unknown
};
Func<HttpRequestMessage, IAsyncPolicy<HttpResponseMessage>> retryFunc = (request) =>
{
return Policy.HandleResult<HttpResponseMessage>(r => {
var grpcStatus = StatusManager.GetStatusCode(r);
var httpStatusCode = r.StatusCode;
return (grpcStatus == null && serverErrors.Contains(httpStatusCode)) || // if the server send an error before gRPC pipeline
(httpStatusCode == HttpStatusCode.OK && gRpcErrors.Contains(grpcStatus.Value)); // if gRPC pipeline handled the request (gRPC always answers OK)
})
.WaitAndRetryAsync(3, (input) => TimeSpan.FromSeconds(3 + input), (result, timeSpan, retryCount, context) =>
{
var grpcStatus = StatusManager.GetStatusCode(result.Result);
Console.WriteLine($"Request failed with {grpcStatus}. Retry");
});
};
services.AddGrpcClient<CountryServiceClient>(o =>
{
o.Address = new Uri("https://localhost:5001");
}).AddPolicyHandler(retryFunc);
var provider = services.BuildServiceProvider();
var client = provider.GetRequiredService<CountryServiceClient>();
try
{
var countries = (await client.GetAllAsync(new EmptyRequest())).Countries.Select(x => new Country
{
CountryId = x.Id,
Description = x.Description,
CountryName = x.Name
}).ToList();
Console.WriteLine("Found countries");
countries.ForEach(x => Console.WriteLine($"Found country {x.CountryName} ({x.CountryId}) {x.Description}"));
}
catch (RpcException e)
{
Console.WriteLine(e.Message);
}
}
}
but at the end WaitAndRetryAsync is never called.
I created a small project available on github in order to reproduce it.
My test is fairly simple. I start the client without a listening back-end, expecting to read 3 times the output from Console.WriteLine($"Request failed with {grpcStatus}. Retry"); on the console. But the policy handler in never fired. I have the following exception instead
Status(StatusCode="Unavailable", Detail="Error connecting to
subchannel.", DebugException="System.Net.Sockets.SocketException
(10061): No connection could be made because the target machine
actively refused it.
without any retry.
This is not working for you because Retry is now built into Grpc. In order to make this work, register your service as follows:
var defaultMethodConfig = new MethodConfig
{
Names = { MethodName.Default },
RetryPolicy = new RetryPolicy
{
MaxAttempts = 3,
InitialBackoff = TimeSpan.FromSeconds(3),
MaxBackoff = TimeSpan.FromSeconds(3),
BackoffMultiplier = 1,
RetryableStatusCodes =
{
// Whatever status codes you want to look for
StatusCode.Unauthenticated, StatusCode.NotFound, StatusCode.Unavailable,
}
}
};
var services = new ServiceCollection();
services.AddGrpcClient<TestServiceClient>(o => {
o.Address = new Uri("https://localhost:5001");
o.ChannelOptionsActions.Add(options =>
{
options.ServiceConfig = new ServiceConfig {MethodConfigs = {defaultMethodConfig}};
});
});
That will add the retry policy to your client. One other thing that you might run into. I didn't realize this at the time, but in my service implementation, I was setting up errors something like this:
var response = new MyServiceResponse()
// something bad happens
context.Status = new Status(StatusCode.Internal, "Something went wrong");
return response;
The retry logic will not kick in if you implement your service like that, you actually have to do something more like this:
// something bad happens
throw new RpcException(new Status(StatusCode.Internal, "Something went wrong"));
The retry logic you configured when registering your client will then work. Hope that helps.
With the help of #PeterCsala I tried some fix.
As a first attempt I tried without DependencyInjection, registering the policy as follows
var policy = Policy
.Handle<Exception>()
.RetryAsync(3, (exception, count) =>
{
Console.WriteLine($"Request {count}, {exception.Message}. Retry");
});
var channel = GrpcChannel.ForAddress("https://localhost:5001");
TestServiceClient client = new TestServiceClient(channel);
await policy.ExecuteAsync(async () => await client.TestAsync(new Empty()));
This way it's working.
Then I came back to DI and used to register the policy as follows
IAsyncPolicy<HttpResponseMessage> policy =
Policy<HttpResponseMessage>.Handle<Exception>().RetryAsync(3, (exception, count) =>
{
Console.WriteLine($"Request {count}, {exception.Exception.Message}. Retry");
});
var services = new ServiceCollection();
services.AddGrpcClient<TestServiceClient>(o => {
o.Address = new Uri("https://localhost:5001");
}).AddPolicyHandler(policy);
var provider = services.BuildServiceProvider();
var client = provider.GetRequiredService<TestServiceClient>();
var testClient = (await client.TestAsync(new Empty()));
And still not working.
At the end it seems AddPolicyHandler is not suitable for grpc clients?
I am writing sample applications in .Net core to interact with Kafka.
I have downloaded Kafka and Zookeeper official docker images to my machine.
I am using Confluent.Kafka nuget package for both producer and consumer. I am able to produce the message to Kafka. But my consumer part is not working.
Below is my producer and consumer code snippet. I am not sure what mistake I'm doing here.
Do we need to Explicitly create a consumer group?
Consumer Code (This code not working. Thread waiting at consumer.Consume(cToken);)
var config = new ConsumerConfig
{
BootstrapServers = "localhost:9092",
GroupId = "myroupidd",
AutoOffsetReset = AutoOffsetReset.Earliest,
};
var cToken = ctokenSource.Token;
var consumerBuilder = new ConsumerBuilder<Null, string>(config);
consumerBuilder.SetPartitionsAssignedHandler((consumer, partitionlist) =>
{
consumer.Assign(new TopicPartition("myactual-toppics", 0) { });
Console.WriteLine("inside SetPartitionsAssignedHandler action");
});
using var consumer = consumerBuilder.Build();
consumer.Subscribe("myactual-toppics");
while (!cToken.IsCancellationRequested)
{
var consumeResult = consumer.Consume(cToken);
if (consumeResult.Message != null)
Console.WriteLine(consumeResult.Message.Value);
}
Producer Code (This is working fine. I am able to see the messages using Conduckto Tool)
var config = new ProducerConfig
{
BootstrapServers = "localhost:9092",
ClientId = Dns.GetHostName(),
};
using (var producer = new ProducerBuilder<Null, string>(config).Build())
{
while (!stoppingToken.IsCancellationRequested)
{
var top = new TopicPartition("myactual-toppics", 0);
var result = await producer.ProduceAsync(top, new Message<Null, string> { Value = "My First Message" });
Console.WriteLine($"Publishedss1234" );
await Task.Delay(5000, stoppingToken);
I am trying to add multiple schemas to the same subject in the schema registry, so I have set ValueSubjectNameStrategy to SubjectNameStrategy.TopicRecord, also set the register automatically to AutomaticRegistrationBehavior.Always. But while auto registering the schema it still using the SubjectNameStrategy.Topic strategy.
var schemaRegistryConfig = new SchemaRegistryConfig { Url = "http://localhost:8081", ValueSubjectNameStrategy = SubjectNameStrategy.TopicRecord };
var registry = new CachedSchemaRegistryClient(schemaRegistryConfig);
var builder = new ProducerBuilder<string, SplitLineKGN>(KafkaConfig.Producer.GetConfig(_config.GetSection("KafkaProducer")))
.SetAvroValueSerializer(registry, registerAutomatically: AutomaticRegistrationBehavior.Always)
.SetErrorHandler((_, error) => Console.Error.WriteLine(error.ToString()));
_producerMsg = builder.Build();
await _producerMsg.ProduceAsync("MyTopic", new Message<string, SampleMessage> { Key = key, Value = line });
how to auto register multiple schemas to a topic?
Ensure that you changed a subject naming strategy for a topic
SchemaRegistryConfig.ValueSubjectNameStrategy is deprecated, it should now be configured using the serializer's configuration: code
For producing multiple event types with a single producer you have to use AvroSerializer<ISpecificRecord> as described below:
var schemaRegistryConfig = new SchemaRegistryConfig { Url = "http://localhost:8081" };
using var schemaRegistryClient = new CachedSchemaRegistryClient(schemaRegistryConfig);
var avroSerializerConfig = new AvroSerializerConfig
{
SubjectNameStrategy = SubjectNameStrategy.TopicRecord,
AutoRegisterSchemas = true // (the default)
};
// Assuming this is your own custom code because the Confluent
// producer doesn't have anything like this.
var producerConfig = KafkaConfig.Producer.GetConfig(_config.GetSection("KafkaProducer"));
using var producer = new ProducerBuilder<string, ISpecificRecord>(producerConfig)
.SetValueSerializer(new AvroSerializer<ISpecificRecord>(schemaRegistryClient, avroSerializerConfig))
.SetErrorHandler((_, error) => Console.Error.WriteLine(error))
.Build();
var deliveryResult = await producer.ProduceAsync("MyTopic", new Message<string, ISpecificRecord>
{
Key = key,
Value = line
});
Console.WriteLine($"Delivered to: {deliveryResult.TopicPartitionOffset}");
We are using masstransit with Request/Response pattern as described here: http://masstransit-project.com/MassTransit/usage/request-response.html
For some reason when the bus is initialized with localhost I receive a MassTransit.EndpointNotFoundException on the respond side, using context.RespondAsync method.
When the bus is initialized with the actual IP it works good without any exception.
We are using the bus also with other patterns of masstransit initialized with localhost without any problems.
We tried using both, Request and GetResponse patterns on the request side thinking maybe the problem source is there, but with no success.
Following is the consumer code the Response side, which generate the exception on the RespondAsync:
public override async Task Consume(ConsumeContext<IImageRequest> context)
{
var msg = context.Message;
try
{
var image = GetImage(msg.Id);
await context.RespondAsync<IImageResult>(new
{
msg.Id,
AImage = image
});
}
catch (Exception e)
{
Logger.Instance.Error($"{e}");
}
}
Following is the request side:
Task.Run(async () =>
{
var reqResClient =
BusControl.GetBusControl.CreateRequestClient<IImageRequest, IImageResult>("requestimage");
var req = new ImageRequest(msg.Id);
var response = await reqResClient.Request(req);
}).Wait();
Bus initialization code:
_bus = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
cfg.Durable = true;
_host = cfg.Host(HostName, port, VirtualHost, h =>
{
h.Username(UserName);
h.Password(Password);
});
cfg.ReceiveEndpoint(host, subscriber.QueueName, ep =>
{
ep.Durable = !subscriber.AutoDelete;
ep.AutoDelete = subscriber.AutoDelete;
ep.Instance(subscriber.Consumer);
if (subscriber.PrefetchCount.HasValue)
ep.PrefetchCount = subscriber.PrefetchCount.Value;
if (subscriber.ConcurrencyLimit.HasValue)
ep.UseConcurrencyLimit(subscriber.ConcurrencyLimit.Value);
});
cfg.Message<T>(x =>
{
x.SetEntityName(entityName);
});
});
public void Start()
{
if (_state == BusState.NotInitialized)
throw new InvalidOperationException("Cannot start bus. Bus is not initialized");
_bus?.Start();
_state = BusState.Started;
}
I wanted to know if there's a solution for it? or is it a known problem?
Thanks,
I'm using kafka 0.8.1.1 on a Red Hat VM with kafka-net plugin. How can I configure my consumer to stop receiving earlier messages from kafka?
My consumer code:
var options = new KafkaOptions(new Uri("tcp://199.53.249.150:9092"), new Uri("tcp://199.53.249.151:9092"));
Stopwatch sp = new Stopwatch();
var router = new BrokerRouter(options);
var consumer = new Consumer(new ConsumerOptions("Test", router));
ThreadStart start2 = () =>
{
while (true)
{
sp.Start();
foreach (var message in consumer.Consume())
{
if (MessageDecoderReceiver.MessageBase(message.Value) != null)
{
PrintMessage(MessageDecoderReceiver.MessageBase(message.Value).ToString());
}
else
{
Console.WriteLine(message.Value);
}
}
sp.Stop();
}
};
var thread2 = new Thread(start2);
thread2.Start();
The Consumer in Kafka-net does not currently auto track the offsets being consumed. You will have to implement the offset tracking manually.
To Store the offset in kafka version 0.8.1:
var commit = new OffsetCommitRequest
{
ConsumerGroup = consumerGroup,
OffsetCommits = new List<OffsetCommit>
{
new OffsetCommit
{
PartitionId = partitionId,
Topic = IntegrationConfig.IntegrationTopic,
Offset = offset,
Metadata = metadata
}
}
};
var commitResponse = conn.Connection.SendAsync(commit).Result.FirstOrDefault();
To set the consumer to start importing at a specific offset point:
var offsets = consumer.GetTopicOffsetAsync(IntegrationConfig.IntegrationTopic).Result
.Select(x => new OffsetPosition(x.PartitionId, x.Offsets.Max())).ToArray();
var consumer = new Consumer(new ConsumerOptions(IntegrationConfig.IntegrationTopic, router), offsets);
Note the above code will set the consumer to start consuming at the very end of the log, effectively only receiving new messages.