How to add multiple consumers to consume from kafka stream? - c#

I have recently started working on a Kafka Streaming Application using .NET core. I have followed the tutorial: https://medium.com/#srigumm/building-realtime-streaming-applications-using-net-core-and-kafka-ad45ed081b31.
I have built a basic producer-consumer application in which the producer takes input data and pushes it into a kafka-topic. A consumer can subscribe to the topic and consume data from it. I am also able to push this data into another topic by using a new producer. But what I am unable to do is initialise multiple consumers to consume from the same topic.
In appsettings.json:
"consumer": {
"bootstrapservers": "localhost:9092", //specify your kafka broker address
"groupid": "csharp-consumer",
"enableautocommit": true,
"statisticsintervalms": 5000,
"sessiontimeoutms": 6000,
"autooffsetreset": 0,
"enablepartitioneof": true,
"SaslMechanism": 0, //0 for GSSAPI
//"SaslKerberosKeytab":"filename.keytab", //specify your keytab file here
"SaslKerberosPrincipal": "youralias#DOMAIN.COM", //specify your alias here
"SaslKerberosServiceName": "kafka"
//"SaslKerberosKinitCmd":"kinit -k -t %{sasl.kerberos.keytab} %{sasl.kerberos.principal}"
},
processOrderServices.cs:
namespace Api.Services
{
public class ProcessOrdersService : BackgroundService
{
private readonly ConsumerConfig consumerConfig;
private readonly ProducerConfig producerConfig;
//----------------------
//private readonly ConsumerConfig consumerConfig2;
public ProcessOrdersService(ConsumerConfig consumerConfig, ProducerConfig producerConfig)
{
this.producerConfig = producerConfig;
this.consumerConfig = consumerConfig;
//----------------------
//this.consumerConfig2 = consumerConfig;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
Console.WriteLine("OrderProcessing Service Started\n\n");
while (!stoppingToken.IsCancellationRequested)
{
var consumerHelper = new ConsumerWrapper(consumerConfig, "orderrequests");
//var consumerHelper2 = new ConsumerWrapper(consumerConfig, "orderrequests");
string orderRequest = consumerHelper.readMessage();
//consumerHelper2.DisplayMessage();
//Deserilaize
OrderRequest order = JsonConvert.DeserializeObject<OrderRequest>(orderRequest);
//TODO:: Process Order
Console.WriteLine($"Info: OrderHandler => Processing the order for {order.productname}\n\n");
order.status = OrderStatus.COMPLETED;
//Write to ReadyToShip Queue
var producerWrapper = new ProducerWrapper(producerConfig,"readytoship");
await producerWrapper.writeMessage(JsonConvert.SerializeObject(order));
//--------------------------
// var consumerHelper2 = new ConsumerWrapper(consumerConfig2, "orderrequests");
//string processedOrder = consumerHelper2.readMessage();
//OrderRequest order2 = JsonConvert.DeserializeObject<OrderRequest>(processedOrder);
//Console.WriteLine($"Info: OrderHandler => Delivered the order for {order2.productname}\n\n");
//order2.status = OrderStatus.DELIVERED;
//----------------------------
}
}
}
}
ConsumerWrapper.cs:
namespace Api
{
public class ConsumerWrapper
{
private string _topicName;
private ConsumerConfig _consumerConfig;
private Consumer<string,string> _consumer;
private static readonly Random rand = new Random();
public ConsumerWrapper(ConsumerConfig config,string topicName)
{
this._topicName = topicName;
this._consumerConfig = config;
this._consumer = new Consumer<string,string>(this._consumerConfig);
this._consumer.Subscribe(topicName);
}
public string readMessage(){
var consumeResult = this._consumer.Consume();
return consumeResult.Value;
}
public void DisplayMessage()
{
var consumeResult = this._consumer.Consume();
Console.WriteLine(consumeResult.Value);
Console.WriteLine($"Info: OrderHandler => Delivered the order for {consumeResult.Value}\n\n");
return;
}
}
}
I want to be able to call the Consumer class multiple times and be able to read from the same topic. I have understood that there is a requirement to create multiple partitions/group-ids in order to be able to do that. But I am unable to figure out where and how to do that.

You can do this by using group id concept in Kafka, just use same group id for multiple consumers to avoid duplicate consumption of the data from the same topic.

Related

How can I adapt this code to use a single instance instead of creating multiple instances of a service?

I am writing a service for sending emails and I would like to send multiple email notifications at the same time. What I currently have is this:
private void SendInstantMailNotification(string notificationId)
{
MailMessage? message = null;
var notifications = _dbContext
.Notifications
.Where(x => x.Id.Equals(notificationId))
.ToList();
var notification = notifications.First();
message = notification.Content;
Smtp.SendMailSync(message, SmtpConfiguration, Smtp.MailTypeEnum.HTML);
}
The last line of the code creates an instance of the "SMTP" service. And for each time I would like to send an email a new instance is created.
How do I achieve this to be only one instance to be created and called multiple times without overloading the system?
This is the constructor:
private readonly NotificationQueueContext _dbContext;
protected NotificationQueueService(NotificationQueueContext dbContext)
{
_dbContext = dbContext;
}
As I understand, you need a mechanism to sequentially run some tasks. So I created a Background service which creates a SMTP client once and a ConcurrentQueue to hold the mail requests and run them one by one.
This service is going to be active through the whole process of your application so it has while(TRUE) in it. and after each email it sends it waits for 500 ms.
If you want to send a mail from other services you just need to call RegisterMailRequest to enqueue a mail request.
you should define this service as a HostedService like this:
services.AddHostedService<SequentialJobHandler>();
using System.Collections.Concurrent;
using System.Threading;
using System.Threading.Tasks;
using MailKit.Net.Smtp;
using Microsoft.Extensions.Hosting;
using MimeKit;
namespace Sample
{
public class SequentialJobHandler : BackgroundService
{
private readonly string MailServerAddress;
private readonly int MailServerPort;
private readonly string AdminEmailAccount;
private readonly string AdminEmailAccountPass;
private readonly string MailUser;
private readonly string MailTitle;
private ConcurrentQueue<MailRequest> queue = new ConcurrentQueue<MailRequest>();
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
using (var client = new SmtpClient())
{
// For demo-purposes, accept all SSL certificates (in case the server supports STARTTLS)
client.ServerCertificateValidationCallback = (s, c, h, e) => true;
await client.ConnectAsync(MailServerAddress, MailServerPort, MailKit.Security.SecureSocketOptions.Auto);
// Note: only needed if the SMTP server requires authentication
await client.AuthenticateAsync(MailUser, AdminEmailAccountPass);
while (true)
{
MailRequest localValue = null;
if (queue.TryDequeue(out localValue))
{
if (localValue != null)
{
SendMail(localValue, client);
}
}
Thread.Sleep(500);
}
//await client.DisconnectAsync(true);
}
}
private async Task SendMail(MailRequest request, SmtpClient client)
{
var message = new MimeMessage();
message.From.Add(new MailboxAddress(MailTitle, AdminEmailAccount));
message.To.Add(new MailboxAddress(request.toUsername, request.toEmail));
message.Subject = request.subject;
message.Body = new TextPart("html")
{
Text = request.body
};
await client.SendAsync(message);
}
public void RegisterMailRequest(MailRequest request)
{
queue.Enqueue(request);
}
public class MailRequest
{
public string toUsername, toEmail, subject, body;
}
}
}
hope this helps.

How do you print MQTT messages to the console?

I have a very simple C# command line app that connects to an MQTT server and prints messages to the console.
using MQTTnet;
using MQTTnet.Client.Options;
using MQTTnet.Extensions.ManagedClient;
using System.Text;
var options = new MqttClientOptionsBuilder()
.WithTcpServer(MqttConfig.Server, MqttConfig.Port)
.WithCredentials(MqttConfig.User, MqttConfig.Password)
.WithClientId("MqttTest")
.WithCleanSession()
.Build();
var MqttClient = new MqttFactory().CreateMqttClient();
var cancellationToken = new CancellationToken();
var subscribeOptions = new MQTTnet.Client.Subscribing.MqttClientSubscribeOptions();
subscribeOptions.TopicFilters.Add(new MqttTopicFilter { Topic = MqttConfig.Topic });
MqttClient.ConnectAsync(options, cancellationToken);
MqttClient.SubscribeAsync(subscribeOptions, cancellationToken);
MqttClient.UseApplicationMessageReceivedHandler(e => { HandleMessageReceived(e.ApplicationMessage); });
while (true)
{
Task.Delay(1000).GetAwaiter().GetResult();
}
static void HandleMessageReceived(MqttApplicationMessage applicationMessage)
{
Console.WriteLine("### RECEIVED MESSAGE ###");
Console.WriteLine($"+ Topic = {applicationMessage.Topic}");
Console.WriteLine($"+ Payload = {Encoding.UTF8.GetString(applicationMessage.Payload)}");
Console.WriteLine();
}
abstract class MqttConfig
{
public static readonly string Server = "servername";
public static readonly int Port = 1883;
public static readonly string User = "user";
public static readonly string Password = "password";
public static readonly string Topic = "#";
}
Putting the MqttConfig class information into an app like MQTT X shows a bunch of incoming messages. But running this C# app just shows a blank console.
I ended up making basing the application on an MQTTnet sample. I'm posting it as an answer here in case anyone else has the same question in the future.
using MQTTnet;
using MQTTnet.Client;
using MQTTnet.Client.Options;
using System.Text.Json;
#region Subscribe to topic & handle incoming messages
var mqttFactory = new MqttFactory();
using (var mqttClient = mqttFactory.CreateMqttClient())
{
var mqttClientOptions = new MqttClientOptionsBuilder()
.WithTcpServer(MqttConfig.Server, MqttConfig.Port)
.WithCredentials(MqttConfig.User, MqttConfig.Password)
.Build();
mqttClient.UseApplicationMessageReceivedHandler(e =>
{
Console.WriteLine("Received application message.");
e.DumpToConsole();
return Task.CompletedTask;
});
await mqttClient.ConnectAsync(mqttClientOptions, CancellationToken.None);
var mqttSubscribeOptions = mqttFactory.CreateSubscribeOptionsBuilder()
.WithTopicFilter(f => f.WithTopic(MqttConfig.Topic))
.Build();
await mqttClient.SubscribeAsync(mqttSubscribeOptions, CancellationToken.None);
Console.WriteLine("MQTT client subscribed to topic.");
Console.ReadLine(); // Prevents app from immediately closing once MQTT subscription is complete.
// Will close if user presses "enter" before any messages are received.
}
static class ObjectExtensions
{
public static TObject DumpToConsole<TObject>(this TObject #object)
{
var output = "NULL";
if (#object != null)
{
output = JsonSerializer.Serialize(#object, new JsonSerializerOptions { WriteIndented = true });
}
Console.WriteLine($"[{#object?.GetType().Name}]:\r\n{output}");
return #object;
}
}
#endregion
static class MqttConfig
{
public static readonly string Server = "servername";
public static readonly int Port = 1883;
public static readonly string User = "user";
public static readonly string Password = "password";
public static readonly string Topic = "#";
}

Sending request reponse message on Artemis using C#

I am trying to implement a request response pattern in C# with the ArtemisNetClient, but having a bit of trouble finding out how to do so in a more generic way in a real solution.
I was able to do something like this in two console applications based on some Java examples:
Sender
static async System.Threading.Tasks.Task Main(string[] args)
{
var connectionFactory = new ConnectionFactory();
var endpoint = Endpoint.Create("localhost", 5672, "guest", "guest");
var connection = await connectionFactory.CreateAsync(endpoint);
string guid = new Guid().ToString();
var requestAddress = "TRADE REQ1";
var responseAddress = "TRADE RESP";
Message message = new Message("BUY AMD 1000 SHARES");
message.SetCorrelationId(guid);
message.ReplyTo = responseAddress;
var producer = await connection.CreateProducerAsync(requestAddress, RoutingType.Anycast);
await producer.SendAsync(message);
var consumer = await connection.CreateConsumerAsync(responseAddress, RoutingType.Anycast);
var responseMessage = await consumer.ReceiveAsync();
Console.WriteLine(responseMessage.GetBody<string>());
}
Receiver
static async System.Threading.Tasks.Task Main(string[] args)
{
// Create connection
var connectionFactory = new ConnectionFactory();
var endpoint = Endpoint.Create("localhost", 5672, "guest", "guest");
var connection = await connectionFactory.CreateAsync(endpoint);
var requestAddress = "TRADE REQ1";
// Create consumer to receive trade request messages
var consumer = await connection.CreateConsumerAsync(requestAddress, RoutingType.Anycast);
var message = await consumer.ReceiveAsync();
Console.WriteLine($"Received message: {message.GetBody<string>()}");
// Confirm trade request and ssend response message
if (!string.IsNullOrEmpty(message.ReplyTo))
{
Message responseMessage = new Message("Confirmed trade request");
responseMessage.SetCorrelationId(message.CorrelationId);
var producer = await connection.CreateProducerAsync(message.ReplyTo);
await producer.SendAsync(responseMessage);
}
}
This worked as expected, but I'd like to have something more down the line of what is described in this article, except it doesn't have any examples of a request response pattern.
To elaborate, I currently have two services that I want to communicate across.
In service 1 I want to create and publish a message and then wait for a response to enrich the instance object and save it to a database. I currently have this, but it lacks the await response message.
public async Task<Instance> CreateInstance(Instance instance)
{
await _instanceCollection.InsertOneAsync(instance);
var #event = new InstanceCreated
{
Id = instance.Id,
SiteUrl = instance.SiteUrl
};
await _messageProducer.PublishAsync(#event);
return instance;
}
I figured I might need to setup a temporary queue/connection or something in the PublishAsync() and change it to e.g. Task<Message> to support returning a response message. But how would I go about doing that? Would I have to do a new connectionfactory + CreateConsumerAsync etc. like in the console app example?
public class MessageProducer
{
private readonly IAnonymousProducer _producer;
public MessageProducer(IAnonymousProducer producer)
{
_producer = producer;
}
public async Task PublishAsync<T>(T message, string replyTo = null, string correlationId = null)
{
var serialized = JsonSerializer.Serialize(message);
var address = typeof(T).Name;
var msg = new Message(serialized);
if (replyTo != null && correlationId != null)
{
msg.CorrelationId = correlationId;
msg.ReplyTo = replyTo;
}
await _producer.SendAsync(address, msg);
}
public async Task PublishAsync<T>(T message, string routeName, string replyTo = null, string correlationId = null)
{
var serialized = JsonSerializer.Serialize(message);
var address = routeName;
var msg = new Message(serialized);
if(replyTo != null && correlationId != null)
{
msg.CorrelationId = correlationId;
msg.ReplyTo = replyTo;
}
await _producer.SendAsync(address, msg);
}
}
In Service 2 I have a InstanceCreatedConsumer which receives messages, but again it lacks a way to return response messages.
public class InstanceCreatedConsumer : ITypedConsumer<InstanceCreated>
{
private readonly MessageProducer _messageProducer;
public InstanceCreatedConsumer(MessageProducer messageProducer)
{
_messageProducer = messageProducer;
}
public async Task ConsumeAsync(InstanceCreated message, CancellationToken cancellationToken)
{
// consume message and return response
}
}
I figured I might be able to extend the ActiveMqExtensions class with a ConsumeAsync and HandleMessage that handles the response message with a return value, but I haven't gotten as far yet.
public static IActiveMqBuilder AddTypedConsumer<TMessage, TConsumer>(this IActiveMqBuilder builder,
RoutingType routingType)
where TConsumer : class, ITypedConsumer<TMessage>
{
builder.Services.TryAddScoped<TConsumer>();
builder.AddConsumer(typeof(TMessage).Name, routingType, HandleMessage<TMessage, TConsumer>);
return builder;
}
private static async Task HandleMessage<TMessage, TConsumer>(Message message, IConsumer consumer, IServiceProvider serviceProvider, CancellationToken token)
where TConsumer : class, ITypedConsumer<TMessage>
{
try
{
var msg = JsonConvert.DeserializeObject<TMessage>(message.GetBody<string>());
using var scope = serviceProvider.CreateScope();
var typedConsumer = scope.ServiceProvider.GetService<TConsumer>();
await typedConsumer.ConsumeAsync(msg, token);
await consumer.AcceptAsync(message);
}
catch(Exception ex)
{
// todo
}
}
Am I totally wrong in what I am trying to achieve here, or is it just not possible with the ArtemisNetClient?
Maybe someone has an example or can confirm whether I am down the right path, or maybe I should be using a different framework.
I am new to this kind of communication through messages like ActiveMQ Artemis, so any guidance is appreciated.
I don't see anything in the ArtemisNetClient that would simplify the request/response pattern from your application's point of view. One might expect something akin to JMS' QueueRequestor, but I don't see anything like that in the code, and I don't see anything like that listed in the documentation.
I recommend you simply do in your application what you did in your example (i.e. manually create the consumer & producer to deal with the responses on each end respectively). The only change I would recommend is to re-use connections so you create as few as possible. A connection pool would be ideal here.
For what it's worth, it looks to me like the first release of ArtemisNetClient was just 3 months ago and according to GitHub all but 2 of the commits to the code-base came from one developer. ArtemisNetClient may grow into a very successful C# client implementation, but at this point it seems relatively immature. Even if the existing code is high quality if there isn't a solid community around the client then chances are it won't have the support necessary to get timely bug fixes, new features, etc. Only time will tell.
With version 2.7.0 ArtemisNetClient introduces IRequestReplyClient interface that can be used to implement a request-response messaging pattern. With ArtemisNetClient.Extensions.DependencyInjection this may look as follows:
Client Side:
First you need to register your typed request-reply client in DI:
public void ConfigureServices(IServiceCollection services)
{
/*...*/
var endpoints = new[] { Endpoint.Create(host: "localhost", port: 5672, "guest", "guest") };
services.AddActiveMq("bookstore-cluster", endpoints)
.AddRequestReplyClient<MyRequestReplyClient>();
/*...*/
}
MyRequestReplyClient is your custom class that expects the IRequestReplyClient to be injected via the constructor. Once you have your custom class, you can either expose the IRequestReplyClient directly or encapsulate sending logic inside of it:
public class MyRequestReplyClient
{
private readonly IRequestReplyClient _requestReplyClient;
public MyRequestReplyClient(IRequestReplyClient requestReplyClient)
{
_requestReplyClient = requestReplyClient;
}
public async Task<TResponse> SendAsync<TRequest, TResponse>(TRequest request, CancellationToken cancellationToken)
{
var serialized = JsonSerializer.Serialize(request);
var address = typeof(TRequest).Name;
var msg = new Message(serialized);
var response = await _requestReplyClient.SendAsync(address, msg, cancellationToken);
return JsonSerializer.Deserialize<TResponse>(response.GetBody<string>());
}
}
That's it regarding the client-side.
Worker side
To implement the worker side you can (as you suggested), change the ITypedConsumer interface to return the message that would be sent back, or you can provide the additional data (ReplyTo and CorrelationId headers) so you can send the response back as part of your consumer logic. I prefer the latter as it's a more flexible option in my opinion.
Modified ITypedConsumer might look like that:
public interface ITypedConsumer<in T>
{
public Task ConsumeAsync(T message, MessageContext context, CancellationToken cancellationToken);
}
Where MessageContext is just a simple dto:
public class MessageContext
{
public string ReplyTo { get; init; }
public string CorrelationId { get; init; }
}
HandleMessage extension method:
private static async Task HandleMessage<TMessage, TConsumer>(Message message, IConsumer consumer, IServiceProvider serviceProvider, CancellationToken token)
where TConsumer : class, ITypedConsumer<TMessage>
{
var msg = JsonSerializer.Deserialize<TMessage>(message.GetBody<string>());
using var scope = serviceProvider.CreateScope();
var typedConsumer = scope.ServiceProvider.GetService<TConsumer>();
var messageContext = new MessageContext
{
ReplyTo = message.ReplyTo,
CorrelationId = message.CorrelationId
};
await typedConsumer.ConsumeAsync(msg, messageContext, token);
await consumer.AcceptAsync(message);
}
MessageProducer has to be slightly changed as well, so you can explicitly pass address and CorrelationId:
public class MessageProducer
{
private readonly IAnonymousProducer _producer;
public MessageProducer(IAnonymousProducer producer)
{
_producer = producer;
}
public async Task PublishAsync<T>(string address, T message, MessageContext context, CancellationToken cancellationToken)
{
var serialized = JsonSerializer.Serialize(message);
var msg = new Message(serialized);
if (!string.IsNullOrEmpty(context.CorrelationId))
{
msg.CorrelationId = context.CorrelationId;
}
await _producer.SendAsync(address, msg, cancellationToken);
}
}
And finally, the exemplary consumer could work like that:
public class CreateBookConsumer : ITypedConsumer<CreateBook>
{
private readonly MessageProducer _messageProducer;
public CreateBookConsumer(MessageProducer messageProducer)
{
_messageProducer = messageProducer;
}
public async Task ConsumeAsync(CreateBook message, MessageContext context, CancellationToken cancellationToken)
{
var #event = new BookCreated
{
Id = Guid.NewGuid(),
Title = message.Title,
Author = message.Author,
Cost = message.Cost,
InventoryAmount = message.InventoryAmount,
UserId = message.UserId,
Timestamp = DateTime.UtcNow
};
await _messageProducer.PublishAsync(context.ReplyTo, #event, new MessageContext
{
CorrelationId = context.CorrelationId
}, cancellationToken);
}
}

gRPC DNS resolution failed when using machine name

I am new to gRPC and trying to learn it by using the chat server/client sample from cactuaroid here. I’ve modified the code to show progress in a WPF app from a long running task. All code is running on .NET 5 and I’m using the latest versions of the gRPC packages.
The process is working fine when using the computer's IP address but when using computer name for the gRPC client, I’m getting a “DNS resolution failed” exception (computer name is “skylake”):
RpcException: Status(StatusCode="Unavailable", Detail="DNS resolution
failed for service: skylake:6001",
DebugException="Grpc.Core.Internal.CoreErrorDetailException:
{"created":"#1615312867.300000000","description":"Resolver transient
failure","file":"......\src\core\ext\filters\client_channel\client_channel.cc","file_line":2138,"referenced_errors":[{"created":"#1615312867.300000000","description":"DNS
resolution failed for service:
skylake:6001","file":"......\src\core\ext\filters\client_channel\resolver\dns\c_ares\dns_resolver_ares.cc","file_line":362,"grpc_status":14,"referenced_errors":[{"created":"#1615312867.300000000","description":"C-ares
status is not ARES_SUCCESS qtype=AAAA name=skylake is_balancer=0:
Could not contact DNS
servers","file":"......\src\core\ext\filters\client_channel\resolver\dns\c_ares\grpc_ares_wrapper.cc","file_line":716,"referenced_errors":[{"created":"#1615312866.142000000","description":"C-ares
status is not ARES_SUCCESS qtype=A name=skylake is_balancer=0: Could
not contact DNS
servers","file":"......\src\core\ext\filters\client_channel\resolver\dns\c_ares\grpc_ares_wrapper.cc","file_line":716}]}]}]}")
I verified that I could reach the port with telnet skylake 6001.
I am testing locally, client and server both on the same machine. Oddly enough, the gRPC server seems to be just fine with the computer name. Its just the client that has an issue with it.
Server code:
[Export(typeof(IService))]
public class ProgressServiceGrpcServer : Progress.ProgressBase, IService
{
[Import]
private Logger m_logger = null;
[Import]
private ProgressService m_progressService = null;
private readonly Empty m_empty = new Empty();
private const int Port = 6001;
private readonly Grpc.Core.Server m_server;
public ProgressServiceGrpcServer()
{
m_server = new Grpc.Core.Server
{
Services =
{
Progress.BindService(this)
.Intercept(new IpAddressAuthenticator())
},
Ports =
{
new ServerPort("skylake", Port, ServerCredentials.Insecure)
}
};
}
public void Start()
{
m_server.Start();
m_logger.Info("Started.");
}
public override async Task Subscribe(ChannelName channelName, IServerStreamWriter<ProgressReport> responseStream, ServerCallContext context)
{
context.CancellationToken.Register(() => m_logger.Info($"{context.Host} cancels subscription."));
try
{
await m_progressService.GetProgressReportsAsObservable(channelName)
.ToAsyncEnumerable()
.ForEachAwaitAsync(async (x) => await responseStream.WriteAsync(x), context.CancellationToken)
.ConfigureAwait(false);
}
catch (TaskCanceledException)
{
m_logger.Info($"{context.Host} unsubscribed.");
}
}
public override Task<Empty> Write(ProgressReport request, ServerCallContext context)
{
m_logger.Info($"{context.Host} {request}");
m_progressService.Add(request);
return Task.FromResult(m_empty);
}
}
Client code:
public class ProgressServiceClient
{
private readonly Progress.ProgressClient m_client =
new Progress.ProgressClient(
new Channel("skylake”, 6001, ChannelCredentials.Insecure));
public async Task Write(ProgressReport progressReport)
{
await m_client.WriteAsync(progressReport);
}
public IAsyncEnumerable<ProgressReport> ProgressReports(ChannelName channelName)
{
var call = m_client.Subscribe(channelName);
return call.ResponseStream
.ToAsyncEnumerable()
.Finally(() => call.Dispose());
}
}
Progress write method:
while (inProgress)
{
progressServiceClient.Write(new GrpcServer.ProgressReport
{
Id = Task.Id.ToString(),
PercentDone = percentDone,
TimeRemain = timeRemain
}).Wait();
Thread.Sleep(500);
}
Progress read method:
m_progressService = new ProgressServiceClient();
ChannelName channelName = new ChannelName() { Id = id };
var cts = new CancellationTokenSource();
_ = m_progressService.ProgressReports(channelName)
.ForEachAsync((x) =>
{
Log.Debug($"id: {x.Id} progress: {x.PercentDone}");
}, cts.Token);
this.Dispatcher.Invoke(() =>
{
Application.Current.Exit += (_, __) => cts.Cancel();
this.Unloaded += (_, __) => cts.Cancel();
});
Thanks to #jdweng for pointing me in the right direction, this was solved by adding the DNS suffix to the hostname (skylake.lan in my case).
We can get the DNS suffix via IPInterfaceProperties.DnsSuffix.
Alternatively, (which might be safer) we can use the correct IP address instead of the host name by using something like that.

RabbitMQ work queues using Dependency Injection in c#

I am using rabbitmq in a "Work Queues" scenario.
I need eg. a pool of 5 consumers, (each with its own channel), so one consumer doing I/O operations, won't block other consumer of the same queue.
Eg.
If I have on my queue:
Message 1, Message 2, Message 3, Message 4. Each instance of (FistConsumerHandler) will take 1 message from the queue using Round Robin (default rabbitmq behavior)
The problem I am facing is I need to do this using Dependency Injection.
Here is what i have so far:
On Windows service start (my consumers are hosted in a windows service):
protected override void OnStart(string[] args)
{
BuildConnections();
// Register the consumers. For simplicity only showing FirstConsumerHandler.
AddConsumerHandlers<FistConsumerHandler>(ConstantesProcesos.Exchange, ConstantesProcesos.QueueForFirstHandler);
BuildStartup();
var logger = GetLogger<ServicioProcesos>();
logger.LogInformation("Windows Service Started");
Console.WriteLine("Press [enter] to exit.");
}
protected virtual void BuildConnections(
string notificationHubPath = "notificationhub_path",
string rabbitMQHostname = "rabbitmq_hostname",
string rabbitMQPort = "rabbitmq_port",
string rabbitMQUserName = "rabbitmq_username",
string rabbitMQPassword = "rabbitmq_password")
{
ContextHelpers.Setup(ConfigurationManager.ConnectionStrings[appContextConnectionString].ConnectionString);
if (_connection == null)
{
var factory = new ConnectionFactory
{
HostName = ConfigurationManager.AppSettings[rabbitMQHostname],
Port = int.Parse(ConfigurationManager.AppSettings[rabbitMQPort]),
UserName = ConfigurationManager.AppSettings[rabbitMQUserName],
Password = ConfigurationManager.AppSettings[rabbitMQPassword],
DispatchConsumersAsync = true,
};
// Create a connection
do
{
try
{
_connection = factory.CreateConnection();
}
catch (RabbitMQ.Client.Exceptions.BrokerUnreachableException e)
{
Thread.Sleep(5000);
}
} while (_connection == null);
}
_startupBuilder = new StartupBuilder(_connection);
}
protected void AddConsumerHandlers<THandler>(string exchange, string queue)
{
var consumerHandlerItem = new ConsumerHandlerItem
{
ConsumerType = typeof(THandler),
Exchange = exchange,
Queue = queue
};
_startupBuilder._consumerHandlerItems.Add(consumerHandlerItem);
}
protected void BuildStartup()
{
ServiceProvider = _startupBuilder.Build();
}
Startup Builder:
using Microsoft.Extensions.DependencyInjection;
using RabbitMQ.Client;
using RabbitMQ.Client.Events;
using System;
using System.Collections.Generic;
public class StartupBuilder
{
private static IConnection _connection;
private IModel _channel;
public List<ConsumerHandlerItem> _consumerHandlerItems;
public IServiceCollection Services { get; private set; }
public StartupBuilder(IConnection connection)
{
_connection = connection;
_consumerHandlerItems = new List<ConsumerHandlerItem>();
Services = new ServiceCollection();
}
public IServiceProvider Build()
{
_channel = _connection.CreateModel();
Services.InitSerilog();
// Add channel as singleton (this is not correct as I need 1 channel per ConsumerHandler)
Services.AddSingleton(_channel);
// Register the ConsumerHandler to DI
foreach (var item in _consumerHandlerItems)
{
// Add FirstHandler to DI
Type consumerType = item.ConsumerType;
Services.AddSingleton(consumerType);
}
// Finish DI Setup
var serviceProvider = Services.BuildServiceProvider();
// Bind the consumer handler to the channel and queue
foreach (var item in _consumerHandlerItems)
{
var consumerHandler = (AsyncEventingBasicConsumer)serviceProvider.GetRequiredService(item.ConsumerType);
_channel.AssignNewProcessor(item, consumerHandler);
}
return serviceProvider;
}
}
Helpers:
public static class QueuesHelpers
{
public static void AssignNewProcessor(this IModel channel, ConsumerHandlerItem item, AsyncEventingBasicConsumer consumerHandler)
{
channel.ExchangeDeclare(item.Exchange, ExchangeType.Topic, durable: true);
channel.QueueDeclare(item.Queue, true, false, false, null);
channel.QueueBind(item.Queue, item.Exchange, item.Queue, null);
channel.BasicConsume(item.Queue, false, consumerHandler);
}
}
Consumer handler:
public class FistConsumerHandler : AsyncEventingBasicConsumer
{
private readonly ILogger<FistConsumerHandler> _logger;
private Guid guid = Guid.NewGuid();
public FistConsumerHandler(
IModel channel,
ILogger<FistConsumerHandler> logger) : base(channel)
{
Received += ConsumeMessageAsync;
_logger = logger;
}
private async Task ConsumeMessageAsync(object sender, BasicDeliverEventArgs eventArgs)
{
try
{
// consumer logic to consume the message
}
catch (Exception ex)
{
}
finally
{
Model.Acknowledge(eventArgs);
}
}
}
The problem with this code is:
There is ony 1 instance of FistConsumerHandler (as is reigstered as singleton). I need, for instance 5.
I have only 1 channel, I need 1 channel per instance.
To sum up, the expected behavior using Microsoft.Extensions.DependencyInjection should be:
Create a connection (share this connection with all consumers)
When a message is received to the queue, it should be consumed by 1 consumer using its own channel
If another message is received to the queue, it should be consumed by another consumer
TL;DR; Create your own scope
I've done something similar in an app I'm working on, albeit not as cleanly as I would like (and thus why I came across this post). The key for me was using IServiceScopeFactory to get injected services and use them in a consumer method. In a typical HTTP request the API will automatically create/close scope for you as the request comes in / response goes out, respectively. But since this isn't an HTTP request, we need to create / close the scope for using injected services.
This is a simplified example for getting an injected DB context (but could be anything), assuming I've already set up the RabbitMQ consumer, deserialized the message as an object (FooEntity in this example):
public class RabbitMQConsumer
{
private readonly IServiceProvider _provider;
public RabbitMQConsumer(IServiceProvider serviceProvider)
{
this._serviceProvider = serviceProvider;
}
public async Task ConsumeMessageAsync()
{
// Using statement ensures we close scope when finished, helping avoid memory leaks
using (var scope = this._serviceProvider.CreateScope())
{
// Get your service(s) within the scope
var context = scope.ServiceProvider.GetRequiredService<MyDBContext>();
// Do things with dbContext
}
}
}
Be sure to register RabbitMQConsumer as a singleton and not a transient in Startup.cs also.
References:
Similar SO post
MS Docs

Categories