We Keep Coding

how Multiple consumer in masstransit consume publish one event

I have my controller publishing events to azure service bus via masstransit.
And multiple instanses of .net core service consuming those event.I want ALL instances to consume the same events.
await _publishEndpoint.Publish<MyPublishEvent>(
new MyPublishEvent
{
Id = 1,
Description = "test"
});
here is one consumer from one microservice.
public async Task Consume(ConsumeContext<MyPublishEvent> context)
{
try
{
// Here add business logic to insert record in to Database 1.
}
await Task.CompletedTask;
}
and the configuration is here.
builder.Services.AddMassTransit(cfg =>
{
cfg.SetKebabCaseEndpointNameFormatter();
cfg.AddConsumersFromNamespaceContaining<Consumers>();
cfg.UsingAzureServiceBus((context, cfg) =>
{
cfg.Host($"Endpoint = endpoint");
cfg.ConfigureEndpoints(context);
});
});
I want ALL instances to consume the same events.
The first one consume the event from one microservice and the rest of the service instances don't get it to consume it. can this be fixed. Am I missing any configuration?Appreciate your help in finding the cause.

MassTransit not subscribing to AzureServiceBus Topic

I'm currently trying to update application that was originally .NET Core 3.1 using MassTransit 6.3.2. It is now configured to use .NET 6.0 and MassTransit 7.3.0
Our application uses MassTransit to send messages via Azure Service Bus, publishing messages to Topics, which then have other Subscribers listening to those Topic.
Cut down, it was implemented like so:
// Program.cs
services.AddMassTransit(config =>
{
config.AddConsumer<AppointmentBookedMessageConsumer>();
config.AddBus(BusControlFactory.ConfigureAzureServiceBus);
});
// BusControlFactory.cs
public static class BusControlFactory
{
public static IBusControl ConfigureAzureServiceBus(IRegistrationContext<IServiceProvider> context)
{
var config = context.Container.GetService<AppConfiguration>();
var azureServiceBus = Bus.Factory.CreateUsingAzureServiceBus(busFactoryConfig =>
{
busFactoryConfig.Host("Endpoint=sb://REDACTED-queues.servicebus.windows.net/;SharedAccessKeyName=MyMessageQueuing;SharedAccessKey=MyKeyGoesHere");
busFactoryConfig.Message<AppointmentBookedMessage>(m => m.SetEntityName("appointment-booked"));
busFactoryConfig.SubscriptionEndpoint<AppointmentBookedMessage>(
"my-subscriber-name",
configurator =>
{
configurator.UseMessageRetry(r => r.Interval(5, TimeSpan.FromSeconds(60)));
configurator.Consumer<AppointmentBookedMessageConsumer>(context.Container);
});
return azureServiceBus;
}
}
}
It has now been changed and upgraded to the latest MassTransit and is implemented like:
// Program.cs
services.AddMassTransit(config =>
{
config.AddConsumer<AppointmentBookedMessageConsumer, AppointmentBookedMessageConsumerDefinition>();
config.UsingAzureServiceBus((context, cfg) =>
{
cfg.Host("Endpoint=sb://REDACTED-queues.servicebus.windows.net/;SharedAccessKeyName=MyMessageQueuing;SharedAccessKey=MyKeyGoesHere");
cfg.Message<AppointmentBookedMessage>(m => m.SetEntityName("appointment-booked"));
cfg.ConfigureEndpoints(context);
});
// AppointmentBookedMessageConsumerDefinition.cs
public class AppointmentBookedMessageConsumerDefinition: ConsumerDefinition<AppointmentBookedMessageConsumer>
{
public AppointmentBookedMessageConsumerDefinition()
{
EndpointName = "testharness.subscriber";
}
protected override void ConfigureConsumer(IReceiveEndpointConfigurator endpointConfigurator, IConsumerConfigurator<AppointmentBookedMessageConsumer> consumerConfigurator)
{
endpointConfigurator.UseMessageRetry(r => r.Interval(5, TimeSpan.FromSeconds(60)));
}
}
The issue if it can be considered one, is that I can't bind to a subscription that already exists.
In the example above, you can see that the EndpointName is set as "testharness.subscriber". There was already a subscription to the Topic "appointment-booked" from prior to me upgrading. However, when the application runs, it does not error, but it receives no messages.
If I change the EndpointName to "testharness.subscriber2". Another subscriber appears in the Azure Service Bus topic (via the Azure Portal) and I start receiving messages. I can see no difference in the names (other than the change that I placed, in this case: the "2" suffix).
Am I missing something here? Is there something else I need to do to get these to bind? Is my configuration wrong? Was it wrong? While I'm sure I can get around this by managing the release more closely and removing unneeded queues once they're using new ones - it feels like the wrong approach.

With Azure Service Bus, ForwardTo on a subscription can be a bit opaque.
While the subscription may indeed visually indicate that it is forwarding to the correctly named queue, it might be that the queue was deleted and recreated at some point without deleting the subscription. This results in a subscription that will build up messages, as it is unable to forward them to a queue that no longer exists.
Why? Internally, a subscription maintains the ForwardTo as an object id, which after the queue is deleted points to an object that doesn't exist – resulting in messages building up in the subscription.
If you have messages in the subscription, you may need to go into the portal and update that subscription to point to the new queue (even though it has the same name), at which point the messages should flow through to the queue.
If there aren't any messages in the subscription (or if they aren't important), you can just delete the subscription and it will be recreated by MassTransit when you restart the bus.

Parallel Handling Kafka messages in a Windows Service using C#

I have Windows service written in C#. Earlier we were using Event hubs with multiple partitions for message queuing. We recently moved to Kafka. For implementing Event hubs in c# , we have IEventProcessor.ProcessEventsAsync , which keeps listening to event hub notifications and is triggered whenever a message is posted to event hub , which runs asynchronously in the background
I did not find any equivalent method in Kafka.
My requirement here is to subscribe to a Kafka topic and continuously consume messages. When a message is consumed, some other operations are also supposed to executed for that message. For each message say the execution time takes around 15 mins, I want the Kafka consumer to consume all messages and keep it in queue as when it receives and writes it into a file. Other process should read the file, pick the message and do other operations. I want all of them to run simultaneously/parallelly.
PS : I have written a console application which can produce and consume one message.What I'm looking for is queuing and parallelism.

For paralellism Kafka implements what's known as consumer groups. Kafka stores the "offsets" (read: key of record across a topic) and also stores the offsets of where a given consumer group is also at in processing the records. This should allow you to create new consumer instances on the fly using the same program, and by changing the group allow two programs to consume the same data in paralell for different tasks.
I found this link helpful when I was creating my first consumer as well, in case you found a way to create it without a groupId: http://cloudurable.com/blog/kafka-tutorial-kafka-consumer/index.html
Hope this helps!

Have look at Silverback: https://silverback-messaging.net. It abstracts many of those concerns and the basic usage is as simple as this:
public class Startup
{
public void ConfigureServices(IServiceCollection services)
{
services
.AddSilverback()
.WithConnectionToMessageBroker(options => options.AddKafka())
.AddKafkaEndpoints(
endpoints => endpoints
.Configure(
config =>
{
config.BootstrapServers = "localhost:9092";
})
.AddInbound(
endpoint => endpoint
.ConsumeFrom("my-topic")
.DeserializeJson(serializer => serializer.UseFixedType<SomeMessage>())
.Configure(
config =>
{
config.GroupId = "test-consumer-group";
config.AutoOffsetReset = AutoOffsetReset.Earliest;
})))
.AddSingletonSubscriber<MySubscriber>();
}
}
public class MySubscriber
{
public Task OnMessageReceived(SomeMessage message)
{
// TODO: process message
}
}

RabbitMQ several consumers in the same process

I have a .NET project that needs to read messaged from a given Queue.
I have several producers writing the same type of message into the queue.
I want my consumer app to have several threads reading messages and handling them so that the load will not be on a single thread.
Any ideas or sample code on how to achieve this?
Again, Note:
Each message should be processed once and not several times. The work should be balanced between the worker threads

You are going to need a bit of plumbing to get that done.
I have an open-source service bus called Shuttle.Esb and there are many other service bus options available that you may wish to consider.
But if you do not want to go that route you could still have a look at some of the code and implementations to get some ideas. I have a RabbitMQ implementation that may be of assistance.

Take a look at masstransit project : http://masstransit-project.com/MassTransit/usage/message-consumers.html
It has configurations like prefetch count and concurrency limit. It brings you to consume messages paralelly.
Also it is very simple to setup:
IBusControl busControl = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
IRabbitMqHost host = cfg.Host(new Uri(RabbitMQConstants.RabbitMQUri),
hst =>
{
hst.Username(RabbitMQConstants.RabbitMQUserName);
hst.Password(RabbitMQConstants.RabbitMQPassword);
});
cfg.ReceiveEndpoint(host,
RabbitMQConstants.YourQueueName,
endPointConfigurator => {
endPointConfigurator.Consumer<SomeConsumer>();
endPointConfigurator.UseConcurrencyLimit(4);
});
});
busControl.Start();
public class SomeConsumer :
IConsumer<YourMessageClass>
{
public async Task Consume(ConsumeContext<YourMessageClass> context)
{
await Console.Out.WriteLineAsync($"Message consumed: {context.Message.YourValue}");
}
}

ASP.NET Core 2.2 SignalR Buffering Calls Instead of Invoking Asynchronously

I'm writing an ASP.NET Core 2.2 C# web application that uses SignalR to take calls from JavaScript in a web browser. On the server side, I initialize SignalR like this:
public static void ConfigureServices(IServiceCollection services)
{
...
// Use SignalR
services.AddSignalR(o =>
{
o.EnableDetailedErrors = true;
});
}
and
public static void Configure(IApplicationBuilder app, Microsoft.AspNetCore.Hosting.IHostingEnvironment env, ILoggerFactory loggerFactory)
{
...
// Route to SignalR hubs
app.UseSignalR(routes =>
{
routes.MapHub<ClientProxySignalR>("/clienthub");
});
...
}
My SignalR Hub class has a method like this:
public class ClientProxySignalR : Hub
{
...
public async Task<IEnumerable<TagDescriptor>> GetRealtimeTags(string project)
{
return await _requestRouter.GetRealtimeTags(project).ConfigureAwait(false);
}
...
}
and on the client side:
var connection = new signalR.HubConnectionBuilder()
.withUrl("/clienthub")
.configureLogging(signalR.LogLevel.Information)
.build();
connection.start().then(function () {
...
// Enable buttons & stuff so you can click
...
}
document.getElementById("tagGetter").addEventListener("click", function (event) {
connection.invoke("GetRealtimeTags", "Project1").then(data => {
...
// use data
...
}
}
This all works as far as it goes, and it does work asynchronously. So if I click the "tagGetter" button, it invokes the "GetRealtimeTags" method on my Hub and the "then" portion is invoked when the data comes back. It is also true that if this takes a while to run, and I click the "tagGetter" button again in the meantime, it makes the .invoke("GetRealtimeTags") call again...at least in the JavaScript.
However...this is where the problem occurs. Although the second call is made in the JavaScript, it will not trigger the corresponding method in my SignalR Hub class until the first call finishes. This doesn't match my understanding of what is supposed to happen. I thought that each invocation of a SignalR hub method back to the server would cause the creation of a new instance of the hub class to handle the call. Instead, the first call seems to be blocking the second.
If I create two different connections in my JavaScript code, then I am able to make two simultaneous calls on them without one blocking the other. But I know that isn't the right way to make this work.
So my question is: what am I doing wrong in this case?

This is by design of websockets to ensure messages are delivered in exact order.
You can refer to this for more information: https://hpbn.co/websocket/
Quoted:
The preceding example attempts to send application updates to the
server, but only if the previous messages have been drained from the
client’s buffer. Why bother with such checks? All WebSocket messages
are delivered in the exact order in which they are queued by the
client. As a result, a large backlog of queued messages, or even a
single large message, will delay delivery of messages queued behind
it—head-of-line blocking!
They also suggest a workaround solution:
To work around this problem, the application can split large messages
into smaller chunks, monitor the bufferedAmount value carefully to
avoid head-of-line blocking, and even implement its own priority queue
for pending messages instead of blindly queuing them all on the
socket.

Interesting question.
I think the button should be disabled and show a loading icon on the first time it is clicked. But maybe your UI enables more than one project to be loaded at once. Just thought for a second we might have an X-Y problem.
Anyways, to answer your question:
One way you can easily deal with this is to decouple the process of "requesting" data from the process of "getting and sending" data to the user when it is ready.
Don't await GetRealtimeTags and instead start a background task noting the connection id of the caller
Return nothing from GetRealtimeTags
Once the result is ready in the background task, call a new RealtimeTagsReady method that will call the JavaScript client with the results using the connection id kept earlier
Let me know if this helps.