I am trying to implement a publish only bus in MassTransit v3 with C# and RabbitMQ where the bus has no consumer. The concept is messages will be published and queued, then a separate microservice will consume messages from the queue. Looking at this SO answer, receive endpoints must be specified so that messages are actually queued. However this appears to contradict the common gotchas in the MassTransit docs, which states If you need to only send or publish messages, don’t create any receive endpoints.
Here is some sample code:
public class Program
{
static void Main(string[] args)
{
var bus = BusConfigurator.ConfigureBus();
bus.Start();
bus.Publish<IItemToQueue>(new ItemToQueue { Text = "Hello World" }).Wait();
Console.ReadKey();
bus.Stop();
}
}
public static class BusConfigurator
{
public static IBusControl ConfigureBus()
{
var bus = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
var host = cfg.Host(new Uri("rabbitmq://localhost/"), hst =>
{
hst.Username("guest");
hst.Password("guest");
});
cfg.ReceiveEndpoint(host, "queuename", e =>
{
e.Consumer<MyConsumer>();
});
});
return bus;
}
}
public interface IItemToQueue
{
string Text { get; set; }
}
public class ItemToQueue : IItemToQueue
{
public string Text { get; set; }
}
public class MyConsumer : IConsumer<IItemToQueue>
{
public async Task Consume(ConsumeContext<IItemToQueue> context)
{
await Console.Out.WriteLineAsync(context.Message.Text);
}
}
In this sample, I receive the message in the RabbitMQ queue as expected, and this is consumed by MyConsumer which writes Hello World to the console and the message is then removed from the Queue.
However, when I remove the following code from the above and re-run the sample:
cfg.ReceiveEndpoint(host, RabbitMqConstants.ValidationQueue, e =>
{
e.Consumer<MyConsumer>();
});
A temporary queue is created (with a generated name) and the message never seems to be placed into the temporary queue. This queue is then removed when the bus is stopped.
The problem I have is with a ReceiveEndpoint specified, the messages will be consumed and removed from the queue in the publisher program (meaning the consumer microservice wouldn't process queued items). Without a RecieveEndpoint specified, a temporary queue is used (and the consumer microservice would not know the name of this temporary queue), the message never seems to get queued and the queue is deleted when the bus is stopped which wouldn't be good if the program went down.
There is an example of a send only bus in the MassTransit docs but it is pretty basic so I was wondering if anyone had any suggestions?
The receive endpoint should be in your service, separate from the publish-only application. That way, the service will have the receive endpoint and consume the messages as they are published by the application.
If you have the receive endpoint in the application, the application will consume the messages since it has the same queue name specified in the receive endpoint.
What you need to do is create another service, with the same configuration (including the receive endpoint) - and take the receive endpoint out of your application. At that point, the service will have the receive endpoint and consume the messages from the queue. Even if the service is stopped, the messages will continue to be delivered to the queue and once the service is started they will begin consuming.
Related
I have my controller publishing events to azure service bus via masstransit.
And multiple instanses of .net core service consuming those event.I want ALL instances to consume the same events.
await _publishEndpoint.Publish<MyPublishEvent>(
new MyPublishEvent
{
Id = 1,
Description = "test"
});
here is one consumer from one microservice.
public async Task Consume(ConsumeContext<MyPublishEvent> context)
{
try
{
// Here add business logic to insert record in to Database 1.
}
await Task.CompletedTask;
}
and the configuration is here.
builder.Services.AddMassTransit(cfg =>
{
cfg.SetKebabCaseEndpointNameFormatter();
cfg.AddConsumersFromNamespaceContaining<Consumers>();
cfg.UsingAzureServiceBus((context, cfg) =>
{
cfg.Host($"Endpoint = endpoint");
cfg.ConfigureEndpoints(context);
});
});
I want ALL instances to consume the same events.
The first one consume the event from one microservice and the rest of the service instances don't get it to consume it. can this be fixed. Am I missing any configuration?Appreciate your help in finding the cause.
I'm currently trying to update application that was originally .NET Core 3.1 using MassTransit 6.3.2. It is now configured to use .NET 6.0 and MassTransit 7.3.0
Our application uses MassTransit to send messages via Azure Service Bus, publishing messages to Topics, which then have other Subscribers listening to those Topic.
Cut down, it was implemented like so:
// Program.cs
services.AddMassTransit(config =>
{
config.AddConsumer<AppointmentBookedMessageConsumer>();
config.AddBus(BusControlFactory.ConfigureAzureServiceBus);
});
// BusControlFactory.cs
public static class BusControlFactory
{
public static IBusControl ConfigureAzureServiceBus(IRegistrationContext<IServiceProvider> context)
{
var config = context.Container.GetService<AppConfiguration>();
var azureServiceBus = Bus.Factory.CreateUsingAzureServiceBus(busFactoryConfig =>
{
busFactoryConfig.Host("Endpoint=sb://REDACTED-queues.servicebus.windows.net/;SharedAccessKeyName=MyMessageQueuing;SharedAccessKey=MyKeyGoesHere");
busFactoryConfig.Message<AppointmentBookedMessage>(m => m.SetEntityName("appointment-booked"));
busFactoryConfig.SubscriptionEndpoint<AppointmentBookedMessage>(
"my-subscriber-name",
configurator =>
{
configurator.UseMessageRetry(r => r.Interval(5, TimeSpan.FromSeconds(60)));
configurator.Consumer<AppointmentBookedMessageConsumer>(context.Container);
});
return azureServiceBus;
}
}
}
It has now been changed and upgraded to the latest MassTransit and is implemented like:
// Program.cs
services.AddMassTransit(config =>
{
config.AddConsumer<AppointmentBookedMessageConsumer, AppointmentBookedMessageConsumerDefinition>();
config.UsingAzureServiceBus((context, cfg) =>
{
cfg.Host("Endpoint=sb://REDACTED-queues.servicebus.windows.net/;SharedAccessKeyName=MyMessageQueuing;SharedAccessKey=MyKeyGoesHere");
cfg.Message<AppointmentBookedMessage>(m => m.SetEntityName("appointment-booked"));
cfg.ConfigureEndpoints(context);
});
// AppointmentBookedMessageConsumerDefinition.cs
public class AppointmentBookedMessageConsumerDefinition: ConsumerDefinition<AppointmentBookedMessageConsumer>
{
public AppointmentBookedMessageConsumerDefinition()
{
EndpointName = "testharness.subscriber";
}
protected override void ConfigureConsumer(IReceiveEndpointConfigurator endpointConfigurator, IConsumerConfigurator<AppointmentBookedMessageConsumer> consumerConfigurator)
{
endpointConfigurator.UseMessageRetry(r => r.Interval(5, TimeSpan.FromSeconds(60)));
}
}
The issue if it can be considered one, is that I can't bind to a subscription that already exists.
In the example above, you can see that the EndpointName is set as "testharness.subscriber". There was already a subscription to the Topic "appointment-booked" from prior to me upgrading. However, when the application runs, it does not error, but it receives no messages.
If I change the EndpointName to "testharness.subscriber2". Another subscriber appears in the Azure Service Bus topic (via the Azure Portal) and I start receiving messages. I can see no difference in the names (other than the change that I placed, in this case: the "2" suffix).
Am I missing something here? Is there something else I need to do to get these to bind? Is my configuration wrong? Was it wrong? While I'm sure I can get around this by managing the release more closely and removing unneeded queues once they're using new ones - it feels like the wrong approach.
With Azure Service Bus, ForwardTo on a subscription can be a bit opaque.
While the subscription may indeed visually indicate that it is forwarding to the correctly named queue, it might be that the queue was deleted and recreated at some point without deleting the subscription. This results in a subscription that will build up messages, as it is unable to forward them to a queue that no longer exists.
Why? Internally, a subscription maintains the ForwardTo as an object id, which after the queue is deleted points to an object that doesn't exist – resulting in messages building up in the subscription.
If you have messages in the subscription, you may need to go into the portal and update that subscription to point to the new queue (even though it has the same name), at which point the messages should flow through to the queue.
If there aren't any messages in the subscription (or if they aren't important), you can just delete the subscription and it will be recreated by MassTransit when you restart the bus.
I have Windows service written in C#. Earlier we were using Event hubs with multiple partitions for message queuing. We recently moved to Kafka. For implementing Event hubs in c# , we have IEventProcessor.ProcessEventsAsync , which keeps listening to event hub notifications and is triggered whenever a message is posted to event hub , which runs asynchronously in the background
I did not find any equivalent method in Kafka.
My requirement here is to subscribe to a Kafka topic and continuously consume messages. When a message is consumed, some other operations are also supposed to executed for that message. For each message say the execution time takes around 15 mins, I want the Kafka consumer to consume all messages and keep it in queue as when it receives and writes it into a file. Other process should read the file, pick the message and do other operations. I want all of them to run simultaneously/parallelly.
PS : I have written a console application which can produce and consume one message.What I'm looking for is queuing and parallelism.
For paralellism Kafka implements what's known as consumer groups. Kafka stores the "offsets" (read: key of record across a topic) and also stores the offsets of where a given consumer group is also at in processing the records. This should allow you to create new consumer instances on the fly using the same program, and by changing the group allow two programs to consume the same data in paralell for different tasks.
I found this link helpful when I was creating my first consumer as well, in case you found a way to create it without a groupId: http://cloudurable.com/blog/kafka-tutorial-kafka-consumer/index.html
Hope this helps!
Have look at Silverback: https://silverback-messaging.net. It abstracts many of those concerns and the basic usage is as simple as this:
public class Startup
{
public void ConfigureServices(IServiceCollection services)
{
services
.AddSilverback()
.WithConnectionToMessageBroker(options => options.AddKafka())
.AddKafkaEndpoints(
endpoints => endpoints
.Configure(
config =>
{
config.BootstrapServers = "localhost:9092";
})
.AddInbound(
endpoint => endpoint
.ConsumeFrom("my-topic")
.DeserializeJson(serializer => serializer.UseFixedType<SomeMessage>())
.Configure(
config =>
{
config.GroupId = "test-consumer-group";
config.AutoOffsetReset = AutoOffsetReset.Earliest;
})))
.AddSingletonSubscriber<MySubscriber>();
}
}
public class MySubscriber
{
public Task OnMessageReceived(SomeMessage message)
{
// TODO: process message
}
}
I have a .NET project that needs to read messaged from a given Queue.
I have several producers writing the same type of message into the queue.
I want my consumer app to have several threads reading messages and handling them so that the load will not be on a single thread.
Any ideas or sample code on how to achieve this?
Again, Note:
Each message should be processed once and not several times. The work should be balanced between the worker threads
You are going to need a bit of plumbing to get that done.
I have an open-source service bus called Shuttle.Esb and there are many other service bus options available that you may wish to consider.
But if you do not want to go that route you could still have a look at some of the code and implementations to get some ideas. I have a RabbitMQ implementation that may be of assistance.
Take a look at masstransit project : http://masstransit-project.com/MassTransit/usage/message-consumers.html
It has configurations like prefetch count and concurrency limit. It brings you to consume messages paralelly.
Also it is very simple to setup:
IBusControl busControl = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
IRabbitMqHost host = cfg.Host(new Uri(RabbitMQConstants.RabbitMQUri),
hst =>
{
hst.Username(RabbitMQConstants.RabbitMQUserName);
hst.Password(RabbitMQConstants.RabbitMQPassword);
});
cfg.ReceiveEndpoint(host,
RabbitMQConstants.YourQueueName,
endPointConfigurator => {
endPointConfigurator.Consumer<SomeConsumer>();
endPointConfigurator.UseConcurrencyLimit(4);
});
});
busControl.Start();
public class SomeConsumer :
IConsumer<YourMessageClass>
{
public async Task Consume(ConsumeContext<YourMessageClass> context)
{
await Console.Out.WriteLineAsync($"Message consumed: {context.Message.YourValue}");
}
}
I don't have very much experience using MSMQ and someone recommended I look at MassTransit to help implement a solution but I am having a hard time trying to figure out if using MassTransit + MSMQ is the right tool for the job.
We have a WPF application (3.5) that is used by multiple users. Persistence is done from the application (via NHibernate) to the database. Up until now, users would periodically refresh there view's in order to ensure they had the latest updates. However, we now want to send notification to each application instance when an entity is persisted using pub/sub messaging. The client applications are all run within the same domain and should be able to fulfill most dependencies required (e.g. installation of MSMQ on client machines).
To summarize: Client1 publishes an update message ---> ????? ----> All other active clients receive it.
As I am new to MSMQ, I'm not even sure what the architecture should look like.
Does each client machine need to have a local MSMQ queue to receive messages?
Do we just need to create a queue on a server and all clients listen for messages there? If so, will just a queue(s) suffice or do we need to create a service in order to distribute the messages correctly?
Is this even the right tool for the job?
I created a little POC hoping that it would work, but I ended up with what I think is termed "Competing Consumer". What I would like to happen is one application instance sends a message, and all application instances receive it.
Any suggestions, direction or advice would be greatly appreciated!
Here is the POC view model code (note - in my mind localhost would be replaced with a server that each app instance would send messages to):
Update: Added Network Key (kittens)
Update: I've uploaded the sample code https://docs.google.com/viewer?a=v&pid=explorer&chrome=true&srcid=0ByDMJXKmYB7zMjBmYzYwNDEtYzMwOC00Y2RhLTk1MDYtZjc0NTI2M2E3Y2Qy&hl=en_US
public class MainViewModel : IDisposable, INotifyPropertyChanged
{
private Guid id;
public MainViewModel()
{
id = Guid.NewGuid();
Publish = new RelayCommand(x => OnExecutePublishCommand(), x => !string.IsNullOrEmpty(Message));
Messages = new ObservableCollection<MessagePayload>();
Bus.Initialize(sbc =>
{
sbc.UseMsmq();
sbc.SetNetwork("Kittens");
sbc.VerifyMsmqConfiguration();
sbc.UseMulticastSubscriptionClient();
sbc.ReceiveFrom(string.Format("msmq://localhost/{0}", ConfigurationManager.AppSettings["queue"]));
sbc.Subscribe(subs => subs.Handler<MessagePayload>(OnReceiveMessage));
});
}
public ICommand Publish { get; private set; }
private string message;
public string Message
{
get { return message; }
set
{
message = value;
SendPropertyChanged("Message");
}
}
public ObservableCollection<MessagePayload> Messages { get; private set; }
private void OnReceiveMessage(MessagePayload msg)
{
Application.Current.Dispatcher.Invoke(DispatcherPriority.Background,
new Action(() => Messages.Add(msg)));
}
private void OnExecutePublishCommand()
{
Bus.Instance.Publish(new MessagePayload{ Sender= id, Message = Message});
Message = null;
}
public event PropertyChangedEventHandler PropertyChanged;
private void SendPropertyChanged(string propertyName)
{
if (PropertyChanged != null)
{
PropertyChanged(this, new PropertyChangedEventArgs(propertyName));
}
}
public void Dispose()
{
Bus.Instance.Dispose();
}
}
Update: Just in case anyone is interested we ended up splitting our "Event Bus" into two. For the server, we are using MassTransit. However, because Mass Transit requires "full profile" (.NET 4.0) and we wanted to stick with "client profile" for our WPF instances we are using SignalR for the client side event bus. An "observer" on the server event bus forwards messages to the client event bus.
All the machines on the same network can subscribe to given messages. They all need a local queue to read off of. Don't read off remote queues unless there's absolutely no other way.
What you described generally seems right. There's an message that gets published to all subscribers, they'll receive it and update their state. I have not worked with WPF in a while but generally how you're handling it seems acceptable. Note that it might take a little time to spin up the MT configuration, so you might want to do that on a background thread so you aren't blocking the UI.
Additionally, using the Multicast Subscription, you need to set a network key. It's automatically set to the machine name if not provided. You'll want to make sure they can talk to each other successfully.