I am creating a simple Publisher/Subscriber using MassTransit and RabbitMQ.
The Publisher has the following code to initialize the bus:
/** create the bus */
var bus = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
var host = cfg.Host(new Uri("rabbitmq://localhost/"), h =>
{
h.Username("guest");
h.Password("guest");
});
});
/** start the bus and publish */
bus.Start();
bus.Publish<IPersonLogin>(new {FirstName = "John", LastName = "Smith"});
And the Subscriber has this code for initialization:
var bus = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
var host = cfg.Host(new Uri("rabbitmq://localhost/"), h =>
{
h.Username("guest");
h.Password("guest");
});
cfg.ReceiveEndpoint(host, "person_login", e =>
{
e.Consumer<PersonLoginConsumer>();
});
});
If I shut-down the Subscriber and publish 2 messages, the messages are not lost and as soon as the Subscriber comes back to life the Messages are processed.
So my questions are:
How do I ensure that a Message stays in the Queue of RabbitMQ until one Subscriber comes up and pick it up?
What happen if the Server is reboot and some Messages were not processed by any Subscriber, do they get lost or do they get processed as soon as the Subscriber come alive after reboot?
Is this the correct pattern to ensure that every single message is processed or should I use a different strategy?
by default, any message sitting in a queue will remain there until one of three things happens:
the message is consumed
the message "time to live" expires (default is to live forever)
the server crashes or restarts
if you have a queue full of messages, the messages will generally stick around until one of those three things happens. hopefully you will have your consumers online soon enough that you can consume the messages and process them.
you would only set a time to live (ttl) if you want messages to be automatically deleted after a period of time (assuming they are not consumed first)
for crashes... a message can survive a crash / restart if you make the message persistent to disk. there's still a chance that the message will be lost if the server crashes before the message is routed from the exchange to the queue, though.
On top of mind.
If there arent any subscribers RabbitMQ wont know to which queue a message should be delivered. Then a message will be undeliverable.(Not sure if this will be moved to a error queue or skipped)
If the exchanges are already there it will be placed in the queue of consumer that has subscribed to the event. So you endpoint hosting your consumer can be down the message will still be delivered.
When the message is delivered to the queue the consumer will pick up your message and process it. If a exception occurs while processing your message it will be moved to the endpoint_error queue. (Depending on your RetryPolicy). Deploy a fix and move you message back in to the main queue and the messages will be processed as if nothing has happend.
Good read for common issues on common gotcha's
Under the Hood
Common Gotcha's
Related
I use Masstransit RabbitMQ. As a consumer, only consume events when they are published in the queue. In the above example, how can I get all the SubmitOrder as list from the queue with a single request(consume)?
var busControl = Bus.Factory.CreateUsingRabbitMq(cfg =>
{
cfg.ReceiveEndpoint("order-service", e =>
{
//here i want to get all SubmitOrder as list
e.Handler<SubmitOrder>(async context =>
{
await Console.Out.WriteLineAsync($"Submit Order Received: {context.Message.OrderId}");
});
});
});
In a job, I need to retrieve all messages from a queue, then use that list to process them.
I have a gateway that publishes messages in a queue, and then I have a job that receives the messages from the queue and processes them.
That isn't generally how messaging works. With MassTransit, you can have messages delivered individually, so that each is consumed atomically. Or you can use a batch consumer to have multiple messages delivered at once.
Thinking that you can just "get all the messages" from a queue is not the way one should ever think, there could literally be millions and millions of messages in the queue. And you definitely would not process multiple SubmitOrder style messages as a batch anyway. Architecturally, it's just a really bad idea.
RabbitMq 3.8.5, C# RabbitMqClient v6.1.0, .Net Core 3.1
I feel that I'm misunderstanding something with RabbitMq so I'm looking for clarification:
If I have a client sending a message to an exchange, and there's no consumer on the other side, what is meant to happen?
I had thought that it should sit in a queue until it's picked up, but the issue I've got is that, right now there is no queue on the other end of the exchange (which may well be my issue).
This is my declaration code:
channel.ExchangeDeclare(name, exchangeType, durable, autoDelete);
var queueName = ret._channel.QueueDeclare().QueueName;
channel.ConfirmSelect();
and this is my publisher:
channel.BasicPublish(exchangeName, routingKeyOrTopicName, messageProperties, message);
However doing that gives me one queue name for the outbound exchange, and another for the inbound consumer.
Would someone help this poor idiot out in understanding how this is meant to work? What is the expected behavior if there's no consumer at the other end? I do have an RPC mechanism that does work, but wasn't sure if that's the right way to handle this, or not.
Everything works find if I have my consumer running first, however if I fire up my Consumer after the client, then the messages are lost.
Edit
To further clarify, I've set up a simple RPC type test; I've two Direct Exchanges on the client side, one for the outbound Exchange, and another for the inbound RPC consumer.
Both those have their own queue.
Exchange queue name = amq.gen-fp-J9-TQxOJ7NpePEnIcGQ
Consumer queue name = amq.gen-wDFEJ269QcMsHMbAz-t3uw
When the Consumer app fires up, it declares its own Direct exchange and its own queue.
Consumer queue name = amq.gen-o-1O2uSczjXQDihTbkgeqA
If I do it that way though, the message gets lost.
If I fire up the consumer first then I still get three queues in total, but the messages are handled correctly.
This is the code I use to send my RPC message:
messageProperties.ReplyTo = _rpcResponder._routingKeyOrTopicName;
messageProperties.Type = "rpc";
messageProperties.Priority = priority;
messageProperties.Persistent = persistent;
messageProperties.Headers = headers;
messageProperties.Expiration = "3600000";
Looking at the management GUI, I see that all three queues end up being marked as Exclusive, but I'm not declaring them as such. In fact, I'm not creating any queues myself, rather letting the Client library handle that for me, for example, this is how I define my Consumer:
channel.ExchangeDeclare(name, exchangeType, durable, autoDelete);
var queueName = ret._channel.QueueDeclare().QueueName;
Console.WriteLine($"Consumer queue name = {queueName}");
channel.QueueBind(ret.QueueName, name, routingKeyOrTopicName, new Dictionary<string, object>());
In RabbitMQ, messages stay in queues, but they are published to exchanges. The way to link an exchange to a queue is through bindings (there are some default bindings).
If there are no queues, or the exchange's policy doesn't find any queue to forward the message, the message is lost.
Once a message is in a queue, the message is sent to one of that queue's consumers.
Maybe you're using exclusive queues? These queues get deleted when their declaring connection is gone.
Found the issue: I was allowing the library to generate the queue names rather than using specific ones. This meant that RabbitMq was always having to deal with a shifting target each time.
If I use 'well defined' queue names AND the consumer has fired up at least once to define the queue on RabbitMq, then I do see the message being dropped into the queue and stay there, even though the consumer isn't running.
Using Azure ServiceBus and the OnMessage call I am looking for a way to determine if the OnMessage event pump stops reading from the queue.
Our connection to OnMessage is configured as below:
protected virtual void DoSubscription(string queueName, Func<QueueRequest, bool> callback)
{
var client = GetClient(queueName, PollingTimeout);
var transformCallback = new Action<BrokeredMessage>((message) =>
{
try
{
var request = message.ToQueueRequest();
if (callback(request))
{
message.Complete();
}
else
{
message.Abandon();
Log.Warn("DoSubscription: Message Failed to Process Gracefully: {0}{1}", Environment.NewLine, JsonConvert.SerializeObject(request));
}
}
catch (Exception ex)
{
Log.Error("DoSubscription: Message Failed to Process With Exception:", ex);
message.Abandon();
}
});
var options = new OnMessageOptions
{
MaxConcurrentCalls = _config.GetInt("MaxThreadsPerQueue"),
AutoComplete = false,
AutoRenewTimeout = new TimeSpan(0,0,1)
};
options.ExceptionReceived += OnMessageError;
client.OnMessage(transformCallback, options);
}
The problem we are encountering is after a period of time with no messages being queued new messages that are queued fail to be picked up by the OnMessage event pump until the application is restarted.
I realize there are ways to do this using Worker Roles, however for monitoring and management purposes we decided to implement this in the Application Start of a web app.
So after a call with Microsoft's Azure support team there is not an event to trap when OnMessage or OnMessageAsync errors. As these are not blocking calls, it starts the event pump and returns to the executing thread, this creates a challenge to determine if OnMessage* is doing it's job.
Suggestions from Microsoft were:
Create your own implementation of the QueueClientBase class which exposes the OnClose, On* methods which you could handle. However, in doing this you have to do the handling of the message envelope yourself.
Use OnReceive in a separate thread loop, which you can trap errors yourself and immediately retry.
However, I did explore some bullet proofing against OnMessage and discovered a few things which have eased my fears.
OnMessage is incredibly fault tolerant
I uplugged the ethernet cable from my laptop with wireless turned off, this broke the OnMessage's connection to the Service Bus queue. After waiting 10 minutes I plugged the ethernet cable back in and the OnMessage immediately began processing queued elements.
On Message surprisingly is fairly stable. It has been running inside the global.asax.cs App Start, to abbreviate, for days on end with a Factory IdleTimeout set to 24 hours without restarting the web application for 72 hours.
All-in-all I'm going to continue using OnMessage/OnMessageAsync for now and keep an eye on it. I will update this if I see issues that change my opinion of OnMessage.
Aside - Make sure if you are using OnMessage for permanent listening in an Azure Web Site that you set the "Always On" configuration option to "On". Otherwise, unless a web request comes in OnMessage will be disposed and messages will no longer be processed until the web application is reawakened by a HTTP request.
I'm using MassTransit + MSMQ as a message passing bus, which seems to be having reasonable success. However, for some tests I want to enqueue messages but never dequeue them. It seems like the right way to do this is to not subscribe to the queue directly. Here is my code:
1) I want to send and receive messages from the same queue in this process [this works]:
var solrMessageBus = ServiceBusFactory.New(sbc =>
{
sbc.UseMsmq();
sbc.VerifyMsmqConfiguration();
sbc.ReceiveFrom("msmq://localhost/my_queue");
sbc.Subscribe(subs =>
{
subs.Handler<MyMessage>(msg => Enqueue(msg));
});
});
2) I want to send messages from this process, but not consume them. MSMQ should build up a large queue of messages [this does not work]
var solrMessageBus = ServiceBusFactory.New(sbc =>
{
sbc.UseMsmq();
sbc.VerifyMsmqConfiguration();
sbc.ReceiveFrom("msmq://localhost/my_queue");
});
I'm not a MassTransit expert, but the above seems like a reasonable way to enqueue without dequeuing messages from that same queue. In 1), I see messages end up in my MSMQ, but in 2) no messages ever get to the queue.
How can I build up the queue without dequeuing the messages?
If you do not register any subscriptions on the bus, the queue will be emptied and all of the message sent to the queue will end up in the _error queue.
If you need to just send messages to a queue, you can use an EndpointCacheFactory (instead of a service bus factory) to get an IEndpointCache, then call GetEndpoint(uri) and use the Send method to send messages to that queue. This has the added benefit of avoiding any thread pool usage for receiving messages that are never consumed.
Also, a quick reminder, every service bus instance must have its own queue.
That sounds reasonable however I've never tried it.
Mass Transit builds the subscription mapping out of your setup and then maps subscriptions to queues (using multicast subscription). Note that messages are never stored in queues assigned to senders, rather they are multicasted to subscribers. No subscribers = nowhere to put your messages.
To queue messages forever, I would add a subscriber but pause its consumer thread until tests are completed.
My application's bottleneck has become sending and receiving messages over MSMQ with MassTransit. The send and receive are both happening within the same application, but there are often too many messages to use an in-memory queue.
Here is my simple queue setup:
messageBus = ServiceBusFactory.New(sbc =>
{
sbc.UseMsmq();
sbc.VerifyMsmqConfiguration();
sbc.UseMulticastSubscriptionClient();
sbc.ReceiveFrom("msmq://localhost/msg_queue");
sbc.Subscribe(subs =>
{
subs.Handler<EventMessage>(msg => Enqueue(msg));
});
});
For my experiments, MSMQ currently has ~1 million messages and we are not pushing any new messages to it. We are not doing any work in Enqueue() except time how quickly messages are being sent.
With that information, we can only grab between 150 and 200 messages per second from MSMQ, when I'd hoped it would be at least 1000 messages per second when there is no network latency. Each messages is <2kb.
How can we speed up how quickly MSMQ passes messages to an application over MassTransit while maintaining the message ordering enforced by a queue?
I did addressed something similar before. If I remember it correctly,
We specified message expiration through TimeToBeReceived (The total time for a sent message to be received from the destination queue. The default is InfiniteTimeout.) . Refer this msdn link.