I have a RabbitMQ queue in which I post thousands of messages. I need a background Service that will:
Retrieve the messages in batches of 100 messages
Store those 100 messages in a Database
Ack all 100 messages at once
Proceed with the next batch of 100 messages
I'm using the RabbitMQ Client to listen for messages, but I don't know how to "batch" them.
Does anyone has a working example on how to get my messagess 100-100 at a time and ACK them all at once after they have been saved in a DB?
Thanx a lot in advance
You want to use the "prefetch" and "multi ack" features -
Prefetch - https://www.rabbitmq.com/consumer-prefetch.html
Multi-ack - https://www.rabbitmq.com/confirms.html#consumer-acks-multiple-parameter
This is what you'll do in your code:
Open a connection and channel
Set the channel prefetch to 100
Start a consumer. The callback you register for your consumer will start being called with messages. You will have to save these messages in a list or other data structure
Once you have received 100 messages, save the delivery tag of the 100th message and do your database work.
Ack the 100 messages by setting "multi ack" to true and use the delivery tag of the 100th message.
RabbitMQ will send the next 100 messages in the same manner
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
Related
I have a message host that executes messages from asb queue. I want to create a new message host (scale up)(.NET core) when it first message host's load is up. In order to do so, I want to send some messages from one queue to another queue and then that (new) ,message host reads it from new asb queue. How is this possible?
There are 2 ways by which you can accomplish this:
Auto-forwarding: You can configure auto-forwarding on the 1st queue to forward messages to the 2nd queue automatically. In this case, as soon as the message arrives in the 1st queue, it will be forwarded to the 2nd queue. It will remove the message from the 1st queue though and both queues must be the part of same Namespace. You can learn more about it here: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-auto-forwarding.
Azure Service Bus Triggered Function with Azure Service Bus Output Binding: In this case, the Function will be triggered when a message arrives in the queue. Because of output binding, this Function will send the message to another queue. You can learn more about it here: https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus?tabs=in-process%2Cextensionv5%2Cextensionv3&pivots=programming-language-csharp.
I have a couple of queues where certain information is queued. Let us say I have "success" and "failed" queues in which Server side component has continuously written some data to these queues for clients.
Clients read this data and display it on a UI for end users. Now, I have a situation to purge any message in these queues older than 30 days. Clients would then only be able to see only 30 days of information at any point of time.
I have searched a lot and could see some command line options to purge whole queue but could not find a relevant suggestion.
Any help in the right direction is appreciated. Thanks
I don't think this is possible; looks like you're trying to use RabbitMq as data storage instead of message server.
The only way to understand if a message is "older" than 30, is to process the message, and by doing this you are removing the messagge from the queue.
Best thing to do here is to process the messages and store them in a long term storage; then you can implement a deletion policy to eliminate the older elements.
If you really want to go down this path, RabbitMQ implements TTL at queue level or message level; take a look at this: https://www.rabbitmq.com/ttl.html
[As discussed in comments]
To keep the message in the queue you can try to use a NACK instead of ACK as confirmation; this way RabbitMQ will consider the message undelivered and it will try to deliver it again and again. Remember to create a durable queue (https://www.rabbitmq.com/confirms.html).
You can also check this answer: Rabbitmq Ack or Nack, leaving messages on the queue
I am attempting to prove that MassTransit delivers messages in the same order (FIFO) that rabbitmq receives them. So far, I am not having luck. MT seems to randomly deliver messages out of a queue. I have tried setting both of these bus configuration options to 1:
SetConcurrentReceiverLimit()
SetConcurrentConsumerLimit()
...seems to make no difference.
How do I ensure FIFO delivery via MassTransit?
If you set the ConcurrentConsumerLimit to 1 (receiver limit is 1 by default), and set the prefetch=1 on the URI, it should be in order FIFO delivery assuming no consumer exceptions are thrown. Honestly, even with a prefetch > 1 (which is important for performance reasons) it should be in-order.
Also, if you're doing this with some sample code, post the sample code and make sure that your producer and consumer processes are listening to separate queues.
x.ReceiveFrom(uri) // uri should be unique per bus instance
There's also a prefetch you need to set with RabbitMQ. MassTransit capping message rates at 10 shows an example of using prefetch configuration on the queue URI.
I am sending a message with AcknowledgeTypes.NegativeReceive with a time to be received set so that if the message isn't received off the destination queue within that time frame, a negative receive message is sent to my administration queue. This works fine. The problem is that I want to know which queue the message couldn't be received from.
When I check the message of the negative acknowledgement, there's no hint of the original queue to which the message was sent.
What I'm trying to accomplish is to treat the negative acknowledgement as an event to divert message traffic to a different queue but I need to know the queue to divert away from and that appears not to be retained in the negative acknowledgement message by default.
Is there a way to get the original destination queue out of an acknowledgment message?
UPDATE
What appears to be happening is the DestinationQueue.Path in the acknowledgement message is set to the Outgoing queue on the target machine which is some GUID\ID, e.g.
I found it. It is the ResponseQueue property of the acknowledgment message. Overlooked that.
My application's bottleneck has become sending and receiving messages over MSMQ with MassTransit. The send and receive are both happening within the same application, but there are often too many messages to use an in-memory queue.
Here is my simple queue setup:
messageBus = ServiceBusFactory.New(sbc =>
{
sbc.UseMsmq();
sbc.VerifyMsmqConfiguration();
sbc.UseMulticastSubscriptionClient();
sbc.ReceiveFrom("msmq://localhost/msg_queue");
sbc.Subscribe(subs =>
{
subs.Handler<EventMessage>(msg => Enqueue(msg));
});
});
For my experiments, MSMQ currently has ~1 million messages and we are not pushing any new messages to it. We are not doing any work in Enqueue() except time how quickly messages are being sent.
With that information, we can only grab between 150 and 200 messages per second from MSMQ, when I'd hoped it would be at least 1000 messages per second when there is no network latency. Each messages is <2kb.
How can we speed up how quickly MSMQ passes messages to an application over MassTransit while maintaining the message ordering enforced by a queue?
I did addressed something similar before. If I remember it correctly,
We specified message expiration through TimeToBeReceived (The total time for a sent message to be received from the destination queue. The default is InfiniteTimeout.) . Refer this msdn link.