Say I have a connection to rabbit, and I've pulled 1000 messages, but have not yet ack'd them, as they are being processed by a single thread out of a Blocking collection.
Now suppose my connection dies and is auto recovered. At this point all of these msgs on the server will be re queued for delivery. But I still have copies of them locally, with the old Delivery tag.
This leads me to believe I should handle connection or channel down events by clearing my local queue out.
Can you confirm this is true?
Yes that is the case. Those messages will be redelivered.
So in addition to clearing our your locally queued messages, you might want to consider your prefetch so that you don't have so many messages queued locally.
Is your strategy is to pull 1000, process them all, then finally ack them all? I can see that due to performance reasons you might do this so you can send a single ack with multiple=true, but it does introduce extra redelivery and duplicate processing risks.
You are right.If you are processing one message at a time you can set prefetch count as 1 and you may not need to clear any messages locally,too.
Related
We are running multiple instances of a windows service that reads messages from a Topic, runs a report, then converts the results into a PDF and emails them to a user. In case of exceptions we simply log the exception and move on.
The use case we want to handle is when the service is shut down we want to preserve the jobs that are currently running so they can be reprocessed by another instance of the service or when the service is restarted.
Is there a way of requeueing a message? The hacky solution would be to just republish the message from the consuming service, but there must be another way.
When incoming messages are processed, their data is put in an internal queue structure (not a message queue) and processed in batches of parallel threads, so the IbmMq transaction stuff seems hard to implement. Is that what I should be using though?
Your requirement seems to be hard to implement if you don't get rid of the "internal queue structure (not a message queue)" if this is not based on a transaction oriented middleware. The MQ queue / topic works well for multi-threaded consumers, so it is not apparent what you gain from this intermediate step of moving the data to just another queue. If you start your transaction with consuming the message from MQ, you can have it being rolled back when something goes wrong.
If I understood your use case correctly, you can use Durable subscriptions:
Durable subscriptions continue to exist when a subscribing application's connection to the queue manager is closed.
The details are explained in DEFINE SUB (create a durable subscription). Example:
DEFINE QLOCAL(THE.REPORTING.QUEUE) REPLACE DEFPSIST(YES)
DEFINE TOPIC(THE.REPORTING.TOPIC) REPLACE +
TOPICSTR('/Path/To/My/Interesting/Thing') DEFPSIST(YES) DURSUB(YES)
DEFINE SUB(THE.REPORTING.SUB) REPLACE +
TOPICOBJ(THE.REPORTING.TOPIC) DEST(THE.REPORTING.QUEUE)
Your service instances can consume now from THE.REPORTING.QUEUE.
While I readily admit that my knowledge is shaky, from what I understood from IBM’s [sketchy, inadequate, obtuse] documentation there really is no good built in solution. With transactions the Queue Manager assumes all is well unless it receives a roll back request and when it does it rolls back to a syncpoint, so if you’re trying to roll back to one message but two other messages have completed in the meantime it will roll back all three.
We ended up coding our own solution updating the way we’re logging messages and marking them as completed in the DB. Then on both startup and shutdown we find the uncompleted messages and programmatically publish them back to the queue, limiting the DB search by machine name so if we have multiple instances of the service running they won’t duplicate message processing.
We use rabbit mq to send messages to a server for processing.
We require the server to ack a message. That way if the server happens to die whilst processing the message, we will retry the message when it restarts, or with a different server.
The problem is, on a very rare occasion, we will get a message that deterministically crashes the server. This is because we call into some open source native dlls, those dlls have bugs, and sometimes these dlls just cause the process to crash with no exception. Of course it would be ideal to fix those bugs, but we don't expect to fix all such issues in pdfium or opencv any time soon. We have to reckon with the fact that whatever we do, we will eventually get such a message.
The result of this is that the message is then retried, the server restarts, picks ups the message, crashes, and so on ad infinitum. Nothing gets processed till we manually stop the server, and purge the message. Not ideal.
What can we do to solve this problem?
What we don't want to do is create another service that monitors the rabbitmq service, looks for such messages and purges them, since that just leads to spiralling complexity. Instead we want to deal with this at the rabbitmq client level. We would be perfectly happy to say that if a message is not processed 3 times, we should just fail the message. We could do this by maintaining a database entry of which messages we've processed, but ideally I wouldn't want to involve anything external, and just contain the solution to this problem in our rabbitmq client library. I'm not sure how to do this though.
One method I have used in my event driven architecture is to use dead letter exchanges (DLXs) or poison queues, that way if we see the same message multiple times due to service failure then it'll be pushed into the DLX instead of being re-queued into the original exchange. These messages then trigger a different type of process within our system to alert us messages are stuck and failing to process, we can then diagnose and fix the consumer. After a fix has been made we trigger another process to move the poison messages back into the original exchange to be then processed as normal.
In your scenario because your process crashes there is two possible options to deal with these messages:
If the message is marked as redelivered then clone the message and add an attempt count to the body or as a header (x-attempt-count) to the message. The copy will then be added to the back of the queue with the attempt count. When the copy is then consumed you can check if it hits the threshold and then move the message into a DLX or store in a database. The major drawback here is that it breaks the order of which the messages are processed.
Use an external services to keep track of the number of delivery attempts, I would recommend using something like redis/memcache where you can increment a counter based on a unique message id. At the start of your process if the message has been marked as redelivered then lookup the counter. If the message has reached the threshold, trigger a different process again like moving it into a DLX.
Here is what I am trying to do:
Dequeue a message
Do an action with the message
If the action fails, put the message back in the queue
If the action succeeds, acknowledge the message
My problem right now is that, if the action fails, the message isn't re-queued, but stays unacknowledged. If I go in RabbitMQ web configuration interface, I see that the messages are flagged as unacknowledged, even though the basic.Nack has been stepped over.
var delivery = subscription.Next();
var messageBody = delivery.Body;
try
{
action.Invoke(messageBody);
subscription.Ack(delivery);
}
catch (Exception ex)
{
subscription.Model.BasicNack(delivery.DeliveryTag, false, true);
throw ex;
}
Update:
So I've noticed that Messages go from Ready to Unacknowledged really fast. A rate way faster then I'm actually calling subscriber.Next(), as if the the .Net client caches all the messages in memory (the memory foot print of my app is actually growing quite fast), and processes those messages from memory and sends the Ack() afterwards, unflagging the message from Unacknowledged.
Update 2:
Seems like the queue being emptied really fast was because I hadn't set BasicQos on my Model. The following fixed everything. Basic.Nack() still doesn't seem to work tho:
Model.BasicQos(0, 1, false)
I suspect you're using:
channel.BasicConsume(your_queue_name, false, consumer); to retrieve messages.
I ran several tests with a RabbitMQ 3.2.4 server and client. I was unable to get either channel.BasickAck(...) or channel.BasicNack(...) to work as expected.
That said, I was able to get the expected Ack | Nack behavior when I used:
BasicGetResult result = channel.BasicGet(your_queue_name, false);
So you may want to consider a different retrieval method to get messages. I realize that the Consume & Dequeue are the "preferred" methods but they weren't working in my case. I wanted fair, one-at-a-time dispatch with acknowledgments. Using BasicGet was the only way I could achieve that.
The downside to that approach is you'll possibly lose the client side event iterator you're using with subscription.Next().
If I had to venture a guess, I think that something about the local Queue collection is messing up the channel's ability to provide an acknowledgement. And it's worth pointing out that creating the consumer with new QueueingBasicConsumer(channel); triggers a call to pre-fetch events from the server's queue. The consumer's Queue is just a SharedQueue<RabbitMQ.Client.Events.BasicDeliverEventArgs> and SharedQueue is just an extension of IEnumerable.
Also keep in mind that the same channel that pulls the message needs to provide the Ack | Nack. You cannot Ack | Nack a message from a different channel. Or at least I haven't figured out how to do so, nor have others. That's a problem if you wrap your RabbitMQ objects within using statements (so you don't leave network resources laying around) and you have long process to run before you can safely acknowledge.
This SO Answer lays out a decent workflow to get around the likely reality that your pulling channel is not going to be the channel that sends the Ack | Nack. The trick is setting a TTL and not bothering with sending a Nack - just let the new message expire and requeue automatically.
I have code that sends a message into a MessageQueue.
_queue.Send(new Message(myData));
This queue is on the local machine and the threads that Receive() from it are in the same process.
What happens if the messages are inserted faster than they are extracted? Will Send() block?
Is there a way for me to know if the MessageQueue is full before sending into it more events?
(I would prefer at this point just to log myData and not send the event).
Thanks,
Sela.
Short answer: Do the simple thing and don't limit on send.
Long answer:
The message queue will only really get full when the disk it's allocated to save to is out of space - which is the same time that your logging will be out of space. The message queue is very good at holding data you're not ready to process. Don't throttle on send. If you're concerned about system management and disk space then you might prefer to rely on Window's excellent system monitoring facilities and disk space usage threshold alerts. You don't need to reinvent this for your application.
That is unless you're running the queue in memory only mode which may not be necessary. If you can't process the messages fast enough then you definitely have enough time to let the queue manager persist the messages to disk. You should only consider running the queue in memory only mode if you're going to scale to many consumer processes on many servers and the disk IO on the queue manager becomes the bottleneck. One process on the same machine is very far away from that scenario. Let the queue manager do what it does best. Don't optimise prematurely.
If you implement a specified quality of service like X messages per second and bill your customers more for processing a higher quality of service then throttle at the receiving end. I've done this successfully using a semaphore initialised with a resource limit equal to the number of messages to consume per second. Each consumer thread took a snapshot of the message start time, processed 1 message and then waited for the end of second before giving up the semaphore. That way the thread pool could grow to accomodate the quality of service if messages took more than 1 second to process but would not exceed the quality of service.
Good luck!
Designing a system so that messages are produced no faster than they are consumed is good and I agree with that. Nevertheless it might happen that a producer of messages would hit a wall because queue is overfilled especially when there are low quotas set.
To prepare for such a situation you need to monitor if Send() method succeeds. If you send a message into a full queue the message is lost and because Send() returns void there is no immediate result of success or failure. But there is a way to detect that. You should request an acknowledgements when working with MSMQ. To receive them you need to use an administrative queue. This way you can be notified about different things that happen including queue being full.
Message msg = new Message
{
Formatter = new BinaryMessageFormatter(),
Body = data,
AdministrationQueue = this.adminQueue,
AcknowledgeType = AcknowledgeTypes.FullReachQueue
};
this.queue.Send(msg);
Message admMsg = this.adminQueue.Receive();
if (admMsg != null && admMsg.Acknowledgment == Acknowledgment.QueueExceedMaximumSize)
{
// queue is full
}
I have an NServiceBus app which receives a particular message when a large database update is required. While this update is happening, I want to either somehow ignore all incoming messages of this type. What is the best way of doing it? Should I put a lock statement in there?
In the config of NServiceBus (App.config) you can set the NumberOfWorkerThreads, if you set this to 1 only one message at a time will be processed.
Then if you keep the time you stopped processing this message in memory. You can drop all messages that have arrived between these two times.
But as Udi probably would say, don't try to solve it technically but find a solution from the business perspective.