I have a MSMQ with hundreds of messages being sent to it a second. To aid processing and provide high availability I am processing the queue over N computers. I have two services per computer, one is a Reader that reads off the queue, the other is a node that actually processes the messages. Note the nodes are arranged in a cluster for HA.
Currently my implementation "reads" off the queue which essentially removes the message from the queue. This is fine in testing but should something happening to the processing nodes that message will be lost.
As such I think need to implement something like this:
Peek off the queue,
Send message to be processed by nodes
Check result
If successful, read (ie. remove) the message from the queue.
Currently I process messages in parallel using TPL but I am unsure how to achieve such a thing whilst also using peek?
Note:
I assume peek and then read on success is the best way to go here?
Related
I have read about RabbitMQ & it's techniques on handling the events which fail to process at consumers (not acknowledged messages, expired ttl-s etc.). RabbitMQ DLX
The way this is working (as I understand) is setting a Dead Letter Exchange to the processing queue.
So far so good, when the events are failing I can see that the Dead letter queue is getting it's messages saved there.
But how do I consume these messages saved there ?!
In my case I want to re-process them after their TTL in dlq has expired, but in .NET Core (C#), I couldn't find a way or an implementation how to achieve this.
I could also agree with other solutions like creating a background worker to check for dead-messages etc.
Can this be achieved ? If yes, please help me understand what I need to do to get this working
You need to configure a "dead letter queue" to handle messages that have been rejected or undelivered. Using the RabbitMQ Client library, you can bind a consumer to that configured queue and retrieve the messages from it. From there you decide in code what you want to do to reprocess/reject them completely.
I've found a useful step by step guide for you to follow with more detail: https://medium.com/nerd-for-tech/dead-letter-exchanges-at-rabbitmq-net-core-b6348122460d
You may want to consider attaching the original q's source exchange as the DL exchange of the dlq, which makes the original q and the dlq the DLQs to each other, forming a re-try loop.
The added dlq serves as a temp store of retry messages, and a retry delay buffer with the TTL mechanism. Messages timing out in the dlq get auto-pushed back to the original q for retry. The original q's consumer handles both first-time and retry messages.
To avoid an infinite loop of retries, a retry counter would be set in the message; and the consumer of the original q needs to eventually break the retry loop based on that counter.
I have an actor that generates UpdateItemMessage , this message is consumed by multiple child actors . my problem is the Producer is faster than the consumer actors and it generates far more messages than the child actors can consume, for that reason i want to pace the producer . What i tried to do is use Become() and Unbecome() to indicate busy/idle status. What is the best practice for the parent/producer to query the status of child actors .
Producer calls Ask for each item in Context.GetChildren() ?
Consumers Tell Producer when they switch their status to busy ?
There are many ways to do this. And doing something like this would arguably be easier once the streaming API has been properly ported to akka.net.
So for example lets say your producer is an actor that reads something from a file on disk. Where each line/item represents some work that needs to be
done. That you distribute across multiple workers.
I would let the workers send an ack message back to the producer (via Sender.Tell).
I would then let the producer read the lines in batches of a fixed size (for example 500). After that it schedules the next batch to be read by scheduling a message to itself. (via scheduler
That way you free up your producer actor by enabling it to recieve ack's from your worker actors.
You could then keep state in your producer actor on how much it has send out, and how much work is done.
At that point its easy to determine if your workers are ready for the next batch of work, or that you need to wait a bit.
An alternative to this is that you communicate with your workers via a proxy actor. That does the throttling for you. (the same way as in the producer example, by keeping record of how much work is done in combination with worker ack's).
The downside to this is that the throttler actor would effectively buffer all messages the producer will throw at it.
I am implementing a component that reads all the messages off a specific queue as they are available but should only remove messages from the queue asynchronously, after the message contents has been processed and persisted. We read messages off faster than we acknowledge them (e.g. we could read have read 10 messages off before we are ready to Ack the first). The current implementation uses the XMS API, but we can switch to MQI if XMS is inappropriate for these purposes.
We have tried two approaches to try solve this problem, but both have drawbacks that make them unacceptable. I was hoping that someone could suggest a better way.
The first implementation uses an IMessageConsumer in a dedicated thread to read all messages and publish their content as they are available. When the message has been processed, the message.Acknowledge() method is called. The Session is created with AcknowledgeMode.ClientAcknowledge. The problem with this approach is that, as per the documentation, this acknowledges (and deletes) ALL unacknowledged messages that have been received. With the example above, that would mean that all 10 read messages would be acked with the first call. So this does not really solve the problem. Because of the reading throughput we need, we cannot really modify this solution to wait for the first message's ack before reading the second, etc.
The second implementation uses an IQueueBrowser in a decided thread to read all messages and publish their content. This does not delete the messages off the queue as it reads. A separate dedicated thread then waits (on a BlockingQueue) for JMS Message IDs of messages that have been processed. For each of these, it then constructs a dedicated IMessageConsumer (using a message selector with JMSMessageID) to read off the message and ack it. (This pairing of an IQueueBrowser with dedicated IMessageConsumer is recommend by the XMS documentation's section on Queue browsers.) This method does work as expected but, as one would imagine, it is too CPU-intensive on the MQ Server.
Both of the methods proposed in the question appear to rely on a single instance of the app. What's wrong with using multiple app instances, transacted sessions and COMMIT? The performance reports (these are the SupportPacs with names like MP**) all show that throughput is maximized with multiple app instances, and horizontal scaling is one of the most used approaches in your scenario.
The design for this would be either multiple application instances or multiple threads within the same application. The key to making it work correctly is to keep in mind that transactions are scoped to a connection handle. The implication is that a multi-threaded app must dispatch a separate thread for each connection instance and the messages are read in the same thread.
The process flow is that, using a transacted session, the app performs a normal MQGet against the queue, processes the message contents as required and then issues an MQCommit. (I'll use the MQ native API names in my examples because this isn't language dependent.) If this is an XA transaction the app would call MQBegin to start the transaction but for single-phase commit the transaction is assumed. In both cases, MQCommit ends the transaction which removes the messages from the queue. While messages are under syncpoint, no other app instance can retrieve them; MQ just delivers the next available message. If a transaction is rolled back, the next MQGet from any thread retrieves it, assuming FIFO delivery.
There are some samples in:
[WMQ install home]\tools\dotnet\samples\cs\xms\simple\wmq\
...and SimpleXAConsumer.cs is one example that shows the XA version of this. The non-XA version is simpler since you don't need the external coordinator, the MQBegin verbs and so forth. If you start with one of these, make sure that they do not specify exclusive use of the queue and you can fire up several instances using the same configuration. Alternatively, take the portion of the sample that includes creation of the connection, message handling, connection close and destroy, and wrap all that in a thread spawner class.
[Insert usual advice about using the latest version of the classes here.]
We have a durable RabbitMQ queue. When consumer gets item from the queue, it processes it and then acknowledges it. If consumer fails to process the item it prints an error expecting someone to fix the problem and stops. No acknowledgement is being sent. When consumer restarts the item it receives is next item in the queue, not the item without ack. Basic.Recover() doesn't help (using .NET client).
Any ideas how to make it work as a queue - always get the first item if it is not acked.
Messages can be consumed in two ways noAck=false or noAck=true
noAck is a parameter on both Model.BasicConsume and Model.BasicGet
When noAck is set to true messages are automatically removed from the queue after being delivered. If noAsk is set to false messages are only removed when you call basicAck.
If noAck=false and you do not call basicAck the message will remain but you will not receive in on other consumers until you restart the application (or close the connection that consumed it first). If you call BasicReject the message will be redelivered to a subscriber.
I hope this helps.
See this entry in the RabbitMQ FAQ. While you may want RabbitMQ to re-queue your unacked messages right back to the head of the queue (where they were before your consumer pulled them down), the reality is likely going to be different, as you've experienced.
So it's not that Basic.Recover() doesn't work (the message was placed back on the queue for future reprocessing) just that it doesn't work the way you expected.
Something in the back of my mind tells me that you may be able to get the behavior you want by setting a prefetch count of 1 and having at most only one consumer connect to the queue at any time, but I can't guarantee that's the case. It's worth trying. However even if it works it's not something to rely on staying the case forever, and with such a low prefetch count your consumer's messages/second performance will probably suffer.
RabbitMQ seems to have fixed part of this issue since 2.7.0, as now messages will be requeued in publication order. Although if you have more than one subscriber to a queue, you may still have message arriving out of the original order.
You can get this behaviour by having a pair of queues, one high priority and one normal priority. Set the prefetch count to 1, and then alternate between the queues using basic.get. Most of the time the priority queue will be empty, but when you want to requeue, publish the message again onto the high priority queue instead.
This works in a scenario where you have multiple processes consuming the message stream, and one process decides to bail out on a message. That message will be picked up almost immediately by another process.
Can someone explain to me the difference between these 3 approaches to processing messages that fail delivery?
Poison Queue service
Dead-Letter Queue service
Using a response service to handle failures
I have "Programming WCF", but I don't really understand when you would use one of these over another, or when it would make sense to use more than one of them. Thanks!
Dead and poison are two different concepts.
Poison messages are messages that can be read from the queue, but your code doesn't know how to handle it so your code gives an exception. If this goes on for some time you want this message to be put on a different queue so your other messages can be handled. A good aproach for this is described on MSDN.
A dead letter is a message that isn't even handled by the queue. The network is broken or the receiving MSMQ computer is shut down. Something like that. The message will automaticly be put on the dead queue after some time by Windows. So it's advisable to write a service that monitors the dead queue.
Poison message / dead letter message queues are used to place messages that have been determined to be undeliverable in a queue that will not try to deliver them anymore. You would do this if you might want to manually take a look at failed messages and process them at a later point. You use these type of queues when you want to keep bad messages from degrading the performance of your system by retrying over and over again.
On the other hand, a response service would be used to notify the sender that there was an error processing the message. Typically in this case you aren't planning on manually processing the bad message and need to let the system that sent the message in that the request has been rejected.
Note that these aren't exclusive. If you are using queues, there is always the chance that the message serialization might change enough to break messages that are in the queue in which case you might still want to have a dead letter queue even if you are using a response service.