Does RabbitMQ (called from a c# client) have the ability to distribute work on a queue and allow the publisher to receive confirmations that that work processed successfully?
It seems like it should be possible without adding an extra queue, but unless I'm missing something acknowledgements/confirms don't tell the original publisher that a message was dealt with successfully. So it has no way of knowing if all of its work was handled.
I'm currently using the standard rabbit c# client, but I know easynetq is also very mature so suggestions for a good way to achieve this with either would be appreciated.
No, absolutely nothing in RabbitMQ will do that. The most you get out of RabbitMQ is an acknowledgment that the message was delivered to a worker, which you may interpret as "someone started work on the task". Your workers will need to find a way to communicate back to the caller with the results of the task, which could well be another exchange-queue mechanism but it is more likely that your workers will put the results of the task in Redis or database and if properly written, even a way to communicate failure codes via the same.
Related
I have read about RabbitMQ & it's techniques on handling the events which fail to process at consumers (not acknowledged messages, expired ttl-s etc.). RabbitMQ DLX
The way this is working (as I understand) is setting a Dead Letter Exchange to the processing queue.
So far so good, when the events are failing I can see that the Dead letter queue is getting it's messages saved there.
But how do I consume these messages saved there ?!
In my case I want to re-process them after their TTL in dlq has expired, but in .NET Core (C#), I couldn't find a way or an implementation how to achieve this.
I could also agree with other solutions like creating a background worker to check for dead-messages etc.
Can this be achieved ? If yes, please help me understand what I need to do to get this working
You need to configure a "dead letter queue" to handle messages that have been rejected or undelivered. Using the RabbitMQ Client library, you can bind a consumer to that configured queue and retrieve the messages from it. From there you decide in code what you want to do to reprocess/reject them completely.
I've found a useful step by step guide for you to follow with more detail: https://medium.com/nerd-for-tech/dead-letter-exchanges-at-rabbitmq-net-core-b6348122460d
You may want to consider attaching the original q's source exchange as the DL exchange of the dlq, which makes the original q and the dlq the DLQs to each other, forming a re-try loop.
The added dlq serves as a temp store of retry messages, and a retry delay buffer with the TTL mechanism. Messages timing out in the dlq get auto-pushed back to the original q for retry. The original q's consumer handles both first-time and retry messages.
To avoid an infinite loop of retries, a retry counter would be set in the message; and the consumer of the original q needs to eventually break the retry loop based on that counter.
I'm trying to run a maintainance task. For this, I want to push a message on application start. When the actual handler runs, it does it's work and defers it to execute again a day later. I don't want to publish this message, if it's already on the queue. Is there a way to peek into the queue? I'm using the SQL transport, so I tried to simply query the DB. However, the table is locked and cannot be read. One other thing to consider is, that there are at least two machines running the same app. This is why I came up with this solutions, since I want to circumvent concurrency issues.
It sounds to me like you're using the message queue as a scheduler.
If I were you, I would use a simple background timer (e.g. System.Timers.Timer) to periodically send a message to yourself, and then you can do your work in the message handler.
If your scheduling requirements are more complex, it might be beneficial to take a look at something like Quartz .NET.
We are using Rebus 4.2.1 and RabbitMQ
What we want to achieve is to have handlers on three (or more) instances all react to the same message.
As far as I understood (which may be wrong) - .Publish on the IBus interface should do exactly that (and we have been running with that on MSMQ).
Is there something I am missing with how RabbitMQ works?
(EDIT: I think the term used in RabbitMQ is a "fanout" style message)
EDIT2: mookid8000 put me on the right track - the issue was that each replica was asking for the same queue. As soon as I made that unique - everything started working as intended (and expected).
With Rebus + RabbitMQ, it's pretty simple, because RabbitMQ has native support for topic-based pub/sub messaging.
In each subscriber, you simply call
await bus.Subscribe<YourEvent>();
which will make Rebus generate a topic string out of your event type, and then bind it to the subscriber's input queue, and then in the publisher you
await bus.Publish(new YourEvent(...));
and then each subscriber will get a copy of the event in its input queue.
Underneath the covers, Rebus uses RabbitMQ's "Topic Exchange" exchange type to make this work.
I have two consumers (different applications) connected to an Azure queue. I can either ReceiveAndDelete or PeekLock the messages and during consumption I can complete() or abandon() the message. Ref: http://msdn.microsoft.com/en-us/library/azure/hh851750.aspx.
I'm sure I want to use PeekLock and then abandon() the messages, as I want them to be received in both applications. I figured I'd set the message lifetime on 10 seconds on the queue as a deletion mechanism.
However, as the messages seem to be deleted after 10 seconds, they keep being published in both applications over and over again during those 10 seconds. Should I create some custom duplication detection or am I using a wrong approach in general?
In my experience, when you use PeeKLock, you will need to almost always finalize using the Complete Method. The Abandon method is creating the duplicity as it's never marked as "done".
Have you considered using the Service Bus Topics/Subscriptions pattern, or perhaps Filters? If I understand your scenario correctly, it may be just what you need. You can send 2 messages to the queue with different topics or filters designating which app it is for.
http://azure.microsoft.com/en-us/documentation/articles/service-bus-dotnet-how-to-use-topics-subscriptions/
Please let me know if this helps, or if this does not match your situation.
Kindest regards...
I am implementing a component that reads all the messages off a specific queue as they are available but should only remove messages from the queue asynchronously, after the message contents has been processed and persisted. We read messages off faster than we acknowledge them (e.g. we could read have read 10 messages off before we are ready to Ack the first). The current implementation uses the XMS API, but we can switch to MQI if XMS is inappropriate for these purposes.
We have tried two approaches to try solve this problem, but both have drawbacks that make them unacceptable. I was hoping that someone could suggest a better way.
The first implementation uses an IMessageConsumer in a dedicated thread to read all messages and publish their content as they are available. When the message has been processed, the message.Acknowledge() method is called. The Session is created with AcknowledgeMode.ClientAcknowledge. The problem with this approach is that, as per the documentation, this acknowledges (and deletes) ALL unacknowledged messages that have been received. With the example above, that would mean that all 10 read messages would be acked with the first call. So this does not really solve the problem. Because of the reading throughput we need, we cannot really modify this solution to wait for the first message's ack before reading the second, etc.
The second implementation uses an IQueueBrowser in a decided thread to read all messages and publish their content. This does not delete the messages off the queue as it reads. A separate dedicated thread then waits (on a BlockingQueue) for JMS Message IDs of messages that have been processed. For each of these, it then constructs a dedicated IMessageConsumer (using a message selector with JMSMessageID) to read off the message and ack it. (This pairing of an IQueueBrowser with dedicated IMessageConsumer is recommend by the XMS documentation's section on Queue browsers.) This method does work as expected but, as one would imagine, it is too CPU-intensive on the MQ Server.
Both of the methods proposed in the question appear to rely on a single instance of the app. What's wrong with using multiple app instances, transacted sessions and COMMIT? The performance reports (these are the SupportPacs with names like MP**) all show that throughput is maximized with multiple app instances, and horizontal scaling is one of the most used approaches in your scenario.
The design for this would be either multiple application instances or multiple threads within the same application. The key to making it work correctly is to keep in mind that transactions are scoped to a connection handle. The implication is that a multi-threaded app must dispatch a separate thread for each connection instance and the messages are read in the same thread.
The process flow is that, using a transacted session, the app performs a normal MQGet against the queue, processes the message contents as required and then issues an MQCommit. (I'll use the MQ native API names in my examples because this isn't language dependent.) If this is an XA transaction the app would call MQBegin to start the transaction but for single-phase commit the transaction is assumed. In both cases, MQCommit ends the transaction which removes the messages from the queue. While messages are under syncpoint, no other app instance can retrieve them; MQ just delivers the next available message. If a transaction is rolled back, the next MQGet from any thread retrieves it, assuming FIFO delivery.
There are some samples in:
[WMQ install home]\tools\dotnet\samples\cs\xms\simple\wmq\
...and SimpleXAConsumer.cs is one example that shows the XA version of this. The non-XA version is simpler since you don't need the external coordinator, the MQBegin verbs and so forth. If you start with one of these, make sure that they do not specify exclusive use of the queue and you can fire up several instances using the same configuration. Alternatively, take the portion of the sample that includes creation of the connection, message handling, connection close and destroy, and wrap all that in a thread spawner class.
[Insert usual advice about using the latest version of the classes here.]