I have a data queue with numbered messages that must be processed in order. When the subscriber receives a message (basicconsume with manual ack) and determines it missed a message I'm trying to do the following
Nack the message with requeue
Stop listening to data queue (basiccancel)
Start listening to error queue to get missed message, process msg (basicconsume)
Stop listening to error queue (basiccancel)
Start listening to data queue (basicconsume)
When I do this, the NACK message (#1) is immediately picked backup before the Stop Listening (#2). It seems like this should be straight forward and I'm just missing something. I'd like to avoid putting the Nack message into a 3rd queue, but it is what it is.
Thanks
RabbitMq 3.8.5, C# RabbitMqClient v6.1.0, .Net Core 3.1
I feel that I'm misunderstanding something with RabbitMq so I'm looking for clarification:
If I have a client sending a message to an exchange, and there's no consumer on the other side, what is meant to happen?
I had thought that it should sit in a queue until it's picked up, but the issue I've got is that, right now there is no queue on the other end of the exchange (which may well be my issue).
This is my declaration code:
channel.ExchangeDeclare(name, exchangeType, durable, autoDelete);
var queueName = ret._channel.QueueDeclare().QueueName;
channel.ConfirmSelect();
and this is my publisher:
channel.BasicPublish(exchangeName, routingKeyOrTopicName, messageProperties, message);
However doing that gives me one queue name for the outbound exchange, and another for the inbound consumer.
Would someone help this poor idiot out in understanding how this is meant to work? What is the expected behavior if there's no consumer at the other end? I do have an RPC mechanism that does work, but wasn't sure if that's the right way to handle this, or not.
Everything works find if I have my consumer running first, however if I fire up my Consumer after the client, then the messages are lost.
Edit
To further clarify, I've set up a simple RPC type test; I've two Direct Exchanges on the client side, one for the outbound Exchange, and another for the inbound RPC consumer.
Both those have their own queue.
Exchange queue name = amq.gen-fp-J9-TQxOJ7NpePEnIcGQ
Consumer queue name = amq.gen-wDFEJ269QcMsHMbAz-t3uw
When the Consumer app fires up, it declares its own Direct exchange and its own queue.
Consumer queue name = amq.gen-o-1O2uSczjXQDihTbkgeqA
If I do it that way though, the message gets lost.
If I fire up the consumer first then I still get three queues in total, but the messages are handled correctly.
This is the code I use to send my RPC message:
messageProperties.ReplyTo = _rpcResponder._routingKeyOrTopicName;
messageProperties.Type = "rpc";
messageProperties.Priority = priority;
messageProperties.Persistent = persistent;
messageProperties.Headers = headers;
messageProperties.Expiration = "3600000";
Looking at the management GUI, I see that all three queues end up being marked as Exclusive, but I'm not declaring them as such. In fact, I'm not creating any queues myself, rather letting the Client library handle that for me, for example, this is how I define my Consumer:
channel.ExchangeDeclare(name, exchangeType, durable, autoDelete);
var queueName = ret._channel.QueueDeclare().QueueName;
Console.WriteLine($"Consumer queue name = {queueName}");
channel.QueueBind(ret.QueueName, name, routingKeyOrTopicName, new Dictionary<string, object>());
In RabbitMQ, messages stay in queues, but they are published to exchanges. The way to link an exchange to a queue is through bindings (there are some default bindings).
If there are no queues, or the exchange's policy doesn't find any queue to forward the message, the message is lost.
Once a message is in a queue, the message is sent to one of that queue's consumers.
Maybe you're using exclusive queues? These queues get deleted when their declaring connection is gone.
Found the issue: I was allowing the library to generate the queue names rather than using specific ones. This meant that RabbitMq was always having to deal with a shifting target each time.
If I use 'well defined' queue names AND the consumer has fired up at least once to define the queue on RabbitMq, then I do see the message being dropped into the queue and stay there, even though the consumer isn't running.
How do I handle a long running tasks on a bot so the client dosnt retry to send the message after 15 seconds again.
I got a bot with the botframework v3 and connect the client with directline
The Direct Line channel connector itself does not retry sending messages. If it does not receive an ack within 15 seconds of sending a message to your bot, it will throw a Gateway Timeout.
If you are using the DirectLineClient, you can override the retry policy, ensuring the client does not retry messages:
DirectLineClientCredentials creds = new DirectLineClientCredentials(directLineSecret);
DirectLineClient directLineClient = new DirectLineClient(new Uri("https://directline.botframework.com"), creds);
directLineClient.SetRetryPolicy(new Microsoft.Rest.TransientFaultHandling.RetryPolicy(new Microsoft.Rest.TransientFaultHandling.HttpStatusCodeErrorDetectionStrategy(), 0));
If you have a long running process, that takes more than 15 seconds, consider queuing the message somewhere, so you can acknowledge the call immediately, then process the message on a background thread. This is conceptually called Proactive Messaging. More information can be found here: https://learn.microsoft.com/en-us/azure/bot-service/dotnet/bot-builder-dotnet-proactive-messages?view=azure-bot-service-3.0
Edit: This blog post also explains one method for handling long operations within a bot, by using Azure Queue storage and an Azure Function which processes the operation and calls the bot when finished:
Manage a long-running operation
Another option is to process incoming messages, or long processing messages, on a background thread. This experimental sample demonstrates some methods using this design:
Immediate Accept Bot
I have a couple of queues where certain information is queued. Let us say I have "success" and "failed" queues in which Server side component has continuously written some data to these queues for clients.
Clients read this data and display it on a UI for end users. Now, I have a situation to purge any message in these queues older than 30 days. Clients would then only be able to see only 30 days of information at any point of time.
I have searched a lot and could see some command line options to purge whole queue but could not find a relevant suggestion.
Any help in the right direction is appreciated. Thanks
I don't think this is possible; looks like you're trying to use RabbitMq as data storage instead of message server.
The only way to understand if a message is "older" than 30, is to process the message, and by doing this you are removing the messagge from the queue.
Best thing to do here is to process the messages and store them in a long term storage; then you can implement a deletion policy to eliminate the older elements.
If you really want to go down this path, RabbitMQ implements TTL at queue level or message level; take a look at this: https://www.rabbitmq.com/ttl.html
[As discussed in comments]
To keep the message in the queue you can try to use a NACK instead of ACK as confirmation; this way RabbitMQ will consider the message undelivered and it will try to deliver it again and again. Remember to create a durable queue (https://www.rabbitmq.com/confirms.html).
You can also check this answer: Rabbitmq Ack or Nack, leaving messages on the queue
I am sending a message with AcknowledgeTypes.NegativeReceive with a time to be received set so that if the message isn't received off the destination queue within that time frame, a negative receive message is sent to my administration queue. This works fine. The problem is that I want to know which queue the message couldn't be received from.
When I check the message of the negative acknowledgement, there's no hint of the original queue to which the message was sent.
What I'm trying to accomplish is to treat the negative acknowledgement as an event to divert message traffic to a different queue but I need to know the queue to divert away from and that appears not to be retained in the negative acknowledgement message by default.
Is there a way to get the original destination queue out of an acknowledgment message?
UPDATE
What appears to be happening is the DestinationQueue.Path in the acknowledgement message is set to the Outgoing queue on the target machine which is some GUID\ID, e.g.
I found it. It is the ResponseQueue property of the acknowledgment message. Overlooked that.