We Keep Coding

Azure Service Bus - Leave message

(FYI - I am new ASB)
A couple of questions around Azure Service Bus:
How do you get a message from a Queue but leave it there until its' TTL expires? I would have thought simply not calling CompleteMessageAsync would do just that, but it appears to get removed regardless.
How do get a message from a Queue, but only dequeue (remove) it when received by a specific receiver?
Message.ApplicationProperties["ReceiverId"].ToString() == "123"
// now you can remove it
Thanks

How do you get a message from a Queue but leave it there until its' TTL expires?
You can peek at messages rather than receive them. The problem is that the message will be picked up again and again until the delivery count exceeds the maximum and the message will dead-letter, which you don't want to happen. I would review what you're trying to achieve here as it's a contradictory setup. You want the message to have a TTL in anticipation that it's not picked up, but then you want to probe it until TTL expires continuedly.
How do get a message from a Queue, but only dequeue (remove) it when received by a specific receiver?
My advice is don't use a queue for that. If you target a specific destination, express it with your entity topology. For example: publish a message on a topic and have different subscriptions based on the subscriber identification. That way you can have messages for specific subscribers, where a logical subscriber can be scaled out.

1-Use the PeekMessage:
You can peek at the messages in the queue without removing them from
the queue by calling the PeekMessages method. If you don't pass a
value for the maxMessages parameter, the default is to peek at one
message.
//-------------------------------------------------
// Peek at a message in the queue
//-------------------------------------------------
public void PeekMessage(string queueName)
{
// Get the connection string from app settings
string connectionString = ConfigurationManager.AppSettings["StorageConnectionString"];
// Instantiate a QueueClient which will be used to manipulate the queue
QueueClient queueClient = new QueueClient(connectionString, queueName);
if (queueClient.Exists())
{
// Peek at the next message
PeekedMessage[] peekedMessage = queueClient.PeekMessages();
// Display the message
Console.WriteLine($"Peeked message: '{peekedMessage[0].Body}'");
}
}
https://learn.microsoft.com/en-us/azure/storage/queues/storage-dotnet-how-to-use-queues?tabs=dotnet
2-you can also use PeekMessage, check for the property you want (ReceiverId), and it case it's the right one, just complete the message:
// ServiceBusReceiver
await receiver.CompleteMessageAsync(receivedMessage);

How to defer a Azure Service Bus message?

Current I'm using Microsoft.Azure.ServiceBus.IQueueClient to RegisterMessageHandler, and then the message I receive is of type Microsoft.Azure.ServiceBus.Message.
According to the documentation:
Message deferral APIs The API is BrokeredMessage.Defer or
BrokeredMessage.DeferAsync in the .NET Framework client,
MessageReceiver.DeferAsync in the .NET Standard client, and
IMessageReceiver.defer or IMessageReceiver.deferAsync in the Java
client.
...but none of those libraries seam to relate to the classes I'm actually using. How do I defer? What classes and stuff do I have to use in order to be able to defer messages? All the samples above dont give enough code snippets to explain it.
Update as requested by #Gaurav
from your answer, I can see my message has that property:
message.ScheduledEnqueueTimeUtc = DateTime.UtcNow.AddHours(1);
but the queueClient also has this method:
queueClient.ScheduleMessageAsync(message, DateTime.UtcNow.AddHours(1));
I'm going to try 'scheduledMessageAsync' as I cant see how to communicate that I've set ScheduledEnqueueTimeUtc without calling the queueClient

Microsoft.Azure.ServiceBus.Message has a property called ScheduledEnqueueTimeUtc. Just set the value of this property to a date/time value in future when you want the message to appear in the queue. Message will be hidden till that time and will only appear in the queue at that date/time.
UPDATE
So I ran a test and confirmed that both ScheduledEnqueueTimeUtc and ScheduleMessageAsync works. I used version 4.1.1 for Microsoft.Azure.ServiceBus SDK.
Here's the code I wrote:
static void Main(string[] args)
{
var connectionString = "my-connection-string";
var queueName = "test";
QueueClient queueClient = new QueueClient(connectionString, queueName);
Message msg1 = new Message()
{
Body = Encoding.UTF8.GetBytes("This message has ScheduledEnqueueTimeUtc property set. It will appear in queue after 2 minutes. Current date/time is: " + DateTime.Now),
ScheduledEnqueueTimeUtc = DateTime.UtcNow.AddMinutes(2)
};
queueClient.SendAsync(msg1).GetAwaiter().GetResult();
Message msg2 = new Message()
{
Body = Encoding.UTF8.GetBytes("This message is sent via ScheduleMessageAsync method. It will appear in queue after 2 minutes. Current date/time is: " + DateTime.Now)
};
queueClient.ScheduleMessageAsync(msg2, new DateTimeOffset(DateTime.UtcNow.AddMinutes(2))).GetAwaiter().GetResult();
Console.ReadLine();
}
And this is what I see when I fetch the messages in Peek-Lock mode:

Using the message deferral APIs like BrokeredMessage.Defer or BrokeredMessage.DeferAsync will defer the message.
Defering a message will change the state of the message from Active to Deferred. The message can be later retrieved based on the sequence number.
ScheduleMessageAsync() is used to schedule the delivery of message (sends a message at specified time). It cannot be used after receiving a message.

I've coded the solution I was looking for, here is the basic outline:
inside an asynchronous method (runs its own thread)
public async Task InitialiseAndRunMessageReceiver()
start an infinite loop that reads the message
receiver = new MessageReceiver(serviceBusConnectionString, serviceBusQueueName, ReceiveMode.PeekLock);
while (true) { var message = await receiver.ReceiveAsync(); ... more code... }
once you know you are about to start your long task, defer the message, but store the message.SystemProperties.SequenceNumber. this keeps it in the queue but prevents it from being re-delivered.
await receiver.DeferAsync(message.SystemProperties.LockToken);
and when you finally done ask for the message again using the message.SystemProperties.SequenceNumber, and complete the message as if it weren't deferred
var message = receiver.ReceiveDeferredMessageAsync(message.SystemProperties.SequenceNumber);
receiver.CompleteAsync(message.Result.SystemProperties.LockToken);
and your message will be removed from the queue.
much of my confusion was caused by the libraries being named similarly with overlapping lifespans.
Microsoft.Azure.ServiceBus.Core.MessageReceiver is the message receiver above

Old question, but what suited my situation was deleting the message and posting a copy using ScheduleMessageAsync (there is a copy method somewhere). Then the message would just come back at the desired time.

Apache NMS using ActiveMQ: How do I use transactional acknowledge mode but still acknowledging/rolling back a single message every time?

I use Apache NMS (in c#) to receive messages from ActiveMQ.
I want to be able to acknowledge every message I received, or roll back a message in case I had an error.
I solved the first part by using the CreateSession(AcknowledgementMode.IndividualAcknowledge), and then for every received message I use message.Acknowledge().
The problem is that in this mode there is no Rollback option. if the message is not acknowledged - I can never receive it again for another trial. It can only be sent to another consumer, but there isn't another consumer so it is just stucked in queue.
So I tried to use AcknowledgementMode.Transactional instead, but here there is another problem: I can only use session.Commit() or session.Rollback(), but there is no way to know which specific message I commit or role back.
What is the correct way to do this?

Stay with INDIVIDUAL_ACKNOWLEDGE and then try session.recover() and session.close(). Both of those should signal to the broker that the messages are not going to be acknowledged.

My solution to this was to throw an exception if (for any reason (exception from db savechanges event for example)) I did not want to acknowledge the message with message.Acknowledge().
When you throw an exception inside your extended method of IMessageConsumer Listener then the message will be sent again to your consumer for about 5 times (it will then moved to default DLQ queue for investigation).
However you can change this using the RedeliveryPolicy in connection object.
Example of Redelivery
Policy redeliveryPolicy = new RedeliveryPolicy
{
InitialRedeliveryDelay = 5000, // every 5 secs
MaximumRedeliveries = 10, // the message will be redelivered 10 times
UseCollisionAvoidance = true, // use this to random calculate the 5 secs
CollisionAvoidancePercent = 50,// used along with above option
UseExponentialBackOff = false
};
If message fails again (after 10 times) then it will be moved to a default DLQ queue. (this queue will be created automatically)
You can use this queue to investigate the messages that have not been acknowledged using an other consumer.

Azure Storage Queue - processing messages on poison queue

I've been using Azure Storage Queues to post messages too, then write the messages to a db table. However I've noticed that when an error occurs processing messages on the queue, the message is written to a poison queue.
Here is some background to the setup of my app:
Azure Web App -> Writes message to the queue
Azure function -> Queue trigger processes the message and writes the contents to a db
There was an issue with the db schema which caused the INSERTS to fail. Each message was retried 5 times, which I believe is the default for retrying queue messages, and after the 5th attempt the message was placed on the poison queue.
The db schema was subsequently fixed but now I've no way of processing the messages on the poison queue.
My question is can we recover messages written to the poison queue in order to process them and INSERT them into the db, and if so how?

For your particular problem, I would recommend solution mentioned in question part of this post: Azure: How to move messages from poison queue to back to main queue?
Please note that name of poison queue == $"{queueName}-poison"

In my current project I've created something what is called: "Support functions" in the FunctionApp. It exposes a special HTTP endpoint with Admin authorization level that can be executed at any time.
Please See the code below, which solves the problem of reprocessing messages from the poison queue:
public static class QueueOperations
{
[FunctionName("Support_ReprocessPoisonQueueMessages")]
public static async Task<IActionResult> Support_ReprocessPoisonQueueMessages([HttpTrigger(AuthorizationLevel.Admin, "put", Route = "support/reprocessQueueMessages/{queueName}")]HttpRequest req, ILogger log,
[Queue("{queueName}")] CloudQueue queue,
[Queue("{queueName}-poison")] CloudQueue poisonQueue, string queueName)
{
log.LogInformation("Support_ReprocessPoisonQueueMessages function processed a request.");
int.TryParse(req.Query["messageCount"], out var messageCountParameter);
var messageCount = messageCountParameter == 0 ? 10 : messageCountParameter;
var processedMessages = 0;
while (processedMessages < messageCount)
{
var message = await poisonQueue.GetMessageAsync();
if (message == null)
break;
var messageId = message.Id;
var popReceipt = message.PopReceipt;
await queue.AddMessageAsync(message); // a new Id and PopReceipt is assigned
await poisonQueue.DeleteMessageAsync(messageId, popReceipt);
processedMessages++;
}
return new OkObjectResult($"Reprocessed {processedMessages} messages from the {poisonQueue.Name} queue.");
}
}
Alternatively it may be a good idea to create a new message with the additional metadata (as information that the message has already been processed in the past with no success - then it may be send to the dead letter queue).

You have two options
Add another function that is triggered by messages added to the poison queue. You can try adding the contents to the db in this function. More details on this approach can be found here. Of course, if this function too fails to process the message you could check the dequeue count and post a notification that needs manual intervention.
Add an int 'dequeueCount' parameter to the function processing the queue and after say 5 retries log the failure instead of letting the message go the poison queue. For example you can send an email to notify that manual intervention is required.

You can use azure management studio(cerulean) and move the message from poison queue to actual queue. Highly recommended tool to access queues and blobs and do any production related activity also. https://www.cerebrata.com/products/cerulean
I am just user of the tool and no way affiliated, i recommended because it is very powerful, very useful and makes you very productive.
Click on move and message can be moved to the actual uploaded queue

Just point your Azure function to the poison queue and the items in that poison queue will be handled. More details here: https://briancaos.wordpress.com/2018/05/03/azure-functions-how-to-retry-messages-in-the-poison-queue/

Azure Storage Explorer(version above 1.15.0) has now added support to move messages from one queue to another. This makes it possible to move all, or a selected set of messages, from the poison queue back to the original queue.
https://github.com/microsoft/AzureStorageExplorer/issues/1064

When does a CloudQueueClient updates?

I have a WCF on a Web Role and then a Worker Role to process the messages added to an azure queue by the WCF.
I am doing the following :
var queue = queueStorage.GetQueueReference("myqueue");
var message = new CloudQueueMessage(string.Format("{0},{1}", pWord,processed));
queue.AddMessage(message);
Then I want to wait until the message has been processed, but I dont know if my queue object will get updated on its own or I have to do something about it.
On my worker role I have the following :
This is my onStart method :
CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();
inputQueue = queueClient.GetQueueReference("myqueue");
And then on my Run method :
while (true)
{
try
{
// Retrieve and process a new message from the queue.
msg = inputQueue.GetMessage();
if (msg != null)
{
result = processMessage(msg);
On my processMessage method :
var messageParts = msg.AsString.Split(new char[] { ',' });
var word = messageParts[0];
var processed = Convert.ToBoolean(messageParts[2]);
word = "recibido";
processed = true;
addMessageToQueue2(userId,processed);
return 1;
Add message to queue is :
var queue = outputQueue.GetQueueReference("myQueue");
var message = new CloudQueueMessage(string.Format("{0},{1}", pWord, pProcessed));
queue.AddMessage(message);
Im fairly new to queues but I think this should work so all I need is just waiting until the message has been processed but dont know how it internally works.

Not quite sure what you mean by waiting until the message has been processed. With Azure queues, the operation is very simple:
Place messages on queue (with message TTL)
Read message(s) from queue (with processing-timeout). This processing timeout says "I promise to finish dealing with this message, and then deleting this message, before this timeout is hit.
The queue message stays in the queue but becomes invisible to all other readers during the timeout period.
The message-reader then deletes the message from the queue.
Assuming the code that read the queue message deletes the message before the promised timeout expires, all is good in QueueLand. However: If the processing goes beyond the timeout period, then the message becomes visible again. And... if someone else then reads that message, the original reader loses rights to delete it (they'll get an exception when making an attempt).
So: Long story short: You can process a message for as long as you want, within the stated timeout period, and then delete it. If your code crashes during processing, the message will eventually reappear for someone else to read. If you want to deal with poison messages, just look at the DequeueCount property of the message to see how many times it's been read (and if over a certain threshold, do something special with the message, like tuck it away in a blob or table row for future inspection by the development team).
See this article for all documented queue limits (and a detailed side-by-side comparison with Service Bus Queues).