We Keep Coding

Azure Service Bus - Leave message

(FYI - I am new ASB)
A couple of questions around Azure Service Bus:
How do you get a message from a Queue but leave it there until its' TTL expires? I would have thought simply not calling CompleteMessageAsync would do just that, but it appears to get removed regardless.
How do get a message from a Queue, but only dequeue (remove) it when received by a specific receiver?
Message.ApplicationProperties["ReceiverId"].ToString() == "123"
// now you can remove it
Thanks

How do you get a message from a Queue but leave it there until its' TTL expires?
You can peek at messages rather than receive them. The problem is that the message will be picked up again and again until the delivery count exceeds the maximum and the message will dead-letter, which you don't want to happen. I would review what you're trying to achieve here as it's a contradictory setup. You want the message to have a TTL in anticipation that it's not picked up, but then you want to probe it until TTL expires continuedly.
How do get a message from a Queue, but only dequeue (remove) it when received by a specific receiver?
My advice is don't use a queue for that. If you target a specific destination, express it with your entity topology. For example: publish a message on a topic and have different subscriptions based on the subscriber identification. That way you can have messages for specific subscribers, where a logical subscriber can be scaled out.

1-Use the PeekMessage:
You can peek at the messages in the queue without removing them from
the queue by calling the PeekMessages method. If you don't pass a
value for the maxMessages parameter, the default is to peek at one
message.
//-------------------------------------------------
// Peek at a message in the queue
//-------------------------------------------------
public void PeekMessage(string queueName)
{
// Get the connection string from app settings
string connectionString = ConfigurationManager.AppSettings["StorageConnectionString"];
// Instantiate a QueueClient which will be used to manipulate the queue
QueueClient queueClient = new QueueClient(connectionString, queueName);
if (queueClient.Exists())
{
// Peek at the next message
PeekedMessage[] peekedMessage = queueClient.PeekMessages();
// Display the message
Console.WriteLine($"Peeked message: '{peekedMessage[0].Body}'");
}
}
https://learn.microsoft.com/en-us/azure/storage/queues/storage-dotnet-how-to-use-queues?tabs=dotnet
2-you can also use PeekMessage, check for the property you want (ReceiverId), and it case it's the right one, just complete the message:
// ServiceBusReceiver
await receiver.CompleteMessageAsync(receivedMessage);

Consume messages from Kafka Topic - no response

var configs = new Dictionary<string, string>
{
{"bootstrap.servers", MY_SERVER},
{"security.protocol", "SASL_PLAINTEXT"},
{"sasl.mechanism", "SCRAM-SHA-256"},
{"sasl.username", "MY_USERNAME"},
{"sasl.password", "MY_PWD"},
{"group.id", "sample_group"} // added
};
var consumerConfig = new ConsumerConfig(configs);
using (var schemaRegistry = new CachedSchemaRegistryClient(schemaRegistryConfig))
using (var consumer = new ConsumerBuilder<string, MyModel>(consumerConfig)
.SetKeyDeserializer(new AvroDeserializer<string>(schemaRegistry, avroSerializerConfig).AsSyncOverAsync())
.SetValueDeserializer(new AvroDeserializer<MyModel>(schemaRegistry, avroSerializerConfig).AsSyncOverAsync())
.Build())
{
consumer.Subscribe(TOPIC_NAME);
while (true)
{
var result = consumer.Consume(); //stuck here
Console.WriteLine(result);
}
}
As stated in the code, there is no response coming from consumer.Consume() . It does not throw any error message even during consumer.Subscribe() What will be the possible reason? (I am new to Kafka Consumer)
Maybe there is no message in Topic, so nothing to receive?
The code asked for missing 'group.id', so I added {"group.id", "sample_group"} in config and wrap with ConsumerConfig. Is random name ("sample_group") allowed for group.id or should it be something retrieved from Topic information?
anything else?

Your code looks fine and the fact that no errors and Exceptions are showing up is also a good sign.
"1. Maybe there is no message in Topic, so nothing to receive?"
Even if there are no messages in the Kafka topic, your observation matches the expected behavior. In the while(true) loop you are continuously trying to fetch data from the topic and if nothing can be fetched the consumer will try again in the next iteration. Consumers on Kafka topics are meant to read a topic sequentially while running continuously. It is totally fine that some times the consumers has consumed all messages and stays idle for some time until new message arrive into the topic. During the waiting time the consumer will not stop or crash.
Keep in mind that messages in a Kafka topic have by default a retention period of 7 days. After that time, the messages will be deleted.
"2. The code asked for missing 'group.id', so I added {"group.id", "sample_group"} in config and wrap with ConsumerConfig. Is random name ("sample_group") allowed for group.id or should it be something retrieved from Topic information?"
Yes, the name "sample_group" is allowed as a ConsumerGroup name. There are no reserved consumer group names so this name will not cause any trouble.
"3. anything else?"
By default, a KafkaConsumer reads the messages from "latest" offset. That means, if you run a ConsumerGroup for the very first time it will not read all messages from beginning but rather from end. Check the consumer configurations in the .net Kafka-API documentation for something like auto_offset_reset. You might set this configuration to "earliest" in case you want to read all messages from beginning. Please note, that as soon as you run your application with a given ConsumerGroup for the first time, the second time you run this application this configuration auto_offset_reset will not have any impact because the ConsumerGroup is now registered within Kafka.
What you can usually do, to ensure that the consumer should actually read messages is if you start your consumer before you start producing messages to that topic. Then, (almost) independent of your configuration you should see data flowing through your application.

How to defer a Azure Service Bus message?

Current I'm using Microsoft.Azure.ServiceBus.IQueueClient to RegisterMessageHandler, and then the message I receive is of type Microsoft.Azure.ServiceBus.Message.
According to the documentation:
Message deferral APIs The API is BrokeredMessage.Defer or
BrokeredMessage.DeferAsync in the .NET Framework client,
MessageReceiver.DeferAsync in the .NET Standard client, and
IMessageReceiver.defer or IMessageReceiver.deferAsync in the Java
client.
...but none of those libraries seam to relate to the classes I'm actually using. How do I defer? What classes and stuff do I have to use in order to be able to defer messages? All the samples above dont give enough code snippets to explain it.
Update as requested by #Gaurav
from your answer, I can see my message has that property:
message.ScheduledEnqueueTimeUtc = DateTime.UtcNow.AddHours(1);
but the queueClient also has this method:
queueClient.ScheduleMessageAsync(message, DateTime.UtcNow.AddHours(1));
I'm going to try 'scheduledMessageAsync' as I cant see how to communicate that I've set ScheduledEnqueueTimeUtc without calling the queueClient

Microsoft.Azure.ServiceBus.Message has a property called ScheduledEnqueueTimeUtc. Just set the value of this property to a date/time value in future when you want the message to appear in the queue. Message will be hidden till that time and will only appear in the queue at that date/time.
UPDATE
So I ran a test and confirmed that both ScheduledEnqueueTimeUtc and ScheduleMessageAsync works. I used version 4.1.1 for Microsoft.Azure.ServiceBus SDK.
Here's the code I wrote:
static void Main(string[] args)
{
var connectionString = "my-connection-string";
var queueName = "test";
QueueClient queueClient = new QueueClient(connectionString, queueName);
Message msg1 = new Message()
{
Body = Encoding.UTF8.GetBytes("This message has ScheduledEnqueueTimeUtc property set. It will appear in queue after 2 minutes. Current date/time is: " + DateTime.Now),
ScheduledEnqueueTimeUtc = DateTime.UtcNow.AddMinutes(2)
};
queueClient.SendAsync(msg1).GetAwaiter().GetResult();
Message msg2 = new Message()
{
Body = Encoding.UTF8.GetBytes("This message is sent via ScheduleMessageAsync method. It will appear in queue after 2 minutes. Current date/time is: " + DateTime.Now)
};
queueClient.ScheduleMessageAsync(msg2, new DateTimeOffset(DateTime.UtcNow.AddMinutes(2))).GetAwaiter().GetResult();
Console.ReadLine();
}
And this is what I see when I fetch the messages in Peek-Lock mode:

Using the message deferral APIs like BrokeredMessage.Defer or BrokeredMessage.DeferAsync will defer the message.
Defering a message will change the state of the message from Active to Deferred. The message can be later retrieved based on the sequence number.
ScheduleMessageAsync() is used to schedule the delivery of message (sends a message at specified time). It cannot be used after receiving a message.

I've coded the solution I was looking for, here is the basic outline:
inside an asynchronous method (runs its own thread)
public async Task InitialiseAndRunMessageReceiver()
start an infinite loop that reads the message
receiver = new MessageReceiver(serviceBusConnectionString, serviceBusQueueName, ReceiveMode.PeekLock);
while (true) { var message = await receiver.ReceiveAsync(); ... more code... }
once you know you are about to start your long task, defer the message, but store the message.SystemProperties.SequenceNumber. this keeps it in the queue but prevents it from being re-delivered.
await receiver.DeferAsync(message.SystemProperties.LockToken);
and when you finally done ask for the message again using the message.SystemProperties.SequenceNumber, and complete the message as if it weren't deferred
var message = receiver.ReceiveDeferredMessageAsync(message.SystemProperties.SequenceNumber);
receiver.CompleteAsync(message.Result.SystemProperties.LockToken);
and your message will be removed from the queue.
much of my confusion was caused by the libraries being named similarly with overlapping lifespans.
Microsoft.Azure.ServiceBus.Core.MessageReceiver is the message receiver above

Old question, but what suited my situation was deleting the message and posting a copy using ScheduleMessageAsync (there is a copy method somewhere). Then the message would just come back at the desired time.

How can I get messages from Services Bus in a specific time?

I have this method that works great to get all the messages from my subscription in Azure Service Bus. But I want the messages that have been in there for the last 60 minutes. Is there a way to do that?
public void GetMessages()
{
var connectionString = ConfigurationManager.ConnectionStrings["ServiceBus.EndPoint"].ConnectionString;
var topic = ConfigurationManager.AppSettings["ServiceBus.Topic"];
var subscription = ConfigurationManager.AppSettings["ServiceBus.Subscription"];
var client = SubscriptionClient.CreateFromConnectionString(connectionString, topic, subscription, ReceiveMode.PeekLock);
client.RetryPolicy = new RetryExponential(TimeSpan.FromSeconds(0.1), TimeSpan.FromSeconds(5), 5);
var messages = client.PeekBatch(100);
foreach (var msg in messages)
{
string body = msg.GetBody<String>();
}
}

I want the messages that have been in there for the last 60 minutes
The short answer is "why?". You can peek at the messages, but when you'll try to get them, you're not promised to have the same messages. Or any messages at all.
It's been said by many multiple times that a happy queue is an empty queue. If a message is sitting on a queue for 60 minutes, then something feels off. Almost as if a queue is used as a storage. Either your processors where all offline, and once they get online they should process regardless of how long the message was on the queue, or you're looking for some functionality that probably should be implemented via additional service/in a different way.

SubscriptionClient.RecieveBatch not retrieving all the brokered messages

I have a console application to read all the brokered messages present in the subscription on the Azure Service Bus. I have around 3500 messages in there. This is my code to read the messages:
SubscriptionClient client = messagingFactory.CreateSubscriptionClient(topic, subscription);
long count = namespaceManager.GetSubscription(topic, subscription).MessageCountDetails.ActiveMessageCount;
Console.WriteLine("Total messages to process : {0}", count.ToString()); //Here the number is showing correctly
IEnumerable<BrokeredMessage> dlIE = null;
dlIE = client.ReceiveBatch(Convert.ToInt32(count));
When I execute the code, in the dlIE, I can see only 256 messages. I have also tried giving the prefetch count like this client.PrefetchCountbut then also it returns 256 messages only.
I think there is some limit to the number of messages that can be retrieved at a time.However there is no such thing mentioned on the msdn page for the RecieveBatch method. What can I do to retrieve all messages at a time?
Note:
I only want to read the message and then let it exist on the service bus. Therefore I do not use message.complete method.
I cannot remove and re-create the topic/subscription from the Service Bus.
Edit:
I used PeekBatch instead of ReceiveBatch like this:
IEnumerable<BrokeredMessage> dlIE = null;
List<BrokeredMessage> bmList = new List<BrokeredMessage>();
long i = 0;
dlIE = subsciptionClient.PeekBatch(Convert.ToInt32(count)); // count is the total number of messages in the subscription.
bmList.AddRange(dlIE);
i = dlIE.Count();
if(i < count)
{
while(i < count)
{
IEnumerable<BrokeredMessage> dlTemp = null;
dlTemp = subsciptionClient.PeekBatch(i, Convert.ToInt32(count));
bmList.AddRange(dlTemp);
i = i + dlTemp.Count();
}
}
I have 3255 messages in the subscription. When the first time peekBatch is called it gets 250 messages. so it goes into the while loop with PeekBatch(250,3225). Every time 250 messages are only received. The final total messages I am having in the output list is 3500 with duplicates. I am not able to understand how this is happening.

I have figured it out. The subscription client remembers the last batch it retrieved and when called again, retrieves the next batch.
So the code would be :
IEnumerable<BrokeredMessage> dlIE = null;
List<BrokeredMessage> bmList = new List<BrokeredMessage>();
long i = 0;
while (i < count)
{
dlIE = subsciptionClient.PeekBatch(Convert.ToInt32(count));
bmList.AddRange(dlIE);
i = i + dlIE.Count();
}
Thanks to MikeWo for guidance
Note: There seems to be some kind of a size limit on the number of messages you can peek at a time. I tried with different subscriptions and the number of messages fetched were different for each.

Is the topic you are writing to partitioned by chance? When you receive messages from a partitioned entity it will only fetch from one of the partitions at a time. From MSDN:
"When a client wants to receive a message from a partitioned queue, or from a subscription of a partitioned topic, Service Bus queries all fragments for messages, then returns the first message that is returned from any of the messaging stores to the receiver. Service Bus caches the other messages and returns them when it receives additional receive requests. A receiving client is not aware of the partitioning; the client-facing behavior of a partitioned queue or topic (for example, read, complete, defer, deadletter, prefetching) is identical to the behavior of a regular entity."
It's probably not a good idea to assume that even with a non partitioned entity that you'd get all messages in one go with really either the Receive or Peek methods. It would be much more efficient to loop through the messages in much smaller batches, especially if your message have any decent size to them or are indeterminate in size.
Since you don't actually want to remove the message from the queue I'd suggest using PeekBatch instead of ReceiveBatch. This lets you get a copy of the message and doesn't lock it. I'd highly suggest a loop using the same SubscriptionClient in conjunction with PeekBatch. By using the same SubscriptionClient with PeekBatch under the hood the last pulled sequence number is kept as as you loop through it should keep track and go through the whole queue. This would essentially let you read through the entire queue.

I came across a similar issue where client.ReceiveBatchAsync(....) would not retrieve any data from the subscription in the azure service bus.
After some digging around I found out that there is a bit for each subscriber to enable batch operations. This can only be enabled through powershell. Below is the command I used:
$subObject = Get-AzureRmServiceBusSubscription -ResourceGroup '#resourceName' -NamespaceName '#namespaceName' -Topic '#topicName' -SubscriptionName '#subscriptionName'
$subObject.EnableBatchedOperations = $True
Set-AzureRmServiceBusSubscription -ResourceGroup '#resourceName' -NamespaceName '#namespaceName' -Topic '#topicName'-SubscriptionObj $subObject
More details can be found here. While it still didn't load all the messages at least it started to clear the queue. As far as I'm aware, the batch size parameter is only there as a suggestion to the service bus but not a rule.
Hope it helps!