We Keep Coding

Does Kafka allow reading a message content asynchronously?

Anybody knows whether the Kafka clients allows sending and reading the content of a message in an asynchronous way.
I am currently using the Confluent.Kafka producers and consumers in C# which allows making an async call containing the whole message payload, however it would be interesting to publish the value of the message or content of several MBs asynchronously, and being able of reading it asynchronously as well, instead of just receiving the message in one shot.
using (var producer = new ProducerBuilder<string, string>(config).Build())
{
await producer.ProduceAsync(_topic, new Message<string, string> { Key = _file, Value = <pass async content here> });
}
Is anyway of achieving this?
Thanks

The producer needs to flush the event, send to the broker, which gets written to disk and (optionally) ack the entire record before consumers can read it.
If you'd like to stream chunks of files, then you should send them as binary, but you will need to chunk it yourself, and deal with potential ordering problems in the consumer (e.g. two clients are streaming the same filename, your key, at the same time, with interwoven values)
The recommendation for dealing with files (i.e. large binary content) is to not send them through Kafka, but rather upload them to a shared filesystem, then send the URI as a string through an event.

How to defer a Azure Service Bus message?

Current I'm using Microsoft.Azure.ServiceBus.IQueueClient to RegisterMessageHandler, and then the message I receive is of type Microsoft.Azure.ServiceBus.Message.
According to the documentation:
Message deferral APIs The API is BrokeredMessage.Defer or
BrokeredMessage.DeferAsync in the .NET Framework client,
MessageReceiver.DeferAsync in the .NET Standard client, and
IMessageReceiver.defer or IMessageReceiver.deferAsync in the Java
client.
...but none of those libraries seam to relate to the classes I'm actually using. How do I defer? What classes and stuff do I have to use in order to be able to defer messages? All the samples above dont give enough code snippets to explain it.
Update as requested by #Gaurav
from your answer, I can see my message has that property:
message.ScheduledEnqueueTimeUtc = DateTime.UtcNow.AddHours(1);
but the queueClient also has this method:
queueClient.ScheduleMessageAsync(message, DateTime.UtcNow.AddHours(1));
I'm going to try 'scheduledMessageAsync' as I cant see how to communicate that I've set ScheduledEnqueueTimeUtc without calling the queueClient

Microsoft.Azure.ServiceBus.Message has a property called ScheduledEnqueueTimeUtc. Just set the value of this property to a date/time value in future when you want the message to appear in the queue. Message will be hidden till that time and will only appear in the queue at that date/time.
UPDATE
So I ran a test and confirmed that both ScheduledEnqueueTimeUtc and ScheduleMessageAsync works. I used version 4.1.1 for Microsoft.Azure.ServiceBus SDK.
Here's the code I wrote:
static void Main(string[] args)
{
var connectionString = "my-connection-string";
var queueName = "test";
QueueClient queueClient = new QueueClient(connectionString, queueName);
Message msg1 = new Message()
{
Body = Encoding.UTF8.GetBytes("This message has ScheduledEnqueueTimeUtc property set. It will appear in queue after 2 minutes. Current date/time is: " + DateTime.Now),
ScheduledEnqueueTimeUtc = DateTime.UtcNow.AddMinutes(2)
};
queueClient.SendAsync(msg1).GetAwaiter().GetResult();
Message msg2 = new Message()
{
Body = Encoding.UTF8.GetBytes("This message is sent via ScheduleMessageAsync method. It will appear in queue after 2 minutes. Current date/time is: " + DateTime.Now)
};
queueClient.ScheduleMessageAsync(msg2, new DateTimeOffset(DateTime.UtcNow.AddMinutes(2))).GetAwaiter().GetResult();
Console.ReadLine();
}
And this is what I see when I fetch the messages in Peek-Lock mode:

Using the message deferral APIs like BrokeredMessage.Defer or BrokeredMessage.DeferAsync will defer the message.
Defering a message will change the state of the message from Active to Deferred. The message can be later retrieved based on the sequence number.
ScheduleMessageAsync() is used to schedule the delivery of message (sends a message at specified time). It cannot be used after receiving a message.

I've coded the solution I was looking for, here is the basic outline:
inside an asynchronous method (runs its own thread)
public async Task InitialiseAndRunMessageReceiver()
start an infinite loop that reads the message
receiver = new MessageReceiver(serviceBusConnectionString, serviceBusQueueName, ReceiveMode.PeekLock);
while (true) { var message = await receiver.ReceiveAsync(); ... more code... }
once you know you are about to start your long task, defer the message, but store the message.SystemProperties.SequenceNumber. this keeps it in the queue but prevents it from being re-delivered.
await receiver.DeferAsync(message.SystemProperties.LockToken);
and when you finally done ask for the message again using the message.SystemProperties.SequenceNumber, and complete the message as if it weren't deferred
var message = receiver.ReceiveDeferredMessageAsync(message.SystemProperties.SequenceNumber);
receiver.CompleteAsync(message.Result.SystemProperties.LockToken);
and your message will be removed from the queue.
much of my confusion was caused by the libraries being named similarly with overlapping lifespans.
Microsoft.Azure.ServiceBus.Core.MessageReceiver is the message receiver above

Old question, but what suited my situation was deleting the message and posting a copy using ScheduleMessageAsync (there is a copy method somewhere). Then the message would just come back at the desired time.

Azure Service Bus "ReceiveAsync"

Is there a way, using the Microsoft.Azure.ServiceBus package, to wait on your current thread to receive a message from a queue?
This may more be a problem with my understanding and a desire to use the technology in a way it is not intended to be used, but what I would like to do is combine the send and receive examples from the following Microsoft example so that you can send message(s) off to various queues, and be able to listen in and handle "replies" (just messages that you're listening to on a queue) and close the connection when you are done receiving messages.
Some pseudo-code here:
// send message(s) that will be consumed by other processes / applications, and by doing so later on we will expect some messages back
await SendMessagesAsync(numberOfMessages);
var receivedMessages = 0;
while (receivedMessages < numberOfMessages)
{
// there is no "ReceiveAsync" method, this is what I would be looking for
Message message = await queueClient.ReceiveAsync(TimeSpan.FromSeconds(30));
receivedMessages++;
// do something with the message here
}
await queueClient.CloseAsync();
Is this possible or am I "doing it wrong"?

In the new library ReceiveAsync method is available on MessageReceiver class:
var messageReceiver = new MessageReceiver(SBConnString, QueueName, ReceiveMode.PeekLock);
Message message = await messageReceiver.ReceiveAsync();
See a full example at Get started sending and receiving messages from Service Bus queues using MessageSender and MessageReceiver.

In Microsoft.Azure.ServiceBus library, there is no such a thing calledReceiveAsync. In this, you can process or receive the message by using RegisterOnMessageHandlerAndReceiveMessages(). With this you can receive the message with an event. With this RegisterOnMessageHandlerAndReceiveMessages() like queueClient.RegisterMessageHandler(ReceiveOrProcessMessagesAsync, messageHandlerOptions); and you have to seprately create this event for receiveMessages, in our case it is ReceiveOrProcessMessagesAsync
static async Task ReceiveOrProcessMessagesAsync(Message message, CancellationToken token)
{
// Process the message
Console.WriteLine($"Received message: SequenceNumber:{message.SystemProperties.SequenceNumber} Body:{Encoding.UTF8.GetString(message.Body)}");
// Complete the message so that it is not received again.
// This can be done only if the queueClient is created in ReceiveMode.PeekLock mode (which is default).
await queueClient.CompleteAsync(message.SystemProperties.LockToken);
// Note: Use the cancellationToken passed as necessary to determine if the queueClient has already been closed.
// If queueClient has already been Closed, you may chose to not call CompleteAsync() or AbandonAsync() etc. calls
// to avoid unnecessary exceptions.
}
and you refer the below link for know about Microsoft.Azure.ServiceBus
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dotnet-get-started-with-queues

Azure Storage Queue - processing messages on poison queue

I've been using Azure Storage Queues to post messages too, then write the messages to a db table. However I've noticed that when an error occurs processing messages on the queue, the message is written to a poison queue.
Here is some background to the setup of my app:
Azure Web App -> Writes message to the queue
Azure function -> Queue trigger processes the message and writes the contents to a db
There was an issue with the db schema which caused the INSERTS to fail. Each message was retried 5 times, which I believe is the default for retrying queue messages, and after the 5th attempt the message was placed on the poison queue.
The db schema was subsequently fixed but now I've no way of processing the messages on the poison queue.
My question is can we recover messages written to the poison queue in order to process them and INSERT them into the db, and if so how?

For your particular problem, I would recommend solution mentioned in question part of this post: Azure: How to move messages from poison queue to back to main queue?
Please note that name of poison queue == $"{queueName}-poison"

In my current project I've created something what is called: "Support functions" in the FunctionApp. It exposes a special HTTP endpoint with Admin authorization level that can be executed at any time.
Please See the code below, which solves the problem of reprocessing messages from the poison queue:
public static class QueueOperations
{
[FunctionName("Support_ReprocessPoisonQueueMessages")]
public static async Task<IActionResult> Support_ReprocessPoisonQueueMessages([HttpTrigger(AuthorizationLevel.Admin, "put", Route = "support/reprocessQueueMessages/{queueName}")]HttpRequest req, ILogger log,
[Queue("{queueName}")] CloudQueue queue,
[Queue("{queueName}-poison")] CloudQueue poisonQueue, string queueName)
{
log.LogInformation("Support_ReprocessPoisonQueueMessages function processed a request.");
int.TryParse(req.Query["messageCount"], out var messageCountParameter);
var messageCount = messageCountParameter == 0 ? 10 : messageCountParameter;
var processedMessages = 0;
while (processedMessages < messageCount)
{
var message = await poisonQueue.GetMessageAsync();
if (message == null)
break;
var messageId = message.Id;
var popReceipt = message.PopReceipt;
await queue.AddMessageAsync(message); // a new Id and PopReceipt is assigned
await poisonQueue.DeleteMessageAsync(messageId, popReceipt);
processedMessages++;
}
return new OkObjectResult($"Reprocessed {processedMessages} messages from the {poisonQueue.Name} queue.");
}
}
Alternatively it may be a good idea to create a new message with the additional metadata (as information that the message has already been processed in the past with no success - then it may be send to the dead letter queue).

You have two options
Add another function that is triggered by messages added to the poison queue. You can try adding the contents to the db in this function. More details on this approach can be found here. Of course, if this function too fails to process the message you could check the dequeue count and post a notification that needs manual intervention.
Add an int 'dequeueCount' parameter to the function processing the queue and after say 5 retries log the failure instead of letting the message go the poison queue. For example you can send an email to notify that manual intervention is required.

You can use azure management studio(cerulean) and move the message from poison queue to actual queue. Highly recommended tool to access queues and blobs and do any production related activity also. https://www.cerebrata.com/products/cerulean
I am just user of the tool and no way affiliated, i recommended because it is very powerful, very useful and makes you very productive.
Click on move and message can be moved to the actual uploaded queue

Just point your Azure function to the poison queue and the items in that poison queue will be handled. More details here: https://briancaos.wordpress.com/2018/05/03/azure-functions-how-to-retry-messages-in-the-poison-queue/

Azure Storage Explorer(version above 1.15.0) has now added support to move messages from one queue to another. This makes it possible to move all, or a selected set of messages, from the poison queue back to the original queue.
https://github.com/microsoft/AzureStorageExplorer/issues/1064

Is it possible to publish multiple messages at once using the RabbitMQ client for C#?

Right now, our publishing code for large amounts of messages looks like so:
foreach (var message in messages)
{
publisher.Publish(message);
}
Does there exist the ability to send more than one message over the channel at once?
publisher.Publish(messages);
or as so if we chunk
var chunks = messages.Chunk(100);
foreach (var chunk in chunks)
{
publisher.Publish(chunk);
}

With current version of RabbitMq(3.8.2) you can send batch messages as below for c# client sdk:
basicPublishBatch = channel.CreateBasicPublishBatch();
basicPublishBatch.Add("exchange", "routeKey", false, null, new byte[]{1});
basicPublishBatch.Add("exchange", "routeKey", false, null, new byte[]{1});
basicPublishBatch.Publish();
Check this PR:
https://github.com/rabbitmq/rabbitmq-dotnet-client/pull/368

For RabbitMQ the AMQP protocol is asynchronous for produce and consume operations so it is not clear how to benefit from batch consumer endpoint given out of the box.
What you can do is to create endpoints for chunked messages and process them inside workflow if you can speed up operations. So one solution would be for example to have batching component included before publisher class and send custom messages.