I am currently retrieving messages from an Azure Service Bus Topic. Using the example provided by Microsoft, I was able to retrieve and read the message(s) sent to my test topic.
I might seem strange, but I do not wish my message(s) being completed upon retrieval, at least not as long as I am testing my present code. I would like to be able to read those messages again and again and not being required to create and send new messages every time completed a cycle.
The standard code sample mentioned above, states the following
// Complete the message so that it is not received again.
// This can be done only if the queueClient is created in ReceiveMode.PeekLock mode (which is default).
await queueClient.CompleteAsync(message.SystemProperties.LockToken);
I would like to add that I modified the sample a tiny bit:
namespace CoreReceiverApp
{
using System;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Azure.ServiceBus;
class Program
{
// Connection String for the namespace can be obtained from the Azure portal under the
// 'Shared Access policies' section.
const string ServiceBusConnectionString = "<your_connection_string>";
const string TopicName = "<your_topic_Name>";
const string SubscriptionName = "<your_subscription_Name>";
static ISubscriptionClient subscriptionClient;
static void Main(string[] args)
{
MainAsync().GetAwaiter().GetResult();
}
static async Task MainAsync()
{
subscriptionClient = new SubscriptionClient(ServiceBusConnectionString, TopicName, SubscriptionName);
Console.WriteLine("======================================================");
Console.WriteLine("Press ENTER key to exit after receiving all the messages.");
Console.WriteLine("======================================================");
// Register MessageHandler and receive messages in a loop
RegisterOnMessageHandlerAndReceiveMessages();
Console.ReadKey();
await subscriptionClient.CloseAsync();
}
static void RegisterOnMessageHandlerAndReceiveMessages()
{
// Configure the MessageHandler Options in terms of exception handling, number of concurrent messages to deliver etc.
var messageHandlerOptions = new MessageHandlerOptions(ExceptionReceivedHandler)
{
// Maximum number of Concurrent calls to the callback `ProcessMessagesAsync`, set to 1 for simplicity.
// Set it according to how many messages the application wants to process in parallel.
MaxConcurrentCalls = 10,
// Indicates whether MessagePump should automatically complete the messages after returning from User Callback.
// False below indicates the Complete will be handled by the User Callback as in `ProcessMessagesAsync` below.
AutoComplete = false
};
// Register the function that will process messages
subscriptionClient.RegisterMessageHandler(ProcessMessagesAsync, messageHandlerOptions);
}
static async Task ProcessMessagesAsync(Message message, CancellationToken token)
{
// Process the message
Console.WriteLine($"Received message: SequenceNumber:{message.SystemProperties.SequenceNumber} Body:{Encoding.UTF8.GetString(message.Body)}");
// Complete the message so that it is not received again.
// This can be done only if the queueClient is created in ReceiveMode.PeekLock mode (which is default).
await subscriptionClient.CompleteAsync(message.SystemProperties.LockToken);
// Note: Use the cancellationToken passed as necessary to determine if the queueClient has already been closed.
// If queueClient has already been Closed, you may chose to not call CompleteAsync() or AbandonAsync() etc. calls
// to avoid unnecessary exceptions.
}
static Task ExceptionReceivedHandler(ExceptionReceivedEventArgs exceptionReceivedEventArgs)
{
Console.WriteLine($"Message handler encountered an exception {exceptionReceivedEventArgs.Exception}.");
var context = exceptionReceivedEventArgs.ExceptionReceivedContext;
Console.WriteLine("Exception context for troubleshooting:");
Console.WriteLine($"- Endpoint: {context.Endpoint}");
Console.WriteLine($"- Entity Path: {context.EntityPath}");
Console.WriteLine($"- Executing Action: {context.Action}");
return Task.CompletedTask;
}
}
}
So what would be the alternative, or to be a bit more precise, the opposite of CompleteAsync?
Is it possible at all, how could I avoid having to create new messages after each run?
I would not advise having your production code altered for debugging purposes. Instead, create a script or a helper program to seed the queue with the necessary messages for your testing/debugging sessions. If you really wish, you could comment out the message completion code (queueClient.CompleteAsync(message.SystemProperties.LockToken) but that will shift from one problem to another as you'll have to increase the MaxDeliveryCount on the queue to ensure messages are not dead-lettered.
If you use the peeklock mode to receive message in Azure sdervice bus, the receiving client will initiate settlement of a received message with a positive acknowledgment when it calls Complete at the API level. This indicates to the broker that the message has been successfully processed and the message is removed from the queue or subscription. So you want the message to be redelivered, you can elapse lock or use Abandon to unlock message. For more details, please refer to here
Related
I'm trying to use activeMQ with an NMS (C#) consumer to get messages, do some processing and then send the contents to a webserivce via HttpClient.PostAsync(), all running within a windows service (via Topshelf).
The downstream system I'm communicating with is extremely touchy and I'm using individual acknowledgement so that I can check the response and act accordingly by acknowledging or triggering a custom retry (i.e. not session.recover).
Since the downstream system is unreliable, I've been trying a few different ways to reduce the throughput of my consumer. I thought I'd be able to accomplish this by converting to be synchronous and using prefetch, but it doesn't appear to have worked.
My understanding is that with an async consumer the prefetch 'limit' will never be hit but using synchronous method the prefetch queue will only be eaten away as messages are acknowledged, meaning that I can tune my listener to pass messages at a rate which the downstream component can handle.
With a queue loaded with 100 messages, and kick off my code using a listener (i.e. asynchronously) then I can successfully log that 100 msgs have been through.
When I change it to use consumer.Receive() (or ReceiveNoWait) then I never get a message.
Here is a snippet of what I'm trying for the synchronous consumer, with the async option included but commented out:
public Worker(LogWriter logger, ServiceConfiguration config, IConnectionFactory connectionFactory, IEndpointClient endpointClient)
{
log = logger;
configuration = config;
this.endpointClient = endpointClient;
connection = connectionFactory.CreateConnection();
connection.RedeliveryPolicy = GetRedeliveryPolicy();
connection.ExceptionListener += new ExceptionListener(OnException);
session = connection.CreateSession(AcknowledgementMode.IndividualAcknowledge);
queue = session.GetQueue(configuration.JmsConfig.SourceQueueName);
consumer = session.CreateConsumer(queue);
// Asynchronous
//consumer.Listener += new MessageListener(OnMessage);
// Synchronous
var message = consumer.Receive(TimeSpan.FromSeconds(5));
while (true)
{
if (!Equals(message, null))
{
OnMessage(message);
}
}
}
public void OnMessage(IMessage message)
{
log.DebugFormat("Message {count} Received. Attempt:{attempt}", message.Properties.GetInt("count"), message.Properties.GetInt("NMSXDeliveryCount"));
message.Acknowledge();
}
I believe you need to call Start() on your connection, e.g.:
connection.Start();
Calling Start() indicates that you want messages to flow.
It's also worth noting that there's no way to break out of your while(true) loop aside from throwing an exception from OnMessage.
What is the proper method of consuming a message, processing it, and then publishing it? I run into a lot of unacknowledged messages and I believe there is some blocking going on. Trying to understand the best practice for something like this.
I'm working on a set of services that will process around 50k requests a day. I have decided to use RabbitMQ and three Windows Services written in Dotnet Core 3.1.
I have diagrammed the process but essentially it works like this:
an external service publishes the message to Queue #1
service A is "listening" on Queue #1 and consumes any messages that arrive in the Queue. A database call is made and then Service A passes message to Queue #2
service B is "listening" on Queue #2 and consumes any messages that arrive in the Queue. Some internal processing is done and then Service B passes message to Queue #3
service C is "listening" on Queue #2 and consumes any messages that arrive in the Queue. Some internal processing is done and then Service C pushes message to database
Code example is below image
protected override void OnStart(string[] args)
{
logger.LogInformation("Starting Service ...");
base.OnStart(args);
string queue = "Queue_StageOne";
this.connection = factory.CreateConnection();
this.channel = connection.CreateModel();
this.publishingChannel = connection.CreateModel();
this.channel.BasicQos(0, 1, false);
consumer = new AsyncEventingBasicConsumer(channel);
consumer.Received += Consumer_Recieved;
this.channel.BasicConsume(queue: queue, autoAck: false, consumer: consumer);
}
private async Task Consumer_Recieved(object sender, BasicDeliverEventArgs #event)
{
var body = #event.Body;
var message = Encoding.UTF8.GetString(body.ToArray());
var inboundTransferObject = PatientObject.ConvertFromJson(message);
//logger.LogInformation("Processed message " + inboundTransferObject.WebhookMessageId);
//ServicePointManager.SecurityProtocol = SecurityProtocolType.SystemDefault;
//X509Certificate2 cert = new X509Certificate2(config["CertificationPath"].ToString(), config["PFXPassword"]);
//JToken access_token = GetAccessToken(cert);
//JObject payerData = GetPractitionerData(inboundTransferObject, cert, access_token);
//inboundTransferObject = ProcessPractitioner(inboundTransferObject, payerData);
var outboundTransferObject = Encoding.ASCII.GetBytes(inboundTransferObject.ConvertToJson());
channel.BasicAck(deliveryTag: #event.DeliveryTag, multiple: false);
publishingChannel.BasicPublish(exchange: "ExchangeA", routingKey: "Queue_StageTwo", basicProperties: null, body:outboundTransferObject);
await Task.Delay(250);
}
It's not clear exactly what you're asking here but one thing that does stand out is that your services should not acknowledge the inbound message unless and until they've completed all their processing steps, and that includes publishing follow-on outbound messages. In your code sample you appear to acknowledge the inbound message before publishing the outbound message.
That however does not explain the symptom you described "I run into a lot of unacknowledged messages". When do you run into these? How many is a lot? Have you set a prefetch limit on your channel? For testing purposes, you could try setting your prefetch count to one to ensure that only one message is in-flight at a time.
channel.BasicQos(1, global: true)
Please see this section of the RabbitMQ documentation:
"Because messages are sent (pushed) to clients asynchronously, there is usually more than one message "in flight" on a channel at any given moment. In addition, manual acknowledgements from clients are also inherently asynchronous in nature. So there's a sliding window of delivery tags that are unacknowledged. Developers would often prefer to cap the size of this window to avoid the unbounded buffer problem on the consumer end. This is done by setting a "prefetch count" value using the basic.qos method. The value defines the max number of unacknowledged deliveries that are permitted on a channel. Once the number reaches the configured count, RabbitMQ will stop delivering more messages on the channel unless at least one of the outstanding ones is acknowledged."
Current I'm using Microsoft.Azure.ServiceBus.IQueueClient to RegisterMessageHandler, and then the message I receive is of type Microsoft.Azure.ServiceBus.Message.
According to the documentation:
Message deferral APIs The API is BrokeredMessage.Defer or
BrokeredMessage.DeferAsync in the .NET Framework client,
MessageReceiver.DeferAsync in the .NET Standard client, and
IMessageReceiver.defer or IMessageReceiver.deferAsync in the Java
client.
...but none of those libraries seam to relate to the classes I'm actually using. How do I defer? What classes and stuff do I have to use in order to be able to defer messages? All the samples above dont give enough code snippets to explain it.
Update as requested by #Gaurav
from your answer, I can see my message has that property:
message.ScheduledEnqueueTimeUtc = DateTime.UtcNow.AddHours(1);
but the queueClient also has this method:
queueClient.ScheduleMessageAsync(message, DateTime.UtcNow.AddHours(1));
I'm going to try 'scheduledMessageAsync' as I cant see how to communicate that I've set ScheduledEnqueueTimeUtc without calling the queueClient
Microsoft.Azure.ServiceBus.Message has a property called ScheduledEnqueueTimeUtc. Just set the value of this property to a date/time value in future when you want the message to appear in the queue. Message will be hidden till that time and will only appear in the queue at that date/time.
UPDATE
So I ran a test and confirmed that both ScheduledEnqueueTimeUtc and ScheduleMessageAsync works. I used version 4.1.1 for Microsoft.Azure.ServiceBus SDK.
Here's the code I wrote:
static void Main(string[] args)
{
var connectionString = "my-connection-string";
var queueName = "test";
QueueClient queueClient = new QueueClient(connectionString, queueName);
Message msg1 = new Message()
{
Body = Encoding.UTF8.GetBytes("This message has ScheduledEnqueueTimeUtc property set. It will appear in queue after 2 minutes. Current date/time is: " + DateTime.Now),
ScheduledEnqueueTimeUtc = DateTime.UtcNow.AddMinutes(2)
};
queueClient.SendAsync(msg1).GetAwaiter().GetResult();
Message msg2 = new Message()
{
Body = Encoding.UTF8.GetBytes("This message is sent via ScheduleMessageAsync method. It will appear in queue after 2 minutes. Current date/time is: " + DateTime.Now)
};
queueClient.ScheduleMessageAsync(msg2, new DateTimeOffset(DateTime.UtcNow.AddMinutes(2))).GetAwaiter().GetResult();
Console.ReadLine();
}
And this is what I see when I fetch the messages in Peek-Lock mode:
Using the message deferral APIs like BrokeredMessage.Defer or BrokeredMessage.DeferAsync will defer the message.
Defering a message will change the state of the message from Active to Deferred. The message can be later retrieved based on the sequence number.
ScheduleMessageAsync() is used to schedule the delivery of message (sends a message at specified time). It cannot be used after receiving a message.
I've coded the solution I was looking for, here is the basic outline:
inside an asynchronous method (runs its own thread)
public async Task InitialiseAndRunMessageReceiver()
start an infinite loop that reads the message
receiver = new MessageReceiver(serviceBusConnectionString, serviceBusQueueName, ReceiveMode.PeekLock);
while (true) { var message = await receiver.ReceiveAsync(); ... more code... }
once you know you are about to start your long task, defer the message, but store the message.SystemProperties.SequenceNumber. this keeps it in the queue but prevents it from being re-delivered.
await receiver.DeferAsync(message.SystemProperties.LockToken);
and when you finally done ask for the message again using the message.SystemProperties.SequenceNumber, and complete the message as if it weren't deferred
var message = receiver.ReceiveDeferredMessageAsync(message.SystemProperties.SequenceNumber);
receiver.CompleteAsync(message.Result.SystemProperties.LockToken);
and your message will be removed from the queue.
much of my confusion was caused by the libraries being named similarly with overlapping lifespans.
Microsoft.Azure.ServiceBus.Core.MessageReceiver is the message receiver above
Old question, but what suited my situation was deleting the message and posting a copy using ScheduleMessageAsync (there is a copy method somewhere). Then the message would just come back at the desired time.
Is there a way, using the Microsoft.Azure.ServiceBus package, to wait on your current thread to receive a message from a queue?
This may more be a problem with my understanding and a desire to use the technology in a way it is not intended to be used, but what I would like to do is combine the send and receive examples from the following Microsoft example so that you can send message(s) off to various queues, and be able to listen in and handle "replies" (just messages that you're listening to on a queue) and close the connection when you are done receiving messages.
Some pseudo-code here:
// send message(s) that will be consumed by other processes / applications, and by doing so later on we will expect some messages back
await SendMessagesAsync(numberOfMessages);
var receivedMessages = 0;
while (receivedMessages < numberOfMessages)
{
// there is no "ReceiveAsync" method, this is what I would be looking for
Message message = await queueClient.ReceiveAsync(TimeSpan.FromSeconds(30));
receivedMessages++;
// do something with the message here
}
await queueClient.CloseAsync();
Is this possible or am I "doing it wrong"?
In the new library ReceiveAsync method is available on MessageReceiver class:
var messageReceiver = new MessageReceiver(SBConnString, QueueName, ReceiveMode.PeekLock);
Message message = await messageReceiver.ReceiveAsync();
See a full example at Get started sending and receiving messages from Service Bus queues using MessageSender and MessageReceiver.
In Microsoft.Azure.ServiceBus library, there is no such a thing calledReceiveAsync. In this, you can process or receive the message by using RegisterOnMessageHandlerAndReceiveMessages(). With this you can receive the message with an event. With this RegisterOnMessageHandlerAndReceiveMessages() like queueClient.RegisterMessageHandler(ReceiveOrProcessMessagesAsync, messageHandlerOptions); and you have to seprately create this event for receiveMessages, in our case it is ReceiveOrProcessMessagesAsync
static async Task ReceiveOrProcessMessagesAsync(Message message, CancellationToken token)
{
// Process the message
Console.WriteLine($"Received message: SequenceNumber:{message.SystemProperties.SequenceNumber} Body:{Encoding.UTF8.GetString(message.Body)}");
// Complete the message so that it is not received again.
// This can be done only if the queueClient is created in ReceiveMode.PeekLock mode (which is default).
await queueClient.CompleteAsync(message.SystemProperties.LockToken);
// Note: Use the cancellationToken passed as necessary to determine if the queueClient has already been closed.
// If queueClient has already been Closed, you may chose to not call CompleteAsync() or AbandonAsync() etc. calls
// to avoid unnecessary exceptions.
}
and you refer the below link for know about Microsoft.Azure.ServiceBus
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dotnet-get-started-with-queues
I'm trying to resubmit a message from a deadletter queue.
I am can replay a message on a dead letter queue, thats fine.
The problem is when I want to now delete this from the deadletter queue.
Here is what I am trying to do:
var subscription = "mySubscription";
var topic = "myTopic";
var connectionString = "connectionStringOnAzure";
var messagingFactory = MessagingFactory.CreateFromConnectionString(connectionString);
var messageReceiver = messagingFactory.CreateMessageReceiver(SubscriptionClient.FormatDeadLetterPath(topic, subscription), ReceiveMode.ReceiveAndDelete);
long messageSequenceNumber = 835;
var brokeredMessage = messageReceiver.Receive(messageSequenceNumber); // this part fails
// mark message as complete to remove from the queue
brokeredMessage.Complete();
I get following error message:
Microsoft.ServiceBus.Messaging.MessageNotFoundException : Failed to lock one or more specified messages. The message does not exist..TrackingId:ae15edcc-06ac-4d2b-9059-009599cf5c4e_G5_B15,TimeStamp:8/13/2013 1:45:42 PM
However, instead of specifying a message sequence number and I just use the ReceiveBatch as shown below, it is fine.
// this works and does not throw any errors
var brokeredMessages = messageReceiver.ReceiveBatch(10);
Am I missing something? Or is there another way of reprocessing deadletters and removing them?
The deadletter queue is processed in sequence just like any other queue.
The Receive(seqNo) method is used in combination with Defer(), which puts the message into a different secondary Queue - the "deferral queue". The deferral queue exists for scenarios where you are getting messages out of the expected order (eg. in a state machine) and need a place to put the messages that arrived early. Those you can park with Defer() and make a note of that (probably even in session state) and then pull the messages once you're ready to do so. The Workflow Manager runtime used by SharePoint uses that feature, for instance.
After creating receiver you can politely start receiving all messages (without being picky) till you encounter message with your SequenceNumber, call Complete() on the message and stop iterating the queue. i.e
while (true)
{
BrokeredMessage message = receiver.Receive();
if (message.SequenceNumber == sequenceNumber)
{
message.Complete();
break;
}
}
Without completing message it remains in the queue and that's what you want (at least in .NET 4.5. Worth to note that if your Sequence Number is not found Receiver will loop the queue indefinitely.