I'm trying to use activeMQ with an NMS (C#) consumer to get messages, do some processing and then send the contents to a webserivce via HttpClient.PostAsync(), all running within a windows service (via Topshelf).
The downstream system I'm communicating with is extremely touchy and I'm using individual acknowledgement so that I can check the response and act accordingly by acknowledging or triggering a custom retry (i.e. not session.recover).
Since the downstream system is unreliable, I've been trying a few different ways to reduce the throughput of my consumer. I thought I'd be able to accomplish this by converting to be synchronous and using prefetch, but it doesn't appear to have worked.
My understanding is that with an async consumer the prefetch 'limit' will never be hit but using synchronous method the prefetch queue will only be eaten away as messages are acknowledged, meaning that I can tune my listener to pass messages at a rate which the downstream component can handle.
With a queue loaded with 100 messages, and kick off my code using a listener (i.e. asynchronously) then I can successfully log that 100 msgs have been through.
When I change it to use consumer.Receive() (or ReceiveNoWait) then I never get a message.
Here is a snippet of what I'm trying for the synchronous consumer, with the async option included but commented out:
public Worker(LogWriter logger, ServiceConfiguration config, IConnectionFactory connectionFactory, IEndpointClient endpointClient)
{
log = logger;
configuration = config;
this.endpointClient = endpointClient;
connection = connectionFactory.CreateConnection();
connection.RedeliveryPolicy = GetRedeliveryPolicy();
connection.ExceptionListener += new ExceptionListener(OnException);
session = connection.CreateSession(AcknowledgementMode.IndividualAcknowledge);
queue = session.GetQueue(configuration.JmsConfig.SourceQueueName);
consumer = session.CreateConsumer(queue);
// Asynchronous
//consumer.Listener += new MessageListener(OnMessage);
// Synchronous
var message = consumer.Receive(TimeSpan.FromSeconds(5));
while (true)
{
if (!Equals(message, null))
{
OnMessage(message);
}
}
}
public void OnMessage(IMessage message)
{
log.DebugFormat("Message {count} Received. Attempt:{attempt}", message.Properties.GetInt("count"), message.Properties.GetInt("NMSXDeliveryCount"));
message.Acknowledge();
}
I believe you need to call Start() on your connection, e.g.:
connection.Start();
Calling Start() indicates that you want messages to flow.
It's also worth noting that there's no way to break out of your while(true) loop aside from throwing an exception from OnMessage.
Related
I am currently retrieving messages from an Azure Service Bus Topic. Using the example provided by Microsoft, I was able to retrieve and read the message(s) sent to my test topic.
I might seem strange, but I do not wish my message(s) being completed upon retrieval, at least not as long as I am testing my present code. I would like to be able to read those messages again and again and not being required to create and send new messages every time completed a cycle.
The standard code sample mentioned above, states the following
// Complete the message so that it is not received again.
// This can be done only if the queueClient is created in ReceiveMode.PeekLock mode (which is default).
await queueClient.CompleteAsync(message.SystemProperties.LockToken);
I would like to add that I modified the sample a tiny bit:
namespace CoreReceiverApp
{
using System;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Azure.ServiceBus;
class Program
{
// Connection String for the namespace can be obtained from the Azure portal under the
// 'Shared Access policies' section.
const string ServiceBusConnectionString = "<your_connection_string>";
const string TopicName = "<your_topic_Name>";
const string SubscriptionName = "<your_subscription_Name>";
static ISubscriptionClient subscriptionClient;
static void Main(string[] args)
{
MainAsync().GetAwaiter().GetResult();
}
static async Task MainAsync()
{
subscriptionClient = new SubscriptionClient(ServiceBusConnectionString, TopicName, SubscriptionName);
Console.WriteLine("======================================================");
Console.WriteLine("Press ENTER key to exit after receiving all the messages.");
Console.WriteLine("======================================================");
// Register MessageHandler and receive messages in a loop
RegisterOnMessageHandlerAndReceiveMessages();
Console.ReadKey();
await subscriptionClient.CloseAsync();
}
static void RegisterOnMessageHandlerAndReceiveMessages()
{
// Configure the MessageHandler Options in terms of exception handling, number of concurrent messages to deliver etc.
var messageHandlerOptions = new MessageHandlerOptions(ExceptionReceivedHandler)
{
// Maximum number of Concurrent calls to the callback `ProcessMessagesAsync`, set to 1 for simplicity.
// Set it according to how many messages the application wants to process in parallel.
MaxConcurrentCalls = 10,
// Indicates whether MessagePump should automatically complete the messages after returning from User Callback.
// False below indicates the Complete will be handled by the User Callback as in `ProcessMessagesAsync` below.
AutoComplete = false
};
// Register the function that will process messages
subscriptionClient.RegisterMessageHandler(ProcessMessagesAsync, messageHandlerOptions);
}
static async Task ProcessMessagesAsync(Message message, CancellationToken token)
{
// Process the message
Console.WriteLine($"Received message: SequenceNumber:{message.SystemProperties.SequenceNumber} Body:{Encoding.UTF8.GetString(message.Body)}");
// Complete the message so that it is not received again.
// This can be done only if the queueClient is created in ReceiveMode.PeekLock mode (which is default).
await subscriptionClient.CompleteAsync(message.SystemProperties.LockToken);
// Note: Use the cancellationToken passed as necessary to determine if the queueClient has already been closed.
// If queueClient has already been Closed, you may chose to not call CompleteAsync() or AbandonAsync() etc. calls
// to avoid unnecessary exceptions.
}
static Task ExceptionReceivedHandler(ExceptionReceivedEventArgs exceptionReceivedEventArgs)
{
Console.WriteLine($"Message handler encountered an exception {exceptionReceivedEventArgs.Exception}.");
var context = exceptionReceivedEventArgs.ExceptionReceivedContext;
Console.WriteLine("Exception context for troubleshooting:");
Console.WriteLine($"- Endpoint: {context.Endpoint}");
Console.WriteLine($"- Entity Path: {context.EntityPath}");
Console.WriteLine($"- Executing Action: {context.Action}");
return Task.CompletedTask;
}
}
}
So what would be the alternative, or to be a bit more precise, the opposite of CompleteAsync?
Is it possible at all, how could I avoid having to create new messages after each run?
I would not advise having your production code altered for debugging purposes. Instead, create a script or a helper program to seed the queue with the necessary messages for your testing/debugging sessions. If you really wish, you could comment out the message completion code (queueClient.CompleteAsync(message.SystemProperties.LockToken) but that will shift from one problem to another as you'll have to increase the MaxDeliveryCount on the queue to ensure messages are not dead-lettered.
If you use the peeklock mode to receive message in Azure sdervice bus, the receiving client will initiate settlement of a received message with a positive acknowledgment when it calls Complete at the API level. This indicates to the broker that the message has been successfully processed and the message is removed from the queue or subscription. So you want the message to be redelivered, you can elapse lock or use Abandon to unlock message. For more details, please refer to here
What is the proper method of consuming a message, processing it, and then publishing it? I run into a lot of unacknowledged messages and I believe there is some blocking going on. Trying to understand the best practice for something like this.
I'm working on a set of services that will process around 50k requests a day. I have decided to use RabbitMQ and three Windows Services written in Dotnet Core 3.1.
I have diagrammed the process but essentially it works like this:
an external service publishes the message to Queue #1
service A is "listening" on Queue #1 and consumes any messages that arrive in the Queue. A database call is made and then Service A passes message to Queue #2
service B is "listening" on Queue #2 and consumes any messages that arrive in the Queue. Some internal processing is done and then Service B passes message to Queue #3
service C is "listening" on Queue #2 and consumes any messages that arrive in the Queue. Some internal processing is done and then Service C pushes message to database
Code example is below image
protected override void OnStart(string[] args)
{
logger.LogInformation("Starting Service ...");
base.OnStart(args);
string queue = "Queue_StageOne";
this.connection = factory.CreateConnection();
this.channel = connection.CreateModel();
this.publishingChannel = connection.CreateModel();
this.channel.BasicQos(0, 1, false);
consumer = new AsyncEventingBasicConsumer(channel);
consumer.Received += Consumer_Recieved;
this.channel.BasicConsume(queue: queue, autoAck: false, consumer: consumer);
}
private async Task Consumer_Recieved(object sender, BasicDeliverEventArgs #event)
{
var body = #event.Body;
var message = Encoding.UTF8.GetString(body.ToArray());
var inboundTransferObject = PatientObject.ConvertFromJson(message);
//logger.LogInformation("Processed message " + inboundTransferObject.WebhookMessageId);
//ServicePointManager.SecurityProtocol = SecurityProtocolType.SystemDefault;
//X509Certificate2 cert = new X509Certificate2(config["CertificationPath"].ToString(), config["PFXPassword"]);
//JToken access_token = GetAccessToken(cert);
//JObject payerData = GetPractitionerData(inboundTransferObject, cert, access_token);
//inboundTransferObject = ProcessPractitioner(inboundTransferObject, payerData);
var outboundTransferObject = Encoding.ASCII.GetBytes(inboundTransferObject.ConvertToJson());
channel.BasicAck(deliveryTag: #event.DeliveryTag, multiple: false);
publishingChannel.BasicPublish(exchange: "ExchangeA", routingKey: "Queue_StageTwo", basicProperties: null, body:outboundTransferObject);
await Task.Delay(250);
}
It's not clear exactly what you're asking here but one thing that does stand out is that your services should not acknowledge the inbound message unless and until they've completed all their processing steps, and that includes publishing follow-on outbound messages. In your code sample you appear to acknowledge the inbound message before publishing the outbound message.
That however does not explain the symptom you described "I run into a lot of unacknowledged messages". When do you run into these? How many is a lot? Have you set a prefetch limit on your channel? For testing purposes, you could try setting your prefetch count to one to ensure that only one message is in-flight at a time.
channel.BasicQos(1, global: true)
Please see this section of the RabbitMQ documentation:
"Because messages are sent (pushed) to clients asynchronously, there is usually more than one message "in flight" on a channel at any given moment. In addition, manual acknowledgements from clients are also inherently asynchronous in nature. So there's a sliding window of delivery tags that are unacknowledged. Developers would often prefer to cap the size of this window to avoid the unbounded buffer problem on the consumer end. This is done by setting a "prefetch count" value using the basic.qos method. The value defines the max number of unacknowledged deliveries that are permitted on a channel. Once the number reaches the configured count, RabbitMQ will stop delivering more messages on the channel unless at least one of the outstanding ones is acknowledged."
I have the following Actor where I am trying to restart and resend the failing message back to the actor :
public class BuildActor : ReceivePersistentActor
{
public override string PersistenceId => "asdad3333";
private readonly IActorRef _nextActorRef;
public BuildActor(IActorRef nextActorRef)
{
_nextActorRef = nextActorRef;
Command<Workload>(x => Build(x));
RecoverAny(workload =>
{
Console.WriteLine("Recovering");
});
}
public void Build(Workload Workload)
{
var context = Context;
var self = Self;
Persist(Workload, async x =>
{
//after this line executes
//application goes into break mode
//does not execute PreStart or Recover
var workload = await BuildTask(Workload);
_nextActorRef.Tell(workload);
context.Stop(self);
});
}
private Task<Workload> BuildTask(Workload Workload)
{
//works as expected if method made synchronous
return Task.Run(() =>
{
//simulate exception
if (Workload.ShowException)
{
throw new Exception();
}
return Workload;
});
}
protected override void PreRestart(Exception reason, object message)
{
if (message is Workload workload)
{
Console.WriteLine("Prestart");
workload.ShowException = false;
Self.Tell(message);
}
}
}
Inside the success handler of Persist I am trying to simulate an exception being thrown but on exception the application goes in to break mode and PreRestart hook is not invoked. But if I make BuildTask method synchronous by removing Task.Run then on exception both PreRestart and Recover<T> methods are invoked.
I would really appreciated if someone can point to me what should be the recommended pattern for this and where I am going wrong.
Most probably, Akka.Persistence is not the good solution for your problem here.
Akka.Persistence uses eventsourcing principles for storing actor's state. Few key points important in this context:
What you're sending to actor, is a command. It describes a job, you want to be done. Executing that command may result in doing some actual processing and eventually may lead to persist actor's linear state change history in form of the events.
In Akka.NET Persist method is used only to store events - they describe the fact, that something has happened: because of that, they cannot be denied and they cannot fail (a thing that you're doing in your Persist callback).
When an actor restarts at any point in time, it will always try to recreate its own state by replaying all events Persisted up to the last known point in time. For this reason it's important that Recover method should only focus on replaying actor's state (it can be called multiple times over the same event) and never result in side effects (example of side effect is sending an email). Any exception thrown there will mean, that actor state is irrecoverably corrupted and that actor will be killed.
If you want to resend the message to your actor, you could:
Put a reliable message queue (i.e. RabbitMQ or Azure Service Bus) or log (Kafka or Event Hub) in front of your actor processing pipeline. This is actually the most reasonable scenario in many cases.
Use at-least-once delivery semantics from Akka.Persistence - but IMHO only if for some reason you cannot use 1st solution.
The most simplistic and unreliable option (since messages are residing only in memory and never persisted) is dead letter queue. Every unhandled message is send there. You can subscribe to it and filter the incoming data to detect which messages should be send again to their recipients.
I am integrating with an ActiveMQ JMS system using the Apache.NMS library. For the response queue listener, it's not clear whether the consumer is disposed after a message is received.
Here are excerpts from the solution:
var destination = getDestination(session, Settings.Instance.OutboundQueue);
// Create a consumer and producer
using (var consumer = session.CreateConsumer(destination))
{
// Start the connection so that messages will be processed.
connection.Start();
consumer.Listener += OnMessage;
var receiveTimeout = TimeSpan.FromSeconds(Settings.Instance.Timeout);
// Wait for the message
semaphore.WaitOne((int)receiveTimeout.TotalMilliseconds, true);
}
// The Listener event
protected void OnMessage(IMessage receivedMsg)
{
var message = receivedMsg as ITextMessage;
semaphore.Set();
// process the message
}
Is the consumer durable?
Do you have to resubscribe after receiving a message?
Is this similar to other queuing/listener implementations (SQL Server service broker or the TCP/IP listener) where you need a while(true) loop to just keep the listener active?
Because of the way you've coded this, I believe (my .NET is rusty) you would need to create a new listener on each message as the using block will dispose of the consumer.
If you coded things such that the consumer was a member variable where it was saved away and only closed when you wanted to stop listening then you would not have this issue. The using block by it's nature will dispose of the resources that you ask it to manage.
I am creating a consumer that runs in an infinite loop to read messages from the queue. I am looking for advice/sample code on how to recover abd continue within my infinite loop even if there are network disruptions. The consumer has to stay running as it will be installed as a WindowsService.
1) Can someone please explain how to properly use these settings? What is the difference between them?
NetworkRecoveryInterval
AutomaticRecoveryEnabled
RequestedHeartbeat
2) Please see my current sample code for the consumer. I am using the .Net RabbitMQ Client v3.5.6.
How will the above settings do the "recovery" for me?
e.g. will consumer.Queue.Dequeue block until it is recovered?
That doesn't seem right
so...
Do I have to code for this manually? e.g. will consumer.Queue.Dequeue throw an exception for which I have to detect and manually re-create my connection, channel, and consumer? Or just the consumer, as "AutomaticRecovery" will recover the channel for me?
Does this mean I should move the consumer creation inside the while loop? what about the channel creation? and the connection creation?
3) Assuming I have to do some of this recovery code manually, are there event callbacks (and how do I register for them) to tell me that there are network problems?
Thanks!
public void StartConsumer(string queue)
{
using (IModel channel = this.Connection.CreateModel())
{
var consumer = new QueueingBasicConsumer(channel);
const bool noAck = false;
channel.BasicConsume(queue, noAck, consumer);
// do I need these conditions? or should I just do while(true)???
while (channel.IsOpen &&
Connection.IsOpen &&
consumer.IsRunning)
{
try
{
BasicDeliverEventArgs item;
if (consumer.Queue.Dequeue(Timeout, out item))
{
string message = System.Text.Encoding.UTF8.GetString(item.Body);
DoSomethingMethod(message);
channel.BasicAck(item.DeliveryTag, false);
}
}
catch (EndOfStreamException ex)
{
// this is likely due to some connection issue -- what am I to do?
}
catch (Exception ex)
{
// should never happen, but lets say my DoSomethingMethod(message); throws an exception
// presumably, I'll just log the error and keep on going
}
}
}
}
public IConnection Connection
{
get
{
if (_connection == null) // _connection defined in class -- private static IConnection _connection;
{
_connection = CreateConnection();
}
return _connection;
}
}
private IConnection CreateConnection()
{
ConnectionFactory factory = new ConnectionFactory()
{
HostName = "RabbitMqHostName",
UserName = "RabbitMqUserName",
Password = "RabbitMqPassword",
};
// why do we need to set this explicitly? shouldn't this be the default?
factory.AutomaticRecoveryEnabled = true;
// what is a good value to use?
factory.NetworkRecoveryInterval = TimeSpan.FromSeconds(5);
// what is a good value to use? How is this different from NetworkRecoveryInterval?
factory.RequestedHeartbeat = 5;
IConnection connection = factory.CreateConnection();
return connection;
}
RabbitMQ features
The documentation on RabbitMQ's site is actually really good. If you want to recover queues, exchanges and consumers, you're looking for topology recovery, which is enabled by default. Automatic Recovery (which is enabled by default) includes:
Reconnect
Restore connection listeners
Re-open channels
Restore channel listeners
Restore channel basic.qos setting, publisher confirms and transaction settings
The NetworkRecoveryInterval is the amount of time before a retry on an automatic recovery is performed (defaults to 5s).
Heartbeat has another purpose, namely to identify dead TCP connections. There are more to read about that at RabbitMQ's site.
Code sample
Writing reliable code for recovery is tricky. The EndOfStreamException is (as you suspect) most likely due to network problems. If you use the management plugin, you can reproduce this by closing the connection from there and see that the exception is triggered. For production-like applications, you might want to have a set of brokers that you alternate between in case of connection failure. If you have several RabbitMQ brokers, you might also want to guard yourself against long-term server failure on one or more of the servers. You might want to implement error strategies, like requeuing the message, or using a dead letter exchange.
I've been thinking a bit of these things and written a thin client, RawRabbit, that handles some of these things. Maybe it could be something for you? If not, I would suggest that you change the QueueingBasicConsumer to an EventingBasicConsumer. It is event driven, rather than thread blocking.
var eventConsumer = new EventingBasicConsumer(channel);
eventConsumer.Received += (sender, args) =>
{
var body = args.Body;
eventConsumer.Model.BasicAck(args.DeliveryTag, false);
};
channel.BasicConsume(queue, false, eventConsumer);
If you have topology recovery activated, the consumer will be restored by the RabbitMQ Client and start receiving messages again.
For more granular control, hook up event handlers for ConsumerCancelled and Shutdown to detect connectivity problems and Registered to know when the consumer can be used again.