I have this in a class called "MessageQueueReceive".
public MessageQueueTransaction BlockingReceive(out Message message)
{
MessageQueueTransaction tran = null;
message = null;
tran = new MessageQueueTransaction();
tran.Begin();
try
{
message = Queue.Receive(new TimeSpan(0, 0, 5), tran);
}
catch (MessageQueueException ex)
{
// If the exception was a timeout, then just continue
// otherwise re-raise it.
if (ex.MessageQueueErrorCode != MessageQueueErrorCode.IOTimeout)
throw ex;
}
return tran;
}
Then my processing loop has this:-
while (!Abort)
{
try
{
tran = this.Queue.BlockingReceive(out msg);
if (msg != null)
{
// Process message here
if (tran != null)
tran.Commit();
}
}
catch (Exception ex)
{
if (tran != null)
tran.Abort();
}
}
The control panel tool shows that the message queues I'm using are transactional. Journal queue is not enabled.
This code creates the queue:-
private static MessageQueue CreateMessageQueue(string queueName, bool transactional = false)
{
MessageQueue messageQueue = MessageQueue.Create(queueName, transactional);
messageQueue.SetPermissions("Administrators", MessageQueueAccessRights.FullControl,
AccessControlEntryType.Allow);
return messageQueue;
}
The transactional parameter is set as "true" when this is called.
What I find is that when an exception occurs during the processing of the message, tran.Abort is called but at that point I'd expect the message to be returned to the queue. However, this is not happening and the messages are lost.
Am I missing something obvious? Can anyone see what I'm doing wrong?
Thanks for all the comments. I did re-organise my code as Russell McClure suggested, and I tried to create simple test cases but could not reproduce the problem.
In the end, the problem was not at all where I was looking (how often does that happen?).
In my pipeline, I had a duplicate message checker. The "messages" my system deals with are from remote devices on a WAN, and occasionally messages on the wire are duplicated.
When a message was pulled from the MSMQ, it would pass via the duplicate checker the database writer. If the database writer failed, the duplicate checked did not remove the hash from its table. When the process tried to loop again, it would get the same message from the queue agan because the MSMQ transaction had been rolled back when the database writer failed. However, on the second attempt, the duplicate checker would spot that it had seen the message before, and swallow it silently.
The fix was to make the duplicate checker spot the exception coming from the next link in the chain, and roll-back anything it had done too.
Your queue needs to be created as a transactional queue to get what you want.
EDIT:
Well, if your queue is transactional then that points to the fact that you are mishandling your transaction, although I can't see specifically how it is happening. I would change your BlockingReceive method to return the message. I would move the creation of the MessageQueueTransaction to the outer method. Your code will be much more maintainable if you have the Begin, Commit and Abort method calls in the same method.
Related
I've got an azure function right now that runs on a service bus trigger (queue trigger) and outputs a SendGridMessage. The trick is I need to do some cleanup in my blob storage after the function has successfully sent a sendgrid message but it seems like I have no way of identifying whether or not the function was successful until after it goes out of scope.
I'm currently attempting to push the message that needs to be cleaned up to a cleanup queue and take care of it after the try catch but I think I'm still running into the same problem. The function could succeed and then fail on the SendGrid output and the message would be cleaned up but thrown back into the queue to be reprocessed on this function and fail. Bleh.
Queue Trigger and Sendgrid Output
[FunctionName("ProcessEmail")]
public static void Run([ServiceBusTrigger("email-queue-jobs", AccessRights.Manage,
Connection = "MicroServicesServiceBus")]OutgoingEmail outgoingEmail, TraceWriter log,
[ServiceBus("email-queue-cleanup", Connection = "MicroServicesServiceBus",
EntityType = Microsoft.Azure.WebJobs.ServiceBus.EntityType.Queue)] IAsyncCollector<OutgoingEmail> cleanupEmailQueue,
[SendGrid] out SendGridMessage message)
{
try
{
log.Info($"Attempting to send the email {outgoingEmail.Id}");
message = SendgridHelper.ConvertToSendgridMessage(outgoingEmail);
log.Info("Successfully sent email:");
log.Info(JsonConvert.SerializeObject(outgoingEmail));
}
catch (Exception ex)
{
message = null;
throw ex;
}
// Add email to the cleanup queue
log.Info("Sending email to the cleanup queue.");
cleanupEmailQueue.AddAsync(outgoingEmail).Wait();
}
You should be able to achieve this by using ICollector or IAsyncCollector
[SendGrid] ICollector<SendGridMessage> messageCollector)
and then
var message = SendgridHelper.ConvertToSendgridMessage(outgoingEmail);
messageCollector.Add(message);
should call SendGrid synchronously and throw exception in case of failure.
If you want to use IAsyncCollector (as you already do for another binding), be sure to call FlushAsync method too:
[SendGrid] IAsyncCollector<SendGridMessage> messageCollector)
and then
var message = SendgridHelper.ConvertToSendgridMessage(outgoingEmail);
await messageCollector.AddAsync(message);
await messageCollector.FlushAsync();
I'm looking for a solid way of re-queuing messages that couldn't be handled properly - at this time.
I've been looking at http://dotnetcodr.com/2014/06/16/rabbitmq-in-net-c-basic-error-handling-in-receiver/ and it seems that it's supported to requeue messages in the RabbitMQ API.
else //reject the message but push back to queue for later re-try
{
Console.WriteLine("Rejecting message and putting it back to the queue: {0}", message);
model.BasicReject(deliveryArguments.DeliveryTag, true);
}
However I'm using EasyNetQ.
So wondering how I would do something similar here.
bus.Subscribe<MyMessage>("my_subscription_id", msg => {
try
{
// do work... could be long running
}
catch ()
{
// something went wrong - requeue message
}
});
Is this even a good approach? Not ACK the message could cause problems if do work exceeds the wait for ACK timeout by the RabbitMQ server.
So I came up with this solution. Which replaces the default error strategy by EasyNetQ.
public class DeadLetterStrategy : DefaultConsumerErrorStrategy
{
public DeadLetterStrategy(IConnectionFactory connectionFactory, ISerializer serializer, IEasyNetQLogger logger, IConventions conventions, ITypeNameSerializer typeNameSerializer)
: base(connectionFactory, serializer, logger, conventions, typeNameSerializer)
{
}
public override AckStrategy HandleConsumerError(ConsumerExecutionContext context, Exception exception)
{
object deathHeaderObject;
if (!context.Properties.Headers.TryGetValue("x-death", out deathHeaderObject))
return AckStrategies.NackWithoutRequeue;
var deathHeaders = deathHeaderObject as IList;
if (deathHeaders == null)
return AckStrategies.NackWithoutRequeue;
var retries = 0;
foreach (IDictionary header in deathHeaders)
{
var count = int.Parse(header["count"].ToString());
retries += count;
}
if (retries < 3)
return AckStrategies.NackWithoutRequeue;
return base.HandleConsumerError(context, exception);
}
}
You replace it like this:
RabbitHutch.CreateBus("host=localhost", serviceRegister => serviceRegister.Register<IConsumerErrorStrategy, DeadLetterStrategy>())
You have to use the AdvancedBus so you have to setup everything up manually.
using (var bus = RabbitHutch.CreateBus("host=localhost", serviceRegister => serviceRegister.Register<IConsumerErrorStrategy, DeadLetterStrategy>()))
{
var deadExchange = bus.Advanced.ExchangeDeclare("exchange.text.dead", ExchangeType.Direct);
var textExchange = bus.Advanced.ExchangeDeclare("exchange.text", ExchangeType.Direct);
var queue = bus.Advanced.QueueDeclare("queue.text", deadLetterExchange: deadExchange.Name);
bus.Advanced.Bind(deadExchange, queue, "");
bus.Advanced.Bind(textExchange, queue, "");
bus.Advanced.Consume<TextMessage>(queue, (message, info) => HandleTextMessage(message, info));
}
This will dead letter a failed message 3 times. After that it'll go to the default error queue provided by EasyNetQ for error handling. You can subscribe to that queue.
A message is dead lettered when an exception propagates out of your consumer method. So this would trigger a dead letter.
static void HandleTextMessage(IMessage<TextMessage> textMessage, MessageReceivedInfo info)
{
throw new Exception("This is a test!");
}
to the best of my knowledge, there is no way to manually ack, nack or reject a message with EasyNetQ.
I see you have opened an issue ticket with the EasyNetQ team, regarding this... but no answer, yet.
FWIW, this is a very appropriate thing to do. All of the libraries that I use support this feature set (in NodeJS) and it is common. I'm surprised EasyNetQ doesn't support this.
I use RabbitMQ as my queue message server, I use .NET C# client.
When there is error in processing message from queue, message will not ackknowleage and still stuck in queue not be processed again as the document I understand.
I don't know if I miss some configurations or block of codes.
My idea now is auto manual ack the message if error and manual push this message to queue again.
I hope to have another better solution.
Thank you so much.
my code
public void Subscribe(string queueName)
{
while (!Cancelled)
{
try
{
if (subscription == null)
{
try
{
//try to open connection
connection = connectionFactory.CreateConnection();
}
catch (BrokerUnreachableException ex)
{
//You probably want to log the error and cancel after N tries,
//otherwise start the loop over to try to connect again after a second or so.
log.Error(ex);
continue;
}
//crate chanel
channel = connection.CreateModel();
// This instructs the channel not to prefetch more than one message
channel.BasicQos(0, 1, false);
// Create a new, durable exchange
channel.ExchangeDeclare(exchangeName, ExchangeType.Direct, true, false, null);
// Create a new, durable queue
channel.QueueDeclare(queueName, true, false, false, null);
// Bind the queue to the exchange
channel.QueueBind(queueName, exchangeName, queueName);
//create subscription
subscription = new Subscription(channel, queueName, false);
}
BasicDeliverEventArgs eventArgs;
var gotMessage = subscription.Next(250, out eventArgs);//250 millisecond
if (gotMessage)
{
if (eventArgs == null)
{
//This means the connection is closed.
DisposeAllConnectionObjects();
continue;//move to new iterate
}
//process message
channel.BasicAck(eventArgs.DeliveryTag, false);
}
}
catch (OperationInterruptedException ex)
{
log.Error(ex);
DisposeAllConnectionObjects();
}
}
DisposeAllConnectionObjects();
}
private void DisposeAllConnectionObjects()
{
//dispose subscription
if (subscription != null)
{
//IDisposable is implemented explicitly for some reason.
((IDisposable)subscription).Dispose();
subscription = null;
}
//dipose channel
if (channel != null)
{
channel.Dispose();
channel = null;
}
//check if connection is not null and dispose it
if (connection != null)
{
try
{
connection.Dispose();
}
catch (EndOfStreamException ex)
{
log.Error(ex);
}
catch (OperationInterruptedException ex)//handle this get error from dispose connection
{
log.Error(ex);
}
catch (Exception ex)
{
log.Error(ex);
}
connection = null;
}
}
I think you may have misunderstood the RabbitMQ documentation. If a message does not get ack'ed from the consumer, the Rabbit broker will requeue the message onto the queue for consumption.
I don't believe your suggested method for ack'ing and then requeuing a message is a good idea, and will just make the problem more complex.
If you want to explicitly "reject" a message because the consumer had a problem processing it, you could use the Nack feature of Rabbit.
For example, within your catch exception blocks, you could use:
subscription.Model.BasicNack(eventArgs.DeliveryTag, false, true);
This will inform the Rabbit broker to requeue the message. Basically, you pass the delivery tag, false to say it is not multiple messages, and true to requeue the message.
If you want to reject the message and NOT requeue, just change true to false.
Additionally, you have created a subscription, so I think you should perform your ack's directly on this, not through the channel.
Change:
channel.BasicAck(eventArgs.DeliveryTag, false);
To:
subscription.Ack();
This method of ack'ing is much cleaner since you are then keeping everything subscription-related on the subscription object, rather than messing around with the channel that you've already subscribed to.
I have a very simple client that I want to be available 24/7 to consume messages. It is running in a Windows process.
I have no issues with the server and receiving messages, it is just the client.
The behavior is as follows:
Works if I start the connection fresh. After some time, perhaps hours, my client is in an odd state; the connection it contains 'holds' unacked messages.
In other words, using the web admin interface, I see that I have a total of, say, 2 unacked messages. Looking at my connections, I see the 2 unacked messages spread out.
But there is no processing going on.
And eventually, my connections get killed, with no exceptions or log messages being triggered. This puts all the messages into the ready state.
My first attempt to solve the problem was to add a simple external loop that checked the state of the i-vars of IModel, IChannel, and QueueingBasicConsumer. However, IModel/IChannel's IsOpen always reports true, even after the web admin reports no connections are active, and QueueingBasicConsumer's IsRunning always reports true as well.
Clearly I need another method to check whether a connection is 'active'.
So to summarize, things work well initially. Eventually, I get into an odd state where my diagnostic checks are meaningless, and messages sent to the server get unacked, and are spread out across any existing connections. Soon, my connections are killed with no debugs or exceptions thrown, and my diagnostic checks still report things are kosher.
Any help or best practices would be appreciated. I have read up on heartbeat, and the IsOpen 'race' condition, where it is suggested to use BasicQos and check for an exception, however I want to first understand what is happening.
Here is where I kick things off:
private void StartMessageLoop(string uri, string queueName) {
this.serverUri = uri;
this.queueName = queueName;
Connect(uri);
Task.Factory.StartNew(()=> MessageLoopTask(queueName));
}
Here is how I connect:
private void Connect(string serverAddress) {
ConnectionFactory cf = new ConnectionFactory();
cf.Uri = serverAddress;
this.connection = cf.CreateConnection();
this.connection.ConnectionShutdown += new ConnectionShutdownEventHandler(LogConnClose);
this.channel = this.connection.CreateModel();
}
Here is where the infinite loop starts:
private void MessageLoopTask(string queueName) {
consumer = new QueueingBasicConsumer(channel);
String consumerTag = channel.BasicConsume(queueName, false, consumer);
while (true) {
try {
BasicDeliverEventArgs e = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
IBasicProperties props = e.BasicProperties;
byte[] body = e.Body;
string messageContent = Encoding.UTF8.GetString(body);
bool result = this.messageProcessor.ProcessMessage(messageContent);
if(result){
channel.BasicAck(e.DeliveryTag, false);
}
else{
channel.BasicNack(e.DeliveryTag, false, true);
// log
}
}
catch (OperationInterruptedException ex) {
// log
break;
}
catch(Exception e) {
// log
break;
}
}
// log
}
Regards,
Dane
In C# ASP.NET 3.5 web application running on Windows Server 2003, I get the following error once in a while:
"Object reference not set to an instance of an object.: at System.Messaging.Interop.MessagePropertyVariants.Unlock()
at System.Messaging.Message.Unlock()
at System.Messaging.MessageQueue.ReceiveCurrent(TimeSpan timeout, Int32 action, CursorHandle cursor, MessagePropertyFilter filter, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)
at System.Messaging.MessageEnumerator.get_Current()
at System.Messaging.MessageQueue.GetAllMessages()".
The line of code that throws this error is:
Message[] msgs = Global.getOutputQueue(mode).GetAllMessages();
where Global.getOutputQueue(mode) gives the messagequeue I want to get messages from.
Update:
Global.getPool(mode).WaitOne();
commonClass.log(-1, "Acquired pool: " + mode, "Report ID: " + unique_report_id);
............../* some code /
..............
lock(getLock(mode))
{
bool yet_to_get = true;
int num_retry = 0;
do
{
try
{
msgs = Global.getOutputQueue(mode).GetAllMessages();
yet_to_get = false;
}
catch
{
Global.setOutputQueue(mode);
msgs = Global.getOutputQueue(mode).GetAllMessages();
yet_to_get = false;
}
++num_retry;
}
while (yet_to_get && num_retry < 2);
}
... / some code*/
....
finally
{
commonClass.log(-1, "Released pool: " + mode, "Report ID: " + unique_report_id);
Global.getPool(mode).Release();
}
Your description and this thread suggests a timing issue. I would create the MessageQueue object infrequently (maybe only once) and have Global.getOutputQueue(mode) return a cached version, seems likely to get around this.
EDIT: Further details suggest you have the opposite problem. I suggest encapsulating access to the message queue, catching this exception and recreating the queue if that exception occurs. So, replace the call to Global.getOutputQueue(mode).GetAllMessages() with something like this:
public void getAllOutputQueueMessages()
{
try
{
return queue_.GetAllMessages();
}
catch (Exception)
{
queue_ = OpenQueue();
return queue_.GetAllMessages();
}
}
You'll notice I did not preserve your mode functionality, but you get the idea. Of course, you have to duplicate this pattern for other calls you make to the queue, but only for the ones you make (not the whole queue interface).
This is an old thread, but google brought me here so I shall add my findings.
I agree with user: tallseth that this is a timing issue.
After the message queue is created it is not instantly available.
try
{
return _queue.GetAllMessages().Length;
}
catch (Exception)
{
System.Threading.Thread.Sleep(4000);
return _queue.GetAllMessages().Length;
}
try adding a pause if you catch an exception when accessing a queue which you know has been created.
On a related note
_logQueuePath = logQueuePath.StartsWith(#".\") ? logQueuePath : #".\" + logQueuePath;
_queue = new MessageQueue(_logQueuePath);
MessageQueue.Create(_logQueuePath);
bool exists = MessageQueue.Exists(_logQueuePath);
running the MessageQueue.Exists(string nameofQ); method immediately after creating the queue will return false. So be careful when calling code such as:
public void CreateQueue()
{
if (!MessageQueue.Exists(_logQueuePath))
{
MessageQueue.Create(_logQueuePath);
}
}
As it is likely to throw an exception stating that the queue you are trying to create already exists.
-edit: (Sorry I don't have the relevant link for this new info)
I read that a newly created MessageQueue will return false on MessageQueue.Exists(QueuePath)until it has received at least one message.
Keeping this and the earlier points i mentioned in mind has gotten my code running reliably.