We are running multiple instances of a windows service that reads messages from a Topic, runs a report, then converts the results into a PDF and emails them to a user. In case of exceptions we simply log the exception and move on.
The use case we want to handle is when the service is shut down we want to preserve the jobs that are currently running so they can be reprocessed by another instance of the service or when the service is restarted.
Is there a way of requeueing a message? The hacky solution would be to just republish the message from the consuming service, but there must be another way.
When incoming messages are processed, their data is put in an internal queue structure (not a message queue) and processed in batches of parallel threads, so the IbmMq transaction stuff seems hard to implement. Is that what I should be using though?
Your requirement seems to be hard to implement if you don't get rid of the "internal queue structure (not a message queue)" if this is not based on a transaction oriented middleware. The MQ queue / topic works well for multi-threaded consumers, so it is not apparent what you gain from this intermediate step of moving the data to just another queue. If you start your transaction with consuming the message from MQ, you can have it being rolled back when something goes wrong.
If I understood your use case correctly, you can use Durable subscriptions:
Durable subscriptions continue to exist when a subscribing application's connection to the queue manager is closed.
The details are explained in DEFINE SUB (create a durable subscription). Example:
DEFINE QLOCAL(THE.REPORTING.QUEUE) REPLACE DEFPSIST(YES)
DEFINE TOPIC(THE.REPORTING.TOPIC) REPLACE +
TOPICSTR('/Path/To/My/Interesting/Thing') DEFPSIST(YES) DURSUB(YES)
DEFINE SUB(THE.REPORTING.SUB) REPLACE +
TOPICOBJ(THE.REPORTING.TOPIC) DEST(THE.REPORTING.QUEUE)
Your service instances can consume now from THE.REPORTING.QUEUE.
While I readily admit that my knowledge is shaky, from what I understood from IBM’s [sketchy, inadequate, obtuse] documentation there really is no good built in solution. With transactions the Queue Manager assumes all is well unless it receives a roll back request and when it does it rolls back to a syncpoint, so if you’re trying to roll back to one message but two other messages have completed in the meantime it will roll back all three.
We ended up coding our own solution updating the way we’re logging messages and marking them as completed in the DB. Then on both startup and shutdown we find the uncompleted messages and programmatically publish them back to the queue, limiting the DB search by machine name so if we have multiple instances of the service running they won’t duplicate message processing.
Related
I'm trying to figure out how to implement a fault-tolerant message publication solution using MassTransit. We'll focus on the simple scenario where we only need to commit a database change, and publish an event indicating that change. Because there is no (built-in) mechanism that allows an atomic "commit and publish", when our process crashes, we will get into an inconsistent state (some messages would only be committed to the database, and some might only be published to the message queue).
This documentation page offers a solution, where because we assume message handling is idempotent, we can rely on the entire operation to be retried in case of failure, and these partial commits will be resolved eventually. This is a great solution, but it only has one caveat: it assumes that the operation we are performing was triggered by a message, and if we won't send an ack, processing will be retried. This is not a reasonable assumption, as messaging is typically used only for internal communication inside the system, not for communication with the outside world. What should I do when I need to save-and-publish when handling an HTTP request from an external client?
One possible solution is to hack our way into the approach presented in the article, by only publishing (or sending) a message, and listening to it ourselves, then in the message handler we do the commit and the publishing of the actual event we want others to listen to. The main problem I have with this is that it assumes we never have to return anything in the HTTP response. What if we need to indicate the success or failure of the database transaction back to the HTTP client? (example: if we rely on a UNIQUE constraint to tell us whether or not we should accept the request, and we want to indicate failure to the client). We could solve it by using request-response over the message queue (with ourselves), but this is ugly and increases latency and complexity considerably, for what is actually a very common scenario.
The approach I see the most over the internet to solve this problem, is to use an outbox that is persisted to the same database we need to write to anyway, and thus we can wrap the two operations in a regular ACID database transaction. Then a background task polls this database for new events and publishes them to the message broker. Unlike other frameworks, I understand that MassTransit does not support this behavior out of the box. So I guess my question boils down to: before rushing to implement this relatively complex mechanism myself (once per database technology), is there another solution I'm missing? what is the accepted solution to this problem in the MassTransit community?
This has been asked several times, in a variety of forms, here and other places. But the short answer is simple.
In your controller, write to the message broker only. Let the consumer deal with the database, in the context of consuming a reliable message, with all the nice retry and redelivery options that are available in that context. Then you get all the benefits of the InMemoryOutbox, without adding extreme complexity related to having a third-party (HTTP, database, and broker) in a single conversation.
I have C# application that only reads messages from IBM MQ (ver 9.x). I'm calling MQQueueManager's AccessQueue() method and then MQQueue.Get(), reading messages in a loop and eventually call queue.Close().
I cannot find explicit information whether should I call MQQueueManager.Commit() after each messages (or before queue.Close() or ever) to be sure that processed messages are not consumed second time by another client, that's connected to the same queueName. can someone clarify this for me?
Here are message options: MQC.MQGMO_WAIT | MQC.MQGMO_FAIL_IF_QUIESCING
and open options: MQC.MQSO_FAIL_IF_QUIESCING | MQC.MQSO_MANAGED | MQC.MQSO_NON_DURABLE | MQC.MQOO_INPUT_AS_Q_DEF.
It is somewhat confusing why you are using MQSO options for a queue (SO = Subscription Options), however these are just bitwise operations, so you would be getting the results of what the equivalent MQOO option is for the same MQSO. See below:
Bits Options
Sub Options
Equivalent Open Options
0x00000000
MQSO_NONE MQSO_NON_DURABLE MQSO_READ_AHEAD_AS_Q_DEF
MQOO_BIND_AS_Q_DEF MQOO_READ_AHEAD_AS_Q_DEF
0x00000001
MQSO_ALTER
MQOO_INPUT_AS_Q_DEF
0x00000020
MQSO_MANAGED
MQOO_INQUIRE
0x00002000
MQSO_FAIL_IF_QUIESCING
MQOO_FAIL_IF_QUIESCING
This means that you are opening the queue with the equivalent of the following options:
MQOO_BIND_AS_Q_DEF | MQOO_READ_AHEAD_AS_Q_DEF | MQOO_INPUT_AS_Q_DEF | MQOO_INQUIRE |MQOO_FAIL_IF_QUIESCING
This may or may not be what you want, but if nothing else it is extremely unclear how you have it currently.
Commit is only required if you do your GET under sync point. This would mean you would add MQGMO_SYNCPOINT or MQGMO_SYNCPOINT_IF_PERSISTENT to your Get Message Options. If you were to do this then the messages are not fully removed from the queue until you issue MQQueueManager.Commit(), it does not mean that other instances are able to read them at the same time, but if one of the following conditions happens the messages will be back on the queue available to be read by another instance of the application:
The application crashes
The application is disconnected from the queue manager
The application issues MQQueueManager.Backout()
When you get a message without MQGMO_SYNCPOINT or MQGMO_SYNCPOINT_IF_PERSISTENT the message is immediately removed from the queue. The message can never be read by another instance of the application.
In IBM MQ you can non-destructively read messages from a queue, this is called browsing the queue and is the result of opening the queue with MQOO_BROWSE and specifying one of the following get message options:
MQGMO_BROWSE_FIRST
MQGMO_BROWSE_NEXT
MQGMO_BROWSE_MSG_UNDER_CURSOR
Most of the time the message you browsed can still be read by another application that opens the same queue. You can however also add MQGMO_LOCK which will lock the message from being read by another application. This allows you to later issue a destructive GET against the message you locked, several things will release the lock including issuing another GET call.
If you open the queue with one of the MQOO_INPUT* options and do not specify a MQGMO_BROWSE* option, then you are destructively reading messages from the queue. This means the message will not be available for another application to read.
You can also issue a destructive GET with MQGMO_SYNCPOINT for any message (persistent or non-persistent) (or MQGMO_SYNCPOINT_IF_PERSISTENT if you only want it to apply to persistent messages) and the message will be read in a unit of work, it is no longer available for another application to get, but it is not removed from the queue until the unit of work is committed.
I could not find any specific statement from IBM that says the following:
When you get a message without MQGMO_SYNCPOINT or MQGMO_SYNCPOINT_IF_PERSISTENT the message is immediately removed from the queue. The message can never be read by another instance of the application.
Here are some references that together paint this picture:
IBM Knowledge center page IBM MQ>About IBM MQ>Introduction to IBM MQ
IBM MQ provides:
Qualities of service that provide once and once only delivery of messages to ensure messages will withstand application and system outages
IBM Knowledge center page IBM MQ>Developing applications>Developing MQI applications with IBM MQ>Writing a procedural application for queuing>Committing and backing out units of work
Commit and back out
Similarly, when a program gets a message from a queue within a unit of work, that message remains on the queue until the program commits the unit of work, but the message is not available to be retrieved by other programs. The message is permanently deleted from the queue when the program commits the unit of work. If the program backs out the unit of work, IBM MQ restores the queue by making the messages available to be retrieved by other programs.
Syncpoint coordination, syncpoint, unit of work
Syncpoint coordination is the process by which units of work are either committed or backed out with data integrity.
The decision to commit or back out the changes is taken, in the simplest case, at the end of a transaction. However, it can be more useful for an application to synchronize data changes at other logical points within a transaction. These logical points are called syncpoints (or synchronization points ) and the period of processing a set of updates between two syncpoints is called a unit of work. Several MQGET calls and MQPUT calls can be part of a single unit of work.
IBM Knowledge center page IBM MQ>Developing applications>Developing MQI applications with IBM MQ>Writing a procedural application for queuing>Getting messages from a queue>Getting messages from a queue using the MQGET call
Specifying MQGET options using the MQGMO structure
...
Options
Within your code, you can select the options in any order; each option is represented by a bit in the Options field.
The Options field controls:
...
Whether the get operation is included in a unit of work.
...
Whether the message is removed from the queue, or merely browsed
We use rabbit mq to send messages to a server for processing.
We require the server to ack a message. That way if the server happens to die whilst processing the message, we will retry the message when it restarts, or with a different server.
The problem is, on a very rare occasion, we will get a message that deterministically crashes the server. This is because we call into some open source native dlls, those dlls have bugs, and sometimes these dlls just cause the process to crash with no exception. Of course it would be ideal to fix those bugs, but we don't expect to fix all such issues in pdfium or opencv any time soon. We have to reckon with the fact that whatever we do, we will eventually get such a message.
The result of this is that the message is then retried, the server restarts, picks ups the message, crashes, and so on ad infinitum. Nothing gets processed till we manually stop the server, and purge the message. Not ideal.
What can we do to solve this problem?
What we don't want to do is create another service that monitors the rabbitmq service, looks for such messages and purges them, since that just leads to spiralling complexity. Instead we want to deal with this at the rabbitmq client level. We would be perfectly happy to say that if a message is not processed 3 times, we should just fail the message. We could do this by maintaining a database entry of which messages we've processed, but ideally I wouldn't want to involve anything external, and just contain the solution to this problem in our rabbitmq client library. I'm not sure how to do this though.
One method I have used in my event driven architecture is to use dead letter exchanges (DLXs) or poison queues, that way if we see the same message multiple times due to service failure then it'll be pushed into the DLX instead of being re-queued into the original exchange. These messages then trigger a different type of process within our system to alert us messages are stuck and failing to process, we can then diagnose and fix the consumer. After a fix has been made we trigger another process to move the poison messages back into the original exchange to be then processed as normal.
In your scenario because your process crashes there is two possible options to deal with these messages:
If the message is marked as redelivered then clone the message and add an attempt count to the body or as a header (x-attempt-count) to the message. The copy will then be added to the back of the queue with the attempt count. When the copy is then consumed you can check if it hits the threshold and then move the message into a DLX or store in a database. The major drawback here is that it breaks the order of which the messages are processed.
Use an external services to keep track of the number of delivery attempts, I would recommend using something like redis/memcache where you can increment a counter based on a unique message id. At the start of your process if the message has been marked as redelivered then lookup the counter. If the message has reached the threshold, trigger a different process again like moving it into a DLX.
I have a subscription model, and want to perform renew-related logic like issue new invoice, send emails, etc. For example, user would purchase the subscription today, and the renewal is in a year's time. I've been using an Azure Queue recently, and think it would apply for such a renewal.
Is it possible to use the Azure Queue by pushing messages using BrokeredMessage.ScheduledEnqueueTimeUtc (http://msdn.microsoft.com/en-us/library/microsoft.servicebus.messaging.brokeredmessage.scheduledenqueuetimeutc.aspx) for such long term scheduled messages?
I've used it for shorter-term, like sending notifications in 1 minute time and it works great.
This way, I can have even multiple processes listening to the queue, and be sure that only one process would perform the renewal logic. This would solve a lot of locking-related problems, as that is kind of built-in the Azure Queue via leasing and related features.
Yes, you can use it for long-term scheduling, scheduled messages have the same guaranties as normal ones. But there are few things you need to be aware of:
ScheduledEnqueueTimeUtc is a time when message is going to be available (within hundreds of miliseconds) on the queue but not necessary delivered, this depends on load and state of the queue. So it's fine for business processes but not for time sensitive (milliseconds) usage. Not a problem in your case, unless your subscription cancellation is really time sensitive.
It affects your storage quota ( Not really a problem with current quotas, but if you think about years this might be a problem)
As far as I'm aware you can't access scheduled messages before ScheduledEnqueueTimeUtc, they are invisible.
Extremely awesome source of informations on azure messaging
From technological perspective it's fine but in your case I would also think about other potential problems if you think about years:
Message versioning
What happens when you would like to change Azure to something else (AWS?)
What if you decide to change in next year Azure Service Bus for NServiceBus
I want to know if there is any elegant way to ensure that Queue always have distinct messages (nothing related to Duplicate Detection Window or any time period for that matter) ?
I know that Service Bus Queue provides session concepts (as I mentioned Duplicate Detection of Service Bus Queue won't help me as it depends on time period), which can serve my purpose, but I don't want my component's dependency on another Azure service, just because of this feature.
Thanks,
This is not possible to do reliably.
There is just no mechanism that can query a Storage queue and find out if a message with the same contents is already there or was there before. You can try to implement your own logic using some storage table, but that will not be reliable - as the entry into the table may succeed and then entry into the queue may fail - and now you would potentially have bad data in the table.
Your code should always assume that it can retrieve a message containing the same data that was already processed. This is because messages can come back to the queue when workers that are working on them crash or take too long.
You can use Service Bus. Is like Azure Storage Queue but it allows messages of 256Kb-1MB and makes duplicate detection