We Keep Coding

Requeue Ibm MQ Message

We are running multiple instances of a windows service that reads messages from a Topic, runs a report, then converts the results into a PDF and emails them to a user. In case of exceptions we simply log the exception and move on.
The use case we want to handle is when the service is shut down we want to preserve the jobs that are currently running so they can be reprocessed by another instance of the service or when the service is restarted.
Is there a way of requeueing a message? The hacky solution would be to just republish the message from the consuming service, but there must be another way.
When incoming messages are processed, their data is put in an internal queue structure (not a message queue) and processed in batches of parallel threads, so the IbmMq transaction stuff seems hard to implement. Is that what I should be using though?

Your requirement seems to be hard to implement if you don't get rid of the "internal queue structure (not a message queue)" if this is not based on a transaction oriented middleware. The MQ queue / topic works well for multi-threaded consumers, so it is not apparent what you gain from this intermediate step of moving the data to just another queue. If you start your transaction with consuming the message from MQ, you can have it being rolled back when something goes wrong.

If I understood your use case correctly, you can use Durable subscriptions:
Durable subscriptions continue to exist when a subscribing application's connection to the queue manager is closed.
The details are explained in DEFINE SUB (create a durable subscription). Example:
DEFINE QLOCAL(THE.REPORTING.QUEUE) REPLACE DEFPSIST(YES)
DEFINE TOPIC(THE.REPORTING.TOPIC) REPLACE +
TOPICSTR('/Path/To/My/Interesting/Thing') DEFPSIST(YES) DURSUB(YES)
DEFINE SUB(THE.REPORTING.SUB) REPLACE +
TOPICOBJ(THE.REPORTING.TOPIC) DEST(THE.REPORTING.QUEUE)
Your service instances can consume now from THE.REPORTING.QUEUE.

While I readily admit that my knowledge is shaky, from what I understood from IBM’s [sketchy, inadequate, obtuse] documentation there really is no good built in solution. With transactions the Queue Manager assumes all is well unless it receives a roll back request and when it does it rolls back to a syncpoint, so if you’re trying to roll back to one message but two other messages have completed in the meantime it will roll back all three.
We ended up coding our own solution updating the way we’re logging messages and marking them as completed in the DB. Then on both startup and shutdown we find the uncompleted messages and programmatically publish them back to the queue, limiting the DB search by machine name so if we have multiple instances of the service running they won’t duplicate message processing.

C# RabbitMQ Design Approach

I am just getting started with RabbitMQ, because on a website I am working on we want to de-couple some of the resource intensive tasks such as sending emails, generating PDF's etc.
I have started by following the very simple C# "Hello world" tutorial on the RabbitMQ website (https://www.rabbitmq.com/tutorials/tutorial-one-dotnet.html). This was useful to get a very brief understanding of how RabbitMQ hangs together, but it has left me with a number of questions that genuinely and surprisingly I can't find answers to online....
The "Hello world" example sends a basic string. In my example of sending emails, would my Publisher send all the data needed to send the email i.e. recipient, subject etc, perhaps in JSon format?
How typically would you structure a consumer to execute a method to DoSomething? Ideally, I would like it to be fluent so that if a message is of a particular type the Consumer executes method SendEmail(), or if the message is of a different type it executes the method GeneratePDF() and so on....
I have a Publisher and a Consumer, however I have a number of different tasks that I want the Consumer to process i.e. send emails or generate PDFS. Should i have multiple consumers i.e. one for each type of task, or, multiple queue's (again one for each task)?
These are some of the basic questions I have that currently are preventing me from seeing how RabbitMQ is used in real-world scenario. Any help would be greatly appreciated.

With messaging, typically you send small-ish packets of data (like simple JSON objects) in the same manner you would as though you were defining an http-based API. The function definitions and input/output specifications can be identical - think of messaging as just a different transport mechanism.
For questions 2-3, yes, you want to set up multiple queues with multiple consumer types. Each consumer type would subscribe to exactly one queue whose messages are intended only for that consumer. Use routing logic on the publisher to determine which queue the messages end up in.

Deleting a message from Azure Queue Service by value

I'm using an Azure Storage Queue Service to track a queue of long running jobs which are en-queued by multiple disparate clients, but I now have a requirement to remove messages from the queue if any exist matching some given criteria.
I realise that this is somewhat anti-pattern for a queue, but the service does provide some functionality in addition to simple queueing (such as Delete Message and Peek Messages) so I thought I'd try to implement it.
The solution I've come up with works, but is not very elegant and is quite inefficient - and I'm wondering if it can be done better - or if I should bin the whole approach and use a mechanism which supports this requirement by design (which would require a fair amount of work across different systems). Here is the simplified code:
var queue = MethodThatGetsTheAppropriateQueueReference();
await queue.FetchAttributesAsync(); //populates the current queue length
if (queue.ApproximateMessageCount.HasValue)
{
// Get all messages and find any messages to be removed.
// this makes those messages unavailable for other clients
// for the visibilityTimeOut period.
// I've set this to the minimum of 1 second - not ideal though
var messages = await queue.GetMessagesAsync(queue.ApproximateMessageCount.Value);
var messagesToDelete = messages.Where(x => x.AsString.Contains(someGuid));
// Delete applicable messages
messagesToDelete.ToList().ForEach(x => queue.DeleteMessageAsync(x));
}
Note originally I tried using PeekMessagesAsync() to avoid affecting messages which do not need to be deleted, but this does not give you a PopReceipt which is required by DeleteMessageAsync().
The questions:
Is there a way to do this without pulling ALL of the messages down? (there could be quite a few)
If 1 isnt possible, is there a way to get the PopReceipt for a message if we use PeekMessagesAsync()?

Is there a way to do this without pulling ALL of the messages down?
(there could be quite a few)
Unfortunately no. You have to Get messages (a maximum of 32 at a time) and analyze the contents of the messages to determine if the message should be deleted.
If 1 isnt possible, is there a way to get the PopReceipt for a message
if we use PeekMessagesAsync()?
Again, No. In order to get PopReceipt, a message must be dequeued which is only possible via GetMessagesAsync(). PeekMessagesAsync() simply returns the message without altering its visibility.
Possible Solution
You may want to look into Service Bus Topics and Subscriptions for this kind of functionality.
What you could do is create a topic where all messages will be sent.
Then you would create 2 subscriptions: In one subscription you will set a rule which checks for message contents for the matching value and in other subscription you will set a rule which checks for message contents for not matching value.
What Azure Service Bus will do is check each message that arrives against the rules and accordingly pushes the message in appropriate subscription. This way you will have a nice separation of messages that should/shouldn't be deleted.

NService Bus - Content based routing & auditing - is my approach ok?

I have a little trouble deciding which way to go for while designing the message flow in our system.
Because the volatile nature of our business processes (i.e. calculating freight costs) we use a workflow framework to be able to change the process on the fly.
The general process should look something like this
The interface is a service which connects to the customers system via whatever interface the customer provides (webservices, tcp endpoints, database polling, files, you name it). Then a command is sent to the executor containing the received data and the id of the workflow to be executed.
The first problem comes at the point where we want to distribute load on multiple worker services.
Say we have different processes like printing parcel labels, calculating prices, sending notification mails. Printing the labels should never be delayed because a ton of mailing workflows is executed. So we want to be able to route commands to different workers based on the work they do.
Because all commands are like "execute workflow XY" we would be required to implement our own content based routing. NServicebus does not support this out of the box, most times because it's an anti pattern.
Is there a better way to do this, when you are not able to use different message types to route your messages?
The second problem comes when we want to add a monitoring. Because an endpoint can only subscribe to one queue for each message type we can not let all executors just publish a "I completed a workflow" message. The current solution would be to Bus.Send the message to a pre configured auditing endpoint. This feels a little like cheating to me ;)
Is there a better way to consolidate published messages of multiple workers into one queue again? If there would not be problem #1 I think all workers could use the same input queue however this is not possible in this scenario.

You can try to make your routing not content-based, but headers-based which should be much easier. You are not interested if the workflow is to print labels or not, you are interested in whether this command is priority or not. So you can probably add this information into the message header...

C# Advice about queueing

I am seeking advice on a project I have been assigned and I'm looking to see how it's done "professionally," or any suggestions that can lead me to the right direction.
I have a server piece that accepts commands from clients and pushes out byte streams to a serial port. Although multiple clients can send commands to this server piece, our hardware only can handle one command at a time. My problem is with queueing in the software end.
I've implemented a Queue<T> helper class that also inserts data into a DataSet containing: the requesting client number, message data (byte array to write to serial port) and message type (command description). It will also list the queue commands in a DataGrid (on the Form). Probably not the way to go, but that's the only thing I can think of as far as retaining the requesting client and the data and showing, visually, the queue.
Where do I handle the processing of the queue? I thought about handling it on a custom event where if the DataGrid list changed (item added/removed), grab the first row of data in the DataSet and send it out to the serial port.
Any comments or suggestions are greatly appreciated.
Thanks.
Edit: I forgot to add that it does require a response from the SerialPort as well, in order for the current executed command to be removed from the queue.

I would use a database table to store the queue of commands. The web app would add records to the queue and display the queue, then a separate process (such as a Windows service or console app) would request the next command from the database and send it to the serial port.

Client requests can come in at any time, they'll probably be handled by some proxy class (WCF?) on its own thread/task. Then that thread/ task needs to coordinate with the task that's 'inside' the model actually processing the requests.
A good class to do this with is the BlockingCollection.
The server-thread will block until there's something in the collection to work on. It can then take it from the collection in a thread safe manner and process it. Doing it this way ensures that the requests can be accepted when they arrive, but they are processed on at a time.
The overall pattern to think of here is producer-consumer.
GJ

If it is a high trasaction web application you might want to look at queueing system such as MSMQ, Service Broker Queue or RabbitMQ. Window service can then pick up the queued items and send it to serial port.