I have created a small class using RabbitMQ that implements a publish/subscribe messaging pattern on a topic exchange. On top of this pub/sub I have the methods and properties:
void Send(Message, Subject) - Publish message to destination topic for any subscribers to handle.
MessageReceivedEvent - Subscribe to message received events on this messaging instance (messaging instance is bound to the desired subscribe topic when created).
SendWaitReply(Message, Subject) - Send a message and block until a reply message is received with a correlation id matching the sent message id (or timeout). This is essentially a request/reply or RPC mechanism on top of the pub/sub pattern.
The messaging patterns I have chosen are somewhat set in stone due to the way the system is to be designed. I realize I could use reply-to queues to mitigate the potential issue with SendWaitReply, but that breaks some requirements.
Right now my issues are:
For the Listen event, the messages are processed synchronously through the event subscribers as the listener runs in a single thread. This causes some serious performance issues when handling large volumes of messages (i.e. in a back-end process consuming events from a web api). I am considering passing in a callback function as opposed to subscribing to an event and then dispatching the collection of callbacks in parallel using Task or Threadpool. Thread safety would obviously now be a concern of the caller. I am not sure if this is a correct approach.
For the SendWaitReply event, I have built what seems to be a hacky solution that takes all inbound messages from the message listener loop and places them in a ConcurrentDictionary if they contain a non-empty correlation guid. Then in the SendWaitReply method, I poll the ConcurrentDictionary for a message containing a key that matches the Id of the sent message (or timeout after a certain period). If there is a faster/better way to do this, I would really like to investigate it. Maybe a way to signal to all of the currently blocked SendWaitReply methods that a new message is available and they should all check their Ids instead of polling continuously?
Update 10/15/2014
After much exhaustive research, I have concluded that there is no "official" mechanism/helper/library to directly handle the particular use-case I have presented above for SendWaitReply in the scope of RabbitMQ or AMQP. I will stick with my current solution (and investigate more robust implementations) for the time being. There have been answers recommending I use the provided RPC functionality, but this unfortunately only works in the case that you want to use exclusive callback queues on a per-request basis. This breaks one of my major requirements of having all messages (request and reply) visible on the same topic exchange.
To further clarify, the typical message pair for a SendWaitReply request is in the format of:
Topic_Exchange.Service_A => some_command => Topic_Exchange.Service_B
Topic_Exchange.Service_B => some_command_reply => Topic_Exchange.Service_A
This affords me a powerful debugging and logging technique where I simply set up a listener on Topic_Exchange.# and can see all of the system traffic for tracing very deep 'call stacks' through various services.
TL; DR - Current Problem Below
Backing down from the architectural level - I still have an issue with the message listener loop. I have tried the EventingBasicConsumer and am still seeing a block. The way my class works is that the caller subscribes to the delegate provided by the instance of the class. The message loop fires the event on that delegate and those subscribers then handle the message. It seems as if I need a different way to pass the message event handlers into the instance such that they don't all sit behind one delegate which enforces synchronous processing.
It's difficult to say why your code is blocking without a sample, but to prevent blocking while consuming, you should use the EventingBasicConsumer.
var consumer = new EventingBasicConsumer;
consumer.Received += (s, delivery) => { /* do stuff here */ };
channel.BasicConsume(queue, false, consumer);
One caveat, if you are using autoAck = false (as I do), then you need to ensure you lock the channel when you do channel.BasicAck or you may hit concurrency issues in the .NET library.
For the SendWaitReply, you may have better luck if you just use the SimpleRpcClient included in the RabbitMQ client library:
var props = channel.CreateBasicProperties();
// Set your properties
var client = new RabbitMQ.Client.MessagePatterns.SimpleRpcClient(channel, exchange, ExchangeType.Direct, routingKey);
IBasicProperties replyProps;
byte[] response = client.Call(props, body, out replyProps);
The SimpleRpcClient will deal with creating a temporary queue, correlation ID's, and so on instead of building your own. If you find you want to do something more advanced, the source is also a good reference.
Related
We are using Rebus 4.2.1 and RabbitMQ
What we want to achieve is to have handlers on three (or more) instances all react to the same message.
As far as I understood (which may be wrong) - .Publish on the IBus interface should do exactly that (and we have been running with that on MSMQ).
Is there something I am missing with how RabbitMQ works?
(EDIT: I think the term used in RabbitMQ is a "fanout" style message)
EDIT2: mookid8000 put me on the right track - the issue was that each replica was asking for the same queue. As soon as I made that unique - everything started working as intended (and expected).
With Rebus + RabbitMQ, it's pretty simple, because RabbitMQ has native support for topic-based pub/sub messaging.
In each subscriber, you simply call
await bus.Subscribe<YourEvent>();
which will make Rebus generate a topic string out of your event type, and then bind it to the subscriber's input queue, and then in the publisher you
await bus.Publish(new YourEvent(...));
and then each subscriber will get a copy of the event in its input queue.
Underneath the covers, Rebus uses RabbitMQ's "Topic Exchange" exchange type to make this work.
I like to add azure service bus to one my projects. And the project does a few different tasks: sending email, processing orders, sending invoices etc.
What I am keen to know is, do I create seperate queues to process all these different tasks ? I understand that a queue has one sender and one reciever. That makes sense, but then I will end up with quite a few number of queues for a project. is that normal?
Based on your description:
The project does a few different tasks: sending email, processing orders, sending invoices etc.
These messages are not related to each other. I like to differentiate between commands and events. Commands are sent specifically to certain destination with an expectation of an outcome, knowing that operation could fail. With events it's different. Events are broadcasted and there are no expectations for success or failure. There's also no knowledge about consumers of events to allow complete decoupling. Events can only be handled using Topics/Subscriptions. Commands can be handled either with Queues or with Topics/Subscriptions (a topic with a single subscription would act as a queue).
If you go with events, you don't create separate consumer input queues. You create a topic and subscriptions on that topic. Let's say you'll have a PublisherApp and a ConsumerApp. PublisherApp could create a topic and send all messages to the events topic. ConsumerApp would create the required subscriptions, where each subscription would have a filter based on type of the message you'd like that subscription to receive. For your sample, it would be the following subscriptions:
SendMail
ProcessOrder
SendInvoice
In order to filter properly, your BrokeredMessages will have to have a header (property) that would indicate the intent. You could either come up with a custom header or use a standard one, like Label.
I wrote a blog post a while ago on topologies with ASB, have a look, it might give you more ideas on how you can set up your entities.
If topology & entities management is not what you'd like to do, there are good frameworks that can abstract it for your and allow your code to work w/o diving into details too much. NServiceBus and MassTransit are two good examples you can have a look at.
Full disclosure: I'm working on Azure Service Bus transport for NServiceBus.
First of all look at Azure Storage queue i just switched to it in almost the same scenario. In Storage queue there is NO monthly payment fee you pay for what you use.
Queue is not limited to receivers or senders. What i mean by that, is that you could have many listeners for a queue (in case your app is scaled) but as soon as listener picked up event then its locked and not visible to others. (By default timeout is around 30 sec in Azure storage queue and 60 sec in Service bus, so be aware if you need more time for processing your message you need renew lock otherwise you will end up with processing same message multiple times)
You can use one queue per all your events and depends on message body you can run different message processors. For instance in my project I am sending message with key Type which identify who is going to process this message. You can also use one queue per type and then in your listeners listen to multiple queues
Look at this link for comparison table
topics and subscriptions suit your scenario the most.
At the subscription end, you can filter the messages based on criteria,in your case, it can be your task i.e sendemail,processorder .
If you want to add more tasks in future, you will be free from making any changes on the service bus itself, and will only have do required changes on sender and receiver code.
If you use service bus queues or storage queues, in future , you have to create more queues for adding another tasks and it can become complicated on the management level of your azure infrastrcture.
There are 2 approaches based on your design.
Queue and Message: Message Body has an indicator tasks: sending email, processing orders, sending invoices etc. The code then process the message accordingly.
Topics and Subscriptions: Define topics for each tasks and brokered messages are processed accordingly. This should be better than the Queues.
I am still learning C# so please be easy on me. I am thinking about my application I am working on and I can't seem to figure out the best approach. This is not a forms application but rather a console. I am listening to a UDP port. I get UDP messages as fast as 10 times per second. I then look for a trigger in the UDP message. I am using an event handler that is raised each time i get a new UDP packet which will then call methods to parse the packet and look for my trigger. So, i have these questions.
With regard to threading, I assume a thread like my thread that listens to the UDP data should be a permanent thread?
Also on threading, when I get my trigger and decide to do something, in this case send a message out, i gather that I should use a thread pool each time I want to perform this task?
On thread pools, I am reading that they are not very high priority, is that true? If the message I need to send out is critical, can i rely on thread pools?
With the event handler which is raised when i get a UDP packet and then calls methods, what is the best way to ensure my methods all complete before the next packet/event is raised? At times I see event queue problems because if any of the methods take a bit longer than they should (for exampe writing to a DB) and the next packet comes in 100ms later, you get event queue growth because you cannot consume events in a timely manner. Is there a good way to address this?
With regard to threading, I assume a thread like my thread that listens to the UDP data should be a permanent thread?
There are no permanent threads. However there should be a thread that is responsible for receiving. Once you start it, let it run until you no longer need to receive any messages.
Also on threading, when I get my trigger and decide to do something, in this case send a message out, i gather that I should use a thread pool each time I want to perform this task?
That depends on how often would you send out messages. If your situation is more like consumer/producer than a separate thread for sending is a good idea. But if you send out a message only rarely, you can use thread pool. I can't define how often rare means in this case, you should watch your app and decide.
On thread pools, I am reading that they are not very high priority, is that true? If the message I need to send out is critical, can i rely on thread pools?
You can, it's more like your message will be delayed because of slow message processing or slow network rather than the thread pool.
With the event handler which is raised when i get a UDP packet and then calls methods, what is the best way to ensure my methods all complete before the next packet/event is raised? At times I see event queue problems because if any of the methods take a bit longer than they should (for exampe writing to a DB) and the next packet comes in 100ms later, you get event queue growth because you cannot consume events in a timely manner. Is there a good way to address this?
Queue is a perfect solution. You can have more queues if some messages are independent of others and their execution won't collide and then execute them in parallel.
I'll adress your points:
your listeting thread must be a 'permanent' thread that gets messages and distribute them.
(2+3) - Look at the TPL libarary you should use it instead of working with threads and thread pools (unless you need some fine control over the operations which, from your question, seems like you dont need) - as MSDN states:
The Task Parallel Library (TPL) is based on the concept of a task, which represents an asynchronous operation. In some ways, a task resembles a thread or ThreadPool work item, but at a higher level of abstraction
Look into using MessageQueues since what you need is a place to receive messages, store them for some time (in memory in your case)and handle them at your own pace.
You could implement this yourself but you'll find it gets complicated quickly,
I recommend looking into NetMQ - it's easy to use, especially for what you describe, and it's in c#.
How do I achieve a background task that waits for events to be raised by other processes, similar to how a Winforms Form does "nothing" until an event is raised? I would like to know how to do this in two cases:
a) as a process to be called by applications
b) as a task in one application
(But an answer to one would be appreciated as well, of course.)
The specific type of usage I'm facing now is to have this task process a queue (FIFO) and when the queue is empty - wait. But please don't restrict your answers to that as I'm interested in a general answer.
I've used the terms task and process, but perhaps the term thread should be used. Please feel free to correct me if I'm wrong.
EDIT
I'm looking for some built-in mechanism, rather than implementing a "message pump". And preferably, built-in to .net, not a library that has to be installed.
Within an application (not across apps) - a simple queue pattern will do it, you can use a thread signalling mechanism. Here is a simple example:
Declare a thread-safe collection to store your command, a task (thread pool thread, effectively) to process the queue, a signal trigger - and a flag to allow exiting later:
private ConcurrentQueue<CommandObject> _queueCommands
private Task _queueProcessorTask;
private AutoResetEvent _trigger;
private bool _isRunning;
The code to initialise and kick off the queue processor:
_queueCommands = new ConcurrentQueue<CommandObject>();
_queueProcessorTask = new Task(ProcessQueue);
_trigger = new AutoResetEvent(false);
_isRunning = true;
_queueProcessorTask.Start();
Your queue processor itself will look like something this:
while (_isRunning)
{
if (_isRunning && _queueCommands .Count != 0)
{
if (_queueCommands.TryDequeue(out command))
{
// do the job, this is FIFO
}
}
// you wanna wait here, but only if there's nothing new to do
if (_isRunning && _queueCommands.Count == 0)
{
_trigger.WaitOne(10000, false);
}
}
And some code to add requests to the queue:
_queueCommands.Enqueue(newCommand);
_trigger.Set(); // this is the bit which does your event / signal to spark queue processor into life
There is also a collection called BlockingCollection that can do the above with the signal part intrinsic, but I like to show this verbose version so you know what's going on.
ADDED:
With the code above we're basically notifying a dedicated thread that it has a message / command to process - so you're "message pump" if you will. The blocking collection does this in less steps but you still have to add something to the collection to "pump" that message!
Your question is very broad and as such a lot of good answers are possible. It really depends on what you are trying to achieve. Many options exist and you just need to carefully look at your requirements and determine what fits best.
The basic pattern that would appear to apply in most cases is that of queued event or message handling.
I will list a few available "out of the box" solutions that address different types of needs, without trying to make this an exhaustive list.
Across processes
There are several "precooked" solutions available that for example could fit if you are doing eventing / messaging across multiple processes.
If you require some form of durable messaging, you could look at one of several message bus implementations, such as e.g. NServiceBus, Kafka, ...
If your messaging needs to be fast but does not require durability, then something like 0mq that supports multiple messaging patterns could be useful, if you want to do this across .NET web apps, you may want to have a look at SignalR.
If you are looking to do any type of complex event processing, or you want to have a permanent memory of the events and do "computatations", such as aggregation or other types of projections over a series of events, you could have a look at the EventStore project.
If you want to do something simple, like get notifications when data in your database has changed, you could look into specific change notification services / trigger analogs for your specific type database (e.g. RavenDB's changes API).
Within a process
If you need to perform (non-durable) messaging between threads in a single process, the two most complete "prebaked" building blocks available to you are the
TPL Dataflow
Allow to set up specific data processing pipelines, with several buffering and transformation options (think pipes and filters).
Reactive extensions
Specifically intended to build non-durable asynchronous event processing, composing and transforming events using observable sequences and LINQ-style query operators.
If your needs are far simpler than what they offer, you could build your own, taking into account the basic type of tried and tested patterns that are used in the solutions already mentioned.
I am implementing a component that reads all the messages off a specific queue as they are available but should only remove messages from the queue asynchronously, after the message contents has been processed and persisted. We read messages off faster than we acknowledge them (e.g. we could read have read 10 messages off before we are ready to Ack the first). The current implementation uses the XMS API, but we can switch to MQI if XMS is inappropriate for these purposes.
We have tried two approaches to try solve this problem, but both have drawbacks that make them unacceptable. I was hoping that someone could suggest a better way.
The first implementation uses an IMessageConsumer in a dedicated thread to read all messages and publish their content as they are available. When the message has been processed, the message.Acknowledge() method is called. The Session is created with AcknowledgeMode.ClientAcknowledge. The problem with this approach is that, as per the documentation, this acknowledges (and deletes) ALL unacknowledged messages that have been received. With the example above, that would mean that all 10 read messages would be acked with the first call. So this does not really solve the problem. Because of the reading throughput we need, we cannot really modify this solution to wait for the first message's ack before reading the second, etc.
The second implementation uses an IQueueBrowser in a decided thread to read all messages and publish their content. This does not delete the messages off the queue as it reads. A separate dedicated thread then waits (on a BlockingQueue) for JMS Message IDs of messages that have been processed. For each of these, it then constructs a dedicated IMessageConsumer (using a message selector with JMSMessageID) to read off the message and ack it. (This pairing of an IQueueBrowser with dedicated IMessageConsumer is recommend by the XMS documentation's section on Queue browsers.) This method does work as expected but, as one would imagine, it is too CPU-intensive on the MQ Server.
Both of the methods proposed in the question appear to rely on a single instance of the app. What's wrong with using multiple app instances, transacted sessions and COMMIT? The performance reports (these are the SupportPacs with names like MP**) all show that throughput is maximized with multiple app instances, and horizontal scaling is one of the most used approaches in your scenario.
The design for this would be either multiple application instances or multiple threads within the same application. The key to making it work correctly is to keep in mind that transactions are scoped to a connection handle. The implication is that a multi-threaded app must dispatch a separate thread for each connection instance and the messages are read in the same thread.
The process flow is that, using a transacted session, the app performs a normal MQGet against the queue, processes the message contents as required and then issues an MQCommit. (I'll use the MQ native API names in my examples because this isn't language dependent.) If this is an XA transaction the app would call MQBegin to start the transaction but for single-phase commit the transaction is assumed. In both cases, MQCommit ends the transaction which removes the messages from the queue. While messages are under syncpoint, no other app instance can retrieve them; MQ just delivers the next available message. If a transaction is rolled back, the next MQGet from any thread retrieves it, assuming FIFO delivery.
There are some samples in:
[WMQ install home]\tools\dotnet\samples\cs\xms\simple\wmq\
...and SimpleXAConsumer.cs is one example that shows the XA version of this. The non-XA version is simpler since you don't need the external coordinator, the MQBegin verbs and so forth. If you start with one of these, make sure that they do not specify exclusive use of the queue and you can fire up several instances using the same configuration. Alternatively, take the portion of the sample that includes creation of the connection, message handling, connection close and destroy, and wrap all that in a thread spawner class.
[Insert usual advice about using the latest version of the classes here.]