I have a pool of RabbitMQ channels under one connection. And I'm trying to implement the Publisher Confirms feature. I've built the pool in a way that it creates new channels as demand for channels increases. But now, I have to manage the closure of these channels too.
I plan to implement a solution like in the RabbitMQ tutorials using a outstandingConfirms list.
I came to the problem that I have to somehow close a channel (when I'm above my 'soft' threshold of objects in the pool) when the last message in a channel is ack-ed or nack-ed.
As you can see in the code below, the sender parameter is actually the channel itself, I thought I could use this directly to close the channel if there are no pending messages anymore. But I faced the fact that channels should not be used by more than one thread simultaneously. This same channel will be available in the pool and can be picked up by the application for usage.
These are the event subscriptions:
protected void OnBasicAcks(object sender, BasicAckEventArgs e)
{
//sender = channel object
//todo close channel after all pending messages (n)acked
}
protected void OnBasicNacks(object sender, BasicNackEventArgs e)
{
//sender = channel object
//todo close channel after all pending messages (n)acked
}
Also here I can read that blocking operations are allowed in callback handlers:
As of version 3.5.0 application callback handlers can invoke blocking
operations (such as IModel.QueueDeclare or IModel.BasicCancel)
But does this also apply to the closure of the channel itself?
In short, this is my problem. My questions are:
Can I use the channel passed through the sender parameter directly to do operations like IModel.Close()?
I just want to prevent getting deadlocks on channels or other weird stuff. Any advice or best practice for my problem?
And something completely different: can I create a new channel while I'm in the OnBasicAcks method?
If you are using the latest RabbitMQ.Client (v5.1, I think), the callbacks are all invoked on the ThreadPool using async so you're generally okay calling those methods in callbacks. I would, however, avoid hijacking the callback thread to do any long-running tasks as they are managed per-channel by the RabbitMQ client.
So, to answer specifically.
Should be fine, looking at the source code, it doesn't lock anything that would prevent close from being called. Just be wary of anything in your code that would deadlock.
It's best to control access to the session in your own code so that you aren't calling a single channel from multiple threads expecting anything to happen in a specific order.
Should work fine, you're on a TPL task thread, so no reason it shouldn't work.
If you want more details on how the client works, you can look at the source, particularly the ModelBase.cs file, and the asynchronous worker service it creates for callbacks.
Related
I am still learning C# so please be easy on me. I am thinking about my application I am working on and I can't seem to figure out the best approach. This is not a forms application but rather a console. I am listening to a UDP port. I get UDP messages as fast as 10 times per second. I then look for a trigger in the UDP message. I am using an event handler that is raised each time i get a new UDP packet which will then call methods to parse the packet and look for my trigger. So, i have these questions.
With regard to threading, I assume a thread like my thread that listens to the UDP data should be a permanent thread?
Also on threading, when I get my trigger and decide to do something, in this case send a message out, i gather that I should use a thread pool each time I want to perform this task?
On thread pools, I am reading that they are not very high priority, is that true? If the message I need to send out is critical, can i rely on thread pools?
With the event handler which is raised when i get a UDP packet and then calls methods, what is the best way to ensure my methods all complete before the next packet/event is raised? At times I see event queue problems because if any of the methods take a bit longer than they should (for exampe writing to a DB) and the next packet comes in 100ms later, you get event queue growth because you cannot consume events in a timely manner. Is there a good way to address this?
With regard to threading, I assume a thread like my thread that listens to the UDP data should be a permanent thread?
There are no permanent threads. However there should be a thread that is responsible for receiving. Once you start it, let it run until you no longer need to receive any messages.
Also on threading, when I get my trigger and decide to do something, in this case send a message out, i gather that I should use a thread pool each time I want to perform this task?
That depends on how often would you send out messages. If your situation is more like consumer/producer than a separate thread for sending is a good idea. But if you send out a message only rarely, you can use thread pool. I can't define how often rare means in this case, you should watch your app and decide.
On thread pools, I am reading that they are not very high priority, is that true? If the message I need to send out is critical, can i rely on thread pools?
You can, it's more like your message will be delayed because of slow message processing or slow network rather than the thread pool.
With the event handler which is raised when i get a UDP packet and then calls methods, what is the best way to ensure my methods all complete before the next packet/event is raised? At times I see event queue problems because if any of the methods take a bit longer than they should (for exampe writing to a DB) and the next packet comes in 100ms later, you get event queue growth because you cannot consume events in a timely manner. Is there a good way to address this?
Queue is a perfect solution. You can have more queues if some messages are independent of others and their execution won't collide and then execute them in parallel.
I'll adress your points:
your listeting thread must be a 'permanent' thread that gets messages and distribute them.
(2+3) - Look at the TPL libarary you should use it instead of working with threads and thread pools (unless you need some fine control over the operations which, from your question, seems like you dont need) - as MSDN states:
The Task Parallel Library (TPL) is based on the concept of a task, which represents an asynchronous operation. In some ways, a task resembles a thread or ThreadPool work item, but at a higher level of abstraction
Look into using MessageQueues since what you need is a place to receive messages, store them for some time (in memory in your case)and handle them at your own pace.
You could implement this yourself but you'll find it gets complicated quickly,
I recommend looking into NetMQ - it's easy to use, especially for what you describe, and it's in c#.
I have created a small class using RabbitMQ that implements a publish/subscribe messaging pattern on a topic exchange. On top of this pub/sub I have the methods and properties:
void Send(Message, Subject) - Publish message to destination topic for any subscribers to handle.
MessageReceivedEvent - Subscribe to message received events on this messaging instance (messaging instance is bound to the desired subscribe topic when created).
SendWaitReply(Message, Subject) - Send a message and block until a reply message is received with a correlation id matching the sent message id (or timeout). This is essentially a request/reply or RPC mechanism on top of the pub/sub pattern.
The messaging patterns I have chosen are somewhat set in stone due to the way the system is to be designed. I realize I could use reply-to queues to mitigate the potential issue with SendWaitReply, but that breaks some requirements.
Right now my issues are:
For the Listen event, the messages are processed synchronously through the event subscribers as the listener runs in a single thread. This causes some serious performance issues when handling large volumes of messages (i.e. in a back-end process consuming events from a web api). I am considering passing in a callback function as opposed to subscribing to an event and then dispatching the collection of callbacks in parallel using Task or Threadpool. Thread safety would obviously now be a concern of the caller. I am not sure if this is a correct approach.
For the SendWaitReply event, I have built what seems to be a hacky solution that takes all inbound messages from the message listener loop and places them in a ConcurrentDictionary if they contain a non-empty correlation guid. Then in the SendWaitReply method, I poll the ConcurrentDictionary for a message containing a key that matches the Id of the sent message (or timeout after a certain period). If there is a faster/better way to do this, I would really like to investigate it. Maybe a way to signal to all of the currently blocked SendWaitReply methods that a new message is available and they should all check their Ids instead of polling continuously?
Update 10/15/2014
After much exhaustive research, I have concluded that there is no "official" mechanism/helper/library to directly handle the particular use-case I have presented above for SendWaitReply in the scope of RabbitMQ or AMQP. I will stick with my current solution (and investigate more robust implementations) for the time being. There have been answers recommending I use the provided RPC functionality, but this unfortunately only works in the case that you want to use exclusive callback queues on a per-request basis. This breaks one of my major requirements of having all messages (request and reply) visible on the same topic exchange.
To further clarify, the typical message pair for a SendWaitReply request is in the format of:
Topic_Exchange.Service_A => some_command => Topic_Exchange.Service_B
Topic_Exchange.Service_B => some_command_reply => Topic_Exchange.Service_A
This affords me a powerful debugging and logging technique where I simply set up a listener on Topic_Exchange.# and can see all of the system traffic for tracing very deep 'call stacks' through various services.
TL; DR - Current Problem Below
Backing down from the architectural level - I still have an issue with the message listener loop. I have tried the EventingBasicConsumer and am still seeing a block. The way my class works is that the caller subscribes to the delegate provided by the instance of the class. The message loop fires the event on that delegate and those subscribers then handle the message. It seems as if I need a different way to pass the message event handlers into the instance such that they don't all sit behind one delegate which enforces synchronous processing.
It's difficult to say why your code is blocking without a sample, but to prevent blocking while consuming, you should use the EventingBasicConsumer.
var consumer = new EventingBasicConsumer;
consumer.Received += (s, delivery) => { /* do stuff here */ };
channel.BasicConsume(queue, false, consumer);
One caveat, if you are using autoAck = false (as I do), then you need to ensure you lock the channel when you do channel.BasicAck or you may hit concurrency issues in the .NET library.
For the SendWaitReply, you may have better luck if you just use the SimpleRpcClient included in the RabbitMQ client library:
var props = channel.CreateBasicProperties();
// Set your properties
var client = new RabbitMQ.Client.MessagePatterns.SimpleRpcClient(channel, exchange, ExchangeType.Direct, routingKey);
IBasicProperties replyProps;
byte[] response = client.Call(props, body, out replyProps);
The SimpleRpcClient will deal with creating a temporary queue, correlation ID's, and so on instead of building your own. If you find you want to do something more advanced, the source is also a good reference.
We are just starting to use RabbitMQ with C#. My current plan is to configure in the database the number and kind of consumers to run on a given server. We have an existing windows service and when that starts I want to spawn all of the RabbitMQ consumers. My question is what is the best way to spwan these from a windows service?
My current plan is to read the configuration out of the database and spawn a long running task for each consumer.
var t = new Task(() =>
{
var instance = LoadConsumerClass(consumerEnum, consumerName);
instance.StartConsuming();//blocking call
}, TaskCreationOptions.LongRunning);
t.Start();
Is this better or worse than creating a thread for each consumer?
var messageConsumer = LoadConsumerClass(consumerEnum, consumerName);
var thread = new Thread(messageConsumer.StartConsuming);
I'm hoping that more than a few others have already tried what I'm doing and can provide me with some ideas for what worked well and what didn't.
In EasyNetQ we have a single dispatcher thread for all consumers on a single connection. We also provide a facility to to return a Task from the message handler, so it's easy to do async IO if you want to make a database call, go to the file system, or make a web service request.
Having said that it's perfectly legitimate to have each consumer consuming on a different thread. I guess it depends on your message throughput, how many consumers you have and the nature of your message handlers.
I'd stick with Tasks as they give you more features and generally allow for less boilerplate code.
And, If I understand your code correctly, you'd be sharing a channel (IModel) in second case. This might cause troubles as the default IModel implementation is not thread safe (or used to be). There're more subtle nuances regarding thread safety you'd have to watch out.
But it depends on your usage patterns. If you don't expect many messages/sec on each consumer, or if your app can handle messages fast then perhaps a single thread for all consumers will be you best option.
Task is great, but you not really going to use all the stuff it can do. The only thing you need is to do work in parallel.
I faced the same question couple of months ago, what I finished with - is a thread per computation type (per queue) which is blocking on message arrival and doesn't consume cpu when waiting for messages.
Open a new channel for each one of the threads.
As for connections - if you application is meant to deal with high load of messages, I suggest you opening connection for every X workers (figure you your X), since only one channel can send the messages through the connection, so assuming one worker is consuming large message the others are blocked on connection level waiting it to be free.
I would like to rephrase my previous question How to create Singleton with async method?
Imagine messaging application (like icq) - something that should be always connected to server and can post messages.
I need to implment class Connection. It should be singleton, because it contains "socket" inside and that socket should persist during entirely application lifetime.
Then I want to implement async method Connection.postMessage
Because postMessage can take significant ammount of time:
postMessage should be async
postMessage should queue messages if neccesary
Note my application posts dozens messages per second, so it is not appropiate to create new Thread for each postMessage call.
I diffenetely need to create exactly one extra thread for messages posting but I don't know where and how.
upd: good example http://msdn.microsoft.com/en-us/library/yy12yx1f(v=vs.80).aspx
No, Postmessage (itself) should not be async .
It should
be Thread-safe
ensure the Processing thread is running
queue the message (ConcurrentQueue)
return
And the Processing Thread should
Wait on the Queue
Process the messages
maybe Terminate itself when idle for xx milliseconds
What you have is a classic Producer/Consumer situation with 1 Consumer and multiple Producers.
PostMessage is the entry-point for all producers.
jp,
You're looking at a classic producer/consumer problem here... During initialisation the Connection should create a MessageQueue start a Sender in it's own background thread.
Then the connection posts just messages to the queue, for the Sender to pickup and forward when ready.
The tricky bit is managing the maximum queue size... If the producer consistently outruns the consumer then queue can grow to an unmanagable size. The simplest approach is to block the producer thread until the queue is no longer full. This can be done with a back-off-ARQ. ie: while(queue.isFull) sleep(100, "milliseconds"); queue.add(message); If you don't require 100% transmission (like a chat-app, for instance) then you can simply throw a MessageQueueFullException, and the poor client will just have to get over it... just allways allow them to resubmit later... allowing the user manage the retrys for you.
That's how I'd tackle it anyway. I'll be interested to see what others suggestions are muted.
Hope things work out for you. Cheers. Keith.
A few words about an ongoing design and implementation
I send a lot of requests to the remote application (running on a different
host, of course), and the application send back data.
About client
Client is a UI that spawn a separate thread to submit and process the requests. Once it submits all the requests, it calls Wait. And the Wait will parse all events coming the app and invoke client's callbacks.
Below is the implementation of Wait.
public void Wait (uint milliseconds)
{
while(_socket.IsConnected)
{
if (_socket.Poll(milliseconds, SelectMode.SelectRead))
{
// read info of the buffer and calls registered callbacks for the client
if(_socket.IsAvailable > 0)
ProcessSocket(socket);
}
else
return; //returns after Poll has expired
}
}
The Wait is called from a separate thread, responsible for managing network connection: both inbound and outbound traffic:
_Receiver = new Thread(DoWork);
_Receiver.IsBackground = true;
_Receiver.Start(this);
This thread is created from UI component of the application.
The issue:
client sometimes sees delays in callbacks even though main application has sent the data on time. Notably, one the message in Poll was delayed until I client disconnected, and internally I called:
_socket.Shutdown(SocketShutdown.Both);
I think something funky is happening in the Poll
Any suggestions on how to fix the issue or an alternative workaround?
Thanks
please let me know if anything is unclear
A couple of things. First, in your example, is there a difference between "_socket" and "socket"? Second, you are using the System.Net.Sockets.Socket class, right? I don't see IsConnected or IsAvailable properties on that class in the MSDN documentation for any .NET version going back to 1.1. I assume these are both typing mistakes, right?
Have you tried putting an "else" clause on the "IsAvailable > 0" test and writing a message to the Console/Output window, e.g.,
if (_socket.IsAvailable > 0) {
ProcessSocket(socket);
} else {
Console.WriteLine("Poll() returned true but there is no data");
}
This might give you an idea of what might be going on in the larger context of your program.
Aside from that, I'm not a big fan of polling sockets for data. As an alternative, is there a reason not to use the asynchronous Begin/EndReceive functions on the socket? I think it'd be straightforward to convert to the asynchronous model given the fact that you're already using a separate thread to send and receive your data. Here is an example from MSDN. Additionally, I've added the typical implementation that I use of this mechanism to this SO post.
What thread is calling the Wait() method? If you're just throwing it into the UI threadpool, that may be why you experience delays sometimes. If this is your problem, then either use the system threadpool, create a new one just for the networking parts of your application, or spawn a dedicated thread for it.
Beyond this, it's hard to help you much without seeing more code.