We Keep Coding

How to correctly finish RabbitMq consumer and wait to RabbitMq consumer threads (library consumer callbacks) after close connection and model?

I have RabbitMq consumer (RabbitMQ.Client.Events.EventingBasicConsumer) that process incoming messages.
But I noticed that if close connection and model they does not wait to finish library processing threads. E.g.:
If I will add thread sleep for several seconds to EventingBasicConsumer::Received callback then I noticed that Close functions (Close() of IModel and IConnection) finished before exit from this consumer callback
After finish Close functions I continue to receive some messages to EventingBasicConsumer::Received callback.
So how to correctly close consumer and wait to finish all processing's in consumer threads of library?
I want to ensure that I will not receive any incoming messages from library for my consumer after close all connections/consumers.
Simplified code:
RunTest()
{
MyConsumer consumer = new MyConsumer();
consumer.Connect();
// Wait before close for process some count of incoming messages
Thread.Sleep(10 * 1000);
consumer.Disconnect();
}
class MyConsumer
{
private RabbitMQ.Client.IConnection m_Connection = null;
private RabbitMQ.Client.IModel m_Channel = null;
public void Connect()
{
//
// ...
//
m_Channel = m_Connection.CreateModel();
m_Consumer = new RabbitMQ.Client.Events.EventingBasicConsumer(m_Channel);
m_Consumer.Received += OnRequestReceived;
m_ConsumerTag = m_Channel.BasicConsume(m_Config.RequestQueue, false, m_Consumer);
}
public void Disconnect()
{
Console.WriteLine("---> IModel::Close()");
m_Channel.Close();
Console.WriteLine("<--- IModel::Close()");
Console.WriteLine("---> RabbitMQ.Client.IConnection::Close()");
m_Connection.Close();
Console.WriteLine("<--- RabbitMQ.Client.IConnection::Close()");
//
// Maybe there is need to do some RabbitMQ API call of channel/model
// for wait to finish of all consumer callbacks?
//
m_Channel = null;
m_Connection = null;
}
private void OnRequestReceived(object sender, RabbitMQ.Client.Events.BasicDeliverEventArgs mqMessage)
{
Console.WriteLine("---> MyConsumer::OnReceived");
Console.WriteLine("MyConsumer: ThreadSleep started");
Thread.Sleep(10000);
Console.WriteLine("MyConsumer: ThreadSleep finished");
if (m_Channel != null)
{
m_Channel.BasicAck(mqMessage.DeliveryTag, false);
}
else
{
Console.WriteLine("MyConsumer: already closed");
}
Console.WriteLine("<--- MyConsumer::OnReceived");
}
}
Result:
---> MyConsumer::OnReceived
MyConsumer: ThreadSleep started
---> IModel::Close()
<--- IModel::Close()
---> RabbitMQ.Client.IConnection::Close()
<--- RabbitMQ.Client.IConnection::Close()
MyConsumer: ThreadSleep finished
MyConsumer: already closed
<--- MyConsumer::OnReceived
---> MyConsumer::OnReceived
MyConsumer: ThreadSleep started
MyConsumer: ThreadSleep finished
MyConsumer: already closed
<--- MyConsumer::OnReceived
How we see MyConsumer::OnReceived was finished after exit from Close() functions of Consumer and Connection. Moreover how we see there is one more message which was income after finish of previous call of OnReceived and close connection (that means that RqbbitMq continues to process consumer messages until the internal library queues are empty ignoring the fact that consumer and connection are already closed).

This is really bug in RabbitMQ.Client (v5.1.2). Source code of ConsumerWorkService.cs:
namespace RabbitMQ.Client
{
public class ConsumerWorkService
{
...
class WorkPool
{
readonly ConcurrentQueue<Action> actions;
readonly AutoResetEvent messageArrived;
readonly TimeSpan waitTime;
readonly CancellationTokenSource tokenSource;
readonly string name;
public WorkPool(IModel model)
{
name = model.ToString();
actions = new ConcurrentQueue<Action>();
messageArrived = new AutoResetEvent(false);
waitTime = TimeSpan.FromMilliseconds(100);
tokenSource = new CancellationTokenSource();
}
public void Start()
{
#if NETFX_CORE
System.Threading.Tasks.Task.Factory.StartNew(Loop, System.Threading.Tasks.TaskCreationOptions.LongRunning);
#else
var thread = new Thread(Loop)
{
Name = "WorkPool-" + name,
IsBackground = true
};
thread.Start();
#endif
}
public void Enqueue(Action action)
{
actions.Enqueue(action);
messageArrived.Set();
}
void Loop()
{
while (tokenSource.IsCancellationRequested == false)
{
Action action;
while (actions.TryDequeue(out action))
{
try
{
action();
}
catch (Exception)
{
}
}
messageArrived.WaitOne(waitTime);
}
}
public void Stop()
{
tokenSource.Cancel();
}
}
}
}
As we see there is no any waitings of thread var thread = new Thread(Loop). So really event RabbitMQ.Client.Events.EventingBasicConsumer::Received can be fired anytime even when there is no consumer or connection for a long time which closed long time ago till internal library queue will empty. As I supposed(( :
Action action;
while (actions.TryDequeue(out action))
{
try
{
action();
}
catch (Exception)
{
}
}
So IModel::Close() will set only CancelationToken without join of thread and there is need some workaround for this Bug.

Just to confirm #Alexander's findings, this issue is still present in v6.2.2 of the .Net Client and the issue is also present with the event driven callback consumer.
I've found that:
The 'Received' callback registered with both the EventingBasicConsumer and AsyncEventingBasicConsumer will be invoked for up to the prefetchCount setting of the BasicQos setup on the channel, after the connection is closed.
In manual Ack mode, the call to BasicAck will throw a RabbitMQ.Client.Exceptions.AlreadyClosedException if the connection is closed, and the message won't be acked from the queue, however, the callback will continue to process subsequent messages. This could lead to idempotence problems with receiving the same message more than once during a disconnect.
This may be another good reason to ensure that the prefetchCount is set to a finite and sane value (in addition to e.g. Memory considerations of unbounded prefetch counts).
Finally, if I deliberately close the Connection unexpectedly (e.g. during testing), I found that I needed to explicitly detach my consumer Received handler AND explicitly call consumerChannel.Close(); (i.e. IModel.Close) before I could create a new connection (the Rabbit client would tend to 'hang').
The connection.ConnectionShutdown event doesn't seem to fire reliably if I have set a ConsumerDispatchConcurrency > 1 with the synchronous EventingBasicConsumer
The Shutdown event on AsyncEventingBasicConsumer doesn't fire either, if I have ConsumerDispatchConcurrency > 1` when using the async consumer.
However, I did find that the ModelShutdown event on the IModel / channel fires reliably for sync / async and concurrency > 1.
consumerChannel.ModelShutdown += (sender, args) =>
{
consumer.Received -= handler; // e.g. AsyncEventingBasicConsumer
consumerChannel.Close(); // IModel
};

Using async socket in Tasks

I have an application that is processing items in a FIFO queue using Tasks in .net 4.0.
I am new to TPL and Tasks in .net and was wondering if there is an easy solution to my problem:
The Action delegate in the Task is assigned to a method that is sending and receiving data on an asynchronous socket. The problem I am having is that the Task ends "prematurely". How do I tell the Task to wait until all communication is completed before processing the next item in the queue?
One solution is to switch to using a synchronous socket, but I was hoping there is a way to do this using the async socket.
EDIT
Added some code:
class Program
{
private BlockingCollection<string> myQueue;
private CancellationTokenSource cancellationSignalForConsumeTask;
private CancellationTokenSource cancellationSignalForProcessCommandTask;
private AsyncSocket mySocket;
public void Main(string[] args)
{
mySocket = new mySocket();
myscoket.ReceiveData += mySocket_ReceiveData;
cancellationSignalForConsumeTask = new CancellationTokenSource();
Task listenerTask = Task.Factory.StartNew((obj) => Consume(),
cancellationSignalForConsumeTask.Token,
TaskCreationOptions.LongRunning);
while (true)
{}
}
private void Consume()
{
while (!myQueue.IsCompleted )
{
string _item = myQueue.Take();
cancellationSignalForProcessCommandTask = new CancellationTokenSource();
Task t = new Task(() =>
{
cancellationSignalForProcessCommandTask.Token.ThrowIfCancellationRequested();
DoSomeWork(_item);
}, cancellationSignalForProcessCommandTask.Token, TaskCreationOptions.LongRunning);
t.Start();
t.Wait();
}
}
private void DoSomeWork(string _item)
{
mySocket.SendData("Data to server that could take a long time to process")
}
private void mySocket_ReceiveData(string dataFromServer)
{
string returnMessage = dataFromServer;
//I want the Task to end here...
}
}
The problem is that the Task ends when the DoSomeWork() method finishes (and I understand why), is there a way I can manually tell the Task to end through the CancellationTokenSource object maybe?

If I understand correctly, you want to wait on the task the receives the data, but you're currently waiting on the task that sends the data. One way to do so would be to use a construct like AutoResetEvent:
private AutoResetEvent autoResetEvent = new AutoResetEvent(false);
private void Consume()
{
while (!myQueue.IsCompleted )
{
string _item = myQueue.Take();
cancellationSignalForProcessCommandTask = new CancellationTokenSource();
Task t = new Task(() =>
{
cancellationSignalForProcessCommandTask.Token.ThrowIfCancellationRequested();
DoSomeWork(_item);
}, cancellationSignalForProcessCommandTask.Token, TaskCreationOptions.LongRunning);
t.Start();
// Wait for data to be received.
// This line will block until autoResetEvent.Set() is called.
autoResetEvent.WaitOne();
}
}
private void mySocket_ReceiveData(string dataFromServer)
{
string returnMessage = dataFromServer;
// Notify other threads that data was received and that processing can continue.
autoResetEvent.Set();
}
This is only an example of using AutoResetEvent - you'd probably want to refine it to suit your needs.

Stopping and restarting HttpListener?

I am working on an app that has an HttpListener. My goal is for the user to turn the listener off and on as they choose. I put the Listener in a new thread and I'm having a problem aborting that thread. I read somewhere that if you try to abort a thread that is in an unmanaged context, then as soon as it re-enters a managed context the ThreadAbortException will be fired. It appears that an HttpListener's GetContext() method is unmanaged because when I try to abort the thread nothing happens until I make a web request against my app. THEN the thread exits. The problem is when I attempt to kill the thread, I may start up the thread again later on the same port and an HttpListenerException goes off saying that the prefix is already registered.
How can I kill a cross thread HttpListener? Is there a managed alternative to GetContext() that will allow the thread to abort? Can I abort the thread in a way that unmanaged code will halt?

What about something like:
public class XListener
{
HttpListener listener;
public XListener(string prefix)
{
listener = new HttpListener();
listener.Prefixes.Add(prefix);
}
public void StartListen()
{
if (!listener.IsListening)
{
listener.Start();
Task.Factory.StartNew(async () =>
{
while (true) await Listen(listener);
}, TaskCreationOptions.LongRunning);
Console.WriteLine("Listener started");
}
}
public void StopListen()
{
if (listener.IsListening)
{
listener.Stop();
Console.WriteLine("Listener stopped");
}
}
private async Task Listen(HttpListener l)
{
try
{
var ctx = await l.GetContextAsync();
var text = "Hello World";
var buffer = Encoding.UTF8.GetBytes(text);
using (var response = ctx.Response)
{
ctx.Response.ContentLength64 = buffer.Length;
ctx.Response.OutputStream.Write(buffer, 0, buffer.Length);
}
}
catch (HttpListenerException)
{
Console.WriteLine("screw you guys, I'm going home!");
}
}
}
Usage:
var x = new XListener("http://locahost:8080");
x.StartListen();
Thread.Sleep(500); // test purpose only
x.StopListen();
Thread.Sleep(500); // test purpose only
x.StartListen();
/* OUTPUT:
=> Listener started
=> Listener stopped
=> screw you guys, I'm going home!
=> Listener started */

You need to signal the thread to call HttpListener.Stop() and wait for the thread to finish by calling Thread.Join()

All you need to do is call stop on the listener. Since your listener thread is blocked on GetContext you will need to do this on another thread. IIRC this will cause GetContext to throw, so you will want to handle that exception and clean up. Calling Thread.Abort should be your last resort and wont cause the listener to stop listening until it is garbage collected anyway.

using System;
using System.Net;
using System.Text;
class Server
{
HttpListener listener = new HttpListener();
public Server(string url)
{
listener.Prefixes.Add(url);
}
void Callback(IAsyncResult result)
{
HttpListenerContext context = listener.EndGetContext(result);
byte[] buffer = Encoding.UTF8.GetBytes("Hello world!");
context.Response.ContentLength64 = buffer.Length;
context.Response.OutputStream.Write(buffer, 0, buffer.Length);
context.Response.OutputStream.Close();
listener.BeginGetContext(new AsyncCallback(Callback), listener);
}
public void Start()
{
listener.Start();
listener.BeginGetContext(new AsyncCallback(Callback), listener);
}
public void Stop()
{
listener.Stop();
}
public void Close()
{
listener.Close();
}
}

How to abort thread that use AcceptTcpClient() inside?

AcceptTcpClient() prevents app from exit after I called thrd.Abort().
How to exit application when in listening?

You should be able to interrupt the call to AcceptTcpClient() by closing the TcpListener (this will result in an exception being thrown by the blocking AcceptTcpClient(). You should not be aborting the thread, which is generally a very bad idea in all but a few very specific circumstances.
Here's a brief example:
class Program
{
static void Main(string[] args)
{
var listener = new TcpListener(IPAddress.Any, 12343);
var thread = new Thread(() => AsyncAccept(listener));
thread.Start();
Console.WriteLine("Press enter to stop...");
Console.ReadLine();
Console.WriteLine("Stopping listener...");
listener.Stop();
thread.Join();
}
private static void AsyncAccept(TcpListener listener)
{
listener.Start();
Console.WriteLine("Started listener");
try
{
while (true)
{
using (var client = listener.AcceptTcpClient())
{
Console.WriteLine("Accepted client: {0}", client.Client.RemoteEndPoint);
}
}
}
catch(Exception e)
{
Console.WriteLine(e);
}
Console.WriteLine("Listener done");
}
}
The code above starts a listener on a separate thread, pressing Enter on the console window will stop the listener, wait for the listener thread to complete, then the application will exit normally, no thread aborts required!

You could:
Use BeginAcceptTcpClient() and End.. instead:
See:https://msdn.microsoft.com/en-us/library/system.net.sockets.tcplistener.beginaccepttcpclient(v=vs.110).aspx
Or your could:
Create a TcpClient and send your listener message:
therefore (I guess you have a loop in your thread):
break the loop wherein the listener.AcceptTcpClient() is running.
(i.e. CancelAsync()) from outside and
Loop While (!Tread.CancellationPending);
Create a TcpClient and send your listener a message (and discard data);
TcpClient see: https://msdn.microsoft.com/en-us/library/system.net.sockets.tcpclient(v=vs.110).aspx
Now your thread can go on with:
client.close() and listener.stop()

Stopping a TcpListener after calling BeginAcceptTcpClient

I have this code...
internal static void Start()
{
TcpListener listenerSocket = new TcpListener(IPAddress.Any, 32599);
listenerSocket.Start();
listenerSocket.BeginAcceptTcpClient(new AsyncCallback(AcceptClient), null);
}
Then my call back function looks like this...
private static void AcceptClient(IAsyncResult asyncResult)
{
MessageHandler handler = new MessageHandler(listenerSocket.EndAcceptTcpClient(asyncResult));
ThreadPool.QueueUserWorkItem((object state) => handler.Process());
listenerSocket.BeginAcceptTcpClient(new AsyncCallback(AcceptClient), null);
}
Now, I call BeginAcceptTcpClient, then some time later I want to stop the server. To do this I have been calling TcpListener.Stop(), or TcpListener.Server.Close(). Both of these however execute my AcceptClient function. This then throws an exception when I call EndAcceptTcpClient. What is the best practice way around this? I could just put a flag in to stop the execution of AcceptClient once I have called stop, but I wonder if I am missing something.
Update 1
Currently I have patched it by changing the code to look like this.
private static void AcceptClient(IAsyncResult asyncResult)
{
if (!shutdown)
{
MessageHandler handler = new MessageHandler(listenerSocket.EndAcceptTcpClient(asyncResult));
ThreadPool.QueueUserWorkItem((object state) => handler.Process());
listenerSocket.BeginAcceptTcpClient(new AsyncCallback(AcceptClient), null);
}
}
private static bool shutdown = false;
internal static void Stop()
{
shutdown = true;
listenerSocket.Stop();
}
Update 2
I changed it to impliment the answer from Spencer Ruport.
private static void AcceptClient(IAsyncResult asyncResult)
{
if (listenerSocket.Server.IsBound)
{
MessageHandler handler = new MessageHandler(listenerSocket.EndAcceptTcpClient(asyncResult));
ThreadPool.QueueUserWorkItem((object state) => handler.Process());
listenerSocket.BeginAcceptTcpClient(new AsyncCallback(AcceptClient), null);
}
}

I just ran into this issue myself, and I believe your current solution is incomplete/incorrect. There is no guarantee of atomicity between the check for IsBound and the subsequent call to EndAcceptTcpClient(). You can still get an exception if the listener is Stop()'d between those two statements. You didn't say what exception you're getting but I assume it's the same one I'm getting, ObjectDisposedException (complaining that the underlying socket has already been disposed).
You should be able to check this by simulating the thread scheduling:
Set a breakpoint on the line after the IsBound check in your callback
Freeze the thread that hits the breakpoint (Threads window -> right click, "Freeze")
Run/trigger the code that calls TcpListener.Stop()
Break in and step through the EndAcceptTcpClient() call. You should see the ObjectDisposedException.
IMO the ideal solution would be for Microsoft to throw a different exception from EndAcceptTcpClient in this case, e.g. ListenCanceledException or something like that.
As it is, we have to infer what's happening from the ObjectDisposedException. Just catch the exception and behave accordingly. In my code I silently eat the exception, since I have code elsewhere that's doing the real shutdown work (i.e. the code that called TcpListener.Stop() in the first place). You should already have exception handling in that area anyway, since you can get various SocketExceptions. This is just tacking another catch handler onto that try block.
I admit I'm uncomfortable with this approach since in principle the catch could be a false positive, with a genuine "bad" object access in there. But on the other hand there aren't too many object accesses in the EndAcceptTcpClient() call that could otherwise trigger this exception. I hope.
Here's my code. This is early/prototype stuff, ignore the Console calls.
private void OnAccept(IAsyncResult iar)
{
TcpListener l = (TcpListener) iar.AsyncState;
TcpClient c;
try
{
c = l.EndAcceptTcpClient(iar);
// keep listening
l.BeginAcceptTcpClient(new AsyncCallback(OnAccept), l);
}
catch (SocketException ex)
{
Console.WriteLine("Error accepting TCP connection: {0}", ex.Message);
// unrecoverable
_doneEvent.Set();
return;
}
catch (ObjectDisposedException)
{
// The listener was Stop()'d, disposing the underlying socket and
// triggering the completion of the callback. We're already exiting,
// so just return.
Console.WriteLine("Listen canceled.");
return;
}
// meanwhile...
SslStream s = new SslStream(c.GetStream());
Console.WriteLine("Authenticating...");
s.BeginAuthenticateAsServer(_cert, new AsyncCallback(OnAuthenticate), s);
}

No you're not missing anything. You can check the IsBound property of the Socket object. At least for TCP connections, while the socket is listening this will be set to true and after you call close it's value will be false. Though, your own implementation can work just as well.

try this one. it works fine for me without catching exceptions.
private void OnAccept(IAsyncResult pAsyncResult)
{
TcpListener listener = (TcpListener) pAsyncResult.AsyncState;
if(listener.Server == null)
{
//stop method was called
return;
}
...
}

i think that all tree things are needed and that the restart of BeginAcceptTcpClient should be placed outside the tryctach of EndAcceptTcpClient.
private void AcceptTcpClientCallback(IAsyncResult ar)
{
var listener = (TcpListener)ar.AsyncState;
//Sometimes the socket is null and somethimes the socket was set
if (listener.Server == null || !listener.Server.IsBound)
return;
TcpClient client = null;
try
{
client = listener.EndAcceptTcpClient(ar);
}
catch (SocketException ex)
{
//the client is corrupt
OnError(ex);
}
catch (ObjectDisposedException)
{
//Listener canceled
return;
}
//Get the next Client
listener.BeginAcceptTcpClient(new AsyncCallback(AcceptTcpClientCallback), listener);
if (client == null)
return; //Abort if there was an error with the client
MyConnection connection = null;
try
{
//Client-Protocoll init
connection = Connect(client.GetStream());
}
catch (Exception ex)
{
//The client is corrupt/invalid
OnError(ex);
client.Close();
}
}

This is a simple example how to start listening, how to process requests asynchronously, and how to stop listening.
Full example here.
public class TcpServer
{
#region Public.
// Create new instance of TcpServer.
public TcpServer(string ip, int port)
{
_listener = new TcpListener(IPAddress.Parse(ip), port);
}
// Starts receiving incoming requests.
public void Start()
{
_listener.Start();
_ct = _cts.Token;
_listener.BeginAcceptTcpClient(ProcessRequest, _listener);
}
// Stops receiving incoming requests.
public void Stop()
{
// If listening has been cancelled, simply go out from method.
if(_ct.IsCancellationRequested)
{
return;
}
// Cancels listening.
_cts.Cancel();
// Waits a little, to guarantee
// that all operation receive information about cancellation.
Thread.Sleep(100);
_listener.Stop();
}
#endregion
#region Private.
// Process single request.
private void ProcessRequest(IAsyncResult ar)
{
//Stop if operation was cancelled.
if(_ct.IsCancellationRequested)
{
return;
}
var listener = ar.AsyncState as TcpListener;
if(listener == null)
{
return;
}
// Check cancellation again. Stop if operation was cancelled.
if(_ct.IsCancellationRequested)
{
return;
}
// Starts waiting for the next request.
listener.BeginAcceptTcpClient(ProcessRequest, listener);
// Gets client and starts processing received request.
using(TcpClient client = listener.EndAcceptTcpClient(ar))
{
var rp = new RequestProcessor();
rp.Proccess(client);
}
}
#endregion
#region Fields.
private CancellationToken _ct;
private CancellationTokenSource _cts = new CancellationTokenSource();
private TcpListener _listener;
#endregion
}