We Keep Coding

How to correctly finish RabbitMq consumer and wait to RabbitMq consumer threads (library consumer callbacks) after close connection and model?

I have RabbitMq consumer (RabbitMQ.Client.Events.EventingBasicConsumer) that process incoming messages.
But I noticed that if close connection and model they does not wait to finish library processing threads. E.g.:
If I will add thread sleep for several seconds to EventingBasicConsumer::Received callback then I noticed that Close functions (Close() of IModel and IConnection) finished before exit from this consumer callback
After finish Close functions I continue to receive some messages to EventingBasicConsumer::Received callback.
So how to correctly close consumer and wait to finish all processing's in consumer threads of library?
I want to ensure that I will not receive any incoming messages from library for my consumer after close all connections/consumers.
Simplified code:
RunTest()
{
MyConsumer consumer = new MyConsumer();
consumer.Connect();
// Wait before close for process some count of incoming messages
Thread.Sleep(10 * 1000);
consumer.Disconnect();
}
class MyConsumer
{
private RabbitMQ.Client.IConnection m_Connection = null;
private RabbitMQ.Client.IModel m_Channel = null;
public void Connect()
{
//
// ...
//
m_Channel = m_Connection.CreateModel();
m_Consumer = new RabbitMQ.Client.Events.EventingBasicConsumer(m_Channel);
m_Consumer.Received += OnRequestReceived;
m_ConsumerTag = m_Channel.BasicConsume(m_Config.RequestQueue, false, m_Consumer);
}
public void Disconnect()
{
Console.WriteLine("---> IModel::Close()");
m_Channel.Close();
Console.WriteLine("<--- IModel::Close()");
Console.WriteLine("---> RabbitMQ.Client.IConnection::Close()");
m_Connection.Close();
Console.WriteLine("<--- RabbitMQ.Client.IConnection::Close()");
//
// Maybe there is need to do some RabbitMQ API call of channel/model
// for wait to finish of all consumer callbacks?
//
m_Channel = null;
m_Connection = null;
}
private void OnRequestReceived(object sender, RabbitMQ.Client.Events.BasicDeliverEventArgs mqMessage)
{
Console.WriteLine("---> MyConsumer::OnReceived");
Console.WriteLine("MyConsumer: ThreadSleep started");
Thread.Sleep(10000);
Console.WriteLine("MyConsumer: ThreadSleep finished");
if (m_Channel != null)
{
m_Channel.BasicAck(mqMessage.DeliveryTag, false);
}
else
{
Console.WriteLine("MyConsumer: already closed");
}
Console.WriteLine("<--- MyConsumer::OnReceived");
}
}
Result:
---> MyConsumer::OnReceived
MyConsumer: ThreadSleep started
---> IModel::Close()
<--- IModel::Close()
---> RabbitMQ.Client.IConnection::Close()
<--- RabbitMQ.Client.IConnection::Close()
MyConsumer: ThreadSleep finished
MyConsumer: already closed
<--- MyConsumer::OnReceived
---> MyConsumer::OnReceived
MyConsumer: ThreadSleep started
MyConsumer: ThreadSleep finished
MyConsumer: already closed
<--- MyConsumer::OnReceived
How we see MyConsumer::OnReceived was finished after exit from Close() functions of Consumer and Connection. Moreover how we see there is one more message which was income after finish of previous call of OnReceived and close connection (that means that RqbbitMq continues to process consumer messages until the internal library queues are empty ignoring the fact that consumer and connection are already closed).

This is really bug in RabbitMQ.Client (v5.1.2). Source code of ConsumerWorkService.cs:
namespace RabbitMQ.Client
{
public class ConsumerWorkService
{
...
class WorkPool
{
readonly ConcurrentQueue<Action> actions;
readonly AutoResetEvent messageArrived;
readonly TimeSpan waitTime;
readonly CancellationTokenSource tokenSource;
readonly string name;
public WorkPool(IModel model)
{
name = model.ToString();
actions = new ConcurrentQueue<Action>();
messageArrived = new AutoResetEvent(false);
waitTime = TimeSpan.FromMilliseconds(100);
tokenSource = new CancellationTokenSource();
}
public void Start()
{
#if NETFX_CORE
System.Threading.Tasks.Task.Factory.StartNew(Loop, System.Threading.Tasks.TaskCreationOptions.LongRunning);
#else
var thread = new Thread(Loop)
{
Name = "WorkPool-" + name,
IsBackground = true
};
thread.Start();
#endif
}
public void Enqueue(Action action)
{
actions.Enqueue(action);
messageArrived.Set();
}
void Loop()
{
while (tokenSource.IsCancellationRequested == false)
{
Action action;
while (actions.TryDequeue(out action))
{
try
{
action();
}
catch (Exception)
{
}
}
messageArrived.WaitOne(waitTime);
}
}
public void Stop()
{
tokenSource.Cancel();
}
}
}
}
As we see there is no any waitings of thread var thread = new Thread(Loop). So really event RabbitMQ.Client.Events.EventingBasicConsumer::Received can be fired anytime even when there is no consumer or connection for a long time which closed long time ago till internal library queue will empty. As I supposed(( :
Action action;
while (actions.TryDequeue(out action))
{
try
{
action();
}
catch (Exception)
{
}
}
So IModel::Close() will set only CancelationToken without join of thread and there is need some workaround for this Bug.

Just to confirm #Alexander's findings, this issue is still present in v6.2.2 of the .Net Client and the issue is also present with the event driven callback consumer.
I've found that:
The 'Received' callback registered with both the EventingBasicConsumer and AsyncEventingBasicConsumer will be invoked for up to the prefetchCount setting of the BasicQos setup on the channel, after the connection is closed.
In manual Ack mode, the call to BasicAck will throw a RabbitMQ.Client.Exceptions.AlreadyClosedException if the connection is closed, and the message won't be acked from the queue, however, the callback will continue to process subsequent messages. This could lead to idempotence problems with receiving the same message more than once during a disconnect.
This may be another good reason to ensure that the prefetchCount is set to a finite and sane value (in addition to e.g. Memory considerations of unbounded prefetch counts).
Finally, if I deliberately close the Connection unexpectedly (e.g. during testing), I found that I needed to explicitly detach my consumer Received handler AND explicitly call consumerChannel.Close(); (i.e. IModel.Close) before I could create a new connection (the Rabbit client would tend to 'hang').
The connection.ConnectionShutdown event doesn't seem to fire reliably if I have set a ConsumerDispatchConcurrency > 1 with the synchronous EventingBasicConsumer
The Shutdown event on AsyncEventingBasicConsumer doesn't fire either, if I have ConsumerDispatchConcurrency > 1` when using the async consumer.
However, I did find that the ModelShutdown event on the IModel / channel fires reliably for sync / async and concurrency > 1.
consumerChannel.ModelShutdown += (sender, args) =>
{
consumer.Received -= handler; // e.g. AsyncEventingBasicConsumer
consumerChannel.Close(); // IModel
};

Ui not updating while connecting to wifi?

I am trying to connect to wifi with the following code:
private static bool ConnectToWifi(string profileName, WlanClient.WlanInterface wlanIface, Wifi wifi, string profile)
{
try
{
wlanIface.SetProfile(Wlan.WlanProfileFlags.AllUser, profile, true);
}
catch (Exception e)
{
var ex = e;
return false;
};
// Task.Run()
wlanIface.Connect(Wlan.WlanConnectionMode.Profile, Wlan.Dot11BssType.Infrastructure, profileName);
Thread.Sleep(5000);
var status = wifi.ConnectionStatus;
var x = wlanIface.GetProfileXml(profileName);
if (status == WifiStatus.Disconnected)
{
return false;
}
return true;
}
I have kept a delay of 5000 ms to ensure the network is connected, but this is causing my UI to not show the loading icon when this code executes.
How I can make my UI also update at same time, instead of waiting for the connection?

You have two options:
(Both of which make it not possible to return a bool that indicates a successful connection without a bit more logic around it.)
Move your code to a separate thread (if the rest of it is thread-safe) and use the synchronous methods instead:
private static void ConnectToWifi(string profileName, WlanClient.WlanInterface wlanIface, Wifi wifi, string profile)
{
new Thread(()=>{
bool result = false;
try
{
wlanIface.SetProfile(Wlan.WlanProfileFlags.AllUser, profile, true);
wlanIface.ConnectSynchronously(Wlan.WlanConnectionMode.Profile, Wlan.Dot11BssType.Infrastructure, profileName, 5000);
var status = wifi.ConnectionStatus;
var x = wlanIface.GetProfileXml(profileName);
result = (status != WifiStatus.Disconnected);
}
catch (Exception e)
{
var ex = e;
}
finally
{
Dispatcher.BeginInvoke(new Action(()=>{WhateverYouDoWithYourResult(result);}));
}
}).Start();
}
Or subscribe to the WlanConnectionNotification (Not being able to connect might not be seen as a change, so you have to test that):
private static bool ConnectToWifi(string profileName, WlanClient.WlanInterface wlanIface, Wifi wifi, string profile)
{
try
{
wlanIface.WlanConnectionNotification += Interface_ConnectionStateChanged;
wlanIface.SetProfile(Wlan.WlanProfileFlags.AllUser, profile, true);
wlanIface.Connect(Wlan.WlanConnectionMode.Profile, Wlan.Dot11BssType.Infrastructure, profileName);
return true; //Just means the attempt was successful, not the connecting itself
}
catch (Exception e)
{
var ex = e;
return false;
}
}
private static void Interface_ConnectionStateChanged(Wlan.WlanNotificationData notifyData, Wlan.WlanConnectionNotificationData connNotifyData)
{
// Do something with that info, be careful - might not be the same thread as before.
}
I don't have access to a Wifi right now, so I haven't tested above code. It should work, but you better consider it pseudo-code instead of an actual ready-to-use solution.

Whenever you execute Thread.Sleep in the UI thread, you interrupt processing all UI messages, which makes your UI unresponsive. Hence, Thread.Sleep and any other long running operations should never be executed in the UI thread.
The solution is to execute these operations in a separate thread to allow the UI thread to continue UI operations. It is generally a good idea to let the UI thread do only UI operations.
In your case it means that the caller should execute the operation in a task:
private static bool ConnectToWifi(string profileName, WlanClient.WlanInterface wlanIface,
Wifi wifi, string profile, Action<bool> resultCallback, Dispatcher dispatcher)
{
//Your connect code
bool result;
if (status == WifiStatus.Disconnected)
{
result = false;
}
else
{
result = true;
}
dispatcher.BeginInvoke(() => resultCallback(result));
return result;
}
Another thing: Thread.Sleep is not a good idea in task, since you don't know which scheduler you are running on. You should use Task.Delay instead. In this case Thread.Sleep is generally not a good idea, since you just wait and hope your task is done in five seconds, which is not guaranteed. Also you might simply waste 5 seconds of the user's time in case it connects immediately. The better solution is to use a wait loop and check regularly if the connection is established. If you expect the connection to happen in rather short time, you can use SpinWait.SpinUntil (with a timeout of five seconds):
SpinWait.SpinUntil(() => wifi.ConnectionStatus == WifiStatus.Connected, 5000);

App freezes after Dispatcher.Invoke

I have this application that freezes when calling the dispatcher.invoke for any control.
When i Call the Dispatcher in radiobutton, Grid, Image..etc the App freezes but without giving errors. any help please !!! thank you
I call the thread Method RunClient
private void RunClient()
{
TcpClient client;
// instantiate TcpClient for sending data to server
try
{
// Step 1: create TcpClient and connect to server
client = new TcpClient();
client.Connect(ip, 5001);
// Step 2: get NetworkStream associated with TcpClient
output = client.GetStream();
// create objects for writing and reading across stream
writer = new BinaryWriter(output);
reader = new BinaryReader(output);
string theReply = "";
do
{
try
{
// read the string sent to the server
theReply = reader.ReadString();
int i = 0;
foreach (var x in theReply.Split('#'))
{
ReadString[i] = x;
i++;
}
CheckConnection(ReadString[0]);
}
catch (Exception)
{
//do nothing
}
} while (ReadString[6].Equals(" ") &&
connection.Connected);
updatelabel = () => GameResult(ReadString[6]);
Dispatcher.Invoke(new Action(updatelabel));
if (!connection.Connected)
{
MessageBox.Show("The connection was lost. The game will be closed automatically.");
writer.Close();
reader.Close();
output.Close();
connection.Close();
this.Close();
}
}
// handle exception if error in establishing connection
catch (Exception error)
{
MessageBox.Show("Check Internet Connectivity. Couldn't connect!");
}
}
when the code enters the method ( check connection ) and calls the dispatcher the app freezes.
void CheckConnection(string ii)
{
try
{
if (ii.Equals("Connected"))
{
MessageBox.Show("A Connection was established");
int x = Convert.ToInt32(ReadString[1]);
if (x == 1)
{
updatelabel = () => char1RadioButton2.IsEnabled = false;
char1RadioButton2.Dispatcher.Invoke(new Action(updatelabel));
}
else
{
updatelabel = () => char5RadioButton2.IsEnabled = false;
char5RadioButton2.Dispatcher.Invoke(new Action(updatelabel));
}
updatelabel = () => CreatingGameGrid.Visibility = System.Windows.Visibility.Visible;
CreatingGameGrid.Dispatcher.Invoke(new Action(updatelabel));
updatelabel = () => JoinGameGrid.Visibility = System.Windows.Visibility.Visible;
JoinGameGrid.Dispatcher.Invoke(new Action(updatelabel));
}
else
{
MessageBox.Show("No Such Game found");
this.Close();
}
}
catch (Exception x)
{
MessageBox.Show(x.ToString());
}
}

The Dispatcher.Invoke attempts to synchronously run the specified action on the Dispatcher Thread.
Assuming the RunClient is run on the Dispatcher Thread, and the while loop continues to run while you are trying to Invoke back onto the Dispatcher Thread, the call will freeze.
The simplest solution is to replace all the Dispatcher.Invoke with Dispatcher.BeginInvoke and give it a priority that will run once your RunClient is finished.
The other solution is to run RunClient on a BackgroundWorker.
Similar questions with answers are
Dispatcher.Invoke loop freeze UI
Dispatcher.Invoke hangs main window.
Response to comment on ReadString freeze
Calling Read on a NetworkStream is a blocking call. Well, in fact, it is the Stream obtained by calling TcpClient.GetStream() that blocks. The documentation on MSDN states 'After you have obtained the NetworkStream, call the Write method to send data to the remote host. Call the Read method to receive data arriving from the remote host. Both of these methods block until the specified operation is performed'.
I used dotPeek to see what ReadString was doing and the first thing it does is read the length of the incoming string off the stream using NetworkStream.ReadByte which will block until it has a value to read.
That means the ReadString will sit there until there is data available to read and the amount of data is the same as or more than is expected. You will need to check if you have anything to read before you do by calling stream.DataAvailable or reader.PeekChar.
Alternatively, you could run your socket code on a separate thread. If you are using .Net 4.5, I would take a good look at the Task Parallel Library. ntziolis says in an answer to this question that 'We have made good experiences with that (long being days rather than minutes or hours).'

IIS CPU goes bezerk after updating web.config in a C# application with a Singleton with a thread

I have a web application that does the following:
You click a button to instantiate a singleton, which creates a Thread. That Thread runs continuously doing some HTTP requests to gather some data. You can click a stop button that calls the Abort() method on the thread and the application stops making HTTP requests. When I start/stop it manually, everything works fine.
My problem occurs when ever I "touch" web.config. The CPU (w3wp.exe process) spikes and the website stops responding. Does anybody know why this is happening? Shouldn't an update to web.config reset everything?
Sample code is below:
private static MyProcessor mp = null;
private Thread theThread = null;
private string status = STOP;
public static string STOP = "Stopped";
public static string START = "Started";
private MyProcessor()
{}
public static MyProcessor getInstance()
{
if (mp == null)
{
mp = new MyProcessor();
}
return mp;
}
public void Start()
{
if (this.status == START)
return;
this.theThread = new Thread(new ThreadStart(this.StartThread));
this.theThread.Start();
this.status = START;
}
public void Stop()
{
if (this.theThread != null)
this.theThread.Abort();
this.status = STOP;
}
private void StartThread()
{
do
{
try
{
//do some work with HTTP requests
Thread.Sleep(1000 * 2);
}
catch (Exception e)
{
//retry - work forever
this.StartThread();
}
} while (this.status == START);
}

I suspect this is the problem:
private void StartThread()
{
do
{
try
{
//do some work with HTTP requests
Thread.Sleep(1000 * 2);
}
catch (Exception e)
{
//The recursive call here is suspect
//at the very least find a way to prevent infinite recursion
//--or rethink this strategy
this.StartThread();
}
} while (this.status == START);
}
When your app domain resets, you'll get a ThreadAbort exception which will be caught here and trigger a recursive call, which will hit another exception, and another recursive call. It's turtles all the way down!

Yes, making any change in web .config resets the application, and asp.net re-build the application.
Same is true for some other files like files under Bin and App_Code folders.

Stopping a TcpListener after calling BeginAcceptTcpClient

I have this code...
internal static void Start()
{
TcpListener listenerSocket = new TcpListener(IPAddress.Any, 32599);
listenerSocket.Start();
listenerSocket.BeginAcceptTcpClient(new AsyncCallback(AcceptClient), null);
}
Then my call back function looks like this...
private static void AcceptClient(IAsyncResult asyncResult)
{
MessageHandler handler = new MessageHandler(listenerSocket.EndAcceptTcpClient(asyncResult));
ThreadPool.QueueUserWorkItem((object state) => handler.Process());
listenerSocket.BeginAcceptTcpClient(new AsyncCallback(AcceptClient), null);
}
Now, I call BeginAcceptTcpClient, then some time later I want to stop the server. To do this I have been calling TcpListener.Stop(), or TcpListener.Server.Close(). Both of these however execute my AcceptClient function. This then throws an exception when I call EndAcceptTcpClient. What is the best practice way around this? I could just put a flag in to stop the execution of AcceptClient once I have called stop, but I wonder if I am missing something.
Update 1
Currently I have patched it by changing the code to look like this.
private static void AcceptClient(IAsyncResult asyncResult)
{
if (!shutdown)
{
MessageHandler handler = new MessageHandler(listenerSocket.EndAcceptTcpClient(asyncResult));
ThreadPool.QueueUserWorkItem((object state) => handler.Process());
listenerSocket.BeginAcceptTcpClient(new AsyncCallback(AcceptClient), null);
}
}
private static bool shutdown = false;
internal static void Stop()
{
shutdown = true;
listenerSocket.Stop();
}
Update 2
I changed it to impliment the answer from Spencer Ruport.
private static void AcceptClient(IAsyncResult asyncResult)
{
if (listenerSocket.Server.IsBound)
{
MessageHandler handler = new MessageHandler(listenerSocket.EndAcceptTcpClient(asyncResult));
ThreadPool.QueueUserWorkItem((object state) => handler.Process());
listenerSocket.BeginAcceptTcpClient(new AsyncCallback(AcceptClient), null);
}
}

I just ran into this issue myself, and I believe your current solution is incomplete/incorrect. There is no guarantee of atomicity between the check for IsBound and the subsequent call to EndAcceptTcpClient(). You can still get an exception if the listener is Stop()'d between those two statements. You didn't say what exception you're getting but I assume it's the same one I'm getting, ObjectDisposedException (complaining that the underlying socket has already been disposed).
You should be able to check this by simulating the thread scheduling:
Set a breakpoint on the line after the IsBound check in your callback
Freeze the thread that hits the breakpoint (Threads window -> right click, "Freeze")
Run/trigger the code that calls TcpListener.Stop()
Break in and step through the EndAcceptTcpClient() call. You should see the ObjectDisposedException.
IMO the ideal solution would be for Microsoft to throw a different exception from EndAcceptTcpClient in this case, e.g. ListenCanceledException or something like that.
As it is, we have to infer what's happening from the ObjectDisposedException. Just catch the exception and behave accordingly. In my code I silently eat the exception, since I have code elsewhere that's doing the real shutdown work (i.e. the code that called TcpListener.Stop() in the first place). You should already have exception handling in that area anyway, since you can get various SocketExceptions. This is just tacking another catch handler onto that try block.
I admit I'm uncomfortable with this approach since in principle the catch could be a false positive, with a genuine "bad" object access in there. But on the other hand there aren't too many object accesses in the EndAcceptTcpClient() call that could otherwise trigger this exception. I hope.
Here's my code. This is early/prototype stuff, ignore the Console calls.
private void OnAccept(IAsyncResult iar)
{
TcpListener l = (TcpListener) iar.AsyncState;
TcpClient c;
try
{
c = l.EndAcceptTcpClient(iar);
// keep listening
l.BeginAcceptTcpClient(new AsyncCallback(OnAccept), l);
}
catch (SocketException ex)
{
Console.WriteLine("Error accepting TCP connection: {0}", ex.Message);
// unrecoverable
_doneEvent.Set();
return;
}
catch (ObjectDisposedException)
{
// The listener was Stop()'d, disposing the underlying socket and
// triggering the completion of the callback. We're already exiting,
// so just return.
Console.WriteLine("Listen canceled.");
return;
}
// meanwhile...
SslStream s = new SslStream(c.GetStream());
Console.WriteLine("Authenticating...");
s.BeginAuthenticateAsServer(_cert, new AsyncCallback(OnAuthenticate), s);
}

No you're not missing anything. You can check the IsBound property of the Socket object. At least for TCP connections, while the socket is listening this will be set to true and after you call close it's value will be false. Though, your own implementation can work just as well.

try this one. it works fine for me without catching exceptions.
private void OnAccept(IAsyncResult pAsyncResult)
{
TcpListener listener = (TcpListener) pAsyncResult.AsyncState;
if(listener.Server == null)
{
//stop method was called
return;
}
...
}

i think that all tree things are needed and that the restart of BeginAcceptTcpClient should be placed outside the tryctach of EndAcceptTcpClient.
private void AcceptTcpClientCallback(IAsyncResult ar)
{
var listener = (TcpListener)ar.AsyncState;
//Sometimes the socket is null and somethimes the socket was set
if (listener.Server == null || !listener.Server.IsBound)
return;
TcpClient client = null;
try
{
client = listener.EndAcceptTcpClient(ar);
}
catch (SocketException ex)
{
//the client is corrupt
OnError(ex);
}
catch (ObjectDisposedException)
{
//Listener canceled
return;
}
//Get the next Client
listener.BeginAcceptTcpClient(new AsyncCallback(AcceptTcpClientCallback), listener);
if (client == null)
return; //Abort if there was an error with the client
MyConnection connection = null;
try
{
//Client-Protocoll init
connection = Connect(client.GetStream());
}
catch (Exception ex)
{
//The client is corrupt/invalid
OnError(ex);
client.Close();
}
}

This is a simple example how to start listening, how to process requests asynchronously, and how to stop listening.
Full example here.
public class TcpServer
{
#region Public.
// Create new instance of TcpServer.
public TcpServer(string ip, int port)
{
_listener = new TcpListener(IPAddress.Parse(ip), port);
}
// Starts receiving incoming requests.
public void Start()
{
_listener.Start();
_ct = _cts.Token;
_listener.BeginAcceptTcpClient(ProcessRequest, _listener);
}
// Stops receiving incoming requests.
public void Stop()
{
// If listening has been cancelled, simply go out from method.
if(_ct.IsCancellationRequested)
{
return;
}
// Cancels listening.
_cts.Cancel();
// Waits a little, to guarantee
// that all operation receive information about cancellation.
Thread.Sleep(100);
_listener.Stop();
}
#endregion
#region Private.
// Process single request.
private void ProcessRequest(IAsyncResult ar)
{
//Stop if operation was cancelled.
if(_ct.IsCancellationRequested)
{
return;
}
var listener = ar.AsyncState as TcpListener;
if(listener == null)
{
return;
}
// Check cancellation again. Stop if operation was cancelled.
if(_ct.IsCancellationRequested)
{
return;
}
// Starts waiting for the next request.
listener.BeginAcceptTcpClient(ProcessRequest, listener);
// Gets client and starts processing received request.
using(TcpClient client = listener.EndAcceptTcpClient(ar))
{
var rp = new RequestProcessor();
rp.Proccess(client);
}
}
#endregion
#region Fields.
private CancellationToken _ct;
private CancellationTokenSource _cts = new CancellationTokenSource();
private TcpListener _listener;
#endregion
}