We Keep Coding

How to detect a network was broken down with ZeroMQ monitor mechanism?

A. Description
I am using ZeroMQ monitor and I find that it works when logic disconnection but does not work when network broken down (unplug my cable).
For example:
I launch client app on an android pad, then launch a server app on my windows laptop. They are connected through a router with cables.
Everything will be OK with monitor if I close or open either client app or server app manually. Namely, the monitor on both sides can receive a 'Connect' or an 'Accept' and a 'Disconnect' event.
But If I unplug cable on the server side, while the client and server are connected and running, the monitors on both sides can not detect the 'Disconnect' event.
Is the monitor designed like this?
If so, are there any solutions to detect network broken down ( a cable unplug event ) except heartbeats?
If not, how to use the ZeroMQ's original monitor mechanism to solve this problem? Can a setTCPKeepAlive() interface be useful?
B. System environment
My scenario is as below.
Client
OS: Android, running on a pad, IDE: Android studio 2.3, lib:jeromq-0.4.3
// Java Code
String monitorAddr = "inproc://client.req";
ZContext ctx = new ZContext();
ZMQ.Socket clientSocket = ctx.createSocket(ZMQ.REQ);
clientSocket.monitor(monitorAddr,ZMQ.EVENT_ALL);
// Then start a montitor thread which is implemented by my own.
Server
OS: Windows 7 ( 64 bit ), running on my laptop, IDE: VS2013, lib: Clrzmq4
// C# Code
const string MonitorEndpoint = "inproc://server.rep";
var ctx = new ZContext();
var serverSocket = new ZSocket(ctx,ZSocketType.REP);
ZError error;
// Create serverSocket pair socket
if (!serverSocket.Monitor(MonitorEndpoint, ZMonitorEvents.AllEvents, out error))
{
if (error == ZError.ETERM)
return ; // Interrupted
throw new ZException(error);
}
// Create a monitor
ZMonitor _monitor = ZMonitor.Create(ctx, MonitorEndpoint);
_monitor.AllEvents += _monitor_AllEvents;
_monitor.Start();

AFAIK there is no built in heartbeat within ZeroMQ. I know there was some discussion on the topic within the ZMQ community some years ago, and that discussion may still be going on.
It is comparatively simple to incorporate your own heartbeat messaging in your application's use of ZeroMQ, especially if you use something like Google Protocol Buffers to encode different message types; the heartbeat is just another message.
Doing heartbeats in your application (rather than relying on some inbuilt mechanism) is ultimately more flexible; you can choose the heartbeat rate, you can choose what to do if the heartbeat fails, you can decide when heartbeating is important and not important, etc.
Consider heartbeats within a PUB/SUB pattern; it's a bit difficult for the ZMQ authors to decide on your behalf what connection / disconnection / connection-break events matter to you. And if they do build in a mechanism, but an application developer didn't want it, then it is a waste of bandwidth.
It's far easier for the ZMQ authors to leave that kind of application architectural issue to the application author (that's you!) to deal with.
With your specific example, an unplugged network cable simply looks (so far as any software can determine) like no traffic is flowing; it's the same as the application not sending anything. ZMQ doesn't send anything if the application hasn't sent anything.
If you look at the events that the socket monitor can report on, they're all the consequence of something flowing over the network connection, or something done to the socket by the application.

Trying to go lower than ZMQ protocol itself and access the TCP connection that specific ZeroMQ sockets use (while others do not) doesn't sound like a good idea; it would required to break encapsulation in multiple classes.
The answer #bazza gave in 2017 was entirely correct at the time.
However, newer versions of ZMQ (specifically ZMTP) include an heartbeat functionality.
Check ZMQ documentation for
socketOpt
Java functions
name
purpose
ZMQ_HEARTBEAT_IVL
get/setHeartbeatLvl()
heartbeat interval
milliseconds between ZMPT PINGs
ZMQ_HEARTBEAT_TIMEOUT
get/setHeartbeatTimeout()
local heartbeat timeout
how long the local socket waits between received packets until it considers the connection timed out
ZMQ_HEARTBEAT_TTL
get/setHeartbeatTtl()
remote heartbeat timeout
if and when remote side shall consider the connection timed out
ZMQ_HEARTBEAT_CONTEXT is still in draft state as of 2022. It is supposed to send an byte[] context with every ping.
Now, by design of ZMQ, quoting from chapter 2 of its documentation,
The network connection itself happens in the background, and ZeroMQ
will automatically reconnect if the network connection is broken
(e.g., if the peer disappears and then comes back).
Thus, answering your main question, I'd expect the monitor to give you ZMQ_EVENT_CONNECT_RETRIED / ZMQ_EVENT_CONNECTED events after the underlying connection was detected as disrupted.

Console app SignalR client Disconnect Timeout

I am trying to set Disconnect Timeout to higher value from the default 30s.
All examples on web are more JS oriented.
var hubConnection = new HubConnection("http://localhost:8087");
var testHubProxy = hubConnection.CreateHubProxy("TestHub");
Error: System.TimeoutException: Couldn't reconnect within the configured timeout of 00:00:30, disconnecting.
This did not work:
GlobalHost.Configuration.DisconnectTimeout = TimeSpan.FromSeconds(35);
Update:
It looks like DisconnectTimeout needs to be set on the server side!?
What is the reason for disallowing different clients to have different Disconnect Timeout?

Disconnect Timeout is configured on server-side. The main reasons could be as follows:
We know the server may take some N-time units to respond so that the all clients may be well aware.
The server should be pinging the clients for connection at regular times. So the server is aware of clients connection and can manage other hubs and eradicate the expired connections from its connection pool.
The client is not supposed to set disconnect timeout because it does not know when could it shutdown e.g. the internet switched off accidentally on client side than the client is not able to tell server that I am not going to connect to you again. Yes but we have some events at client-side which tells it that it is not connected to the signalr hub anymore. Please see the reconnecting and disconnected events.
Summary:
Disconnect timeout is to inform the server that its client is not connected anymore even if it disconnects disgracefully .

.NET WebSockets forcibly closed despite keep-alive and activity on the connection

We have written a simple WebSocket client using System.Net.WebSockets. The KeepAliveInterval on the ClientWebSocket is set to 30 seconds.
The connection is opened successfully and traffic flows as expected in both directions, or if the connection is idle, the client sends Pong requests every 30 seconds to the server (visible in Wireshark).
But after 100 seconds the connection is abruptly terminated due to the TCP socket being closed at the client end (watching in Wireshark we see the client send a FIN). The server responds with a 1001 Going Away before closing the socket.
After a lot of digging we have tracked down the cause and found a rather heavy-handed workaround. Despite a lot of Google and Stack Overflow searching we have only seen a couple of other examples of people posting about the problem and nobody with an answer, so I'm posting this to save others the pain and in the hope that someone may be able to suggest a better workaround.
The source of the 100 second timeout is that the WebSocket uses a System.Net.ServicePoint, which has a MaxIdleTime property to allow idle sockets to be closed. On opening the WebSocket if there is an existing ServicePoint for the Uri it will use that, with whatever the MaxIdleTime property was set to on creation. If not, a new ServicePoint instance will be created, with MaxIdleTime set from the current value of the System.Net.ServicePointManager MaxServicePointIdleTime property (which defaults to 100,000 milliseconds).
The issue is that neither WebSocket traffic nor WebSocket keep-alives (Ping/Pong) appear to register as traffic as far as the ServicePoint idle timer is concerned. So exactly 100 seconds after opening the WebSocket it just gets torn down, despite traffic or keep-alives.
Our hunch is that this may be because the WebSocket starts life as an HTTP request which is then upgraded to a websocket. It appears that the idle timer is only looking for HTTP traffic. If that is indeed what is happening that seems like a major bug in the System.Net.WebSockets implementation.
The workaround we are using is to set the MaxIdleTime on the ServicePoint to int.MaxValue. This allows the WebSocket to stay open indefinitely. But the downside is that this value applies to any other connections for that ServicePoint. In our context (which is a Load test using Visual Studio Web and Load testing) we have other (HTTP) connections open for the same ServicePoint, and in fact there is already an active ServicePoint instance by the time that we open our WebSocket. This means that after we update the MaxIdleTime, all HTTP connections for the Load test will have no idle timeout. This doesn't feel quite comfortable, although in practice the web server should be closing idle connections anyway.
We also briefly explore whether we could create a new ServicePoint instance reserved just for our WebSocket connection, but couldn't see a clean way of doing that.
One other little twist which made this harder to track down is that although the System.Net.ServicePointManager MaxServicePointIdleTime property defaults to 100 seconds, Visual Studio is overriding this value and setting it to 120 seconds - which made it harder to search for.

I ran into this issue this week. Your workaround got me pointed in the right direction, but I believe I've narrowed down the root cause.
If a "Content-Length: 0" header is included in the "101 Switching Protocols" response from a WebSocket server, WebSocketClient gets confused and schedules the connection for cleanup in 100 seconds.
Here's the offending code from the .Net Reference Source:
//if the returned contentlength is zero, preemptively invoke calldone on the stream.
//this will wake up any pending reads.
if (m_ContentLength == 0 && m_ConnectStream is ConnectStream) {
((ConnectStream)m_ConnectStream).CallDone();
}
According to RFC 7230 Section 3.3.2, Content-Length is prohibited in 1xx (Informational) messages, but I've found it mistakenly included in some server implementations.
For additional details, including some sample code for diagnosing ServicePoint issues, see this thread: https://github.com/ably/ably-dotnet/issues/107

I set the KeepAliveInterval for the socket to 0 like this:
theSocket.Options.KeepAliveInterval = TimeSpan.Zero;
That eliminated the problem of the websocket shutting down when the timeout was reached. But then again, it also probably turns off the send of ping messages altogether.

I studied this issue these days, compared capture packages in Wireshark(webclient-client of python and WebSocketClient of .Net), and found what happened. In WebSocketClient, "Options.KeepAliveInterval" only send one packet to the server when no message received from server in these period. But some server only judge if there is active message from client. So we have to manually send arbitrary packets (not necessarily ping packets,and WebSocketMessageType has no ping type) to the server at regular intervals,even if the server side continuously sends packets. That's the solution.

Windows 8 - .NET TCP AcceptAsync callback not firing (blocked by Console.ReadLine())

I'm experiencing an issue specific to Windows 8 and VS2012.
I have a TCP socket server and client and am doing some testing on the local network. With sysinternals TCPView, I can see that packets are sent from the TCP client and arrive at the TCP Server (I see the packet counters increase).
However, it appears as if the data is not making it to the application stack?
The very same build runs without issues on Windows 7.
I have the Windows 8 firewall turned off and run both process with elevated permissions on a domain admin users with UAC turned off.
When I connect the client to a an outside server (running on a separate machine), everything works fine.
Is there anything else in Windows 8 that could prohibit TCP data communication between local processes?
Thanks,
EDIT
To make sure nothing in my server application is causing this issue, I built a quick TCP server in a console application, with the following code for the socket constructor:
listenerSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
and listen on the same local IP/Port as my server application.
I'm experiencing the same issue, I can telnet to the port but listenerSocket.AcceptAsync is never hit.
EDIT 2
Upon further testing, it appers my issue has something to do with the use of the Async socket calls, i.e. if I use the synchronous calls like socket.Accept(), the test application is performing normally. However, when I use Async socket calls, i.e. socket.AcceptAsync(), I'm experiencing the issues mentioned. So far I couldn't find any mention of differences between win7 & 8 in regards to async socket calls.
Here's my quick sample app that shows that the async callback is never triggered. This snippet works fine in Windows 7 but does not work in Windows 8 (try to telnet to 127.0.0.1 : 7000).
class Program
{
private static SocketAsyncEventArgs socketAsyncEventArgs = new SocketAsyncEventArgs();
static void Main(string[] args)
{
var listenerSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
listenerSocket.Bind(new IPEndPoint(IPAddress.Parse("127.0.0.1"), 7000));
listenerSocket.Listen(100);
socketAsyncEventArgs.Completed += AcceptEventArg_Completed;
listenerSocket.AcceptAsync(socketAsyncEventArgs);
Console.ReadLine();
}
private static void AcceptEventArg_Completed(object sender, SocketAsyncEventArgs e)
{
Console.WriteLine("AcceptEventArg_Completed");
}
}
EDIT 3
I found 2 others reporting the same issue on Microsoft Connect:
https://connect.microsoft.com/VisualStudio/feedback/details/759913/socketasynceventargs-completed-doesnt-fire-in-net-framework-4-5
and
http://connect.microsoft.com/VisualStudio/feedback/details/747218/saea-not-working-in-net-4-5-rp
Whereas the 2nd one is interesting as it seems to conclude there is a Windows bug in the Console.ReadLine() call and it is causing the issue and blocking the async callback. If I replace Console.ReadLine() in my snippet with:
while (true)
{
System.Threading.Thread.Sleep(10);
}
everything works fine.

See this: GetQueuedCompletionStatus can't dequeue IO from IOCP if the thread which originally issued the IO is blocking in ReadFile under windows 8
It's a bug in windows 8 and 2012 and affect all programs which used AcceptEx and ReadFile. As for now, only these two functions are known affected.

I meet the same thing when I was developing Tcp server and client applications with SocketAsyncEventArgs
I suggest you try this first.
open Windows firewall with Advanced Security
check the inbound / outbound rules to see if your application is blocked.
open AssemblyInfo.cs and change the
[assembly: Guid("06985fe3-80eb-48b4-940a-fd926e2f2053")]
to any other guid value.
By changing this, windows will think this is a new application and if there were any restrictions towards the old application, it wont be on the new one.

Sounds like this windows bug relating to the IOCP processing (possibly just of AcceptEx) on Windows 8 while other blocking I/O is in progress on the same thread:
http://social.technet.microsoft.com/Forums/en-GB/winserver8gen/thread/5764cd0f-fda1-4cfa-ae35-808210bae77e
So the socket connection is accepted, but your app never recieves notification of it.
Maybe Windows 8 does some weird, slighly broken, voodoo to convert synchronous IO like Console.Read into async internally.
You could just move your server code into a different thread, other workarounds to try might be to perform Accept synchonously or to change the console processing to be asynchronous (I can't really try that as I don't have Windows 8).

Proper implementation of C# TCP reconnecting client

I have to write a TCP Client that will have ability to reconnect to server. The server can be unavailable due to poor network connection quality or some maintenance issues. I'm searching for quality solutions in this area.
My current solutions is following:
keep connection state in ConnectionState enum {Offline, Online, Connecting}
create client with TcpClient class.
create two timers called ConnectionCheckTimer, and ReconnectTimer
connect to server
start reader thread and connection check timer
reading is performed with tcpClient.GetStream() and then reading from this stream
when Exception is caught in readerLoop client state is changed to offline and ReconnectTimer is launched
ConnectionCheckTimer periodically checks lastMessageTimestamp and compares it with current time if the interval is greater then maxValue it launches ReconnectTimer
Currently i'm not satisfied with this solution because it still generates exceptions for instance ObjectDisposedException on TcpClient.NetworkStream. I'm looking for some clean and reusable Tcp reconnecting client implementation that is able to cope with all sockets problems that can occur during connecting, disconnecting, reading data.

If you have connection issues, you will always have exceptions. I think you have a sound outline, you just need to handle the exceptions. You could start with your own Socket class implemenation and write the TCPIP Server. Starter code is at MS:
http://msdn.microsoft.com/en-us/library/fx6588te(VS.71).aspx
The C# code is half way down the VB page.

The class you should use is "SocketAsyncEventArgs".
I've used it in this project:
http://ts3querylib.codeplex.com/
Have a look at the AsyncTcpDispatcher class.