We Keep Coding

How to detect a network was broken down with ZeroMQ monitor mechanism?

A. Description
I am using ZeroMQ monitor and I find that it works when logic disconnection but does not work when network broken down (unplug my cable).
For example:
I launch client app on an android pad, then launch a server app on my windows laptop. They are connected through a router with cables.
Everything will be OK with monitor if I close or open either client app or server app manually. Namely, the monitor on both sides can receive a 'Connect' or an 'Accept' and a 'Disconnect' event.
But If I unplug cable on the server side, while the client and server are connected and running, the monitors on both sides can not detect the 'Disconnect' event.
Is the monitor designed like this?
If so, are there any solutions to detect network broken down ( a cable unplug event ) except heartbeats?
If not, how to use the ZeroMQ's original monitor mechanism to solve this problem? Can a setTCPKeepAlive() interface be useful?
B. System environment
My scenario is as below.
Client
OS: Android, running on a pad, IDE: Android studio 2.3, lib:jeromq-0.4.3
// Java Code
String monitorAddr = "inproc://client.req";
ZContext ctx = new ZContext();
ZMQ.Socket clientSocket = ctx.createSocket(ZMQ.REQ);
clientSocket.monitor(monitorAddr,ZMQ.EVENT_ALL);
// Then start a montitor thread which is implemented by my own.
Server
OS: Windows 7 ( 64 bit ), running on my laptop, IDE: VS2013, lib: Clrzmq4
// C# Code
const string MonitorEndpoint = "inproc://server.rep";
var ctx = new ZContext();
var serverSocket = new ZSocket(ctx,ZSocketType.REP);
ZError error;
// Create serverSocket pair socket
if (!serverSocket.Monitor(MonitorEndpoint, ZMonitorEvents.AllEvents, out error))
{
if (error == ZError.ETERM)
return ; // Interrupted
throw new ZException(error);
}
// Create a monitor
ZMonitor _monitor = ZMonitor.Create(ctx, MonitorEndpoint);
_monitor.AllEvents += _monitor_AllEvents;
_monitor.Start();

AFAIK there is no built in heartbeat within ZeroMQ. I know there was some discussion on the topic within the ZMQ community some years ago, and that discussion may still be going on.
It is comparatively simple to incorporate your own heartbeat messaging in your application's use of ZeroMQ, especially if you use something like Google Protocol Buffers to encode different message types; the heartbeat is just another message.
Doing heartbeats in your application (rather than relying on some inbuilt mechanism) is ultimately more flexible; you can choose the heartbeat rate, you can choose what to do if the heartbeat fails, you can decide when heartbeating is important and not important, etc.
Consider heartbeats within a PUB/SUB pattern; it's a bit difficult for the ZMQ authors to decide on your behalf what connection / disconnection / connection-break events matter to you. And if they do build in a mechanism, but an application developer didn't want it, then it is a waste of bandwidth.
It's far easier for the ZMQ authors to leave that kind of application architectural issue to the application author (that's you!) to deal with.
With your specific example, an unplugged network cable simply looks (so far as any software can determine) like no traffic is flowing; it's the same as the application not sending anything. ZMQ doesn't send anything if the application hasn't sent anything.
If you look at the events that the socket monitor can report on, they're all the consequence of something flowing over the network connection, or something done to the socket by the application.

Trying to go lower than ZMQ protocol itself and access the TCP connection that specific ZeroMQ sockets use (while others do not) doesn't sound like a good idea; it would required to break encapsulation in multiple classes.
The answer #bazza gave in 2017 was entirely correct at the time.
However, newer versions of ZMQ (specifically ZMTP) include an heartbeat functionality.
Check ZMQ documentation for
socketOpt
Java functions
name
purpose
ZMQ_HEARTBEAT_IVL
get/setHeartbeatLvl()
heartbeat interval
milliseconds between ZMPT PINGs
ZMQ_HEARTBEAT_TIMEOUT
get/setHeartbeatTimeout()
local heartbeat timeout
how long the local socket waits between received packets until it considers the connection timed out
ZMQ_HEARTBEAT_TTL
get/setHeartbeatTtl()
remote heartbeat timeout
if and when remote side shall consider the connection timed out
ZMQ_HEARTBEAT_CONTEXT is still in draft state as of 2022. It is supposed to send an byte[] context with every ping.
Now, by design of ZMQ, quoting from chapter 2 of its documentation,
The network connection itself happens in the background, and ZeroMQ
will automatically reconnect if the network connection is broken
(e.g., if the peer disappears and then comes back).
Thus, answering your main question, I'd expect the monitor to give you ZMQ_EVENT_CONNECT_RETRIED / ZMQ_EVENT_CONNECTED events after the underlying connection was detected as disrupted.

Console app SignalR client Disconnect Timeout

I am trying to set Disconnect Timeout to higher value from the default 30s.
All examples on web are more JS oriented.
var hubConnection = new HubConnection("http://localhost:8087");
var testHubProxy = hubConnection.CreateHubProxy("TestHub");
Error: System.TimeoutException: Couldn't reconnect within the configured timeout of 00:00:30, disconnecting.
This did not work:
GlobalHost.Configuration.DisconnectTimeout = TimeSpan.FromSeconds(35);
Update:
It looks like DisconnectTimeout needs to be set on the server side!?
What is the reason for disallowing different clients to have different Disconnect Timeout?

Disconnect Timeout is configured on server-side. The main reasons could be as follows:
We know the server may take some N-time units to respond so that the all clients may be well aware.
The server should be pinging the clients for connection at regular times. So the server is aware of clients connection and can manage other hubs and eradicate the expired connections from its connection pool.
The client is not supposed to set disconnect timeout because it does not know when could it shutdown e.g. the internet switched off accidentally on client side than the client is not able to tell server that I am not going to connect to you again. Yes but we have some events at client-side which tells it that it is not connected to the signalr hub anymore. Please see the reconnecting and disconnected events.
Summary:
Disconnect timeout is to inform the server that its client is not connected anymore even if it disconnects disgracefully .

MSMQ: Messages occasionally not being sent or received without error

I have the following setup and problem with MSMQ. Based on previous experience with MSMQ I'm betting that it is something simple I'm missing but I just don't know what it is.
The Setup
I have 3 load-balanced web servers (lets call them Servers W1, W2 and W3) and 1 server which processes certain events/data away from web requests (which I'll call P). All 3 of the web servers, once a particular event occurs within the web application, will send a message to a remote private queue on Server P, which will then process each message from the queue and carry out some task.
The Problem
For the most part - at a guess 95% of the time - everything runs fine, but occasionally Server P does not receive messages from the web servers. This is either because W1, W2 or W3 are not sending them or they are not being received by P, I just can't tell. This means I'm missing vital events happening from the users on the web application but I cannot find any errors listed in my own logs.
The Details
Here are all the details I can think of which may help explain my setup and what I've figured out so far:
The private queue on Server P is non-transactional.
The private queue has permissions setup for Everyone to both Send and Receive Messages.
This is the code I use (C#) to send the message to the remote private queue:
var queue = new MessageQueue(#"FormatName:DIRECT=OS:ServerP\PRIVATE$\MyMessageQueue");
var defaultProperties = queue.DefaultPropertiesToSend;
defaultProperties.AcknowledgeType = AcknowledgeTypes.FullReachQueue | AcknowledgeTypes.FullReceive;
defaultProperties.Recoverable = true;
defaultProperties.UseDeadLetterQueue = true;
defaultProperties.UseJournalQueue = true;
queue.Send(requestData);
Sending the message using the code above does not appear to throw an exception - if it did my error handler in the web application would have caught and logged it, so I'm assuming it is sent.
There are outgoing queues on W1, W2 and W3 all pointing to the private queue on P - all these are empty.
On W1, W2 and W3 I cannot see any "dead-letter" messages.
On P the private queue is empty so messages are being processed (which I can verify from my database).
On P there are no "dead-letter" messages. There are journal messages but they don't seem to correspond to any recent date/times.
All servers are running Windows Server 2012.
Most of the time messages are sent, received and processed just fine but, without any pattern visible to me, sometimes they are not. Can anyone see what is going wrong? Or explain to me how I can try and figure out what is happening?

Are you sure that the receiver on P does not crash/lose the message somehow? Because your queue is not transactional, if somehow processing fails then that's one lost message.
Anyway, there are many possible causes why this could fail.
What kind of logging do you have (DEBUG/INFO levels)?
I think the following will help tracking down the issue:
When an event is generated in the web app.
Right before you send an event from the web app, via MSMQ.
In the receiver when you get a message from the queue.
This way you could at least match sent messages to received messages and to processed messages.
As a side note, when you check for dead-letter messages you do so on the source computer and on any intermediary hops, not on the destination one. If you don't have any hops, then they will be relayed to the non-transactional dead-letter queue on the web servers.

Problem about checking a WCF connection is opened

I have a problem about checking a WCF connection is opened. My WCF Connection is bi-directional. I use State property to check the connection's state at client. My function:
private bool ConnectionIsOpen()
{
if (m_Service != null && (m_Service.State | CommunicationState.Opened) == CommunicationState.Opened)
{
return true;
}
return false;
}
I create a service which is a thread running every 10 seconds to check the connection's state. I use the method ConnectionIsOpen() for checking. Everything is well on running on Windows XP. However there is a problem when running on Windows 7.
When I unplug the network cable to create to disconnect, If running application on Windows XP, checking connection's State is Faulted but if running on Windows 7, checking connection' State is still Opened.
Anyone can help me how to check a connection is openned or not in this case. Thanks.

This will always be true:
(m_Service.State | CommunicationState.Opened) == CommunicationState.Opened
Example, m_Service.State = 0:
0 | CommuncationState.Opened == CommuncationState.Opened
You want to use & (AND) instead.

We ran into a similar problem in our own system; disconnecting the network cable or placing either the client machine or the server in sleep mode does not generate a channel fault.
From what I can tell, it seems that the connection state only indicates the state of the connection after the last call and not the current connection state. The only way to know the current state is to actually call the service.
If your client doesn’t need to call the service that often but must react if the connection is lost one solution is to implement a dummy call on the client side which periodically polls the service. If the connection is unavailable when the dummy call is made you’ll get a channel fault that you can then deal with.
The catch is you can’t simply use the dummy call to guarantee that the next call to the service will work:
public void SomeMethode()
{
if (ConnectionIsOpen())
{
m_Service.Dummy();
// Connection is lost here
m_Service.SomeMethode();
}
}
To get around this problem, we implemented a system that automatically re-executes any failed service calls which generate a channel fault after the connection has been restored.

The best and asured way to confirm the Communication state is Open or not is to call the Faulted event like below :
proxyInstance.InnerChannel.Faulted -= new EventHandler(ProxyChannelFaulted);
But this works only with those bindings that support ReliableMessaging like WsHttpBinding.
For detail refer the link : WCF Proxy Client taking time to create, any cache or singleton solution for it
Thanks,
Jai Kumar

The fact that you are getting completely different results on windows 7 is not surprising. Microsoft completely re-engineered the TCP stack with windows vista, so the functionality is quite different from xp in the core networking functionality.
The first thing that I would do is use wireshark to see what is actually going across the wire. See if your TCP connection actually terminates when you pull the plug. Windows might be doing some kind of connection persistence / buffering in case the connection comes back quickly.

WCF - Client callback vs. polling for "keep list of subscribers"

I want to create a simple client-server example in WCF. I did some testing with callbacks, and it works fine so far. I played around a little bit with the following interface:
[ServiceContract(SessionMode = SessionMode.Required, CallbackContract = typeof(IStringCallback))]
public interface ISubscribeableService
{
[OperationContract]
void ExecuteStringCallBack(string value);
[OperationContract]
ServerInformation Subscribe(ClientInformation c);
[OperationContract]
ServerInformation Unsubscribe(ClientInformation c);
}
Its a simple example. a little bit adjusted. You can ask the server to "execute a string callback" in which case the server reversed the string and calls all subscribed client callbacks.
Now, here comes the question: If I want to implement a system where all clients "register" with the server, and the server can "ask" the clients if they are still alive, would you implement this with callbacks (so instead of this "stringcallback" a kind of TellTheClientThatIAmStillHereCallback). By checking the communication state on the callback I can also "know" if a client is dead. Something similar to this:
Subscribers.ForEach(delegate(IStringCallback callback)
{
if (((ICommunicationObject)callback).State == CommunicationState.Opened)
{
callback.StringCallbackFunction(new string(retVal));
}
else
{
Subscribers.Remove(callback);
}
});
My problem, put in another way:
The server might have 3 clients
Client A dies (I pull the plug of the laptop)
The server dies and comes back online
A new client comes up
So basically, would you use callbacks to verify the "still living state" of clients, or would you use polling and keep track "how long I havent heard of a client"...

You can detect most changes to the connection state via the Closed, Closing, and Faulted events of ICommunicationObject. You can hook them at the same time that you set up the callback. This is definitely better than polling.
IIRC, the Faulted event will only fire after you actually try to use the callback (unsuccessfully). So if the Client just disappears - for example, a hard reboot or power-off - then you won't be notified right away. But do you need to be? And if so, why?
A WCF callback might fail at any time, and you always need to keep this in the back of your mind. Even if both the client and server are fine, you might still end up with a faulted channel due to an exception or a network outage. Or maybe the client went offline sometime between your last poll and your current operation. The point is, as long as you code your callback operations defensively (which is good practice anyway), then hooking the events above is usually enough for most designs. If an error occurs for any reason - including a client failing to respond - the Faulted event will kick in and run your cleanup code.
This is what I would refer to as the passive/lazy approach and requires less coding and network chatter than polling or keep-alive approaches.

If you enable reliable sessions, WCF internally maintains a keep-alive control mechanism. It regularly checks, via hidden infrastructure test messages, if the other end is still there. The time interval of these checks can be influenced via the ReliableSession.InactivityTimeout property. If you set the property to, say, 20 seconds, then the ICommunicationObject.Faulted event will be raised about 20 to 30 (maximum) seconds after a service breakdown has occurred on the other side.
If you want to be sure that client applications always remain "auto-connected", even after temporary service breakdowns, you may want to use a worker thread (from the thread pool) that repeatedly tries to create a new proxy instance on the client side, and calls a session-initiating operation, after the Faulted event has been raised there.
As a second approach, since you are implementing a worker thread mechanism anyway, you might also ignore the Faulted event and let the worker thread loop during the whole lifetime of the client application. You let the thread repeatedly check the proxy state, and try to do its repair work whenever the state is faulted.
Using the first or the second approach, you can implement a service bus architecture (mediator pattern), guaranteeing that all client application instances are constantly ready to receive "spontaneous" service messages whenever the service is running.
Of course, this only works if the reliable session "as such" is configured correctly to begin with (using a session-capable binding, and applying the ServiceContractAttribute.SessionMode, ServiceBehaviorAttribute.InstanceContextMode, OperationContractAttribute.IsInitiating, and OperationContractAttribute.IsTerminating properties in meaningful ways).

I had a similar situation using WCF and callbacks. I did not want to use polling, but I was using a "reilable" protocol, so if a client died, then it would hang the server until it timed out and crashed.
I do not know if this is the most correct or elegant solution, but what I did was create a class in the service to represent the client proxy. Each instance of this class contained a reference to the client proxy, and would execute the callback function whenever the server set the "message" property of the class. By doing this, when a client disconnected, the individual wrapper class would get the timeout excetpion, and remove itself from the server's list of listeners, but the service would not have to wait for it. This doesn't actually answer your question about determining if the client is alive, but it is another way of structuring the service to addrss the issue. If you needed to know when a client died, you would be able to pick up when the client wrapper removed itself from the listener list.

I have not tried to use WCF callbacks over the wire but i have used them for interprocess communication. I was having a problem where call of the calls that were being sent were ending up on the same thread and making the service dead lock when there were calls that were dependant on the same thread.
This may apply to the problem that you are currently have so here is what I had to do to fix the problem.
Put this attribute onto the server and client of the WCF server implemetation class
[ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Multiple)]
public class WCFServerClass
The ConcurrencyMode.Multiple makes each call process on its own thread which should help you with the server locking up when a client dies until it timesout.
I also made sure to use a Thread Pool on the client side to make sure that there were no threading issues on the client side