I have to write a TCP Client that will have ability to reconnect to server. The server can be unavailable due to poor network connection quality or some maintenance issues. I'm searching for quality solutions in this area.
My current solutions is following:
keep connection state in ConnectionState enum {Offline, Online, Connecting}
create client with TcpClient class.
create two timers called ConnectionCheckTimer, and ReconnectTimer
connect to server
start reader thread and connection check timer
reading is performed with tcpClient.GetStream() and then reading from this stream
when Exception is caught in readerLoop client state is changed to offline and ReconnectTimer is launched
ConnectionCheckTimer periodically checks lastMessageTimestamp and compares it with current time if the interval is greater then maxValue it launches ReconnectTimer
Currently i'm not satisfied with this solution because it still generates exceptions for instance ObjectDisposedException on TcpClient.NetworkStream. I'm looking for some clean and reusable Tcp reconnecting client implementation that is able to cope with all sockets problems that can occur during connecting, disconnecting, reading data.
If you have connection issues, you will always have exceptions. I think you have a sound outline, you just need to handle the exceptions. You could start with your own Socket class implemenation and write the TCPIP Server. Starter code is at MS:
http://msdn.microsoft.com/en-us/library/fx6588te(VS.71).aspx
The C# code is half way down the VB page.
The class you should use is "SocketAsyncEventArgs".
I've used it in this project:
http://ts3querylib.codeplex.com/
Have a look at the AsyncTcpDispatcher class.
Related
I have a TCP client and I set the network stream timeout as follows.
stream.ReadTimeout = 60000;
It works. But I would like to know how to test if the stream timed out. The class doesn't provide this method.
A little more detail to the question.
I am sending data to a TCPListener, about 33KB every 30 minutes. Typically, the transmission lasts about 10s and the client issues a manual "DISCONNECT" command to causes the Listener to start again. The client is an embedded system using a 3G module and sometimes the network connectivity causes the link to break. Right now, I am simply setting a read timeout of 60s. If we do not get data during that time, we simply restart the listener and wait for the next connection.
I am logging the performance of the system and would like to know how many timeouts typically occur in, say, one week. It'd have been good for the listener to simply check if the read operation time out, but I do not see a way of doing it easily in C#.
Will appreciate any help.
I do not really understand the problem about logging. I would look for the Read operation's return value, because if that is 0, a timeout occured for sure. And before I reinitialized the listener I would put a logging logic that logs the fact of the timeout. Please tell me if I misunderstood the concept of your program.
A. Description
I am using ZeroMQ monitor and I find that it works when logic disconnection but does not work when network broken down (unplug my cable).
For example:
I launch client app on an android pad, then launch a server app on my windows laptop. They are connected through a router with cables.
Everything will be OK with monitor if I close or open either client app or server app manually. Namely, the monitor on both sides can receive a 'Connect' or an 'Accept' and a 'Disconnect' event.
But If I unplug cable on the server side, while the client and server are connected and running, the monitors on both sides can not detect the 'Disconnect' event.
Is the monitor designed like this?
If so, are there any solutions to detect network broken down ( a cable unplug event ) except heartbeats?
If not, how to use the ZeroMQ's original monitor mechanism to solve this problem? Can a setTCPKeepAlive() interface be useful?
B. System environment
My scenario is as below.
Client
OS: Android, running on a pad, IDE: Android studio 2.3, lib:jeromq-0.4.3
// Java Code
String monitorAddr = "inproc://client.req";
ZContext ctx = new ZContext();
ZMQ.Socket clientSocket = ctx.createSocket(ZMQ.REQ);
clientSocket.monitor(monitorAddr,ZMQ.EVENT_ALL);
// Then start a montitor thread which is implemented by my own.
Server
OS: Windows 7 ( 64 bit ), running on my laptop, IDE: VS2013, lib: Clrzmq4
// C# Code
const string MonitorEndpoint = "inproc://server.rep";
var ctx = new ZContext();
var serverSocket = new ZSocket(ctx,ZSocketType.REP);
ZError error;
// Create serverSocket pair socket
if (!serverSocket.Monitor(MonitorEndpoint, ZMonitorEvents.AllEvents, out error))
{
if (error == ZError.ETERM)
return ; // Interrupted
throw new ZException(error);
}
// Create a monitor
ZMonitor _monitor = ZMonitor.Create(ctx, MonitorEndpoint);
_monitor.AllEvents += _monitor_AllEvents;
_monitor.Start();
AFAIK there is no built in heartbeat within ZeroMQ. I know there was some discussion on the topic within the ZMQ community some years ago, and that discussion may still be going on.
It is comparatively simple to incorporate your own heartbeat messaging in your application's use of ZeroMQ, especially if you use something like Google Protocol Buffers to encode different message types; the heartbeat is just another message.
Doing heartbeats in your application (rather than relying on some inbuilt mechanism) is ultimately more flexible; you can choose the heartbeat rate, you can choose what to do if the heartbeat fails, you can decide when heartbeating is important and not important, etc.
Consider heartbeats within a PUB/SUB pattern; it's a bit difficult for the ZMQ authors to decide on your behalf what connection / disconnection / connection-break events matter to you. And if they do build in a mechanism, but an application developer didn't want it, then it is a waste of bandwidth.
It's far easier for the ZMQ authors to leave that kind of application architectural issue to the application author (that's you!) to deal with.
With your specific example, an unplugged network cable simply looks (so far as any software can determine) like no traffic is flowing; it's the same as the application not sending anything. ZMQ doesn't send anything if the application hasn't sent anything.
If you look at the events that the socket monitor can report on, they're all the consequence of something flowing over the network connection, or something done to the socket by the application.
Trying to go lower than ZMQ protocol itself and access the TCP connection that specific ZeroMQ sockets use (while others do not) doesn't sound like a good idea; it would required to break encapsulation in multiple classes.
The answer #bazza gave in 2017 was entirely correct at the time.
However, newer versions of ZMQ (specifically ZMTP) include an heartbeat functionality.
Check ZMQ documentation for
socketOpt
Java functions
name
purpose
ZMQ_HEARTBEAT_IVL
get/setHeartbeatLvl()
heartbeat interval
milliseconds between ZMPT PINGs
ZMQ_HEARTBEAT_TIMEOUT
get/setHeartbeatTimeout()
local heartbeat timeout
how long the local socket waits between received packets until it considers the connection timed out
ZMQ_HEARTBEAT_TTL
get/setHeartbeatTtl()
remote heartbeat timeout
if and when remote side shall consider the connection timed out
ZMQ_HEARTBEAT_CONTEXT is still in draft state as of 2022. It is supposed to send an byte[] context with every ping.
Now, by design of ZMQ, quoting from chapter 2 of its documentation,
The network connection itself happens in the background, and ZeroMQ
will automatically reconnect if the network connection is broken
(e.g., if the peer disappears and then comes back).
Thus, answering your main question, I'd expect the monitor to give you ZMQ_EVENT_CONNECT_RETRIED / ZMQ_EVENT_CONNECTED events after the underlying connection was detected as disrupted.
I'm writing a service that needs to maintain a long running SSL connection to a remote server. I need this server to be self-healing, that is if it's disconnected for any reason then the next time it's written to it will reconnect. I've tried this:
bool isConnected = client.Connected && client.Client.Poll(0, SelectMode.SelectWrite) && stream.CanWrite;
if (!isConnected )
{
this.connected = false;
GetConnection();
}
stream.Write(bytes, 0, bytes.Length);
stream.Flush();
But I find it doesn't act as I would expect it. If I simulate a network outage by disabling my wifi, I'm still able to write to the stream with stream.Write() for approximately 20 seconds. Then next time I try to write to it, none of client.Connected, client.Client.Poll(), or stream.CanWrite() return false, but when I go to write to the stream I get a socket exception. Finally, if I try to recreate the connection, I get this exception: An existing connection was forcibly closed by the remote host.
I would appreciate any help create a long running SslStream that can withstand network failure. Thanks!
From a 10.000 feet point of view:
The reason you can still write to the stream after shutting down your wifi is because there is a network buffer that is holding the data for transmission, stream.Write/stream.Flush success means the network interface (TCP/IP stack) has accepted the data and has been buffered for transmission, not that the data has reach its target.
It takes time to the TCP/IP Stack to notice a full media disconnection, (connection lost/reset) because even if there is no physical link TCP/IP will see this as a temporary issue in the network and will keep retrying for a while (the network could be dropping packets at some point and the stack will keep retrying)
If you think about this in the reverse way, you won't like all your programs to fail if there is a network hiccup (this happen too often on internet), so TCP/IP takes its time to notify to the app layer that the connection has become invalid (after retry several times and wait a reasonable amount of time)
You can always reconnect to the server when the SslStream fails and continue sending data, although you will find is not as easy as this because there are several scenarios where you send and data is not received by server and others where server receive the data and you do not receive any ACK from server at all... So depending on your needs, self-healing alone could be not enough.
Self-Healing is simple to implement, data consistency and reliability is harder and usually requires the server to be ready to support some kind of reliable messaging mechanism to ensure all data has been sent and received.
The underlying protocol for SSL is TCP. TCP will usually only send data if the application wants it to deliver data, or if it needs to reply to data received from the other side by sending an ACK. This means, that a broken connection like a lost link will not be noticed until you are trying to send any data. But you will not notice immediatly, because:
A write to the socket will only deliver the data to the OS kernel and return success if this delivery was successful.
The kernel will then try to deliver the data to the peer and will wait for the ACK from the client.
If it does not get any ACK it will retry again to deliver the data and only after some unsuccessful retries the kernel will declare the connection broken.
Only after the connection is marked broken by the kernel the next write or read will return the error from kernel to user space, like with returning EPIPE when doing a write.
This means, if you want to know up-front if the connection is still alive you have to make sure that you get a regular data exchange on the connection. At the TCP level you might set TCP_KEEPALIVE, but this might use an interval of some hours between exchanges packets. At the SSL layer you might try to use the infamous heartbeat extension, but most peers will not understand it. The last choice is to implement some kind of heartbeat in your own application.
As for the self healing: When reconnecting you get a new TCP connection and you also need to do a full SSL handshake, because the last SSL connection was not cleanly closed and thus cannot be resumed. The server has no idea that this new connection is just a continuation of the old one so you have to implement some kind of meta-connection spanning multiple TCP connections inside your application layer on both client and server. Inside this meta-connection you need to have your own data tracking to detect, which data are really accepted from the peer and which were only send but never explicitly accepted because the connection broke. Sound like a kind of TCP on top of TCP.
I'm using TcpListener to accept & read from TcpClient.
The problem is that when reading from a TcpClient, TcpClient.BeginRead / TcpClient.EndRead doesn't throw exception when the internet is disconnected. It throws exception only if client's process is ended or connection is closed by server or client.
The system generally has no chance to know that connection is broken. The only reliable way to know this is to attempt to send something. When you do this, the packet is sent, then lost or bounced and your system knows that connection is no longer available, and reports the problem back to you by error code or exception (depending on environment). Reading is usually not enough cause reading only checks the state of input buffer, and doesn't send the packet to the remote side.
As far as I know, low level sockets doesn't notify you in such cases. You should provide your own time out implementation or ping the server periodically.
If you want to know about when the network status changes you can subscribe to the System.Net.NetworkInformation.NetworkChange.NetworkAvailabilityChanged event. This is not specific to the internet, just the local network.
EDIT
Sorry, I misunderstood. The concept of "connected" really doesn't exist the more you think about it. This post does a great job of going into more details about that. There is a Connected property on the TcpClient but MSDN says (emphasis mine):
Because the Connected property only
reflects the state of the connection
as of the most recent operation, you
should attempt to send or receive a
message to determine the current
state. After the message send fails,
this property no longer returns true.
Note that this behavior is by design.
You cannot reliably test the state of
the connection because, in the time
between the test and a send/receive,
the connection could have been lost.
Your code should assume the socket is
connected, and gracefully handle
failed transmissions.
Basically the only way to check for a client connection it to try to send data. If it goes through, you're connected. If it fails, you're not.
I don't think you'd want BeginRead and EndRead throwing exceptions as these should be use in multi threaded scenarios.
You probably need to implement some other mechanism to respond to the dropping of a connection.
in C# when a sockets connection is terminated the other node is informed of this before terminating the link thus the remaning node can update the connection status.
in Java when i terminate a communication link the other node keeps reporting the connection as valid.
do i need to implement a read cycle (makes sense) that reports the connection as lost when it recieves a -1 during read (in C# this is 0 i think)?
thank you for your insight.
EDIT: thanks to you both. as i suspected and mentioned in my post that an additional check is required to confirm the connected state of a connection.
If the remote side of the connection goes away, normally you'll get an IOException from the InputStream/InputChannel if the disconnection can be detected. If it can't detect the disconnect an IOException will eventually be thrown when the socket read times out. The length of time it waits for a timeout can be adjusted using Socket.setSoTimeout().
In java, you find out about the other end of the socket being closed only when you read/write to/from the socket, or query the input stream state (e.g. InputStream.available()). I don't think there is any asynchronous notification that the other end is closed.
How are you testing that the socket is still open?
You can poll the InputStream.available() method and if that returns -1, you know the socket is closed. Of course, you can also read data, if that fits with your usage.
See InputStream.available()