We Keep Coding

What's a reliable way to verify that a socket is open?

I'm developing a C# application, working with TCP sockets.
While debugging, I arrive in this piece of source code:
if ((_socket!= null) && (_socket.Connected))
{
Debug.WriteLine($"..."); <= my breakpoint is here.
return true;
}
In my watch-window, the value of _socket.RemoteEndPoint is:
_socket.RemoteEndPoint {10.1.0.160:50001} System.Net.EndPoint {...}
Still, in commandline, when I run netstat -aon | findstr /I "10.1.0.160", I just see this:
TCP 10.1.13.200:62720 10.1.0.160:3389 ESTABLISHED 78792
TCP 10.1.13.200:63264 10.1.0.160:445 ESTABLISHED 4
=> the remote endpoint "10.1.0.160:50001" is not visible in netstat result.
As netstat seems not reliable for testing TCP sockets, what tool can I use instead?
(For your information: even after having run further, there still is no "netstat" entry.)

Documentation of Socket.Connected, says:
The value of the Connected property reflects the state of the
connection as of the most recent operation. If you need to determine
the current state of the connection, make a nonblocking, zero-byte
Send call. If the call returns successfully or throws a WAEWOULDBLOCK
error code (10035), then the socket is still connected; otherwise, the
socket is no longer connected.
So if it returns true - socket has been "connected" some time in the past, but not necessary is still alive right now.
That's because it's not possible to detect if your TCP connection is still alive with certainly without contacting the other side in one way or another. The TCP connection is kind of "virtual", two sides just exchange packets but there is no hard link between them. When one side decides to finish communication - it sends a packet and waits for response from the other side. If all goes well two sides will both agree that connection is closed.
However, if side A does NOT send this close packet, for example because it crashed, or internet died and so on - the other side B has no way to figure out that connection is no longer active UNTIL it tries to send some data to A. Then this send will fail and now B knows connection is dead.
So if you really need to know if other side is still alive - then you have to send some data there. You can use keepalive which is available on TCP sockets (which basically does the same - sends some data from time to time). Or if you always write on this connection first (say other side is server and you do requests to it from time to time, but do not expect any data between those requests) - then just don't check if the other side is alive - you will know that when you will attempt to write next time.

Self-healing SslStream

I'm writing a service that needs to maintain a long running SSL connection to a remote server. I need this server to be self-healing, that is if it's disconnected for any reason then the next time it's written to it will reconnect. I've tried this:
bool isConnected = client.Connected && client.Client.Poll(0, SelectMode.SelectWrite) && stream.CanWrite;
if (!isConnected )
{
this.connected = false;
GetConnection();
}
stream.Write(bytes, 0, bytes.Length);
stream.Flush();
But I find it doesn't act as I would expect it. If I simulate a network outage by disabling my wifi, I'm still able to write to the stream with stream.Write() for approximately 20 seconds. Then next time I try to write to it, none of client.Connected, client.Client.Poll(), or stream.CanWrite() return false, but when I go to write to the stream I get a socket exception. Finally, if I try to recreate the connection, I get this exception: An existing connection was forcibly closed by the remote host.
I would appreciate any help create a long running SslStream that can withstand network failure. Thanks!

From a 10.000 feet point of view:
The reason you can still write to the stream after shutting down your wifi is because there is a network buffer that is holding the data for transmission, stream.Write/stream.Flush success means the network interface (TCP/IP stack) has accepted the data and has been buffered for transmission, not that the data has reach its target.
It takes time to the TCP/IP Stack to notice a full media disconnection, (connection lost/reset) because even if there is no physical link TCP/IP will see this as a temporary issue in the network and will keep retrying for a while (the network could be dropping packets at some point and the stack will keep retrying)
If you think about this in the reverse way, you won't like all your programs to fail if there is a network hiccup (this happen too often on internet), so TCP/IP takes its time to notify to the app layer that the connection has become invalid (after retry several times and wait a reasonable amount of time)
You can always reconnect to the server when the SslStream fails and continue sending data, although you will find is not as easy as this because there are several scenarios where you send and data is not received by server and others where server receive the data and you do not receive any ACK from server at all... So depending on your needs, self-healing alone could be not enough.
Self-Healing is simple to implement, data consistency and reliability is harder and usually requires the server to be ready to support some kind of reliable messaging mechanism to ensure all data has been sent and received.

The underlying protocol for SSL is TCP. TCP will usually only send data if the application wants it to deliver data, or if it needs to reply to data received from the other side by sending an ACK. This means, that a broken connection like a lost link will not be noticed until you are trying to send any data. But you will not notice immediatly, because:
A write to the socket will only deliver the data to the OS kernel and return success if this delivery was successful.
The kernel will then try to deliver the data to the peer and will wait for the ACK from the client.
If it does not get any ACK it will retry again to deliver the data and only after some unsuccessful retries the kernel will declare the connection broken.
Only after the connection is marked broken by the kernel the next write or read will return the error from kernel to user space, like with returning EPIPE when doing a write.
This means, if you want to know up-front if the connection is still alive you have to make sure that you get a regular data exchange on the connection. At the TCP level you might set TCP_KEEPALIVE, but this might use an interval of some hours between exchanges packets. At the SSL layer you might try to use the infamous heartbeat extension, but most peers will not understand it. The last choice is to implement some kind of heartbeat in your own application.
As for the self healing: When reconnecting you get a new TCP connection and you also need to do a full SSL handshake, because the last SSL connection was not cleanly closed and thus cannot be resumed. The server has no idea that this new connection is just a continuation of the old one so you have to implement some kind of meta-connection spanning multiple TCP connections inside your application layer on both client and server. Inside this meta-connection you need to have your own data tracking to detect, which data are really accepted from the peer and which were only send but never explicitly accepted because the connection broke. Sound like a kind of TCP on top of TCP.

TcpClient.BeginRead/TcpClient.EndRead doesn't throw exception when internet disconnected

I'm using TcpListener to accept & read from TcpClient.
The problem is that when reading from a TcpClient, TcpClient.BeginRead / TcpClient.EndRead doesn't throw exception when the internet is disconnected. It throws exception only if client's process is ended or connection is closed by server or client.

The system generally has no chance to know that connection is broken. The only reliable way to know this is to attempt to send something. When you do this, the packet is sent, then lost or bounced and your system knows that connection is no longer available, and reports the problem back to you by error code or exception (depending on environment). Reading is usually not enough cause reading only checks the state of input buffer, and doesn't send the packet to the remote side.

As far as I know, low level sockets doesn't notify you in such cases. You should provide your own time out implementation or ping the server periodically.

If you want to know about when the network status changes you can subscribe to the System.Net.NetworkInformation.NetworkChange.NetworkAvailabilityChanged event. This is not specific to the internet, just the local network.
EDIT
Sorry, I misunderstood. The concept of "connected" really doesn't exist the more you think about it. This post does a great job of going into more details about that. There is a Connected property on the TcpClient but MSDN says (emphasis mine):
Because the Connected property only
reflects the state of the connection
as of the most recent operation, you
should attempt to send or receive a
message to determine the current
state. After the message send fails,
this property no longer returns true.
Note that this behavior is by design.
You cannot reliably test the state of
the connection because, in the time
between the test and a send/receive,
the connection could have been lost.
Your code should assume the socket is
connected, and gracefully handle
failed transmissions.
Basically the only way to check for a client connection it to try to send data. If it goes through, you're connected. If it fails, you're not.

I don't think you'd want BeginRead and EndRead throwing exceptions as these should be use in multi threaded scenarios.
You probably need to implement some other mechanism to respond to the dropping of a connection.

Can Socket.Receive() return 0 if the connection is still open?

I am writing a C# .NET server application that sends and receives data over Sockets, and I am having some issues if the client application crashes without closing the socket the right way.
I have set the 'receive timeout' to a given value, and I expected the Socket.Receive() to throw an exception after that amount of time. But instead the method just returns 0.
So my question is: Is it possible that the socket is still open if the Socket.Receive() returns 0? Or can I safely assume that it is down?
(Might be a bit tricky to understand. If so, please let me know in the comments)

Nope. It's down if you receive 0.
From MSDN:
If the remote host shuts down the Socket connection with the Shutdown method, and all available data has been received, the Receive method will complete immediately and return zero bytes.
http://msdn.microsoft.com/en-us/library/8s4y8aff.aspx

Your question is a little confused. The Socket is open until you close it. The connection is open until either end closes it. The read side of a connection can be closed by the sender shutting it down for output. So when reading a connection you will receive zero if the peer has either closed his socket or shut it down for output. At that point, your socket is still open, although you should now close it.
It will also return zero if the local application has shutdown the socket for input.

Proper implementation of C# TCP reconnecting client

I have to write a TCP Client that will have ability to reconnect to server. The server can be unavailable due to poor network connection quality or some maintenance issues. I'm searching for quality solutions in this area.
My current solutions is following:
keep connection state in ConnectionState enum {Offline, Online, Connecting}
create client with TcpClient class.
create two timers called ConnectionCheckTimer, and ReconnectTimer
connect to server
start reader thread and connection check timer
reading is performed with tcpClient.GetStream() and then reading from this stream
when Exception is caught in readerLoop client state is changed to offline and ReconnectTimer is launched
ConnectionCheckTimer periodically checks lastMessageTimestamp and compares it with current time if the interval is greater then maxValue it launches ReconnectTimer
Currently i'm not satisfied with this solution because it still generates exceptions for instance ObjectDisposedException on TcpClient.NetworkStream. I'm looking for some clean and reusable Tcp reconnecting client implementation that is able to cope with all sockets problems that can occur during connecting, disconnecting, reading data.

If you have connection issues, you will always have exceptions. I think you have a sound outline, you just need to handle the exceptions. You could start with your own Socket class implemenation and write the TCPIP Server. Starter code is at MS:
http://msdn.microsoft.com/en-us/library/fx6588te(VS.71).aspx
The C# code is half way down the VB page.

The class you should use is "SocketAsyncEventArgs".
I've used it in this project:
http://ts3querylib.codeplex.com/
Have a look at the AsyncTcpDispatcher class.