We Keep Coding

C# TCP Socket receive part of data on slow cellular network connection while connection still not broken

I am using Console application as a server on windows 10 Pro.
Android application made with Unity Engine as Client.
For both application I use TCP socket on the .Net framework.
Note: the server is always running on wire connection.
Everything work pretty well on both wire and Wi-Fi connections with no problem no meter how packet length is.
The Problem
Each evening the cellular network became too bad and the client app receive only part of the data when the data is little bigger about 50 kb
The code on the server side:
_socket.Send(Encoding.ASCII.GetBytes(s));
_socket: is an instance System.Net.Sockets.Socket.
s: is string.
The code on the client side:
public static void ReceiveData()
{
message = string.Empty;
_socket.BeginReceive(globalBuffer, 0, globalBuffer.Length, SocketFlags.None, ReceiveCallBack, null);
}
public static void ReceiveCallBack(IAsyncResult ar)
{
int internalBuffer = _socket.EndReceive(ar);
byte[] subtractedBuffer = new byte[internalBuffer];
Array.Copy(globalBuffer, subtractedBuffer, internalBuffer);
string stmp = Encoding.ASCII.GetString(subtractedBuffer);
message += stmp;
while (message.Contains("</Cargo>"))
{
string stringOne = message.Substring(0, message.IndexOf("</Cargo>") + "</Cargo>".Length);
string stringLeft = message.Substring(message.IndexOf("</Cargo>") + "</Cargo>".Length);
message = stringLeft;
Thread thread = new Thread(TreatOrder);
thread.Start(stringOne);
}
if (!(_socket.Poll(1000, SelectMode.SelectRead) && _socket.Available == 0))
{
NetWorkScript.LastConnectedTime = DateTime.Now;
_socket.BeginReceive(globalBuffer, 0, globalBuffer.Length, SocketFlags.None, ReceiveCallBack, null);
}
else
{
Debug.log("Connection lost");
}
}
What I have notice:
The next line of code never executed
else
{
Debug.log("Connection lost");
}
The server keep receiving messages from the client on the same Socket instance by another thread and that is how came to know the connection is not broken unless TCP connection could be broken on one way only.
What I understand so far
The TCP will make sure the packets will arrive at the exact order.
The next line of code in my server application end its responsibility and the Operation System will carry on the mission to send the data.
_socket.Send(Encoding.ASCII.GetBytes(s));
What I am confused about
How the Operation System will deliver the data to the client?, why it is not keep trying until send all the data if the connection still exist?
Should I created my own protocol and send small packet then retransmit if client does not confirm receiving the packet?

When I send a command to server from client , the client receives the response only if the request is sent twice

I am trying to send commands to the server , like for example requesting the server to send back the list of files in it's directory. The problem is that when I send the "list" command to the server, I have to send it twice in order for the server to send back the list of files to the client. I am sure that the server receives the command in both times as on the server side I print the result that is supposed to be sent to the client on the console and it appears both times.
I am using C# and TCPListeners to listen for incoming responses or commands, and TCPClient to send responses or commands between the server and the client.
The client code
private TcpListener tcpListener = new TcpListener(9090);
private void button3_Click(object sender, EventArgs e)
{
Byte[] bytesToSend = ASCIIEncoding.ASCII.GetBytes("list");
try
{
TcpClient clientSocket = new TcpClient(serverIPFinal, 8080);
if (clientSocket.Connected)
{
NetworkStream networkStream = clientSocket.GetStream();
networkStream.Write(bytesToSend, 0, bytesToSend.Length);
// networkStream.Close();
// clientSocket.Close();
thdListener = new Thread(new ThreadStart(listenerThreadList));
thdListener.Start();
}
}
catch
{
isConnectedLbl.Text = "Server not running";
}
}
//Listener Thread to receive list of files.
public void listenerThreadList()
{
tcpListener.Start();
while (true)
{
handlerSocket = tcpListener.AcceptSocket();
if (handlerSocket.Connected)
{
Control.CheckForIllegalCrossThreadCalls = false;
lock (this)
{
if (handlerSocket != null)
{
nSockets.Add(handlerSocket);
}
}
ThreadStart thdstHandler = new
ThreadStart(handlerThreadList);
Thread thdHandler = new Thread(thdstHandler);
thdHandler.Start();
}
}
}
//Handler Thread to receive list of files.
public void handlerThreadList()
{
Socket handlerSocketList = (Socket)nSockets[nSockets.Count - 1];
NetworkStream networkStreams = new NetworkStream(handlerSocketList);
int requestRead = 0;
string dataReceived;
byte[] buffer = new byte[1024];
//int iRx = soc.Receive(buffer);
requestRead = networkStreams.Read(buffer, 0, 1024);
char[] chars = new char[requestRead];
System.Text.Decoder d = System.Text.Encoding.UTF8.GetDecoder();
int charLen = d.GetChars(buffer, 0, requestRead, chars, 0);
dataReceived = new System.String(chars);
Console.WriteLine(dataReceived);
MessageBox.Show(dataReceived);
//tcpListener.Stop();
thdListener.Abort();
}
The Server code:
TcpListener tcpListener = new TcpListener(8080);
public void listenerThreadCommands()
{
tcpListener.Start();
while (true)
{
handlerSocket = tcpListener.AcceptSocket();
if (handlerSocket.Connected)
{
Control.CheckForIllegalCrossThreadCalls = false;
connections.Items.Add(
handlerSocket.RemoteEndPoint.ToString() + " connected.");
// clientIP = handlerSocket.RemoteEndPoint.ToString();
lock (this)
{
nSockets.Add(handlerSocket);
}
ThreadStart thdstHandler = new
ThreadStart(handlerThreadCommands);
Thread thdHandler = new Thread(thdstHandler);
thdHandler.Start();
//tcpListener.Stop();
//handlerSocket.Close();
}
}
}
//Handler Thread to receive commands
public void handlerThreadCommands()
{
Socket handlerSocketCommands = (Socket)nSockets[nSockets.Count - 1];
NetworkStream networkStream = new NetworkStream(handlerSocketCommands);
int requestRead = 0;
string dataReceived;
byte[] buffer = new byte[1024];
requestRead = networkStream.Read(buffer, 0, 1024);
char[] chars = new char[requestRead];
System.Text.Decoder d = System.Text.Encoding.UTF8.GetDecoder();
int charLen = d.GetChars(buffer, 0, requestRead, chars, 0);
dataReceived = new System.String(chars);
//connections.Items.Add(dataReceived);
if (dataReceived.Equals("list"))
{
localDate = DateTime.Now;
Files = Directory.GetFiles(System.IO.Directory.GetCurrentDirectory())
.Select(Path.GetFileName)
.ToArray();
String FilesString = "";
for (int i = 0; i < Files.Length; i++)
{
FilesString += Files[i] + "\n";
}
String clientIP = handlerSocketCommands.RemoteEndPoint.ToString();
int index = clientIP.IndexOf(":");
clientIP = clientIP.Substring(0, index);
WriteLogFile(logFilePath, clientIP, localDate.ToString(), " ", "list");
Console.WriteLine(clientIP);
Console.WriteLine(FilesString);
Byte[] bytesToSend = ASCIIEncoding.ASCII.GetBytes(FilesString);
try
{
WriteLogFile(logFilePath, clientIP, localDate.ToString(), " ", "list-response");
TcpClient clientSocket = new TcpClient(clientIP, 9090);
if (clientSocket.Connected)
{
NetworkStream networkStreamS = clientSocket.GetStream();
networkStreamS.Write(bytesToSend, 0, bytesToSend.Length);
networkStreamS.Close();
clientSocket.Close();
networkStream.Close();
//tcpListener.Stop();
// handlerSocketAuthenticate.Close();
}
}
catch
{
Console.WriteLine("Cant send");
}
}
else if (dataReceived.Equals("downloadfile"))
{
// handlerSocketAuthenticate.Close();
// tcpListener.Stop();
networkStream.Close();
thdListenerDownload = new Thread(new ThreadStart(listenerThreadDownloading));
thdListenerDownload.Start();
}
else
{
String clientIP1 = handlerSocketCommands.RemoteEndPoint.ToString();
int index = clientIP1.IndexOf(":");
clientIP1 = clientIP1.Substring(0, index);
// handlerSocketAuthenticate.Close();
CommandExecutor(dataReceived, clientIP1);
}
}

There are so many different things wrong with the code you posted, it's hard to know where to start, and it's impossible to have confidence that in the context of a Stack Overflow, one could sufficiently address all of the deficiencies. That said, in the interest of helping, it seems worth a try:
Sockets are bi-directional. There is no need for the client to use TcpListener at all. (By convention, the "server" is the endpoint that "listens" for new connections, and the "client" is the endpoint that initiates new connections, by connecting to a listening server.)You should just make a single connection from client to server, and then use that socket both for sending to and receiving from the server.
You are setting the CheckForIllegalCrossThreadCalls property to false. This is evil. The exceptions that occur are there to help you. Setting that property to false disables the exceptions, but does nothing to prevent the problems that the exceptions are designed to warn you about.You should use some mechanism to make sure that when you access UI objects, you do so only in the thread that owns those objects. The most primitive approach to this is to use Control.Invoke(). In modern C#, you are better off using async/await. With TcpClient, this is easy: you already are using GetStream() to get the NetworkStream object that represents the socket, so just use the asynchronous methods on that object, such as ReadAsync(), or if you wrap the stream in a StreamWriter and StreamReader, use the asynchronous methods on that object, such as ReadLineAsync().
You are checking the Connected property of the TcpClient object. This is pointless. When the Connect() method returns, you are connected. If you weren't, an exception would have been thrown.
You are not sufficiently synchronizing access to your nSockets object. In particular, you use its indexer in the handlerThreadList() method. This is safe when using the object concurrently only if you have guaranteed that no other thread is modifying the object, which is not the case in your code.
You are writing to the stream using ASCII encoding, but reading using UTF8 encoding. In practice, this is not really a problem, because ASCII includes only the code points 0-127, and those map exactly to the same character code points in UTF8. But it's really bad form. Pick one encoding, stick with it.
You are accepting using AcceptSocket(), but then just wrapping that in a NetworkStream anyway. Why not just use AcceptTcpClient() and call GetStream() on that? Both Socket and TcpClient are fine APIs, but it's a bit weird to mix and match in the same program, and will likely lead to some confusion later on, trying to keep straight which you're using where and why.
Your code assumes that the handlerThreadCommands() method will always be called in exactly the same order in which connections are accepted. That is, you retrieve the current socket with nSockets[nSockets.Count - 1]. But, due to the way Windows thread scheduling works, it is entirely possible that two or more connections could be accepted before any one of the threads meant to handle the connection is started, with the result that only the most recent connection is handled, and it is handled by those multiple threads.
You are assuming that command strings will be received as complete units. But this isn't how TCP works. TCP guarantees only that if you receive a byte, it will be in order relative to all the bytes sent before it. But you can receive any number of bytes. In particular, you can receive just a single byte, or you can receive multiple commands concatenated with each other, or you can receive half a command string, then the other half later, or the second half of one command and the first half of the next, etc. In practice, these problems don't show up in early testing because the server isn't operating under load, but later on they very well may be. And the code needs to be designed from the outset to work properly under these conditions; trying to patch bad code later is much more difficult.
I can't say that's the above are the only things wrong with the code, but they are most glaring, and in any case I think the above is sufficient food for thought for you at the moment.
Bottom line: you really should spend more time looking at good networking examples, and really getting to understand how they work and why they are written the way they do. You'll need to develop a good mental model for yourself of how the TCP protocol works, and make sure you are being very careful to follow the rules.
One resource I recommend highly is The Winsock Programmer's FAQ. It was written long ago, for a pre-.NET audience, but most of the information contained within is still very much relevant when using the higher-level networking APIs.
Alternatively, don't try to write low-level networking code yourself. There are a number of higher-level APIs that use various serialization techniques to encode whole objects and handle all of the lower-level network transport mechanics for you, allowing you to concentrate on the value-added features in your own program, instead of trying to reinvent the wheel.

Reset TCP connection if server closes/crashes mid connection

I have a tcp connection like follows:
public void ConnectToServer()
{
string mac = GetUID();
while(true)
{
try
{
tcpClient = new TcpClient("xx.x.xx.xxx", xxxx);
networkstream = new SslStream(tcpClient.GetStream());
networkstream.AuthenticateAsClient("xx.x.xx.xxx");
networkstream.Write(Encoding.UTF8.GetBytes("0002:" + mac + "\r\n"));
networkstream.Flush();
string serverMessage = ReadMessage(networkstream);
Console.WriteLine("MESSAGE FROM SERVER: " + serverMessage);
}
catch (Exception e)
{
tcpClient.GetStream().Close();
tcpClient.Close();
}
}
}
This works fine and can send a receive data to/from the server.
What I need help with, if the server isn't running when the client starts, it'll wait and then connect once the server is up. But, if both the client and server are running and everything is working, if I close the server, the client will not reconnect(because I don't have anything to handle the event yet).
I have seen some answers on here that suggest polling and such. Is that the only way? The ReadMessage method that I call get into an infinite loop as well. I can post that code if need be.
I would really like to detect when the server closes/crashes and close the stream and the tcpclient and reconnect ASAP.
Here is my readmessage:
static string ReadMessage(SslStream sslStream)
{
if (sslStream.CanRead)
{
byte[] buffer = new byte[2048];
StringBuilder messageData = new StringBuilder();
int bytes = -1;
string message_type = null;
string actual_message = null;
do
{
try
{
Console.WriteLine("LENGTH: " + buffer.Length);
bytes = sslStream.Read(buffer, 0, buffer.Length);
Decoder decoder = Encoding.UTF8.GetDecoder();
char[] chars = new char[decoder.GetCharCount(buffer, 0, bytes)];
decoder.GetChars(buffer, 0, bytes, chars, 0);
messageData.Append(chars);
message_type = messageData.ToString().Substring(0, 5);
actual_message = messageData.ToString().Substring(5);
if (message_type.Equals("0001:"))
{
m_Window pop = new m_Window();
pop.callHttpPost(null, new EventArgs());
}
if (messageData.ToString().IndexOf("\r\n") != -1)
{
break;
}
}
catch (Exception e)
{
Console.WriteLine("ERROR: " + e.Message);
}
} while (bytes != 0);
return messageData.ToString();
}
return("CONNECTION HAS BEEN LOST");
}

With TCP you have 2 kinds of a server disconnect:
the server is closed
the server crashes
When the server is closed, you are going to receive 0 bytes on your client socket, this is the way you know that the peer has closed its end of the socket, which is called a half close.
But thing get more ugly if the server crashes.
When that happens again you have several possibilities.
If you don't send anything from the client to the server, the you have not way to find out that the server has indeed crashed.
The only way to find out that the server crashed is by letting the client send something or by activating keep alive. If you send something to a server socket that does not exist, you will have to wait a rather long period, because TCP is going to try several times, with retransmits untill there is a server response. When TCP has retried several times, then it will finally bail out and if you have a blocking socket you will see that the send failed, which means you should close your socket.
Actually there is a third possible server disconnect, that is a reset, but this is exceptionally used. I assume here that if there is a gracefull server shutdown, a normal close on the socket on the server end is executed. Which will end up in a FIN being sent instead of a RST, which is the exceptional case.
Now back to your situation, if the server crashes, it is inherently in the design of TCP, because of all those retransmission timeouts and increasing delays, that you will have to wait some time to actually detect that there is a problem. If the server is gracefully closed and startup again, this is not the case, this you detect immediately by receiving 0 bytes.

How to create a TcpClient in C# that can recover from a lost network connection?

I am trying to write a TCP client thread that will connect to a server and continue to process data as it receives it. Sometimes the client will lose connection to the server. When the connection is re-established, the thread should resume/recover and automatically start processing data again. I can't seem to get this to work. The basic algorithm is below. My problem is that I just don't quite understand how the TcpClient behaves when the network connection is lost. How can I tell that the connection has been lost? Do I then need to close the connection? How do I reestablish the connection and continue on?
TcpClient _tcpClient;
IPEndPoint _ipEndPoint;
bool _cancelled = false;
bool _error = true;
while (!_cancelled)
{
try
{
if(_error)
{
_ipEndPoint = new IPEndPoint(_myAddress, _myPort);
_tcpClient.Connect(_ipEndPoint);
_networkStream = _tcpClient.GetStream();
_error = false;
}
else
{
_data = new byte[10025];
if(_networkStream.CanRead)
{
_bytesRead = _networkStream.Read(_data, 0, (int)_tcpClient.ReceiveBufferSize);
if(_bytesRead > 0)
{
...process the data...
}
else
{
_error = true;
}
}
else
{
_error = true;
}
}
}
catch(Exception ex)
{
...log error...
_error = true;
}
}

there is a Connected property on the TcpClient, but it is only updated when a Write or Read is executed on the NetworkStream.
I think you can just execute a Read(null,0,0) or Write(null,0,0) if you want to force a connectivity check. But in the example you have you can check the Connected or CanRead properties after your _networkStream.Read is completed.
As for reestablishing the link what you have will work. I would suggest Disposing of the old network stream before getting a new one. Something like this:
if(_networkStream != null)
{
_networkStream.Dispose();
}
_networkStream = _tcpClient.GetStream();
What I do is start a reconnect timer that will attempt to reconnect on a configured interval. Depending on the TcpHost your connecting to you may even want to start trying at a small interval 500-1000ms and increment is after a given number of failed retries so your not wasting a lot of time trying to connect to a host that is gone. Then after a max number of tries just give up unless the user explicitly requests to try again. But that also depends on what else your app is doing, if the connection is its sole purpose or if its just one piece.
The wrapper class I use to interact with TcpClient is around 700 lines of code and it handles reconnects, sending data as well as reading it. I work in a closed shop so i cant post it but if you have any other specific question I'd be happy to help.
Good luck.

TcpClient communication with server to keep alive connection in c#?

I've this TcpClient code which works fine. It connects to perl server on linux system and receives anything that server sents to it. Works nicely.
public static void Main() {
foreach (ProtocolConnection tcpConnection in TcpConnectionsList) {
ProtocolConnection connection = tcpConnection;
ThreadPool.QueueUserWorkItem(_ => {
ThreadTcpClient(connection);
ManualResetEventTcp.Set();
});
}
... Some code...
}
public static void TcpConnect(ProtocolConnection varConnection) {
int retryCountSeconds = varConnection.RetryEverySeconds*Program.MilisecondsMultiplier;
int count = 0;
while (true) {
try {
using (var client = new TcpClient(varConnection.IpAddress.ToString(), varConnection.Port) { NoDelay = true })
using (var stream = client.GetStream()) {
var data = new Byte[256];
while (!Program.PrepareExit) {
Int32 bytes = stream.Read(data, 0, data.Length);
string varReadData = Encoding.ASCII.GetString(data, 0, bytes).Trim();
if (varReadData != "" && varReadData != "PONG") {
VerificationQueue.EnqueueData(varReadData);
Logging.AddToLog("[TCP][" + varConnection.Name + "][DATA ARRIVED]" + varReadData);
} else {
Logging.AddToLog("[TCP]" + varReadData);
}
}
}
} catch (Exception e) {
if (e.ToString().Contains("No connection could be made because the target machine actively refused it")) {
Logging.AddToLog("[TCP][ERROR] Can't connect to server (" + varConnection.Name + ") " + varConnection.IpAddress + ":" + varConnection.Port );
} else {
Logging.AddToLog(e.ToString());
}
}
DateTime startTimeFunction = DateTime.Now;
do {
Thread.Sleep(1000);
} while (((DateTime.Now - startTimeFunction).TotalSeconds < retryCountSeconds));
}
}
However in certain conditions I'm having some problems with it:
My work connection often drops connection after some idle time so I've implemented in server so when it receives PING it responds with PONG. I can send PING with UDP to server and it will respond with PONG on tcp but i would prefer built-in way into tcp client so it does send PING every 60 seconds or so. Even if UDP solution would be acceptable I have no timeout on string varReadData = Encoding.ASCII.GetString(data, 0, bytes).Trim(); so when PONG doesn't arrive my client doesn't even notice it anyway. It just keeps waiting ... which brings me to..
My other problem is that at some point string varReadData = Encoding.ASCII.GetString(data, 0, bytes).Trim(); this is waiting for data all the time. When server crashes or disconnects my client i don't even notice that. I would like server to have some kind of timeout or check if connection is active. If it's not active it should try to reconnect.
What would be simplest way to fix this TcpClient ? How do i implement both way communication making sure that if server drops my connections or my net gets disconnected client will notice it and reestablish connection ?

It's not Encoding.ASCII.GetString(data, 0, bytes).Trim(); that blocks forever, it's the stream.Read()
If you're reading, you can't easily distinguish between the server(or any NAT gateway inbetween) dropping your connection , and the case where the server simply doesn't have anything to send you. Atleast in the case where the TCP FIN/RST packets doesn't reach your client in case of failure, or a NAT gateway silently dropping your connection.
What you can do;
Set a Send/ReceiveTimeout , and ping the server if a timeout occurs, or implement your own heartbeat messages over your TCP connection. Reestablish or take other actions if you don't receive a heartbeat within a reasonable time.
Set the TCP keepalive option, and rely on that to tell you if the server is gone. See code here.
The last point will tell you if the tcp connection fails, it won't tell you if the server has somewhat failed - e.g. if you CTRL+Z your perl server, it'll just sit there not doing anything as the tcp window closes , so you might need to implement your own heatbeat messges to cover such a case too if you need to.

You should get rid of the UDP heartbeat attempt and put in a real TCP heartbeat. "Pinging" the server using UDP is almost meaningless.
Your protocol is also missing message framing.
Read both of those linked articles carefully (especially message framing). The protocol you're currently using does need serious revision.