I am writing an app that will require to make hundreds of socket connections over tcp to read/write data.
I have come across this code snippet here and I'm wondering how I can make this more robust.
This is currently how I am calling the code:
foreach (var ip in listofIps)
{
IPEndPoint remoteEP = new IPEndPoint(IPAddress.Parse(ip), 4001);
Socket client = new Socket(AddressFamily.InterNetwork,
SocketType.Stream, ProtocolType.Tcp);
client.Connect(remoteEP);
await ReadAsync(client);
}
Is there anything wrong with the above, and how can it be optimized such that it runs concurrently?
In the code snippet, the buffer size is set to 1000. Just as a simple illustration, if I were to attempt to print out only the bytes received, and not the remaining 0x00s, I have to do something like this:
while (true)
{
await s.ReceiveAsync(awaitable);
int bytesRead = args.BytesTransferred;
if (bytesRead <= 0) break;
var hex = new StringBuilder(bytesRead * 2);
var msg = new byte[bytesRead];
for (int i = 0; i < bytesRead; i++)
msg[i] = args.Buffer[i];
foreach (byte b in msg)
hex.AppendFormat("{0:x2} ", b);
AppendLog(string.Format("RX: {0}", hex));
}
Is there a more efficient way of doing this? Previously, I would iterate the whole buffer and print out the data, but that will give me a whole bunch of trailing 0x00s as my protocol is anywhere between 60 to 70 bytes long.
I am writing an app that will require to make hundreds of socket connections over tcp to read/write data.
You don't need "high-performance sockets" for that. The code is far simpler with regular-performance sockets.
For starters, don't use the custom awaitables from the link you posted. They are perfectly fine for some people (and completely "robust"), but you don't need them and your code will be simpler without them.
Is there anything wrong with the above, and can it be further optimized?
Yes. You shouldn't mix blocking (Connect) and asynchronous (ReadAsync) code. I would recommend something like this:
foreach (var ip in listofIps)
{
IPEndPoint remoteEP = new IPEndPoint(IPAddress.Parse(ip), 4001);
Socket client = new Socket(AddressFamily.InterNetwork,
SocketType.Stream, ProtocolType.Tcp);
await client.ConnectTaskAsync(remoteEP);
...
}
Where ConnectTaskAsync is a standard TAP-over-APM wrapper:
public static Task ConnectTaskAsync(this Socket socket, EndPoint endpoint)
{
return TaskFactory.FromAsync(socket.BeginConnect, socket.EndConnect, endpoint, null);
}
As Marc Gravell pointed out, this code (and your original code) is connecting the sockets one at a time. You could use Task.WhenAll to connect them all simultaneously.
2) Is there a more efficient way of doing this?
First, you should define a TAP-over-APM ReceiveTaskAsync wrapper similar to the above. When dealing with binary data, I also like to have an extension method on byte arrays for dumping:
public string DumpHex(this ArraySegment<byte> data)
{
return string.Join(" ", data.Select(b => b.ToString("X2")));
}
Then you can have code like this:
while (true)
{
int bytesRead = await socket.ReceiveTaskAsync(buffer);
if (bytesRead == 0) break;
var data = new ArraySegment<byte>(buffer, 0, bytesRead);
AppendLog("RX: " + data.HexDump());
...
}
If you do a lot of binary manipulation, you may find my ArraySegments library helpful.
3) Where and how should I include the logic to check if my whole data has arrived within a single read
Oh, it's more complex than that. :) Sockets are a stream abstraction, not a message abstraction. So if you want to define "messages" in your protocol, you need to include a length prefix or delimiter byte so you can detect the message boundaries. Then you need to write code that will parse out your messages, keeping in mind that blocks of data read from the socket may contain only a partial message (so you have to buffer it), a complete message, multiple complete messages, and may also end with a partial message (again, buffering). And you have to also consider your existing buffer when receiving the new block.
I have a TCP/IP .NET Sockets FAQ on my blog that addresses this specifically and has some example code using my personal default preference for message framing (4-byte little-endian length prefixing).
4) How should I include a writeasync method such that I can send data through the socket in the middle of reads.
That one's surprisingly tricky:
public static Task<int> SendTaskAsync(this Socket socket, byte[] buffer, int offset, int size, SocketFlags flags)
{
return Task<int>.Factory.FromAsync(socket.BeginSend, socket.EndSend, buffer, offset, size, flags, null);
}
public static Task WriteAsync(this Socket socket, byte[] buffer)
{
int bytesSent = 0;
while (bytesSent != buffer.Length)
{
bytesSent += await socket.SendTaskAsync(data, bytesSent, buffer.Length - bytesSent, SocketFlags.None);
}
}
Related
I am trying to send commands to the server , like for example requesting the server to send back the list of files in it's directory. The problem is that when I send the "list" command to the server, I have to send it twice in order for the server to send back the list of files to the client. I am sure that the server receives the command in both times as on the server side I print the result that is supposed to be sent to the client on the console and it appears both times.
I am using C# and TCPListeners to listen for incoming responses or commands, and TCPClient to send responses or commands between the server and the client.
The client code
private TcpListener tcpListener = new TcpListener(9090);
private void button3_Click(object sender, EventArgs e)
{
Byte[] bytesToSend = ASCIIEncoding.ASCII.GetBytes("list");
try
{
TcpClient clientSocket = new TcpClient(serverIPFinal, 8080);
if (clientSocket.Connected)
{
NetworkStream networkStream = clientSocket.GetStream();
networkStream.Write(bytesToSend, 0, bytesToSend.Length);
// networkStream.Close();
// clientSocket.Close();
thdListener = new Thread(new ThreadStart(listenerThreadList));
thdListener.Start();
}
}
catch
{
isConnectedLbl.Text = "Server not running";
}
}
//Listener Thread to receive list of files.
public void listenerThreadList()
{
tcpListener.Start();
while (true)
{
handlerSocket = tcpListener.AcceptSocket();
if (handlerSocket.Connected)
{
Control.CheckForIllegalCrossThreadCalls = false;
lock (this)
{
if (handlerSocket != null)
{
nSockets.Add(handlerSocket);
}
}
ThreadStart thdstHandler = new
ThreadStart(handlerThreadList);
Thread thdHandler = new Thread(thdstHandler);
thdHandler.Start();
}
}
}
//Handler Thread to receive list of files.
public void handlerThreadList()
{
Socket handlerSocketList = (Socket)nSockets[nSockets.Count - 1];
NetworkStream networkStreams = new NetworkStream(handlerSocketList);
int requestRead = 0;
string dataReceived;
byte[] buffer = new byte[1024];
//int iRx = soc.Receive(buffer);
requestRead = networkStreams.Read(buffer, 0, 1024);
char[] chars = new char[requestRead];
System.Text.Decoder d = System.Text.Encoding.UTF8.GetDecoder();
int charLen = d.GetChars(buffer, 0, requestRead, chars, 0);
dataReceived = new System.String(chars);
Console.WriteLine(dataReceived);
MessageBox.Show(dataReceived);
//tcpListener.Stop();
thdListener.Abort();
}
The Server code:
TcpListener tcpListener = new TcpListener(8080);
public void listenerThreadCommands()
{
tcpListener.Start();
while (true)
{
handlerSocket = tcpListener.AcceptSocket();
if (handlerSocket.Connected)
{
Control.CheckForIllegalCrossThreadCalls = false;
connections.Items.Add(
handlerSocket.RemoteEndPoint.ToString() + " connected.");
// clientIP = handlerSocket.RemoteEndPoint.ToString();
lock (this)
{
nSockets.Add(handlerSocket);
}
ThreadStart thdstHandler = new
ThreadStart(handlerThreadCommands);
Thread thdHandler = new Thread(thdstHandler);
thdHandler.Start();
//tcpListener.Stop();
//handlerSocket.Close();
}
}
}
//Handler Thread to receive commands
public void handlerThreadCommands()
{
Socket handlerSocketCommands = (Socket)nSockets[nSockets.Count - 1];
NetworkStream networkStream = new NetworkStream(handlerSocketCommands);
int requestRead = 0;
string dataReceived;
byte[] buffer = new byte[1024];
requestRead = networkStream.Read(buffer, 0, 1024);
char[] chars = new char[requestRead];
System.Text.Decoder d = System.Text.Encoding.UTF8.GetDecoder();
int charLen = d.GetChars(buffer, 0, requestRead, chars, 0);
dataReceived = new System.String(chars);
//connections.Items.Add(dataReceived);
if (dataReceived.Equals("list"))
{
localDate = DateTime.Now;
Files = Directory.GetFiles(System.IO.Directory.GetCurrentDirectory())
.Select(Path.GetFileName)
.ToArray();
String FilesString = "";
for (int i = 0; i < Files.Length; i++)
{
FilesString += Files[i] + "\n";
}
String clientIP = handlerSocketCommands.RemoteEndPoint.ToString();
int index = clientIP.IndexOf(":");
clientIP = clientIP.Substring(0, index);
WriteLogFile(logFilePath, clientIP, localDate.ToString(), " ", "list");
Console.WriteLine(clientIP);
Console.WriteLine(FilesString);
Byte[] bytesToSend = ASCIIEncoding.ASCII.GetBytes(FilesString);
try
{
WriteLogFile(logFilePath, clientIP, localDate.ToString(), " ", "list-response");
TcpClient clientSocket = new TcpClient(clientIP, 9090);
if (clientSocket.Connected)
{
NetworkStream networkStreamS = clientSocket.GetStream();
networkStreamS.Write(bytesToSend, 0, bytesToSend.Length);
networkStreamS.Close();
clientSocket.Close();
networkStream.Close();
//tcpListener.Stop();
// handlerSocketAuthenticate.Close();
}
}
catch
{
Console.WriteLine("Cant send");
}
}
else if (dataReceived.Equals("downloadfile"))
{
// handlerSocketAuthenticate.Close();
// tcpListener.Stop();
networkStream.Close();
thdListenerDownload = new Thread(new ThreadStart(listenerThreadDownloading));
thdListenerDownload.Start();
}
else
{
String clientIP1 = handlerSocketCommands.RemoteEndPoint.ToString();
int index = clientIP1.IndexOf(":");
clientIP1 = clientIP1.Substring(0, index);
// handlerSocketAuthenticate.Close();
CommandExecutor(dataReceived, clientIP1);
}
}
There are so many different things wrong with the code you posted, it's hard to know where to start, and it's impossible to have confidence that in the context of a Stack Overflow, one could sufficiently address all of the deficiencies. That said, in the interest of helping, it seems worth a try:
Sockets are bi-directional. There is no need for the client to use TcpListener at all. (By convention, the "server" is the endpoint that "listens" for new connections, and the "client" is the endpoint that initiates new connections, by connecting to a listening server.)You should just make a single connection from client to server, and then use that socket both for sending to and receiving from the server.
You are setting the CheckForIllegalCrossThreadCalls property to false. This is evil. The exceptions that occur are there to help you. Setting that property to false disables the exceptions, but does nothing to prevent the problems that the exceptions are designed to warn you about.You should use some mechanism to make sure that when you access UI objects, you do so only in the thread that owns those objects. The most primitive approach to this is to use Control.Invoke(). In modern C#, you are better off using async/await. With TcpClient, this is easy: you already are using GetStream() to get the NetworkStream object that represents the socket, so just use the asynchronous methods on that object, such as ReadAsync(), or if you wrap the stream in a StreamWriter and StreamReader, use the asynchronous methods on that object, such as ReadLineAsync().
You are checking the Connected property of the TcpClient object. This is pointless. When the Connect() method returns, you are connected. If you weren't, an exception would have been thrown.
You are not sufficiently synchronizing access to your nSockets object. In particular, you use its indexer in the handlerThreadList() method. This is safe when using the object concurrently only if you have guaranteed that no other thread is modifying the object, which is not the case in your code.
You are writing to the stream using ASCII encoding, but reading using UTF8 encoding. In practice, this is not really a problem, because ASCII includes only the code points 0-127, and those map exactly to the same character code points in UTF8. But it's really bad form. Pick one encoding, stick with it.
You are accepting using AcceptSocket(), but then just wrapping that in a NetworkStream anyway. Why not just use AcceptTcpClient() and call GetStream() on that? Both Socket and TcpClient are fine APIs, but it's a bit weird to mix and match in the same program, and will likely lead to some confusion later on, trying to keep straight which you're using where and why.
Your code assumes that the handlerThreadCommands() method will always be called in exactly the same order in which connections are accepted. That is, you retrieve the current socket with nSockets[nSockets.Count - 1]. But, due to the way Windows thread scheduling works, it is entirely possible that two or more connections could be accepted before any one of the threads meant to handle the connection is started, with the result that only the most recent connection is handled, and it is handled by those multiple threads.
You are assuming that command strings will be received as complete units. But this isn't how TCP works. TCP guarantees only that if you receive a byte, it will be in order relative to all the bytes sent before it. But you can receive any number of bytes. In particular, you can receive just a single byte, or you can receive multiple commands concatenated with each other, or you can receive half a command string, then the other half later, or the second half of one command and the first half of the next, etc. In practice, these problems don't show up in early testing because the server isn't operating under load, but later on they very well may be. And the code needs to be designed from the outset to work properly under these conditions; trying to patch bad code later is much more difficult.
I can't say that's the above are the only things wrong with the code, but they are most glaring, and in any case I think the above is sufficient food for thought for you at the moment.
Bottom line: you really should spend more time looking at good networking examples, and really getting to understand how they work and why they are written the way they do. You'll need to develop a good mental model for yourself of how the TCP protocol works, and make sure you are being very careful to follow the rules.
One resource I recommend highly is The Winsock Programmer's FAQ. It was written long ago, for a pre-.NET audience, but most of the information contained within is still very much relevant when using the higher-level networking APIs.
Alternatively, don't try to write low-level networking code yourself. There are a number of higher-level APIs that use various serialization techniques to encode whole objects and handle all of the lower-level network transport mechanics for you, allowing you to concentrate on the value-added features in your own program, instead of trying to reinvent the wheel.
I have a Socket code which is communicating through TCP/IP.The machine to which i am communicating has buffer data in its buffer.At present i am trying to get the buffer data using this code.
byte data = new byte[1024];
int recv = sock.Receive(data);
stringData = Encoding.ASCII.GetString(data, 0, recv);
But this code retrieves only 11 lines of data whereas more data is there in the machines buffer.Is this because i have used int recv = sock.Receive(data); and data is 1024 ?
If yes ,How to get the total buffer size and retrieve it into string.
If you think you are missing some data, then you need to check recv and almost certainly: loop. Fortunately, ASCII is always single byte - in most other encodings you would also have to worry about receiving partial characters.
A common approach is basically:
int recv;
while((recv = sock.Receive(data)) > 0)
{
// process recv-many bytes
// ... stringData = Encoding.ASCII.GetString(data, 0, recv);
}
Keep in mind that there is no guarantee that stringData will be any particular entire unit of work; what you send is not always what you receive, and that could be a single character, 14 lines, or the second half of one word and the first half of another. You generally need to maintain your own back-buffer of received data until you have a complete logical frame to process.
Note, however, Receive always tries to return something (at least one byte), unless the inbound stream has closed - and will block to do so. If this is a problem, you may need to check the available buffer (sock.Available) to decide whether to do synchronous versus asynchronous receive (i.e. read synchronously while data is available, otherwise request an asynchronous read).
Try something along these lines:
StringBuilder sbContent=new StringBuilder();
byte data = new byte[1024];
int numBytes;
while ((numBytes = sock.Receive(data))>0)
{
sbContent.Append(Encoding.UTF8.GetString(data));
}
// use sbContent.ToString()
Socket tcpSocket = new Socket(ipe.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
Console.WriteLine(" ReceiveBufferSize {0}", tcpSocket.ReceiveBufferSize);
For actual data you can put below condition:-
int receiveBytes;
while((receiveBytes = tcpSocket.Receive.Data(receiveBytes)) > 0)
{
}
I am writing a telnet server using the async Begin/End methods. The issue that I am having is determining what within my buffer is actual data and what is not. Network coding is a bit new to me, but I've tried to research this and have not been able to find a answer.
public bool Start(IGame game)
{
// Get our server address information.
IPHostEntry serverHost = Dns.GetHostEntry(Dns.GetHostName());
IPEndPoint serverEndPoint = new IPEndPoint(IPAddress.Any, this.Port);
// Instance the server socket, bind it to a port and begin listening for connections.
this._ServerSocket = new Socket(serverEndPoint.Address.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
this._ServerSocket.Bind(serverEndPoint);
this._ServerSocket.Listen(this.MaxQueuedConnections);
this._ServerSocket.BeginAccept(new AsyncCallback(Connect), this._ServerSocket);
return true;
}
private void Connect(IAsyncResult result)
{
var player = new BasePlayer();
try
{
player.Game = this.Game;
player.Connection = this._ServerSocket.EndAccept(result);
lock (this.Connections)
{
this.Connections.Add(player);
}
// Pass all of the data handling for the player to itself.
player.Connection.BeginReceive(player.Buffer, 0, player.BufferSize, SocketFlags.None, new AsyncCallback(player.ReceiveData), player);
// Fetch the next incoming connection.
this._ServerSocket.BeginAccept(new AsyncCallback(Connect), this._ServerSocket);
}
catch (Exception)
{
}
}
and then the player.ReceiveData..
public void ReceiveData(IAsyncResult result)
{
int bytesRead = this.Connection.EndReceive(result);
if (bytesRead > 0)
{
// TODO: Parse data received by the user.
//Queue the next receive data.
this.Connection.BeginReceive(this.Buffer, 0, this.BufferSize, SocketFlags.None, new AsyncCallback(ReceiveData), this);
var str = System.Text.Encoding.Default.GetString(Buffer);
}
else
{
this.Disconnect(result);
}
}
So when I call BeginReceive, I need to provide a buffer of a predetermined size. In doing that, I end up with unused bytes in my buffer array. They all have the value of 0, so I am assuming that I can loop through the array and build a new one starting at index 0 and working until I hit a value of 0.
I imagine there is a better way to do this? Can someone please point me in the right direction as to how I should determine what the data is within my buffer or perhaps a way that I can do this without having to use a predetermined buffer size.
So when call BeginReceive, I need to provide a buffer of a predetermined size. In doing that, I end up with unused bytes in my buffer array. They all have the value of 0, so I am assuming that I can loop through the array and build a new one starting at index 0 and working until I hit a value of 0.
No, that's not what you should do. Instead, in your callback (ReceiveData) you're already calling EndReceive - and the result of that is the number of bytes you read. That's how much of the buffer you should use.
However, you should copy the data you've read out of the buffer before you call BeginReceive again, otherwise you may end up with the next bit of data overwriting the just-read data before you get to use it.
So something like:
string text = Encoding.ASCII.GetString(Buffer, 0, bytesRead);
Connection.BeginReceive(this.Buffer, 0, this.BufferSize, SocketFlags.None,
new AsyncCallback(ReceiveData), this);
I would not suggest that you use Encoding.Default to convert the bytes to text - instead, you should decide which encoding you're using, and stick to that. If you use an encoding which isn't always one-byte-per-character, you'll end up in a slightly trickier situation, as then you might end up receiving a buffer with part of a character. At that point you need to keep a Decoder which can maintain state about partial characters read.
There is a code of async server. Client sends Header - size of Data Block + Data Block.
Server reads asynchronously first Header and then Data Block.
I need, after I read Data Block run the BeginRead for Header reading part, to make threads async.
PROBLEM:
When I got DataCallBack, in line:
int bytesRead = ns.EndRead(result);
I get not all buffer i asked to read in
mc.Client.GetStream().BeginRead(mc.DataBuffer, 0, size, new AsyncCallback(DataCallBack), mc);
If client send 1MB of Data I can get different number of "bytesRead".
QUESTION:
How to force "BeginRead" to read all data from connection. It should cause the new loop of Header - Data.
MyClient - simply wrapper over TcpClient;
CODE:
public void DoAcceptTcpClientCallback(IAsyncResult ar)
{
TcpListener listener = (TcpListener)ar.AsyncState;
TcpClient client = listener.EndAcceptTcpClient(ar);
client.NoDelay = false;
// client.ReceiveBufferSize = 1024*1024;
listener.BeginAcceptTcpClient(new AsyncCallback(DoAcceptTcpClientCallback), listener);
MyClient mc = new MyClient(client);
ContinueRead(0,mc);
}
public void ContinueRead(int size, MyClient mc)
{
if (size != 0)
{
mc.DataBuffer = new byte[size];
mc.Client.GetStream().BeginRead(mc.DataBuffer, 0, size, new AsyncCallback(DataCallBack), mc);
}
mc.Client.GetStream().BeginRead(mc.HeaderBuffer, 0, 4, new AsyncCallback(HeaderCallBack), mc);
}
private void HeaderCallBack(IAsyncResult result)
{
MyClient mc = (MyClient)result.AsyncState;
NetworkStream ns = mc.Stream;
int bytesRead = ns.EndRead(result);
if (bytesRead == 0)
throw new Exception();
mc.TotalLengs = BitConverter.ToInt32(mc.HeaderBuffer, 0);
ContinueRead(mc.TotalLengs, mc);
}
private void DataCallBack(IAsyncResult result)
{
MyClient mc = (MyClient)result.AsyncState;
NetworkStream ns = mc.Stream;
int bytesRead = ns.EndRead(result);
if (bytesRead == 0)
throw new Exception();
BAD CODE - MAKES ASYNC READING - SYNC
while (bytesRead < mc.TotalLengs)
{
bytesRead += ns.Read(mc.DataBuffer, bytesRead, mc.TotalLengs - bytesRead);
}
END BAD CODE
ContinueRead(0, mc);
ProcessPacket(mc.DataBuffer, mc.IP);
}
"If client send 1MB of Data I can get different number of "bytesRead"."
Yes...this is simply how TCP works under the hood. You can't change this. TCP guarantees the order of packets, not how they are grouped. The hardware and traffic conditions along the route the packets travel determine how that data is grouped (or un-grouped).
"How to force "BeginRead" to read all data from connection."
TCP has no idea how much data is being sent. As far as it is concerned, the connection is simply an endless stream of bytes; therefore it cannot read "all data" since there is no end to the data (from its perspective). TCP also has no notion of what a "complete message" is with respect to your application. It is up to you, the programmer, to develop a protocol that allows your application to know when all data has been sent.
If you are expecting a certain number of bytes, then keep a running sum of the values returned by EndRead() and stop when that magic number is hit.
im new to C# and any help/feedback would be appreciated. im trying to develop a a client-server program in c#, however i do have different clients sending information to the server side. Is there any function similar to the c language select() such that can help to get all the information from every client side in C#?
here is my server side code:
// Create the listening socket...
m_mainSocket = new Socket(AddressFamily.InterNetwork,
SocketType.Stream, ProtocolType.Tcp);
IPEndPoint ipLocal = new IPEndPoint(IPAddress.Any, 9051);
// Bind to local IP Address...
m_mainSocket.Bind(ipLocal);
// Start listening...
m_mainSocket.Listen(10);
Socket clientSock = m_mainSocket.Accept();
byte[] clientData = new byte[1024];
int receivedBytesLen = clientSock.Receive(clientData);
string clientDataInString =
Encoding.ASCII.GetString(clientData, 0, receivedBytesLen);
string clientStr = "Client Data Received: " + clientDataInString;
byte[] sendData = new byte[1024];
sendData = Encoding.ASCII.GetBytes(clientStr);
clientSock.Send(sendData);
clientSock.Close();
There are higher level constructs, but if you want to get pretty low level, you are probably looking for Socket.Accept:
http://msdn.microsoft.com/en-us/library/system.net.sockets.socket.accept.aspx
You want to accept connections from more than one client, so you have to call the blocking Accept method again after acception a connection:
while (true)
{
var clientSocket = s.Accept();
beginReceive(clientSocket);
}
After accepting you might want to start receiving data in an async manner:
private static void beginReceive(Socket clientSocket)
{
byte[] buffer = new byte[1000];
clientSocket.BeginReceive(
buffer, 0, 1000, SocketFlags.None, OnReceiveData, clientSocket);
}
And finally here is the callback method which is called by the framework on another thread when data arrives. You have to finish the async call with EndReceive:
private static void OnReceiveData(IAsyncResult ar)
{
int bytesReceived = ((Socket) ar.AsyncState).EndReceive(ar);
// process data...
}
Of cause you have to store your buffer somewhere else and maintain one receive buffer per client. Maybe you should write an own class for managing a connected client.
Here is a code project with some nice diagrams and examples. The underlying "select" is handled by the .NET framework so you have to think / work at a higher level.
At a high level you socket.accept create a thread to process the connection.
http://msdn.microsoft.com/en-us/library/system.net.sockets.socket.accept.aspx
http://www.codeproject.com/KB/IP/dotnettcp.aspx