I have written some code to get a webpage through a proxy using sockets. In essence, it works but reading the response has some strange behavior that is really tripping me up.
When I go to read the response after sending the GET command it is 0 bytes. It takes a few ticks before there is data to read. I don't want to hard code a delay in here as I am trying to write performant reliable code so I have coded a while loop that keeps reading the response until it more than 0.
This works for the first chunk but trying to read subsequent chunks is a problem. If i instantly try to read the response it will be 0 bytes so I need to check the subsequent reads also if they are greater than 0.
So to read the whole response I tried to check if the response is equal to the size of the buffer. If it is equal to the size of the buffer then I carry on and try to read another chunk. This has a few issues also. Sometimes the response will read less than the size of the buffer but there is still more to come, i guess I am reading it faster than they are sending it because if I add a Thread.Sleep() then the buffer will always be full but I don't think it is good practice to hardcode this because I don't know how fast they will be sending. This code will be used for multiple things and will be running on hundreds of threads so performance is everything.
Also if the last chunk just happens to be the size of my buffer then I think the loop will lock, This whole approach I have taken is horrible but I can't see how I should be reading it. I have seen the asynchronous examples but I think that will add to the overall complexity of my code as I just have 1 set process which I will run in many threads.
How do I efficiently read the response when I can't guarantee the next chunk will have data or be full even if there is more data to come?
Sorry for long text but I wanted to explain my thinking. Here is my code:
// Data buffer for incoming data.
byte[] bytes = new byte[1024];
// Connect to a remote device.
try
{
var proxyIpAddress = IPAddress.Parse("123.123.123.123"); //omitted
IPEndPoint remoteEP = new IPEndPoint(proxyIpAddress, 60099);
// Create a TCP/IP socket.
Socket sender = new Socket(proxyIpAddress.AddressFamily,
SocketType.Stream, ProtocolType.Tcp);
// Connect the socket to the remote endpoint. Catch any errors.
try
{
sender.Connect(remoteEP);
Console.WriteLine("Socket connected to {0}",
sender.RemoteEndPoint.ToString());
sender.Send(Encoding.ASCII.GetBytes($"CONNECT google.com:80 HTTP/1.0\r\n\r\n"));
int bytesRec = 0;
while (bytesRec == 0)
{
// Receive the response from the remote device.
bytesRec = sender.Receive(bytes);
Console.WriteLine("{0}",
Encoding.ASCII.GetString(bytes, 0, bytesRec));
}
//clear buffer
bytes = new byte[1024];
bytesRec = 0;
sender.Send(Encoding.ASCII.GetBytes("GET / HTTP/1.0\r\n\r\n"));
//wait for response
while (bytesRec == 0) //if i dont add this it returns before it actually gets data
{
// Receive the response from the remote device.
bytesRec = sender.Receive(bytes);
Console.WriteLine("{0}",
Encoding.ASCII.GetString(bytes, 0, bytesRec));
}
if(bytes.Length == bytesRec) //full buffer so likely more but maybe not if final packet exactly 1024?
{
while (bytes.Length == bytesRec) //again if i miss this it returns too early
{
int subsequentBytes = 0;
while(subsequentBytes == 0) //this can get stuck if last packet exactly size of buffer i think
{
subsequentBytes = sender.Receive(bytes);
Console.WriteLine("{0}",
Encoding.ASCII.GetString(bytes, 0, subsequentBytes));
//this doesn't work. even when there are subsequent bytes sometimes it reads less
//than the size of the buffer so it exits prematurely. If I add a Thread.Sleep() here
// then it works but I don't want to hardcode the delay. How do I read this buffer properly?
Thread.Sleep(1000);
if (subsequentBytes > 0) bytesRec = subsequentBytes;
}
}
}
// Release the socket.
sender.Shutdown(SocketShutdown.Both);
sender.Close();
}
catch (Exception e)
{
Console.WriteLine("Unexpected exception : {0}", e.ToString());
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
I understand this is difficult to follow and a lot of writing so if anyone perseveres with this they have my gratitude as the only option I can see is hardcoded pauses which will hurt performance and may still have issues.
EDIT
I have done some experiementing with different servers. If I ping the server then set a Thread.Sleep(pingValue) it works fine but if i set the sleep to lower than ping i get same issue.
Is there some good way with the .net libraries to account for this latency so I am not under/overestimating?
Related
I've been working with windows app store programming in c# recently, and I've come across a problem with sockets.
I need to be able to read data with an unknown length from a DataReader().
It sounds simple enough, but I've not been able to find a solution after a few days of searching.
Here's my current receiving code (A little sloppy, need to clean it up after I find a solution to this problem. And yes, a bit of this is from the Microsoft example)
DataReader reader = new DataReader(args.Socket.InputStream);
try
{
while (true)
{
// Read first 4 bytes (length of the subsequent string).
uint sizeFieldCount = await reader.LoadAsync(sizeof(uint));
if (sizeFieldCount != sizeof(uint))
{
// The underlying socket was closed before we were able to read the whole data.
return;
}
reader.InputStreamOptions
// Read the string.
uint stringLength = reader.ReadUInt32();
uint actualStringLength = await reader.LoadAsync(stringLength);
if (stringLength != actualStringLength)
{
// The underlying socket was closed before we were able to read the whole data.
return;
}
// Display the string on the screen. The event is invoked on a non-UI thread, so we need to marshal
// the text back to the UI thread.
//MessageBox.Show("Received data: " + reader.ReadString(actualStringLength));
MessageBox.updateList(reader.ReadString(actualStringLength));
}
}
catch (Exception exception)
{
// If this is an unknown status it means that the error is fatal and retry will likely fail.
if (SocketError.GetStatus(exception.HResult) == SocketErrorStatus.Unknown)
{
throw;
}
MessageBox.Show("Read stream failed with error: " + exception.Message);
}
You are going down the right lines - read the first INT to find out how many bytes are to be sent.
Franky Boyle is correct - without a signalling mechanism it is impossible to ever know the length of a stream. Thats why it is called a stream!
NO socket implementation (including the WinSock) will ever be clever enough to know when a client has finished sending data. The client could be having a cup of tea half way through sending the data!
Your server and its sockets will never know! What are they going to do? Wait forever? I suppose they could wait until the client had 'closed' the connection? But your client could have had a blue screen and the server will never get that TCP close packet, it will just be sitting there thinking it is getting more data one day?
I have never used a DataReader - i have never even heard of that class! Use NetworkStream instead.
From my memory I have written code like this in the past. I am just typing, no checking of syntax.
using(MemoryStream recievedData = new MemoryStream())
{
using(NetworkStream networkStream = new NetworkStream(connectedSocket))
{
int totalBytesToRead = networkStream.ReadByte();
// This is your mechanism to find out how many bytes
// the client wants to send.
byte[] readBuffer = new byte[1024]; // Up to you the length!
int totalBytesRead = 0;
int bytesReadInThisTcpWindow = 0;
// The length of the TCP window of the client is usually
// the number of bytes that will be pushed through
// to your server in one SOCKET.READ method call.
// For example, if the clients TCP window was 777 bytes, a:
// int bytesRead =
// networkStream.Read(readBuffer, 0, int.Max);
// bytesRead would be 777.
// If they were sending a large file, you would have to make
// it up from the many 777s.
// If it were a small file under 777 bytes, your bytesRead
// would be the total small length of say 500.
while
(
(
bytesReadInThisTcpWindow =
networkStream.Read(readBuffer, 0, readBuffer.Length)
) > 0
)
// If the bytesReadInThisTcpWindow = 0 then the client
// has disconnected or failed to send the promised number
// of bytes in your Windows server internals dictated timeout
// (important to kill it here to stop lots of waiting
// threads killing your server)
{
recievedData.Write(readBuffer, 0, bytesReadInThisTcpWindow);
totalBytesToRead = totalBytesToRead + bytesReadInThisTcpWindow;
}
if(totalBytesToRead == totalBytesToRead)
{
// We have our data!
}
}
}
I've implemented an Asynchronous Client Socket extremely similar to this example. Is there any reason why I cannot dramatically increase this buffer size? In this example the buffer size is 256 bytes. In many cases, my application ends up receiving data that is 5,000++ bytes of data. Should I increase the buffer size? Are there any reasons why I should NOT increase the buffer size?
Every once in a long while I'll get some issue where the data comes in out of order or a chunk is missing (yet to be confirmed exactly which it is). For example, one time I received some corrupt data that looks like this
Slice Id="0" layotartX='100'
The attribute called layotartX does not exist in my data, it was supposed to say layout=... but instead the layout got cut off and other data was appended to it later. I counted the bytes and noticed that it was cut off at exactly 256 bytes which just so happens to be my buffer size. It's very possible that increasing my buffer size would prevent this problem from happening (data coming in out of order??). Anyways, as stated in the 1st paragraph, I'm just asking if there is any reason I should NOT increase the buffer size to be say like 5,000 bytes or even 10,000 bytes.
Adding some code. Below is my modified ReceiveCallback function (see the linked example code above for the rest of the classes. When the ReceiveCallback receives data, it calls the "ReceiveSomeData" function which I've also posted below. For some reason every once in a while I get data out of order or pieces missing. The "ReceiveSomeData" function is in a class called "MyChitterChatter" and the "ReceiveCallback" function is in a class called "AsyncClient". So when you see the ReceiveSomeData function locking "this", it's locking the MyChitterChatter class. Is there were my problem could by lying?
private static void ReceiveCallback( IAsyncResult ar )
{
AppDelegate appDel = (AppDelegate)UIApplication.SharedApplication.Delegate;
try {
// Retrieve the state object and the client socket
// from the asynchronous state object.
StateObject state = (StateObject) ar.AsyncState;
Socket client = state.workSocket;
// Read data from the remote device.
int bytesRead = client.EndReceive(ar);
if (bytesRead > 0) {
// There might be more data, so store the data received so far.
string stuffWeReceived = Encoding.ASCII.GetString(state.buffer,0,bytesRead);
string debugString = "~~~~~ReceiveCallback~~~~~~ " + stuffWeReceived + " len = " + stuffWeReceived.Length + " bytesRead = " + bytesRead;
Console.WriteLine(debugString);
// Send this data to be received
appDel.wallInteractionScreen.ChitterChatter.ReceiveSomeData(stuffWeReceived);
// Get the rest of the data.
client.BeginReceive(state.buffer,0,StateObject.BufferSize,0,
new AsyncCallback(ReceiveCallback), state);
} else {
// Signal that all bytes have been received.
receiveDone.Set();
}
}
catch (Exception e) {
Console.WriteLine("Error in AsyncClient ReceiveCallback: ");
Console.WriteLine(e.Message);
Console.WriteLine(e.StackTrace);
}
}
public void ReceiveSomeData ( string data )
{
lock(this)
{
DataList_New.Add(data);
// Update the keepalive when we receive ANY data at all
IsConnected = true;
LastDateTime_KeepAliveReceived = DateTime.Now;
}
}
Yes, you absolutely should increase the buffer size to something much closer to what you expect to get in a single read. 32k or 64k would be fine choice for most uses.
Having said that, data never comes in "out of order" or "missing a chunk" if you're using a TCP/IP socket; if you see something like that, it's a bug in your code, not a bug in the socket. Share your code if you want help.
I have a Socket code which is communicating through TCP/IP.The machine to which i am communicating has buffer data in its buffer.At present i am trying to get the buffer data using this code.
byte data = new byte[1024];
int recv = sock.Receive(data);
stringData = Encoding.ASCII.GetString(data, 0, recv);
But this code retrieves only 11 lines of data whereas more data is there in the machines buffer.Is this because i have used int recv = sock.Receive(data); and data is 1024 ?
If yes ,How to get the total buffer size and retrieve it into string.
If you think you are missing some data, then you need to check recv and almost certainly: loop. Fortunately, ASCII is always single byte - in most other encodings you would also have to worry about receiving partial characters.
A common approach is basically:
int recv;
while((recv = sock.Receive(data)) > 0)
{
// process recv-many bytes
// ... stringData = Encoding.ASCII.GetString(data, 0, recv);
}
Keep in mind that there is no guarantee that stringData will be any particular entire unit of work; what you send is not always what you receive, and that could be a single character, 14 lines, or the second half of one word and the first half of another. You generally need to maintain your own back-buffer of received data until you have a complete logical frame to process.
Note, however, Receive always tries to return something (at least one byte), unless the inbound stream has closed - and will block to do so. If this is a problem, you may need to check the available buffer (sock.Available) to decide whether to do synchronous versus asynchronous receive (i.e. read synchronously while data is available, otherwise request an asynchronous read).
Try something along these lines:
StringBuilder sbContent=new StringBuilder();
byte data = new byte[1024];
int numBytes;
while ((numBytes = sock.Receive(data))>0)
{
sbContent.Append(Encoding.UTF8.GetString(data));
}
// use sbContent.ToString()
Socket tcpSocket = new Socket(ipe.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
Console.WriteLine(" ReceiveBufferSize {0}", tcpSocket.ReceiveBufferSize);
For actual data you can put below condition:-
int receiveBytes;
while((receiveBytes = tcpSocket.Receive.Data(receiveBytes)) > 0)
{
}
I am writing a telnet server using the async Begin/End methods. The issue that I am having is determining what within my buffer is actual data and what is not. Network coding is a bit new to me, but I've tried to research this and have not been able to find a answer.
public bool Start(IGame game)
{
// Get our server address information.
IPHostEntry serverHost = Dns.GetHostEntry(Dns.GetHostName());
IPEndPoint serverEndPoint = new IPEndPoint(IPAddress.Any, this.Port);
// Instance the server socket, bind it to a port and begin listening for connections.
this._ServerSocket = new Socket(serverEndPoint.Address.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
this._ServerSocket.Bind(serverEndPoint);
this._ServerSocket.Listen(this.MaxQueuedConnections);
this._ServerSocket.BeginAccept(new AsyncCallback(Connect), this._ServerSocket);
return true;
}
private void Connect(IAsyncResult result)
{
var player = new BasePlayer();
try
{
player.Game = this.Game;
player.Connection = this._ServerSocket.EndAccept(result);
lock (this.Connections)
{
this.Connections.Add(player);
}
// Pass all of the data handling for the player to itself.
player.Connection.BeginReceive(player.Buffer, 0, player.BufferSize, SocketFlags.None, new AsyncCallback(player.ReceiveData), player);
// Fetch the next incoming connection.
this._ServerSocket.BeginAccept(new AsyncCallback(Connect), this._ServerSocket);
}
catch (Exception)
{
}
}
and then the player.ReceiveData..
public void ReceiveData(IAsyncResult result)
{
int bytesRead = this.Connection.EndReceive(result);
if (bytesRead > 0)
{
// TODO: Parse data received by the user.
//Queue the next receive data.
this.Connection.BeginReceive(this.Buffer, 0, this.BufferSize, SocketFlags.None, new AsyncCallback(ReceiveData), this);
var str = System.Text.Encoding.Default.GetString(Buffer);
}
else
{
this.Disconnect(result);
}
}
So when I call BeginReceive, I need to provide a buffer of a predetermined size. In doing that, I end up with unused bytes in my buffer array. They all have the value of 0, so I am assuming that I can loop through the array and build a new one starting at index 0 and working until I hit a value of 0.
I imagine there is a better way to do this? Can someone please point me in the right direction as to how I should determine what the data is within my buffer or perhaps a way that I can do this without having to use a predetermined buffer size.
So when call BeginReceive, I need to provide a buffer of a predetermined size. In doing that, I end up with unused bytes in my buffer array. They all have the value of 0, so I am assuming that I can loop through the array and build a new one starting at index 0 and working until I hit a value of 0.
No, that's not what you should do. Instead, in your callback (ReceiveData) you're already calling EndReceive - and the result of that is the number of bytes you read. That's how much of the buffer you should use.
However, you should copy the data you've read out of the buffer before you call BeginReceive again, otherwise you may end up with the next bit of data overwriting the just-read data before you get to use it.
So something like:
string text = Encoding.ASCII.GetString(Buffer, 0, bytesRead);
Connection.BeginReceive(this.Buffer, 0, this.BufferSize, SocketFlags.None,
new AsyncCallback(ReceiveData), this);
I would not suggest that you use Encoding.Default to convert the bytes to text - instead, you should decide which encoding you're using, and stick to that. If you use an encoding which isn't always one-byte-per-character, you'll end up in a slightly trickier situation, as then you might end up receiving a buffer with part of a character. At that point you need to keep a Decoder which can maintain state about partial characters read.
I am trying to write network part for my game in C# using System.Net.Sockets and TcpClient class.
Each update server is sending information to client.
All information is built into 2kb packets, so in 1 update 1-2-3-5-10 packets can be sent.
Client is checking information and if information has right format - then reading it.
Everything is working fine, until server starts trying to send too many packets.
When it happens client time to time is receiving packets with wrong data 1 of 20-50 packets usually.
For example, 1-2 packets for 1 update usually are received fine, 3-10 packets for update giving wrong data streams.
If I am starting several clients in 1 time, that should get same data streams from server - they get different numbers of success and fail data streams.
What am I doing wrong, and how can I evade this wrong data streams?
Am I just sending too much data in 1 ms and it is needed to send it over time?
This is the sending information:
TcpClient client;
public void SendData(byte[] b)
{
//Try to send the data. If an exception is thrown, disconnect the client
try
{
lock (client.GetStream())
{
client.GetStream().BeginWrite(b, 0, b.Length, null, null);
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
This is the receiving information:
byte[] readBuffer;
int byfferSize = 2048;
private void StartListening()
{
client.GetStream().BeginRead(readBuffer, 0, bufferSize, StreamReceived, null);
}
private void StreamReceived(IAsyncResult ar)
{
int bytesRead = 0;
try
{
lock (client.GetStream())
{
bytesRead = client.GetStream().EndRead(ar); // просмотр длины сообщения
}
}
catch (Exception ex)
{ MessageBox.Show(ex.Message); }
//An error happened that created bad data
if (bytesRead == 0)
{
Disconnect();
return;
}
//Create the byte array with the number of bytes read
byte[] data = new byte[bytesRead];
//Populate the array
for (int i = 0; i < bytesRead; i++)
data[i] = readBuffer[i];
//Listen for new data
StartListening();
//Call all delegates
if (DataReceived != null)
DataReceived(this, data);
}
It is main network code.
I don't know what you do with the data after you've received it, but it's quite possible that you're not reading all of the data from the connection. You have:
bytesRead = client.GetStream().EndRead(ar);
There's no guarantee that the number of bytes you've read are all of the bytes that the server sent. For example, the server could have sent 2,048 bytes, but when you called Read, there were only 1,024 bytes available. The rest of them are still "in transit." As the documentation for NetworkStream.Read says:
The Read operation reads as much data as is available, up to the number of bytes specified by the size parameter
You could be getting partial packets. If your DataReceived handlers assume that the data buffer contains a complete packet, then you're going to have problems.
To reliably read from a network stream, you need to know how much data you're supposed to read, or you need a record separator. Something has to make sure that if you're expecting a complete packet that you get a complete packet before you try to process it. Your code just checks to see if bytesRead is not 0. If it's anything else, you just pass it on. This is going to be a problem unless your DataReceived handlers know how to buffer partial packets.
On another note, you really don't need to lock the network stream. Unless you can have several threads reading from the same stream. And that would be disastrous. Ditch the lock. You don't need it.