My Server will send a message using networkStream.Write(messageBytes);
My Client will receive the message using networkStream.Read(). The client will read byte by byte for a sequence, when it has found the sequence it reads the rest of the header.
The header contains a payload length. Once I have this I read the stream using the payload length to get the payload.
My problem is when I write the data, my client will try to read straight away and all the data may not have been written by this point.
So I end up with a message with the end effectively chopped off:
"blablablablabl\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"
Is there anyway to wait for all the information to be received?
Thanks.
This doesn't really make sense. Socket read functions tell you how much data is available / has been read into the buffer, so if you're processing a bunch of NULs in the buffer because the rest of the data hasn't been received, that's on you for not checking how much data has been received. If you're expecting more data, keep receiving until you get as much as you're expecting (avoid overwriting the previously-received chunks by advancing the pointer into the receive buffer or copying to a new buffer, etc).
System network buffers are not infinite, if you dont read them from windows socket to your internal memory, they will stop receiving.
In the case, you might want to create a memory storage for your data:
byte[] data = new byte[dataLength];
And keep pushing data into the array until it is all read up.
Related
I want to send huge buffers (from 100MB to 1GB) of data by TCP. I solved it by dividing buffer to smaller (approximately 1MB buffers) and sending by socket.send(). Each call of socket.send() method, send part of data (smaller buffer) packed in specific structure: [start byte(1B), Timestamp(4B), Command(4B), Length of data(4B), Data to send(?B), CRC(1B), End byte(1B)]. Everything works fine, when only one huge buffer is sent by specific port. But when I try to send in the same time buffer with another data (very small, e.g. 20 bytes) using the same TCP port, then TCP mixed data in buffers and it's not possible to decode buffer any more. 'Start byte' and 'end byte' in the buffer are not useful to find start and end of the buffer, because it's probable that these bytes appear in data.
EDIT: Issue does not affect the order or IDs between packages but bytes in the packages. At the beginning everything works fine and each buffer is decoded properly. After a while it is not possible to decode buffer, because it contains incorrect data. It looks like bytes in the buffer were moved or changed. Fields at the beginning of the buffer (timestamp, command, length) contain impossible values. So when I want to get length of sent data, I get e.g. value like -1534501133 instead of 1048556 (1048556 is a correct maximum size of the sent data in one package). It happens randomly but it is always connected with the moment when smaller independent buffer is sent. The additional information is, that the smaller buffers are sent repetitively using timers and the problem happens in random moments. Sometimes it is even possible to send whole data (e.g. 300 MB) without problem but it happens very rarely.
I hope, I described it clearly enough.
Do you have any suggestions how to avoid this problem?
Tag your data with a unique id so you know what data relates to what message. Also, separate the packet header from the packet payload.
So, your first request would be [ID][PACKETTYPE][TIME][COMMAND][LENGTH][CRC]
Second would be [ID][PACKETTYPE][DATA]
You can then match up the IDs with the types of packet. So packet type would be 'HEADER' or 'PAYLOAD', the header would contain the meta data for the payload allowing you to make sure that it doesnt get mixed up with other data.
I am using asynchronous socket API in C#. In the client side, I need a buffer to store the binary data read from the server. And other client logic will check the buffer, unpack the head to see the length, if the length is less than that indicated by the header, continue. And next time we check the buffer again. For the network logic, I need to maintain this buffer, and I want to know what data type should I use.
In python we use a string as a buffer, but I don't think this is gonna work in C#. Inefficient, Encoding problem (I need to parse the binary data my own, not necessarily to a string), Frequently changed. What about stringbuilder? Any other suggestions?
I would use byte[]. It will get the job done.
I am working on a project that involves client server communication via TCP and Google Protocol Buffer. On the client side, I am basically using NetworkStream.Read() to do blocking read from server via a byte array buffer.
According to MSDN documentation,
This method reads data into the buffer parameter and returns the number of bytes successfully read. If no data is available for reading, the Read method returns 0. The Read operation reads as much data as is available, up to the number of bytes specified by the size parameter. If the remote host shuts down the connection, and all available data has been received, the Read method completes immediately and return zero bytes.
It is the same case with async read (NetworkStream.BeginRead and EndRead). My question is that when does Read()/EndRead() return? It seems like it will return after all the bytes in the buffer have been filled. But in my own testing, that is not the case. The bytes read in one operation vary a lot. I think it makes sense because if there is a pause on the server side when sending messages, the client should not wait until the read buffer has been filled. Does the Read()/EndRead() inherently have some timeout mechanism?
I was trying to find out how Mono implements Read() in NetworkStream and traced until a extern method Receive_internal() is called.
It reads all the data that is available on the networkstream or when the buffer is full. Whichever comes first. You have already noticed this behaviour.
So you will need to process all the bytes and see whether the message is complete. You do this by framing a message. See .NET question about asynchronous socket operations and message framing on how you can do this.
As for the timeout question, if assuming you are asking whether a beginread has a timeout, I would say no, because it is just waiting for data to arrive on the stream and put it into a buffer, after which you can process the incoming bytes.
The number of bytes available on the read action depends on things like your network (e.g. latency, proxy throttling) and the client sending the data.
BeginRead behaviour summary:
Call BeginRead(); -> Waiting for bytes to arrive on the stream......
1 byte or more have arrived on the stream
Start putting the byte(s) from step 2 into the buffer that was given
Call EndRead(); -> The byte(s) within the buffer can be processed by EndRead();
Most common practice is to repeat all these steps again.
If Read was waiting for a full buffer of data, you could easily deadlock if the remote party expects your response but you are waiting for a full buffer which will never come.
According to this logic it must return without ever blocking if data is available. Even if it is just a single byte that is available.
assume that server sends one message (100 bytes) every 50 ms, what is the bytes read on client side on one NetworkStream.Read() call?
Each call will return between one byte and the number of bytes available without blocking. Nothing, nothing, nothing else is guaranteed. In practice you will get one or multiple network packets at once. It doesn't make sense for the stack to withhold available bytes.
I am trying to bind Sound and Image Sequence Data through ArrayList in order to get it synchronized and serializing it through Binary formatter to be send over Network stream.
The Server end threw an exception:
THE STREAM CAN NOT SUPPORT SEEK OPERATION.
What should I have to do in order to sync Objects to be sent over a single Network stream Instance
TCP is stream based and not message based (as UDP is). That means that there is no telling when a message starts or ends. TCP only guarantees that all bytes are received and in the correct order. It does not guarantee that everything sent with one Send() will be received with one Receive().
Hence you need to specify some kind of message identification mechanism. In this case, a header is the way to go as Jon suggested.
However, you need to understand that the entire header might not be received at once. And that two messages might arrive at once. So you need to parse the received buffer before sending anything to the BinaryFormatter for deserialization.
I would split each object you want to send out into a separate "message" where a message consists of (say) 4 bytes indicating the body length, and then the body itself.
When you want to send a serialized object, you serialize to a byte array, write out the length, then write out the data.
At the server side, you read the length, read that much data into a byte array, then deserialize from that message. The incoming stream is only used to read messages, not objects.
I have a TCP connection , which client is PHP and server is C#
this socket connection transfers a image to the socket server , but
randomly some times the transfer get corrupted [image hash is different]
PHP Client
$file = file_get_contents('img.bmp');
socket_write($socket,$file.$cordinates); it sends //image + sme other data
$recv = socket_read ($socket, 500, PHP_BINARY_READ) // read the server response
This stream always transfer a Bitmap image + some data .
C#
this.DataSocket = this.Listner.Accept();
int filelength = this.DataSocket.Receive(this.buffer, this.buffer.Length, SocketFlags.None)
i investigated that in a fresh-browser [newly opened ] this never failed. but when i using this created service several times frequently in the same browser this intended to fail.
when i check with a different browser or new instance of the browser it never failed in first few attempts.
i thought it was some problem with caching but i disable caching using headers but same problem exists
You can't simply expect to write an entire file to the socket at once, nor can you expect to read the file from the socket in one operation. The socket read and write APIs for just about any network programming API from BSD sockets to WinSock to .NET network classes are all going to transmit or receive data up to the desired byte count.
If you look at the documentation for PHP socket_write for example:
Returns the number of bytes successfully written to the socket or FALSE on failure. The error code can be retrieved with socket_last_error(). This code may be passed to socket_strerror() to get a textual explanation of the error.
Note:
It is perfectly valid for socket_write() to return zero which means no bytes have been written. Be sure to use the == operator to check for FALSE in case of an error.
You will typically want to choose a block size like 4096 or 16384 and loop transmitting or receiving that block size until you get the desired number of bytes transmitted or received. Your code will have to check the return value of the send or receive function you're calling and adjust your file pointer accordingly. If transmit returns 0, that could just mean the send buffer is full (not fatal) so continue sending (might want a Sleep(0) delay). If receive returns 0, this usually means the other side has cleanly closed the connection.
One of the most critical flaws in your simple network code usage is that you're not sending the size of the file before you send the file data, so there's no way for the receiver to know how much to read before sending their response. For a simple operation like this, I'd suggest just sending a binary 32bit integer (4 bytes). This would be part of the schema for your operation. So the receiver would first read 4 bytes and from that know how many more bytes need to be read (one buffer size at a time). The receiver keeps reading until they have that many bytes.
I hope this helps. It would be great if socket code were as simple as the usage you attempted, but unfortunately it isn't. You have to select a buffer size, and then keep reading or writing buffers of that size until you get what you want, and you have to convey to the other side how much data you plan on sending.
That you think caching has anything to do with the problem implies that either there is a lot of functionality outside of the code you've published which is affecting the result or that you are a very long way from understanding the problem.
Without knowing the structure of bmp files, my first concern would be how you separate the file from the additional info sent. A few things you could try...
If '$cordinates' (sic) is a fixed size, then put this at the front of the message, not the back
Log the size sent from PHP and the size received.
base64 encode the binary file before sending it (and decode at the receiving end)
Non of above solutions didn't work for me , but i found out that create a new instance every time after a one request will solve the problem. but i don't think its a reliable way.
i tried the client using ASP.NET but same results. i think its not a problem with client PHP its surely a problem of the socket server