i'm trying to learn to use sockets in c# and i've a doubt, i'm using a code something like this:
byte[] data = new byte[64];
int length = 0;
length = sock.Receive(data);
//more code...
So, the byte[] data is filled with the recived data and the left space in the array is filled with 0s, Is the byte[] allocated into memory completely (all 64bytes)? If it's so, is there a way to make the byte[] the same size as the actual sent data?
You can check sock.Available to see what has already come in. (so far)
byte[] data = new byte[sock.Available];
int length = sock.Receive(data);
//more code...
Note: Since you may or may not know what is coming in next on the network it usually makes more sense to read only a header (with size info) first or to allocate more space than necessary, and call .Recieve() multiple times until the end of a record is reached.
Note: This code assumes you already know there is some data to receive and you've waited long enough for some useful amount of data to be ready.
If you do choose to use length headers, .Available can help you avoid reading a partial header and having to re-assemble it which is nice. (Only large messages may need manual reassembly in that case)
You simply need to use the return value from Receive to understand how much data has arrived. You can shorten the buffer using Array.Resize if you want but that normally would be a sign that something is wrong.
Also note, that TCP is a stream of bytes and does not preserve message boundaries.
As noted normally read may return fewer bytes then it was told. See a workaround function below which ensures it reads as many bytes as it was told - basically size of the passed buffer. Function is from here.
/// Reads data into a complete array, throwing an EndOfStreamException
/// if the stream runs out of data first, or if an IOException
/// naturally occurs.
/// </summary>
/// <param name="stream">The stream to read data from</param>
/// <param name="data">The array to read bytes into. The array
/// will be completely filled from the stream, so an appropriate
/// size must be given.</param>
public static void ReadWholeArray (Stream stream, byte[] data)
{
int offset=0;
int remaining = data.Length;
while (remaining > 0)
{
int read = stream.Read(data, offset, remaining);
if (read <= 0)
throw new EndOfStreamException
(String.Format("End of stream reached with {0} bytes left to read", remaining));
remaining -= read;
offset += read;
}
}
You can use this method first to read say a 2 byte integer which should represent the number of bytes that will follow. Then you read once again however now read as many bytes as specified in that two byte integer.
But for this to work, clearly the sender first has to send a two byte integer which represents length of data that will follow - and then the data itself.
So basically you call above function on a byte array of size two first (to get data length), and then on a byte array with size as indicated in that 2 byte integer (to get data).
You can use this to read from NetworkStream. Some more reading on this topic.
Related
My code
private static int readinput(byte[] buff, BinaryReader reader)
{
int size = reader.ReadInt32();
reader.Read(buff, 0, size);
return size;
}
Exception in reader.Read(buff, 0, size);
The exception is offset and length were out of bounds for the array or count is greater than the number of elements from index to the end of source collection
Take a step back and think about your code
You've written a method that takes an array of bytes. We don't know how big this array is, but it's controlled by the code calling the method. Let's assume it is 1000 bytes long
Then you read an int from somewhere else, let's assume 2000 is read
Then you attempt to read 2000 bytes into an array that can only hold 1000 bytes, you perform no checking to make sure your array is big enough, nor do you attempt to read in chunks and concatenate if it's not big enough
That's why you get the error you're getting, but as to what you should be coding, I think you need to think about that some more- maybe make the size to the buffer in response to the size int you read, or read in chunks..
The buffer buff that you passed into your function to read the data is too small. buff.Length should be bigger than or equal to your variable called size.
Set a breakpoint on "reader.Read(buff, 0, size);" and hover over buff and size and you'll see what I mean.
Make sure when you call your function, the buff you pass in is sufficient size. If you don't know what size to create a buffer for ahead of time, then change your function to look something like this:
private static byte[] ReadInput(BinaryReader reader)
{
int size = reader.ReadInt32();
return reader.ReadBytes(size);
}
Especially since you're just reading it into the beginning of a provided buffer anyways.
Summary to frame what you're currently doing:
You provided us a function which takes a binary reader (whatever position it's already at, if it's new, then position 0), it reads a 32-bit integer (4 bytes) to figure out the size of some data after it. Then you read that data of that size into a buffer. You do all of this with a buffer provided called buff. You need to be sure that whatever size data you're going to read in, the buffer provided to the function is of sufficient size. If you make the size of the buffer too large, then "reader.Read(buff, 0, size)" is only reading it into the beginning of the buffer. So if your intention was just to read the data the way you coded that function into a perfectly sized buffer, I suggest using the code above.
Just thought I'd explain it a bit more in case that helps you understand what's going on.
I want to read very large file (4GBish) chunk by chunk.
I am currently trying to use a StreamReader and the Read() read method. The syntax is:
sr.Read(char[] buffer, int index, int count)
Because the index is an int it will overflow in my case. What should i use instead?
The index is the starting index of buffer not the index of file pointer, usually it would be zero. On each Read call you will read characters equal to the count parameter of Read method. You would not read all the file at once rather read in chunks and use that chunk.
The index of buffer at which to begin writing, reference.
char[] c = null;
while (sr.Peek() >= 0)
{
c = new char[1024];
sr.Read(c, 0, c.Length);
//The output will look odd, because
//only five characters are read at a time.
Console.WriteLine(c);
}
The above example will ready 1024 bytes and will write to console. You can use these bytes, for instance sending these bytes to other application using TCP connection.
When using the Read method, it is more efficient to use a buffer that
is the same size as the internal buffer of the stream, where the
internal buffer is set to your desired block size, and to always read
less than the block size. If the size of the internal buffer was
unspecified when the stream was constructed, its default size is 4
kilobytes (4096 bytes), MSDN.
You could try the simpler version of Read which doesn't chunk the stream but instead reads it character by character. You would have to implement the chunking your self, but it would give you more control allowing you to use a Long instead.
http://msdn.microsoft.com/en-us/library/ath1fht8(v=vs.110).aspx
I would like to be able to get the length of the data available from a TCP network stream in C# to set the size of the buffer before reading from the network stream. There is a NetworkStream.Length property but it isn't implemented yet, and I don't want to allocate an enormous size for the buffer as it would take up too much space. The only way I though of doing it would be to precede the data transfer with another telling the size, but this seems a little messy. What would be the best way for me to go about doing this.
When accessing Streams, you usually read and write data in small chunks (e.g. a kilobyte or so), or use a method like CopyTo that does that for you.
This is an example using CopyTo to copy the contents of a stream to another stream and return it as a byte[] from a method, using an automatically-sized buffer.
using (MemoryStream ms = new MemoryStream())
{
networkStream.CopyTo(ms);
return ms.ToArray();
}
This is code that reads data in the same way, but more manually, which might be better for you to work with, depending on what you're doing with the data:
byte[] buffer = new byte[2048]; // read in chunks of 2KB
int bytesRead;
while((bytesRead = networkStream.Read(buffer, 0, buffer.Length)) > 0)
{
//do something with data in buffer, up to the size indicated by bytesRead
}
(the basis for these code snippets came from Most efficient way of reading data from a stream)
There is no inherent length of a network stream. You will either have to send the length of the data to follow from the other end or read all of the incoming data into a different stream where you can access the length information.
The thing is, you can't really be sure all the data is read by the socket yet, more data might come in at any time. This is try even if you somehow do know how much data to expect, say if you have a package header that contains the length. the whole packet might not be received yet.
If you're reading arbitrary data (like a file perhaps) you should have a buffer of reasonable size (like 1k-10k or whatever you find to be optimal for your scenario) and then write the data to a file as its read from the stream.
var buffer = byte[1000];
var readBytes = 0;
using(var netstream = GetTheStreamSomhow()){
using(var fileStream = (GetFileStreamSomeHow())){
while(netstream.Socket.Connected) //determine if there is more data, here we read until the socket is closed
{
readBytes = netstream.Read(buffer,0,buffer.Length);
fileStrem.Write(buffer,0,buffer.Length);
}
}
}
Or just use CopyTo like Tim suggested :) Just make sure that all the data has indeed been read, including data that hasn't gotten across the network yet.
You could send the lenght of the incoming data first.
For example:
You have data = byte[16] you want to send. So at first you send the 16 and define on the server, that this length is always 2 (because 16 has two characters). Now you know that the incomingLength = 16. You can wait now for data of the lenght incomingLength.
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Is the received stream from a socket limited to a single send command?
Note: I see this question very complicated (hopefully not for you guys, that's why Im asking here lol) and I tried my best to explain it as simple and clear as possible.
In my application, I'm continually receiving byte arrays in a fix sized buffer.
These series of byte arrays that I'm receiving has been serialized 'binarily'.
However, sometimes the byte array received will be bigger than the fix sized buffer so I would need to store the current received byte array into a container and loop again to receive the remaining byte arrays coming in.
My question now is how to "concatenate" or "combine" or "join" all the "batches" of byte arrays I received ( and is stored in a container, possibly a queue of byte arrays) to form a single byte array and then de-serialize them?
int bytesRead = client.EndReceive(ar);
if (bytesRead > 0)
{
// There might be more data, so store the data received so far.
// If the buffer was not filled, I have to get the number of bytes received as Thorsten Dittmar was saying, before queuing it
dataReceivedQueue.Enqueue(state.buffer);
// Get the rest of the data.
client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReceiveCallback_onQuery), state);
}
else
{
// All the data has arrived; put it in response.
response_onQueryHistory = ByteArrayToObject(functionThatCombinesBytes(dataReceivedQueue));
// Signal that all bytes have been received.
receiveDoneQuery.Set();
}
state.buffer is buffer where data are received. buffer is a byte array of size 4096. state is of type StateObject.
ByteArrayToObject(byte []) takes care of deserializing the data received and converting it back to its object form
functionThatCombinesBytes(Queue) this function will receive a Queue of bytes and will "combine" all the bytes into one byte array
Just because you are calling BeginReceive with a buffer of a particular size, doesn't mean that it will necessarily entirely fill the buffer, so it's very likely that some of your queued buffers will actually only be partially filled with received data, and the remainder being zeros, this will almost certainly corrupt your combined stream if you simply concatenate them together since you're not also storing the number of bytes actually read into the buffer. You also appear to be reusing the same buffer each time, so you'll just be overwriting already-read data with new data.
I would therefore suggest replacing your dataReceivedQueue with a MemoryStream, and using something like:
if (bytesRead > 0)
{
// There might be more data, so store the data received so far.
memoryStream.Write(state.buffer, 0, bytesRead);
// Get the rest of the data.
client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReceiveCallback_onQuery), state);
}
else
{
// All the data has arrived; put it in response.
response_onQueryHistory = ByteArrayToObject(memoryStream.ToArray());
// Signal that all bytes have been received.
receiveDoneQuery.Set();
}
First of all, unless your dataReceivedQueue's type implements its own (or overrides Queue's) Enqueue method, your state.buffer would be rewritten with each client.BeginReceive call.
You can simply add a MemoryStream member to your StateObject and append bytes to it as they come:
state.rawData.Seek(0, SeekOrigin.End);
state.rawData.Write(state.buffer, 0, bytesRead);
First of all, you need to not only store the byte array, but also the number of bytes in the arrays that are actually valid. For example, each receive may not fully fill the buffer, thus the number of bytes is returned (bytesRead in your code).
If you had this, you could calculate the size of the final buffer by summing up the number of received bytes for each "batch".
After that you can - in a loop - use Array.Copy to copy a "batch" to a specified position with a specified length into the target array.
For example, this could look like this:
// Batch is a class that contains the batch byte buffer and the number of bytes valid
int destinationPos = 0;
byte[] destination = new byte[<number of bytes in total>];
foreach (Batch b in batches)
{
Array.Copy(b.Bytes, 0, destination, destinationPos, b.ValidLength);
}
I am using the server-client model for communicating with a hardware board using socket programing.
I receive data from board using "read()" method of "NetworkStream" class which reads a buffer with specified maximum size and returns the length of valid data in buffer. I have considered the maximum size of buffer with a enough big number.
The board sends a set of messages every 100ms. Each message consists a 2-byte constant header and a variable number of bytes as its data after the header bytes.
The problem is that I do not receive the messages one by one! Instead, I receive a buffer may contains 2 or 3 messages or one message is scattered between two buffer.
Currently, I am using a DFA which gather the content of messages using the constant header bytes (We do not know the length of messages, we just know the header bytes) but the problem is that the data bytes may contains the header bytes randomly !!
Is there any efficient way to gather the bytes of each message from buffers using any specific stream or class? How can I overcome to this problem?!
You need to add an additional buffer component between your consumer DFA and the socket client.
Whenever data is avaliable from the NetworkStream the buffer component will read it and append it to its own private buffer, incrementing an "available bytes" counter. The buffer component needs to expose at least the following functionality to its users:
a BytesAvailable property -- this returns the value of the counter
a PeekBytes(int count) method -- this returns the first count bytes of the buffer, if that much is available at least, and does not modify the counter or the buffer
a ReadBytes(int count) method -- as above, but it decrements the counter by count and removes the bytes read from the buffer so that subsequent PeekBytes calls will never read them again
Keep in mind that you don't need to be able to service an arbitrarily high count parameter; it is enough if you can service a count as long as the longest message it would be possible to receive at all times.
Obviously the buffer component needs to keep some kind of data structure that allows "wrapping around" of some kind; you might want to look into a circular (ring) buffer implementation, or you can just use two fixed buffers of size N where N is the length of the longest message and switch from one to the other as they become full. You should be careful so that you stop pulling in data from the NetworkStream if your buffers become full and only continue pulling after the DFA has called ReadBytes to free up some buffer space.
Whenever your DFA needs to read data, it will first ask your buffer stage how much data it has accumulated and then proceed accordingly. It would look something like this:
if BytesAvailable < 2
return; // no header to read, cannot do anything
// peek at the header -- do not remove it from the buffer!
header = PeekBytes(2);
// calculate the full message length based on the header
// if this is not possible from just the header, you might want to do this
// iteratively, or you might want to change the header so that it is possible
length = X;
if BytesAvailable < X
return; // no full message to read, cannot continue
header = ReadBytes(2); // to remove it from the buffer
message = ReadBytes(X); // remove the whole message as well
This way your DFA will only ever deal with whole messages.