We Keep Coding

buffer byte array size for NetworkStream.Read [duplicate]

I see several times in some code here for TCP communication the following line:
byte[] bytesFrom = new byte[10025];
Therefore I was wondering if this 10025 value has a special reason or if it is just arbitrary chosen.
Thanks

As far as I can tell, 10025 doesn't have any specific meaning. It's probably a result of random tweaking by someone who doesn't understand how to use buffers (hey, I received a 10000 B packet, I didn't expect that, let me increase the buffer size...).
Less arbitrary values would be:
Powers of two are often used, because they're quite handy in computing (which is based on binary numbers). So you'll often see buffer sizes like 256 or 4096.
65536 - Apart from being a power of two, it's also the maximum size of a TCP payload without window scaling (which can increase the possible payload size to a crazy value of 1 GiB - that's one big packet).
Actual known maximum size of the payload. This can be useful if the payload size is significantly smaller than the usual buffer sizes. For example, if you know that the largest payload you can receive is 100 B, you could use a byte array of 100 B, and you can even reuse it without issues (provided you don't reference the buffer anywhere, but you shouldn't really be doing that anyway).
1460 - This is usually the default TCP send buffer size (if you send anything less than this, TCP will wait for some time (say 200ms) before sending the "incomplete" buffer; this allows TCP to work relatively well if you're writing eg. individual bytes to the network stream without buffering them first). So sending a 4 kiB packet would mean that the first 2920 B would be sent immediately, while the remaining 1176 B would wait for the say 200ms "timeout". Not taking this into account can cause significant delays even though the network is actually not busy at all.
There's also some extra possible reasons in more specific environments. For example, on .NET, you may want to force the byte array to end up on the large object heap. While not entirely reliable, it should probably store objects larger than 85,000 B on the LOH (it would be nice if this could be enforced in some way). This can be handy if you really know what you're doing, especially if you need to keep pinned handles on the array (or its part), which is often the case in eg. asynchronous networking - pinned handles can cause significant issues on the main managed heap, because it relies on compaction to work (it always allocates on the end, while LOH has a table of free spaces).
On an even lower level, you might want to for example restrict the array size to fit well into CPU cache, or a single memory page to improve performance (this is also significant with .NET arrays, which store the array size at the beginning of the array - this means that when accessing the array's items, bounds checking will need to load the start of the array, instead of just the requested item). However, by the time you start with optimizations like this, you're probably a bit of a specialist :)
In other words, well chosen buffer size can be simply a good practice leading to less issues. In the end, though, it's all about profiling - if you find a performance problem in your networking code, a badly chosen buffer size is one of the possible culprits.
Also: WOW. So many google results on the new byte[10025] snippet. I wonder where that value originated, because it's obvious that a lot of people just blindly copied it without understanding it at all, best evidenced by snippets like this:
byte[] inStream = new byte[10025];
bufferSize = clientSocket.ReceiveBufferSize;
serverStream.Read(inStream, 0, bufferSize);
Why the hell would you allocate a 10 025 byte buffer and then only ever read ReceiveBufferSize bytes into it? Not to mention that if ReceiveBufferSize (which has nothing to do with the data being sent) is bigger than 10 025 B, you're possibly going to get an out of bounds error. If you care about ReceiveBufferSize at all (and you probably shouldn't), why not create a new byte[clientSocket.ReceiveBufferSize] buffer in the first place?

FileStream.Read() - bytes read

FileStream.Read() returns the amount of bytes read, but... is there any situation other than having reached the end of file, that it will read less bytes than the number of bytes requested and not throw an exception?
the documentation says:
The Read method returns zero only after reaching the end of the stream. Otherwise, Read always reads at least one byte from the stream before returning. If no data is available from the stream upon a call to Read, the method will block until at least one byte of data can be returned. An implementation is free to return fewer bytes than requested even if the end of the stream has not been reached.
But this doesn't quite explain in what situations data would be unavailable and cause the method to block until it can read again. I mean, shouldn't most situations where data is unavailable force an exception?
What are real situations where comparing the number of bytes read against the number of expected bytes could differ (assuming that we're already checking for end of file when we mention number of bytes expected)?
EDIT: A bit more information, reason why I'm asking this is because I've come across a bit of code where the developer pretty much did something like this:
bytesExpected = (remainingBytesInFile > 94208 ? 94208 : remainingBytesInFile
while (bytesRead < bytesExpected)
{
bytesRead += fileStream.Read(buffer, bytesRead, bytesExpected - bytesRead)
}
Now, I can't see any advantage to having this while at all, I'd expect it to throw an exception if it can't read the number of bytes expected (bearing in mind it's already taking into account that there are those many bytes left to read)
What would the reason one could possibly have for something like this? I'm sure I'm missing something

The documentation is for Stream.Read, from which FileStream is derived. Since FileStream is a stream, it should obey the stream contract. Not all streams do, but unless you have a very good reason, you should stick to that.
In a typical file stream, you'll only get a return value smaller than count when you reach the end of file (and it's a pretty simple way of checking for the end of file).
However, in a NetworkStream, for example, you keep reading in a loop until the method returns zero - signalling the end of stream. The same works for file streams - you know you're at the end of the file when Read returns zero.
Most importantly, FileStream isn't just for what you'd consider files - it's also for pseudo-files like standard input/output pipes and COM ports, for example (try opening a file stream on PRN, for example). In that case, you're not reading a file with a fixed length, and the behaviour is the same as with NetworkStream.
Finally, don't forget that FileStream isn't sealed. It's perfectly fine for you to implement a virtualized file system, for example - and it's perfectly fine if your virtualized file system doesn't support seeking, or checking the length of file.
EDIT:
To address your edit, this is exactly how you're supposed to read any stream. Nothing wrong with it. If there's nothing else to read in a stream, the Read method will simply return 0, and you know the stream is over. The only thing is, it seems that he tries to fill his buffer to full, one buffer at a time - this only makes sense if you explicitly need to partition the file by 94208 bytes, and pass that byte[] for further processing somewhere.
If that's not the case, you don't really need to fill the full buffer - you just keep reading (and probably writing on some other side) until Read returns 0. And indeed, by default, FileStream will always fill the whole buffer unless it's built around a pipe handle - but since that's a possibility, you shouldn't rely on the "real file" behaviour, so as long as you need those byte[] for something non-stream (e.g. parsing messages), this is entirely fine. If you're only using the stream as an actual stream, and you're streaming the data somewhere else, it doesn't have a point, really - you only need one while to read the file.

Your expectations would only apply to the case when the stream is reading data off of a no-latency source. Other I/O sources can be slow, which is why the Read method might will not always be able to return immediately. That doesn't mean that there is an error (so no exception), just that it has to wait for data to arrive.
Examples: network stream, file stream on slow disk, etc.
(UPDATE, HDD example) To give an example specific to files (since your case is FileStream, although Read is defined on Stream and so all implementations should fulfill the requirements): mechanical hard-drives go to "sleep" when not active (specially on battery-powered devices, read laptops). Spinning up can take a second or so. That is not an IOException, but your read would have to wait for a second before any data is read.

Simple answer is that on a FileStream it probably never happens.
However keep in mind that the Read method is inherited from Stream which serves as base for many other streams like NetworkStream and in this case you may not be able to read has many bytes as you requested simple because they havent been received from the network yet.
So like the documentation says it all depends on the implementation of the specific type of stream - FileStream, NetworkStream, etc.

Why write to Stream in chunks?

I am wondering why so many examples read byte arrays into streams in chucks and not all at once... I know this is a soft question, but I am interested.
I understand a bit about hardware and filling buffers can be very size dependent and you wouldn't want to write to the buffer again until it has been flushed to wherever it needs to go etc... but with the .Net platform (and other modern languages) I see examples of both. So when use which and when, or is the second an absolute no no?
Here is the thing (code) I mean:
var buffer = new byte[4096];
while (true)
{
var read = this.InputStream.Read(buffer, 0, buffer.Length);
if (read == 0)
break;
OutputStream.Write(buffer, 0, read);
}
rather than:
var buffer = new byte[InputStream.Length];
var read = this.InputStream.Read(buffer, 0, buffer.Length);
OutputStream.Write(buffer, 0, read);
I believe both are legal? So why go through all the fuss of the while loop (in whatever for you decide to structure it)?
I am playing devils advocate here as I want to learn as much as I can :)

In the first case, all you need is 4kB of memory. In the second case, you need as much memory as the input stream data takes. If the input stream is 4GB, you need 4GB.
Do you think it would be good if a file copy operation required 4GB of RAM? What if you were to prepare a disk image that's 20GB?
There is also this thing with pipes. You don't often use them on Windows, but a similar case is often seen on other operating systems. The second case waits for all data to be read, and only then writes them to the output. However, sometimes it is advisable to write data as soon as possible—the first case will start writing to the output stream as soon as the first 4kB of input is read. Think of serving web pages: it is advisable for a web server to send data as soon as possible, so that client's web browser will start rendering headers and first part of the content, not waiting for the whole body.
However, if you know that the input stream won't be bigger than 4kB, then both cases are equivalent.

Sometimes, InputStream.Length is not valid for some source, e.g from the net transport, or the buffer maybe huge, e.g read from a huge file. IMO.

It protects you from the situation where your input stream is several gigabytes long.

You have no idea how much data Read might return. This could create major performance problems if you're reading a very large file.
If you have control over the input, and are sure the size is reasonable, then you can certainly read the whole array in at once. But be especially careful if the user can supply an arbitrary input.

FileStream.Seek vs. Buffered Reading

Motivated by this answer I was wondering what's going on under the curtain if one uses lots of FileStream.Seek(-1).
For clarity I'll repost the answer:
using (var fs = File.OpenRead(filePath))
{
fs.Seek(0, SeekOrigin.End);
int newLines = 0;
while (newLines < 3)
{
fs.Seek(-1, SeekOrigin.Current);
newLines += fs.ReadByte() == 13 ? 1 : 0; // look for \r
fs.Seek(-1, SeekOrigin.Current);
}
byte[] data = new byte[fs.Length - fs.Position];
fs.Read(data, 0, data.Length);
}
Personally I would have read like 2048 bytes into a buffer and searched that buffer for the char.
Using Reflector I found out that internally the method is using SetFilePointer.
Is there any documentation about windows caching and reading a file backwards? Does Windows buffer "backwards" and consult the buffer when using consecutive Seek(-1) or will it read ahead starting from the current position?
It's interesting that on the one hand most people agree with Windows doing good caching, but on the other hand every answer to "reading file backwards" involves reading chunks of bytes and operating on that chunk.

Going forward vs backward doesn't usually make much difference. The file data is read into the file system cache after the first read, you get a memory-to-memory copy on ReadByte(). That copy isn't sensitive to the file pointer value as long as the data is in the cache. The caching algorithm does however work from the assumption that you'd normally read sequentially. It tries to read ahead, as long as the file sectors are still on the same track. They usually are, unless the disk is heavily fragmented.
But yes, it is inefficient. You'll get hit with two pinvoke and API calls for each individual byte. There's a fair amount of overhead in that, those same two calls could also read, say, 65 kilobytes with the same amount of overhead. As usual, fix this only when you find it to be a perf bottleneck.

Here is a pointer on File Caching in Windows
The behavior may also depends on where physically resides the file (hard disk, network, etc.) as well as local configuration/optimization.
An also important source of information is the CreateFile API documentation: CreateFile Function
There is a good section named "Caching Behavior" that tells us at least how you can influence file caching, at least in the unmanaged world.

When to use StreamReader.ReadBlock()?

I would like to know of a situation Read(char[],int,int) fails to return all chars requested while ReadBlock() returns all chars as expected (say when StreamReader works with an instance of a FileStream object).

In practice, when using a StreamReader it is only likely to happen with a stream which may delay for some time - e.g. a network stream as people have mentioned here.
However, with a TextReader generally, you can expect it to happen at any time (including possibly with a future version of .NET in a case where it doesn't currently happen - that it doesn't happen with StreamReader backed on FileStream isn't documented, so there's no guarantee it won't happen in the future).
In particular there are a lot of cases where it's easier (with a knock on effect of simpler, more reliable and probably more efficient code) for the implementer to not return the requested amount if they can partially fulfil the call with just emptying the current buffer, or by using the amount requested as the amount to pass to a backing source (a stream or another TextReader) before doing an operation that could only return a lower number of characters.
Now, to answer the actual question as to "When to use StreamReader.ReadBlock()?" (or more generally, when to use TextReader.ReadBlock()). The following should be borne in mind:
Both Read() and ReadBlock() are guaranteed to return at least one character unless the entire source has been read. Neither will return 0 if there is pending content.
Calling ReadBlock() when Read() will do is wasteful, as it loops needlessly.
But on the other hand it's not that wasteful.
But on the third hand, the cases where Read() will return fewer than requested characters are often cases where another thread is engaged in getting the content that will fill the buffer for the next call, or where that content doesn't exist yet (e.g. user input or a pending operation on another machine) - there's better overall concurrency to be found in processing the partial result and then calling Read() again when that's finished.
So. If you can do something useful with a partial result, then call Read() and work on what you get. In particular if you are looping through and working on the result of each Read() then do this rather than with ReadBlock().
A notable case is if you are building your own TextReader that is backed by another. There's no point calling ReadBlock() unless the algorithm really needs a certain number of characters to work - just return as much as you can from a call to Read() and let the calling code call ReadBlock() if it needs to.
In particular, note that the following code:
char buffer = char[4096];
int len = 0;
while((len = tr.ReadBlock(buffer, 0 , 4096)) != 0)
DoSomething(buffer, 0, len);
Can be rewritten as:
char buffer = char[4096];
for(int len = tr.Read(buffer, 0, 4096); len != 0; len = tr.Read(buffer, 0, 4096))
DoSomething(buffer, 0, len);
It may call DoSomething() with smaller sizes sometimes, but it can also have better concurrency if there's another thread involved in providing the data for the next call to Read().
However, the gain is not major in most cases. If you really need a certain number of characters, then do call ReadBlock(). Most importantly in those cases where Read() will have the same result as ReadBlock() the overhead of ReadBlock() checking that it has done so is very slight. Don't try to second-guess whether Read() is safe or not in a given case; if it needs the guarantee of ReadBlock() then use ReadBlock().

It's a major issue on network streams particularly if the Streamed data is quite large and the server providing the Stream does so as chunked output (i.e. quite typical with HTTP) since a call to just Read() i.e. the single character read gives -1 once the EOS is reached, calls to Read(char[], int, int) with arguments gets up to the number of characters asked for or less if the EOS is reached and then returns the number of characters read or zero if the EOS has been reached
Whereas ReadBlock() waits for data to be available from the Stream so you never run into this issue.
One other thing to note is that both forms reads a maximum number of characters and are not guaranteed to return that many characters if there aren't that many available
I asked a question on a similar topics some time ago - Reading from a HttpResponseStream fails - when I had issues reading from HTTP streams