We Keep Coding

How to copy a Stream from the begining irrespective its current position

I got a file stream which has content read from a disk.
Stream input = new FileStream("filename");
This stream is to be passed to a third party library which after reading the stream, keeps the Stream's position pointer at the end of the file (as ususal).
My requirement is not to load the file from the desk everytime, instead I want to maintain MemoryStream, which will be used everytime.
public static void CopyStream(Stream input, Stream output)
{
byte[] buffer = new byte[32768];
int read;
while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
{
output.Write(buffer, 0, read);
}
}
I have tried the above code. It works for the first very time to copy the input stream to output stream, but subsequent calls to CopyStream will not work as the source's Position will be at the end of the stream after the first call.
Are there other alternatives which copy the content of the source stream to another stream irrespective of the source stream's current Position.
And this code needs to run in thread safe manner in a multi threaded environment.

You can use .NET 4.0 Stream.CopyTo to copy your steam to a MemoryStream. The MemoryStream has a Position property you can use to move its postition to the beginning.
var ms = new MemoryStream();
using (Stream file = File.OpenRead(#"filename"))
{
file.CopyTo(ms);
}
ms.Position = 0;
To make a thread safe solution, you can copy the content to a byte array, and make a new MemoryStream wrapping the byte array for each thread that need access:
byte[] fileBytes = ms.ToArray();
var ms2 = new MemoryStream(fileBytes);

You should check the input stream's CanSeek property. If that returns false, you can only read it once anyway. If CanSeek returns true, you can set the position to zero and copy away.
if (input.CanSeek)
{
input.Position = 0;
}
You may also want to store the old position and restore it after copying.
ETA: Passing the same instance of a Stream around is not the safest thing to do. E.g. you can't be sure the Stream wasn't disposed when you get it back. I'd suggest to copy the FileStream to a MemoryStream in the beginning, but only store the byte content of the latter by calling ToArray(). When you need to pass a Stream somewhere, just create a new one each time with new MemoryStream(byte[]).

C# TCP NetworkStream missing a few bytes of data

I'm having two problems and after trying a few techniques I've read on stackoverflow, the problem persists. I'm trying to send a file from the server to client with the following code below but the problem is that the file is always a few bytes short, causing file corruption.. The second problem is that the stream doesn't close despite implementing a zero length packet at the end to indicate the transfer is finished without closing the connection.
Server code snippet:
/*
* Received request from client for file, sending file to client.
*/
//open file to send to client
FileStream fs = new FileStream(fileLocation, FileMode.Open, FileAccess.Read);
byte[] data = new byte[1024];
long fileSize = fs.Length;
long sent = 0;
int count = 0;
while (sent < fileSize)
{
count = fs.Read(data, 0, data.Length);
netStream.Write(data, 0, count);
sent += count;
}
netStream.Write(new byte[1024], 0, 0); //send zero length byte stream to indicate end of file.
fs.Flush();
netStream.Flush();
Client code snippet:
TcpClient client;
NetworkStream serverStream;
/*
* [...] client connect
*/
//send request to server for file
byte[] dataToSend = SerializeObject(obj);
serverStream.Write(dataToSend, 0, dataToSend.Length);
//create filestream to save file
FileStream fs = new FileStream(fileName, FileMode.Create, FileAccess.Write);
//handle response from server
byte[] response = new byte[client.ReceiveBufferSize];
byte[] bufferSize = new byte[1024];
int bytesRead;
while ((bytesRead = serverStream.Read(bufferSize, 0, bufferSize.Length)) > 0 && client.ReceiveBufferSize > 0)
{
Debug.WriteLine("Bytes read: " + bytesRead);
fs.Write(response, 0, bytesRead);
}
fs.Close();

With UDP you can transmit an effectively empty packet, but TCP won't allow you to do that. At the application layer the TCP protocol is a stream of bytes, with all of the packet-level stuff abstracted away. Sending zero bytes will not result in anything happening at the stream level on the client side.
Signalling the end of a file transfer can be as simple as having the server close the connection after sending the last block of data. The client will receive the final data packet then note that the socket has been closed, which indicates that the data has been completely delivered. The flaw in this method is that the TCP connection can be closed for other reasons, leaving a client in a state where it believes that it has all the data even though the connection was dropped for another reason.
So even if you are going to use the 'close on complete' method to signal end of transfer, you need to have a mechanism that allows the client to identify that the file is actually complete.
The most common form of this is to send a header block at the start of the transfer that tells you something about the data being transferred. This might be as simple as a 4-byte length value, or it could be a variable-length descriptor structure that includes various metadata about the file such as its length, name, create/modify times and a checksum or hash that you can use to verify the received content. The client reads the header first, then processes the rest of the data in the stream as content.
Let's take the simplest case, sending a 4-byte length indicator at the start of the stream.
Server Code:
public void SendStream(Socket client, Stream data)
{
// Send length of stream as first 4 bytes
byte[] lenBytes = BitConverter.GetBytes((int)data.Length);
client.Send(lenBytes);
// Send stream data
byte[] buffer = new byte[1024];
int rc;
data.Position = 0;
while ((rc = data.Read(buffer, 0, 1024)) > 0)
client.Send(buffer, rc, SocketFlags.None);
}
Client Code:
public bool ReceiveStream(Socket server, Stream outdata)
{
// Get length of data in stream from first 4 bytes
byte[] lenBytes = new byte[4];
if (server.Receive(lenBytes) < 4)
return false;
long len = (long)BitConverter.ToInt32(lenBytes, 0);
// Receive remainder of stream data
byte[] buffer = new byte[1024];
int rc;
while ((rc = server.Receive(buffer)) > 0)
outdata.Write(buffer, 0, rc);
// Check that we received the expected amount of data
return len == outdata.Position;
}
Not much in the way of error checking and so on, and blocking code in all directions, but you get the idea.

There is no such thing as sending "zero bytes" in a stream. As soon as the stream sees you're trying to send zero bytes it can just return immediately and will have done exactly what you asked.
Since you're using TCP, it is up to you to use an agreed-upon protocol between the client and server. For example:
The server could close the connection after sending all its data. The client would see this as a "Read" that completes with zero bytes returned.
The server could send a header of a fixed size (maybe 4 bytes) that includes the length of the upcoming data. The client could then read those 4 bytes and would then know how many more bytes to wait for.
Finally, you might need a "netStream.Flush()" in your server code above (if you intended to keep the connection open).

File upload with c# and streaming

Looking at the different ways to upload a file in .NET, e.g. HttpPostedFile, and using a HttpHandler, I'm trying to understand how the process works in a bit more details.
Specifically how it writes the information to a file.
Say I have the following:
HttpPostedFile file = context.Request.Files[0];
file.SaveAs("c:\temp\file.zip");
The actual file does not get created until the full stream seems to be processed.
Similarly:
using (Stream output = File.OpenWrite("c:\temp\file.zip"))
using (Stream input = file.InputStream)
{
byte[] buffer = new byte[8192];
int bytesRead;
while ((bytesRead = input.Read(buffer, 0, buffer.Length)) > 0)
{
output.Write(buffer, 0, bytesRead);
}
}
I would have thought that this would "progressively" write the file as it reads the stream. Looking at the filesystem, it does not seems to do this at all. If I breakpoint inside the while, it does though.
What I'm trying to do, is have it so you upload a file (using a javascript uploader), and poll alongside, whereby the polling ajax request tries to get the fileinfo(file size) of the uploaded file every second. However, it always returns 0 until the upload is complete.
Vimeo seems to be able to do this type of functionality (for IE)?? Is this a .NET limitation, or is there a way to progressively write the file from the stream?

Two points:
First, in Windows, the displayed size of a file is not updated constantly. The file might indeed be growing continually, but the size only increases once.
Second (more likely in this case), the stream might not be flushing to the disk. You could force it to by adding output.Flush() after the call to output.Write(). You might not want to do that, though, since it will probably have a negative impact on performance.
Perhaps you could poll the Length property of the output stream directly, instead of going through the file system.
EDIT:
To make the Length property of the stream accessible to other threads, you could have a field in your class and update it with each read/write:
private long _uploadedByteCount;
void SomeMethod()
{
using (Stream output = File.OpenWrite("c:\temp\file.zip"))
using (Stream input = file.InputStream)
{
byte[] buffer = new byte[8192];
int bytesRead;
while ((bytesRead = input.Read(buffer, 0, buffer.Length)) > 0)
{
output.Write(buffer, 0, bytesRead);
Interlocked.Add(ref _uploadedByteCount, bytesRead);
}
}
}
public long GetUploadedByteCount()
{
return _uploadedByteCount;
}

Under what conditions does a NetworkStream not read in all the data at once?

In the callback for NetworkStream.BeginRead I seem to notice that all bytes are always read. I see many tutorials check to see if the BytesRead is less than the total bytes and if so, read again, but this never seems to be the case.
The condition if (bytesRead < totalBytes) never fires, even if a lot of data is sent at once (thousands of characters) and even if the buffer size is set to a very small value (16 or so).
I have not tested this with the 'old-fashioned way' as I am using Task.Factory.FromAsync instead of calling NetworkStream.BeginRead and providing a callback where I call EndRead. Perhaps Tasks automatically include this functionality of not returning until all data is read? I'm not sure.
Either way, I am still curious as to when all data would not be read at once. Is it even required to check if not all data was read, and if so, read again? I cannot seem to get the conditional to ever run.
Thanks.

Try sending megabytes of data over a slow link. Why would the stream want to wait until it was all there before giving the caller any of it? What if the other side hadn't closed the connection - there is no concept of "all the data" at that point.
Suppose you open a connection to another server and call BeginRead (or Read) with a large buffer, but it only sends 100 bytes, then waits for your reply - what would you expect NetworkStream to do? Never give you the data, because you gave it too big a buffer? That would be highly counterproductive.
You should absolutely not assume that any stream (with the arguable exception of MemoryStream) will fill the buffer you give it. It's possible that FileStream always will for local files, but I'd expect it not to for shared files.
EDIT: Sample code which shows the buffer not being filled - making an HTTP 1.1 request (fairly badly :)
// Please note: this isn't nice code, and it's not meant to be. It's just quick
// and dirty to demonstrate the point.
using System;
using System.IO;
using System.Net;
using System.Net.Sockets;
using System.Text;
class Test
{
static byte[] buffer;
static void Main(string[] arg)
{
TcpClient client = new TcpClient("www.yoda.arachsys.com", 80);
NetworkStream stream = client.GetStream();
string text = "GET / HTTP/1.1\r\nHost: yoda.arachsys.com:80\r\n" +
"Content-Length: 0\r\n\r\n";
byte[] bytes = Encoding.ASCII.GetBytes(text);
stream.Write(bytes, 0, bytes.Length);
stream.Flush();
buffer = new byte[1024 * 1024];
stream.BeginRead(buffer, 0, buffer.Length, ReadCallback, stream);
Console.ReadLine();
}
static void ReadCallback(IAsyncResult ar)
{
Stream stream = (Stream) ar.AsyncState;
int bytesRead = stream.EndRead(ar);
Console.WriteLine(bytesRead);
Console.WriteLine("Asynchronous read:");
Console.WriteLine(Encoding.ASCII.GetString(buffer, 0, bytesRead));
string text = "Bad request\r\n";
byte[] bytes = Encoding.ASCII.GetBytes(text);
stream.Write(bytes, 0, bytes.Length);
stream.Flush();
Console.WriteLine();
Console.WriteLine("Synchronous:");
StreamReader reader = new StreamReader(stream);
Console.WriteLine(reader.ReadToEnd());
}
}

Stream with openWrite not writing until it is closed

I have a stream writer that opens using a WebClient.OpenWrite call. For this simplified case, assume that reader is reading a multiple of dataChunkSize.
using (Stream writer = myWebClient.OpenWrite(myURIString)
{
using (FileStream reader = new FileStream(myFileName, FileMode.Open, FileAccess.Read)
{
for(int i = 0; i < reader.Length; i += dataChunkSize)
{
byte[] data = new byte[dataChunkSize];
reader.Read(data, 0, dataChunkSize);
writer.Write(data, 0, dataChunkSize);
}
reader.Close();
reader.Dispose();
}
writer.Close();
writer.Dispose();
}
My data is the size of 2 dataChunkSizes. However, it does not send any data (no data is received) until the writer.Close() call is called, where it only sends the first dataChunkSize worth of data...the second dataChunkSize of data is never sent.
How can I get it to send after every Write call? I tried adding writer.Flush() but this did not help.
Thanks.

I think your issue is because your last chunk is perhaps not the full length you expect (dataChunkSize). Also, I would add Flush to force each write there and then if needed (thou I am not sure if the flush will work). Try changing you for loop contents to this...
byte[] data = new byte[dataChunkSize];
int bytesRead = reader.Read(data, 0, dataChunkSize);
writer.Write(data, 0, bytesRead);
writer.Flush();

I guess the write is buffered. It won't write until the buffer is full or the writer is closed.

WebClient may use inner buffered stream (network etc.).
And about the reader. It can read less.
So better use
int amountRead = reader.Read(data, 0, dataChunkSize);
writer.Write(data, 0, amountRead);
i += amountRead;

You will need to call the stream.Flush() method if you want to write before closing