I've been banging my head around this for some while (and know it's something silly).
I'm downloading files with a ProgressBar which shows fine, but how do I get the data from the ReadAsync Stream to save?
public static readonly int BufferSize = 4096;
int receivedBytes = 0;
int totalBytes = 0;
WebClient client = new WebClient();
byte[] result;
using (var stream = await client.OpenReadTaskAsync(urlToDownload))
{
byte[] buffer = new byte[BufferSize];
totalBytes = Int32.Parse(client.ResponseHeaders[HttpResponseHeader.ContentLength]);
for (;;)
{
result = new byte[stream.Length];
int bytesRead = await stream.ReadAsync(buffer, 0, buffer.Length);
if (bytesRead == 0)
{
await Task.Yield();
break;
}
receivedBytes += bytesRead;
if (progessReporter != null)
{
DownloadBytesProgress args =
new DownloadBytesProgress(urlToDownload, receivedBytes, totalBytes);
progessReporter.Report(args);
}
}
}
I was trying via the result var, but that is obviously wrong. I'd appreciate any pointers on this long Sunday afternoon.
The content that was downloaded is inside your byte[] buffer variable:
int bytesRead = await stream.ReadAsync(buffer, 0, buffer.Length);
From Stream.ReadAsync:
buffer:
Type: System.Byte[]
The buffer to write the data into.
You never use your result variable at all. Not sure why its there.
Edit
So the problem is how to read the full content of your stream. You can do the following:
public static readonly int BufferSize = 4096;
int receivedBytes = 0;
WebClient client = new WebClient();
using (var stream = await client.OpenReadTaskAsync(urlToDownload))
using (MemoryStream ms = new MemoryStream())
{
var buffer = new byte[BufferSize];
int read = 0;
totalBytes = Int32.Parse(client.ResponseHeaders[HttpResponseHeader.ContentLength]);
while ((read = await stream.ReadAsync(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, read);
receivedBytes += read;
if (progessReporter != null)
{
DownloadBytesProgress args =
new DownloadBytesProgress(urlToDownload, receivedBytes, totalBytes);
progessReporter.Report(args);
}
}
return ms.ToArray();
}
}
The data you read should be in the buffer array. Actually the beginning bytesRead bytes of the array. Check ReadAsync method on MSDN.
Related
When I send a file with the code below, some data (small amout) is missing. The file size doess not match on the receiver side. Sending a regular string is fine so theres no connection issue here. Im just looking for a minimal improvement to fix the issue, I will add error checking etc later. Thanks! The code is mostly copied from some tutorial but i dont remember which though...
Client is the std .Net TcpClient class
Client.Client is it's socket
public void SendFile2(string fileName)
{
using (FileStream fs = File.OpenRead(fileName))
{
byte[] lenBytes = BitConverter.GetBytes((int)fs.Length);
Client.Client.Send(lenBytes);
byte[] buffer = new byte[1024];
int bytesRead;
fs.Position = 0;
while ((bytesRead = fs.Read(buffer, 0, 1024)) > 0)
Client.Client.Send(buffer, bytesRead, SocketFlags.None);
}
}
public bool ReceiveFile2(string fileName)
{
using (FileStream fs = File.Create(fileName))
{
byte[] lenBytes = new byte[4];
if (Client.Client.Receive(lenBytes) < 4)
return false;
long len = BitConverter.ToInt32(lenBytes, 0);
byte[] buffer = new byte[1024];
int bytesRead;
while ((bytesRead = Client.Client.Receive(buffer)) > 0)
fs.Write(buffer, 0, bytesRead);
return len == fs.Position;
}
}
SOLUTION:
public void SendFile(string fileName)
{
using (FileStream fs = File.OpenRead(fileName))
{
byte[] lenBytes = BitConverter.GetBytes((int)fs.Length);
Client.Client.Send(lenBytes);
byte[] buffer = new byte[1024];
int bytesRead;
fs.Position = 0;
while ((bytesRead = fs.Read(buffer, 0, 1024)) > 0)
Client.Client.Send(buffer, bytesRead, SocketFlags.None);
}
}
public bool ReceiveFile(string fileName)
{
using (FileStream fs = File.Create(fileName))
{
byte[] lenBytes = new byte[4];
if (Client.Client.Receive(lenBytes) < 4)
return false;
long len = BitConverter.ToInt32(lenBytes, 0);
byte[] buffer = new byte[1024];
int bytesRead;
// Changed from here
while (fs.Position < len)
{
bytesRead = Client.Client.Receive(buffer);
fs.Write(buffer, 0, bytesRead);
}
// To here
return len == fs.Position;
}
}
I think this line can be a problem.
if (Client.Client.Receive(lenBytes) < 4)
and
while ((bytesRead = Client.Client.Receive(buffer)) > 0)
You have two receives in your code.
So you drop first bytes.
That can explain the differences you see in files sizes.
I need to download a binary file and access the raw data as it arrives.
private void Downloadfile(string url)
{
WebClient client = new WebClient();
client.DownloadDataCompleted += DownloadDataCompleted;
client.DownloadProgressChanged += DownloadProgressCallback;
client.DownloadDataAsync(new Uri(url));
}
public void DownloadProgressCallback(object sender, DownloadProgressChangedEventArgs e)
{
long bytes = e.BytesReceived;
long total = e.TotalBytesToReceive;
int progress = e.ProgressPercentage;
string userstate = (string)e.UserState;
byte[] received = ?
}
Alternatively, writing to a stream would also be helpful. I don't mind using another download method either, the primary goal is to read a download on the fly.
You can use the WebClient.OpenRead as suggested by Søren Lorentzen
using (var client = new WebClient())
using (var stream = client.OpenRead(address))
{
byte[] readBuffer = new byte[4096];
int totalBytesRead = 0;
int bytesRead;
while ((bytesRead = stream.Read(readBuffer, totalBytesRead, readBuffer.Length - totalBytesRead)) > 0)
{
totalBytesRead += bytesRead;
if (totalBytesRead == readBuffer.Length)
{
int nextByte = stream.ReadByte();
if (nextByte != -1)
{
byte[] temp = new byte[readBuffer.Length];
Buffer.BlockCopy(readBuffer, 0, temp, 0, readBuffer.Length);
Buffer.SetByte(temp, totalBytesRead, (byte)nextByte);
readBuffer = temp;
totalBytesRead++;
}
}
}
}
}
At my company we've run into an issue that we cannot seem to explain. We're reading from a NetworkStream using a buffer of 2048 bytes. When both the client and server use that buffersize to write and read with, it's hideously slow (1 second extra on top of other processing times). If we use a buffersize that's very large, like 32k everything works really fast. The message we're sending is usually much larger than 32k so the entire message does not entirely fit in that buffer. The way we're reading is shown in the code below in the HandleDataLargeBuffer method.
To build a testcase for this issue I've written the code below, and I've posted my output below that.
My questions:
Any idea why a larger buffer size would be faster (less read calls but those shouldn't take too much time)?
In the code below, HandleDataVariableBuffer is much much faster than the other methods of reading, why?
Which is the best way of reading anyway?
Using a Proxy class to simulate slowdowns, pasted below the main file.
class Program
{
private StreamWriter Writer;
private int Count = 0;
static void Main(string[] args)
{
new Program();
}
public Program()
{
Writer = new StreamWriter(new FileStream("C:\\test.txt", FileMode.Create));
Start();
}
public void Start()
{
Proxy p = new Proxy();
new Thread(p.Start).Start();
Thread.Sleep(500);
new Thread(SetupServer).Start();
Thread.Sleep(1000); // Wait for TCP Server setup.
for (int i = 0; i != 3; ++i)
{
TcpClient client = new TcpClient("127.0.0.1", 50001);
Send(client.GetStream());
client.Close();
Thread.Sleep(1000);
}
WriteLine("Tests done.");
Console.ReadLine();
Writer.Close();
}
private void SetupServer()
{
WriteLine("[Server] Starting server.");
TcpListener listener = new TcpListener(IPAddress.Any, 50000);
listener.Start();
WriteLine("[Server] Started listening on port 50000.");
while (true) // We'll just forcibly end, obviously you'd use the callback methods for this normally.
{
TcpClient client = listener.AcceptTcpClient();
WriteLine(String.Format("[Server] Accepted client with IP: {0}", client.Client.RemoteEndPoint.ToString()));
new Thread(HandleClient).Start(client);
}
}
private void HandleClient(object argument)
{
TcpClient client = (TcpClient)argument;
NetworkStream stream = client.GetStream();
// Now there are multiple ways to handle this data, however first we read the int at the start.
byte[] length = new byte[4];
stream.Read(length, 0, 4);
int lengthInt = BitConverter.ToInt32(length, 0);
if (lengthInt <= 0) return; // Shouldn't happen.
// Test data read in multiple ways.
Stopwatch watch = new Stopwatch();
watch.Start();
string handler = "";
if (Count == 0)
{
handler = "LargeBuffer";
HandleDataLargeBuffer(lengthInt, stream);
++Count;
}
else if (Count == 1)
{
handler = "SmallBuffer";
HandleDataSmallBuffer(lengthInt, stream);
++Count;
}
else if (Count == 2)
{
handler = "VariableBuffer";
HandleDataVariableBuffer(lengthInt, stream);
Count = 0;
}
watch.Stop();
WriteLine(String.Format("\t[Server] [{3}] Read {0} bytes from client {1} in {2} ms", lengthInt.ToString(), client.Client.RemoteEndPoint.ToString(), watch.ElapsedMilliseconds.ToString(), handler));
}
private void HandleDataLargeBuffer(int length, NetworkStream stream)
{
int read = 0;
int totalRead = 0;
MemoryStream dataBuffer = new MemoryStream(); // I'm writing to a memory stream because in my real life situation I write (stream) to another NetworkStream
byte[] buffer = new byte[8192 * 4];
while ((read = stream.Read(buffer, 0, 8192 * 4)) != 0 && totalRead < length)
{
totalRead += read;
dataBuffer.Write(buffer, 0, read);
}
}
private void HandleDataSmallBuffer(int length, NetworkStream stream)
{
int read = 0;
int totalRead = 0;
MemoryStream dataBuffer = new MemoryStream(); // I'm writing to a memory stream because in my real life situation I write (stream) to another NetworkStream
byte[] buffer = new byte[512];
while ((read = stream.Read(buffer, 0, 512)) != 0 && totalRead < length)
{
totalRead += read;
dataBuffer.Write(buffer, 0, read);
}
}
private void HandleDataVariableBuffer(int length, NetworkStream stream)
{
int read = 0;
int totalRead = 0;
MemoryStream dataBuffer = new MemoryStream(); // I'm writing to a memory stream because in my real life situation I write (stream) to another NetworkStream
while (totalRead < length)
{
byte[] buffer = new byte[length - totalRead]; // You'd obviously do some additional checks on 'length' to make sure no foul play is involved.
read = stream.Read(buffer, 0, length - totalRead);
totalRead += read;
dataBuffer.Write(buffer, 0, read);
}
}
public void Send(NetworkStream stream)
{
// Generate some random data
Random r = new Random();
byte[] buf = new byte[1024 * 420]; // Buffer of 420k to simulate a decently large message
byte[] length = BitConverter.GetBytes(1024 * 420);
r.NextBytes(buf);
// Create the total message array: [Length][Message]
byte[] totalMessage = new byte[4 + 1024 * 420];
System.Buffer.BlockCopy(length, 0, totalMessage, 0, 4);
System.Buffer.BlockCopy(buf, 0, totalMessage, 4, buf.Length);
// Use a memory stream for ease of use
Stopwatch watch = new Stopwatch();
watch.Start();
using (MemoryStream memStream = new MemoryStream(totalMessage))
{
int read = -1;
byte[] buffer = new byte[8192];
while ((read = memStream.Read(buffer, 0, 8192)) != 0)
{
stream.Write(buffer, 0, read);
}
}
stream.Flush();
watch.Stop();
WriteLine("[Send] Took " + watch.ElapsedMilliseconds + " ms to send " + totalMessage.Length + " bytes of data.");
}
public void WriteLine(string str)
{
Writer.WriteLine(str);
Writer.Flush();
}
}
Proxy.cs
public class Proxy
{
public void Start()
{
TcpListener listener = new TcpListener(IPAddress.Any, 50001);
listener.Start();
while (true)
{
TcpClient client = listener.AcceptTcpClient();
new Thread(HandleClient).Start(client);
}
}
private void HandleClient(object argument)
{
TcpClient client = (TcpClient)argument;
Thread.Sleep(1000);
NetworkStream clientStream = client.GetStream();
int read = 0;
byte[] length = new byte[4];
clientStream.Read(length, 0, 4);
int lengthInt = BitConverter.ToInt32(length, 0);
if (lengthInt <= 0)
{
client.Close();
return;
}
TcpClient serverClient = new TcpClient("127.0.0.1", 50000);
NetworkStream serverStream = serverClient.GetStream();
int totalRead = 0;
serverStream.Write(length, 0, 4);
while (totalRead < lengthInt)
{
Thread.Sleep(50);
byte[] buffer = new byte[lengthInt - totalRead]; // You'd obviously do some additional checks on 'length' to make sure no foul play is involved.
read = clientStream.Read(buffer, 0, lengthInt - totalRead);
totalRead += read;
serverStream.Write(buffer, 0, read);
}
serverStream.Flush();
Thread.Sleep(1000);
client.Close();
serverClient.Close();
}
}
I'm sorry for the heap of code posted, when I first started I hoped it'd be smaller. Here's the output on my machine.
[Server] Starting server.
[Server] Started listening on port 50000.
[Server] Accepted client with IP: 127.0.0.1:3985
[Send] Took 1047 ms to send 430084 bytes of data.
[Server] [LargeBuffer] Read 430080 bytes from client 127.0.0.1:3985 in 1148 ms
[Server] Accepted client with IP: 127.0.0.1:3987
[Send] Took 1049 ms to send 430084 bytes of data.
[Server] [SmallBuffer] Read 430080 bytes from client 127.0.0.1:3987 in 1150 ms
[Server] Accepted client with IP: 127.0.0.1:3989
[Send] Took 1051 ms to send 430084 bytes of data.
[Server] [VariableBuffer] Read 430080 bytes from client 127.0.0.1:3989 in 100 ms
Tests done.
//convert photo to baos
var memoryStream = new System.IO.MemoryStream();
e.ChosenPhoto.CopyTo(memoryStream);
//string baos = memoryStream.ToString();
byte[] result = memoryStream.ToArray();
String base64 = System.Convert.ToBase64String(result);
String post_data = "&image=" + base64;
...
wc.UploadStringAsync(imgur_api,"POST",post_data);
I am using this code to upload an image to the Imgur API v3 using WebClient. The image being selected is either one of the 7 photos provided by the Windows Phone 7.1 emulator, or the simulated camera images. When I try to load the images, they are a largely-grey corrupted mess. Am I generating the base64 properly and/or do I need to render a Bitmap of the picture first before creating the byte[] and base64?
Thanks in advance!
Use something like Uri.EscapeDataString to escape the data so that special URL characters are not interpreted.
I use this
private void PhotoChooserTaskCompleted(object sender, PhotoResult e)
{
if (e.TaskResult != TaskResult.OK) return;
var bimg = new BitmapImage();
bimg.SetSource(e.ChosenPhoto);
var sbytedata = ReadToEnd(e.ChosenPhoto);
}
public static byte[] ReadToEnd(System.IO.Stream stream)
{
long originalPosition = stream.Position;
stream.Position = 0;
try
{
byte[] readBuffer = new byte[4096];
int totalBytesRead = 0;
int bytesRead;
while ((bytesRead = stream.Read(readBuffer, totalBytesRead, readBuffer.Length - totalBytesRead)) > 0)
{
totalBytesRead += bytesRead;
if (totalBytesRead == readBuffer.Length)
{
int nextByte = stream.ReadByte();
if (nextByte != -1)
{
byte[] temp = new byte[readBuffer.Length * 2];
Buffer.BlockCopy(readBuffer, 0, temp, 0, readBuffer.Length);
Buffer.SetByte(temp, totalBytesRead, (byte)nextByte);
readBuffer = temp;
totalBytesRead++;
}
}
}
byte[] buffer = readBuffer;
if (readBuffer.Length != totalBytesRead)
{
buffer = new byte[totalBytesRead];
Buffer.BlockCopy(readBuffer, 0, buffer, 0, totalBytesRead);
}
return buffer;
}
finally
{
stream.Position = originalPosition;
}
}
And upload byte[] to server. Hope it's help
I am trying to download a file to my computer and in the same time save it to Byte Array:
try
{
var req = (HttpWebRequest)HttpWebRequest.Create(url);
var fileStream = new FileStream(filePath,
FileMode.Create, FileAccess.Write, FileShare.Write);
using (var resp = req.GetResponse())
{
using (var stream = resp.GetResponseStream())
{
byte[] buffer = new byte[0x10000];
int len;
while ((len = stream.Read(buffer, 0, buffer.Length)) > 0)
{
//Do with the content whatever you want
// ***YOUR CODE***
MemoryStream memoryStream = new MemoryStream();
if (len > 0)
{
memoryStream.Write(buffer, 0, len);
len = stream.Read(buffer, 0, buffer.Length);
}
file = memoryStream.ToArray();
fileStream.Write(buffer, 0, len);
}
}
}
fileStream.Close();
}
catch (Exception exc) { }
And i noticed that it's not download all the file with this.
I wan to do it because i want to download a file and in the same time work with it.
Any idea why this problem happen?
There is a much easier way to get the file bytes by using the System.Net.WebClient.WebClient():
private static byte[] DownloadFile(string absoluteUrl)
{
using (var client = new System.Net.WebClient())
{
return client.DownloadData(absoluteUrl);
}
}
Usage:
var bytes = DownloadFile(absoluteUrl);
The problem looks to be double-reading - you are putting different things into the memory-stream / file-stream - it should be more like:
// declare file/memory stream here
while ((len = stream.Read(buffer, 0, buffer.Length)) > 0)
{
memoryStream.Write(buffer, 0, len);
fileStream.Write(buffer, 0, len);
// if you need to process "len" bytes, do it here
}
You might be able to lose "memoryStream" completely if you are processing the "len" bytes immediately. If it fits in-memory, it may be easier to just use WebClient.DownloadData and then File.WriteAllBytes.