We Keep Coding

Asynchronous streaming of large files using ASP.Net Framework 2.0

I am working on an ASP.NET framework 2.0 application. On a particular page I am providing a link to user. By clicking on this link a window opens with another aspx page. This page actually sends http request to a third-party url which points to a file(like - mirror urls to download file from cloud). The http response is sent back to user on the very first page using response.write from where user click the link.
Now, the problem I am facing is if the file size is low then it works fine. But, if the file is large (i.e., more than 1 GB), then my application waits until whole file is downloaded from the URL. I have tried using response.flush() to send chunk by chunk data to user, but still user is unable to use application because the worker process is busy getting streams of data from third party URL.
Is there any way by which large files can be downloaded asynchronously so that my pop-up window finishes its execution(download will be in progress) and also user can do other activities on application parallely.
Thanks,
Suvodeep

Use WebClient to read the remote file. Instead of downloading you can take the Stream from the WebClient. Put that in while() loop and push the bytes from the WebClient stream in the Response stream. On this way, you will be async downloading and uploading at the same time.
HttpRequest example:
private void WriteFileInDownloadDirectly()
{
//Create a stream for the file
Stream stream = null;
//This controls how many bytes to read at a time and send to the client
int bytesToRead = 10000;
// Buffer to read bytes in chunk size specified above
byte[] buffer = new byte[bytesToRead];
// The number of bytes read
try
{
//Create a WebRequest to get the file
HttpWebRequest fileReq = (HttpWebRequest)HttpWebRequest.Create("Remote File URL");
//Create a response for this request
HttpWebResponse fileResp = (HttpWebResponse)fileReq.GetResponse();
if (fileReq.ContentLength > 0)
fileResp.ContentLength = fileReq.ContentLength;
//Get the Stream returned from the response
stream = fileResp.GetResponseStream();
// prepare the response to the client. resp is the client Response
var resp = HttpContext.Current.Response;
//Indicate the type of data being sent
resp.ContentType = "application/octet-stream";
//Name the file
resp.AddHeader("Content-Disposition", $"attachment; filename=\"{ Path.GetFileName("Local File Path - can be fake") }\"");
resp.AddHeader("Content-Length", fileResp.ContentLength.ToString());
int length;
do
{
// Verify that the client is connected.
if (resp.IsClientConnected)
{
// Read data into the buffer.
length = stream.Read(buffer, 0, bytesToRead);
// and write it out to the response's output stream
resp.OutputStream.Write(buffer, 0, length);
// Flush the data
resp.Flush();
//Clear the buffer
buffer = new byte[bytesToRead];
}
else
{
// cancel the download if client has disconnected
length = -1;
}
} while (length > 0); //Repeat until no data is read
}
finally
{
if (stream != null)
{
//Close the input stream
stream.Close();
}
}
}
WebClient Stream reading:
using (WebClient client = new WebClient())
{
Stream largeFileStream = client.OpenRead("My Address");
}

HttpWebRequest Content Length Error

When downloading a file with HttpWebResponse the content length sent by the server is wrong and causes the HttpWebResponse to stop downloading the file mid-way through. IE seems to not have this issue when you browse. Any idea on how to get HttpWebResponse to ignore the content length the sever sent or would that even make sense.
Any help that could be given would be greatly appreciated.
--Example
class Program
{
static void Main(string[] args)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://localhost:59771/Default.aspx");
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Console.WriteLine("Content length: " + response.ContentLength);
int bytesRead = 0;
long totalBytesRead = 0;
byte[] data = new byte[1024 * 64];
StringBuilder output = new StringBuilder();
Stream responseStream = response.GetResponseStream();
do
{
bytesRead = responseStream.Read(data, 0, 1024 * 64);
totalBytesRead += bytesRead;
output.Append(Encoding.ASCII.GetString(data, 0, bytesRead));
}
while (bytesRead > 0);
Console.WriteLine("total read: " + totalBytesRead);
Console.WriteLine("last content read: " + output.ToString());
}
protected void Page_Load(object sender, EventArgs e)
{
Response.Clear();
Response.ClearHeaders();
Response.AddHeader("Content-Length", "13");
Response.Write("ABCDEFGHIJKLMNOPQRSTUVWXYZ");
}
}
Problem SOLVED!
The server we are pulling the data down from is a Cognos server and it was calculating the content length as if the string was to be compressed, but we were not sending in the code to state we could accept compression, so it would send back uncompressed data but only to the length of the compression. IE did not have this issue as it stated it could accept compression.
Code to correct issue:
request2.Headers.Add("Accept-Encoding", "gzip,deflate");

Problem SOLVED!
The server we are pulling the data down from is a Cognos server and it was calculating the content length as if the string was to be compressed, but we were not sending in the code to state we could accept compression, so it would send back uncompressed data but only to the length of the compression. IE did not have this issue as it stated it could accept compression. Code to correct issue:
request2.Headers.Add("Accept-Encoding", "gzip,deflate");

HttpWebRequest has nothing to do with data sent from the server, only with the data you send, so I'll assume you meant HttpWebResponse.
HttpWebResponse don't care at all about the Content-Length sent by the server, it only provides it in the Headers property to the client for informational purposes.
You shouldn't rely on the server's content length when reading from the response stream, just keep reading until the Stream.Read method returns 0.

Read several byte from file via http c#

How to download first 200 bytes of a file via HTTP protocol using C#?
I believed it could be done like this:
WebClient wc = new WebClient();
byte[] buffer = new byte[200];
using (var stream = wc.OpenRead(fileName))
{
stream.Read(buffer, 0, 200);
}
but when wc.OpenRead it called it downloads the whole file.

You need to set a Range Header on your WebClient before you invoke the OpenRead method.
See: http://msdn.microsoft.com/en-us/library/system.net.webclient.headers.aspx

Slow performance in reading from stream .NET

I have a monitoring system and I want to save a snapshot from a camera when alarm trigger.
I have tried many methods to do that…and it’s all working fine , stream snapshot from the camera then save it as a jpg in the pc…. picture (jpg format,1280*1024,140KB)..That’s fine
But my problem is in the application performance...
The app need about 20 ~30 seconds to read the steam, that’s not acceptable coz that method will be called every 2 second .I need to know what wrong with that code and how I can get it much faster than that. ?
Many thanks in advance
Code:
string sourceURL = "http://192.168.0.211/cgi-bin/cmd/encoder?SNAPSHOT";
byte[] buffer = new byte[200000];
int read, total = 0;
WebRequest req = (WebRequest)WebRequest.Create(sourceURL);
req.Credentials = new NetworkCredential("admin", "123456");
WebResponse resp = req.GetResponse();
Stream stream = resp.GetResponseStream();
while ((read = stream.Read(buffer, total, 1000)) != 0)
{
total += read;
}
Bitmap bmp = (Bitmap)Bitmap.FromStream(new MemoryStream(buffer, 0,total));
string path = JPGName.Text+".jpg";
bmp.Save(path);

I very much doubt that this code is the cause of the problem, at least for the first method call (but read further below).
Technically, you could produce the Bitmap without saving to a memory buffer first, or if you don't need to display the image as well, you can save the raw data without ever constructing a Bitmap, but that's not going to help in terms of multiple seconds improved performance. Have you checked how long it takes to download the image from that URL using a browser, wget, curl or whatever tool, because I suspect something is going on with the encoding source.
Something you should do is clean up your resources; close the stream properly. This can potentially cause the problem if you call this method regularly, because .NET will only open a few connections to the same host at any one point.
// Make sure the stream gets closed once we're done with it
using (Stream stream = resp.GetResponseStream())
{
// A larger buffer size would be benefitial, but it's not going
// to make a significant difference.
while ((read = stream.Read(buffer, total, 1000)) != 0)
{
total += read;
}
}

I cannot try the network behavior of the WebResponse stream, but you handle the stream twice (once in your loop and once with your memory stream).
I don't thing that's the whole problem but I'd give it a try:
string sourceURL = "http://192.168.0.211/cgi-bin/cmd/encoder?SNAPSHOT";
WebRequest req = (WebRequest)WebRequest.Create(sourceURL);
req.Credentials = new NetworkCredential("admin", "123456");
WebResponse resp = req.GetResponse();
Stream stream = resp.GetResponseStream();
Bitmap bmp = (Bitmap)Bitmap.FromStream(stream);
string path = JPGName.Text + ".jpg";
bmp.Save(path);

Try to read bigger pieces of data, than 1000 bytes per time. I can see no problem with, for example,
read = stream.Read(buffer, 0, buffer.Length);

Try this to download the file.
using(WebClient webClient = new WebClient())
{
webClient.DownloadFile("http://192.168.0.211/cgi-bin/cmd/encoder?SNAPSHOT", "c:\\Temp\myPic.jpg");
}
You can use a DateTime to put a unique stamp on the shot.

Protocol buffer net with windows phone 7

I'm trying to download a response from a server wich is in Protocol Buffer format from a Windows Phone 7 application.
I'm trying to do this with a WebClient, my problem is the following.
The WebClient has only two method for downloading
DownloadStringAsync(new Uri(url));
and
OpenReadAsync(new Uri(url));
but this two method are not good to retrieve the response because, the response size should have 16 hexadecimal caracteres ( 080118CBDDF0E104 ), but the size of the string and the stream get by the two methods are only 8.
So I'm searching a way to resolve my problem.
I found one for C#
public static T DownloadProto<T>(this WebClient client, string address)
{
byte[] data = client.DownloadData(address);
using (var ms = new MemoryStream(data))
{
return Serializer.Deserialize<T>(ms);
}
}
on
http://code.google.com/p/protobuf-net/source/browse/trunk/BasicHttp/HttpClient/ProtoWebClient.cs?spec=svn340&r=340
But this method was deleted or not yet implemented on the Windows Phone 7 development kit.

How are you reading from the stream?
If you are reading it as a string then its perhaps reading two bytes per character - instead use
var buf = new byte[16];
var actual = stream.Read(buf, 0, buf.Length);