We Keep Coding

How to cancel large file download yet still get page source in C#?

I'm working in C# on a program to list all course resources for a MOOC (e.g. Coursera). I don't want to download the content, just get a listing of all the resources (e.g. pdf, videos, text files, sample files, etc...) which are made available to the course.
My problem lies in parsing the html source (currently using HtmlAgilityPack) without downloading all the content.
For example, if you go to this intro video for a banking course on Coursera and check the source (F12 in Chrome for Developer Tools), you can see the page source. I can stop the video download which autoplays, but still see the source.
How can I get the source in C# without download all the content?
I've looked in the HttpWebRequest headers (problem: time out), and DownloadDataAsync with Cancel (problem: the Completed Result object is invalid when cancelling the async request). I've also tried various Loads from HtmlAgilityPack but with no success.
Time out:
HttpWebRequest postRequest = (HttpWebRequest)WebRequest.Create(url);
postRequest.Timeout = TIMEOUT * 1000000; //Really long
postRequest.Referer = "https://www.coursera.org";
if (headers != null)
{ //headers here }
//Deal with cookies
if (cookie != null)
{ cookieJar.Add(cookie); }
postRequest.CookieContainer = cookiejar;
postRequest.Method = "GET";
postRequest.AllowAutoRedirect = allowRedirect;
postRequest.ServicePoint.Expect100Continue = true;
HttpWebResponse postResponse = (HttpWebResponse)postRequest.GetResponse();
Any tips on how to proceed?

There are at least two ways to do what you're asking. The first is to use a range get. That is, specify the range of the file you want to read. You do that by calling AddRange on the HttpWebRequest. So if you want, say, the first 10 kilobytes of the file, you'd write:
request.AddRange(-10240);
Read carefully what the documentation says about the meaning of that parameter. If it's negative, it specifies the ending point of the range. There are also other overloads of AddRange that you might be interested in.
Not all servers support range gets, though. If that doesn't work, you'll have to do it another way.
What you can do is call GetResponse and then start reading data. Once you've read as much data as you want, you can stop reading and close the stream. I've modified your sample slightly to show what I mean.
string url = "https://www.coursera.org/course/money";
HttpWebRequest postRequest = (HttpWebRequest)WebRequest.Create(url);
postRequest.Method = "GET";
postRequest.AllowAutoRedirect = true; //allowRedirect;
postRequest.ServicePoint.Expect100Continue = true;
HttpWebResponse postResponse = (HttpWebResponse) postRequest.GetResponse();
int maxBytes = 1024*1024;
int totalBytesRead = 0;
var buffer = new byte[maxBytes];
using (var s = postResponse.GetResponseStream())
{
int bytesRead;
// read up to `maxBytes` bytes from the response
while (totalBytesRead < maxBytes && (bytesRead = s.Read(buffer, 0, maxBytes)) != 0)
{
// Here you can save the bytes read to a persistent buffer,
// or write them to a file.
Console.WriteLine("{0:N0} bytes read", bytesRead);
totalBytesRead += bytesRead;
}
}
Console.WriteLine("total bytes read = {0:N0}", totalBytesRead);
That said, I ran this sample and it downloaded about 6 kilobytes and stopped. I don't know why you're having trouble with timeouts or too much data.
Note that sometimes trying to close the stream before the entire response is read will cause the program to hang. I'm not sure why that happens at all, and I can't explain why it only happens sometimes. But you can solve it by calling request.Abort before closing the stream. That is:
using (var s = postResponse.GetResponseStream())
{
// do stuff here
// abort the request before continuing
postRequest.Abort();
}

Serializing Alternative views For MSMQ

My concept is downloading a image from url and sending the image(Linked Resource)to mail message to MSMQ!, I can sucessfully download the image , but i cannot able to send it to MSMQ, i need to serialize the Alternative Views, which i could not able to do?
Here is the code
MailMessage m = new MailMessage();
string strBody="<h1>This is sample</h1><image src=\"cid:image1\">";
m.Body = strBody;
AlternateView av1 = AlternateView.CreateAlternateViewFromString(strBody, null, MediaTypeNames.Text.Html);
Here I am Downloading the Image from url
Stream DownloadStream = ReturnImage();
LinkedResource lr = new LinkedResource(DownloadStream, MediaTypeNames.Image.Gif);
lr.ContentId = "image1";
av1.LinkedResources.Add(lr);
m.AlternateViews.Add(av);
private Stream ReturnImage()
{
try
{
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(urlForImage);
webRequest.ProtocolVersion = HttpVersion.Version10;
webRequest.KeepAlive = false;
webRequest.Timeout = 1000000000;
webRequest.ReadWriteTimeout = 1000000000;
using (HttpWebResponse webResponse = (HttpWebResponse)webRequest.GetResponse())
{
Stream k = webResponse.GetResponseStream();
MemoryStream ms = new MemoryStream();
int count = 0;
do
{
byte[] buf = new byte[1024];
count = k.Read(buf, 0, 1024);
ms.Write(buf, 0, count);
} while (k.CanRead && count > 0);
return ms;
}
}
catch (WebException e)
{
return null;
}
Can you guys give solution for serializing the Alternative views so that i can able to send and Receive MSMQ !

I do not think that you should take this approach.
MSMQ is designed to be ligtwheight, so sending huge data like images was not the intention - although this is technically possible, of course.
Also bear in mind that MSQM has a limit of 4 MB per message. Depending on the size of your images this might become problematic (or might not).
Instead I suggest that you save the images to a place that can be accessed by all participating application/ services/ etc., e.g. a network share, file server, or web server, or ...
Then you send only the URIs in your MSMQ message. This will be very fast to process on both sender and receiver side. Also, this will be much, much lighter on the MSMQ infrastructure.

HttpWebRequest Content Length Error

When downloading a file with HttpWebResponse the content length sent by the server is wrong and causes the HttpWebResponse to stop downloading the file mid-way through. IE seems to not have this issue when you browse. Any idea on how to get HttpWebResponse to ignore the content length the sever sent or would that even make sense.
Any help that could be given would be greatly appreciated.
--Example
class Program
{
static void Main(string[] args)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://localhost:59771/Default.aspx");
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Console.WriteLine("Content length: " + response.ContentLength);
int bytesRead = 0;
long totalBytesRead = 0;
byte[] data = new byte[1024 * 64];
StringBuilder output = new StringBuilder();
Stream responseStream = response.GetResponseStream();
do
{
bytesRead = responseStream.Read(data, 0, 1024 * 64);
totalBytesRead += bytesRead;
output.Append(Encoding.ASCII.GetString(data, 0, bytesRead));
}
while (bytesRead > 0);
Console.WriteLine("total read: " + totalBytesRead);
Console.WriteLine("last content read: " + output.ToString());
}
protected void Page_Load(object sender, EventArgs e)
{
Response.Clear();
Response.ClearHeaders();
Response.AddHeader("Content-Length", "13");
Response.Write("ABCDEFGHIJKLMNOPQRSTUVWXYZ");
}
}
Problem SOLVED!
The server we are pulling the data down from is a Cognos server and it was calculating the content length as if the string was to be compressed, but we were not sending in the code to state we could accept compression, so it would send back uncompressed data but only to the length of the compression. IE did not have this issue as it stated it could accept compression.
Code to correct issue:
request2.Headers.Add("Accept-Encoding", "gzip,deflate");

Problem SOLVED!
The server we are pulling the data down from is a Cognos server and it was calculating the content length as if the string was to be compressed, but we were not sending in the code to state we could accept compression, so it would send back uncompressed data but only to the length of the compression. IE did not have this issue as it stated it could accept compression. Code to correct issue:
request2.Headers.Add("Accept-Encoding", "gzip,deflate");

HttpWebRequest has nothing to do with data sent from the server, only with the data you send, so I'll assume you meant HttpWebResponse.
HttpWebResponse don't care at all about the Content-Length sent by the server, it only provides it in the Headers property to the client for informational purposes.
You shouldn't rely on the server's content length when reading from the response stream, just keep reading until the Stream.Read method returns 0.

Incomplete HttpWebResponse with large data sets

I have some code that downloads the content of a webpage that I've been using for a while. This code works fine and has never provided an issue and still doesn't... However, there is a page that is rather large (2MB, no images) with 4 tables with 4, 20, 100, 600 rows respectively and about 20 columns wide.
When trying to get all the data it completes without any apparent errors or exceptions but only returns up to about row 60 in the 4th table - sometimes more, sometimes less. The broswer completes loading in about 20-30 seconds with constant, what seems like flushes, to the page until complete.
I've tried a number of solutions from SO and searches without any different results. Below is the current code, but I've: proxy, async, no timeouts, false keepalive...
I can't use WebClient (as another far-fetch attempt) because I need to login using the cookiecontainer.
HttpWebRequest pageImport = (HttpWebRequest)WebRequest.Create(importUri);
pageImport.ReadWriteTimeout = Int32.MaxValue;
pageImport.Timeout = Int32.MaxValue;
pageImport.UserAgent = "User-Agent Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3";
pageImport.Accept = "Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
pageImport.KeepAlive = true;
pageImport.Timeout = Int32.MaxValue;
pageImport.ReadWriteTimeout = Int32.MaxValue;
pageImport.MaximumResponseHeadersLength = Int32.MaxValue;
if (null != LoginCookieContainer)
{
pageImport.CookieContainer = LoginCookieContainer;
}
Encoding encode = System.Text.Encoding.GetEncoding("utf-8");
using (WebResponse response = pageImport.GetResponse())
using (Stream stream = response.GetResponseStream())
using (StreamReader reader = new StreamReader(stream, encode))
{
stream.Flush();
HtmlRetrieved = reader.ReadToEnd();
}

Try to read block wise instead of reader.ReadToEnd();
Just to give you an idea:
// Pipe the stream to a higher level stream reader with the required encoding format.
StreamReader readStream = new StreamReader( ReceiveStream, encode );
Console.WriteLine("\nResponse stream received");
Char[] read = new Char[256];
// Read 256 charcters at a time.
int count = readStream.Read( read, 0, 256 );
Console.WriteLine("HTML...\r\n");
while (count > 0)
{
// Dump the 256 characters on a string and display the string onto the console.
String str = new String(read, 0, count);
Console.Write(str);
count = readStream.Read(read, 0, 256);
}

I suspect this is handled as a configuration setting on the server side. Incidentally, I think you may be setting your properties incorrectly. Remove the "user-agent" and "accept" from the literals, as such:
pageImport.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3";
pageImport.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";

While I'm still going to try the suggestions provided and will change my answer if it works, it seems that in this case, the problem IS the proxy. I got in front of the proxy and the code works as expected and much quicker.
I'll have to look at some proxy optimizations since this code must run behind the proxy.

My post request to https://qrng.physik.hu-berlin.de/ failed, why?

the page at https://qrng.physik.hu-berlin.de/ provides a high bit rate quantum number generator web service and I'm trying to access that service.
However I could not manage to do so. This is my current code:
using System;
using System.Collections.Generic;
using System.Linq;
using S=System.Text;
using System.Security.Cryptography;
using System.IO;
namespace CS_Console_App
{
class Program
{
static void Main()
{
System.Net.ServicePointManager.Expect100Continue = false;
var username = "testuser";
var password = "testpass";
System.Diagnostics.Debug.WriteLine(Post("https://qrng.physik.hu-berlin.de/", "username="+username+"&password="+password));
Get("http://qrng.physik.hu-berlin.de/download/sampledata-1MB.bin");
}
public static void Get(string url)
{
var my_request = System.Net.WebRequest.Create(url);
my_request.Credentials = System.Net.CredentialCache.DefaultCredentials;
var my_response = my_request.GetResponse();
var my_response_stream = my_response.GetResponseStream();
var stream_reader = new System.IO.StreamReader(my_response_stream);
var content = stream_reader.ReadToEnd();
System.Diagnostics.Debug.WriteLine(content);
stream_reader.Close();
my_response_stream.Close();
}
public static string Post(string url, string data)
{
string vystup = null;
try
{
//Our postvars
byte[] buffer = System.Text.Encoding.ASCII.GetBytes(data);
//Initialisation, we use localhost, change if appliable
System.Net.HttpWebRequest WebReq = (System.Net.HttpWebRequest)System.Net.WebRequest.Create(url);
//Our method is post, otherwise the buffer (postvars) would be useless
WebReq.Method = "POST";
//We use form contentType, for the postvars.
WebReq.ContentType = "application/x-www-form-urlencoded";
//The length of the buffer (postvars) is used as contentlength.
WebReq.ContentLength = buffer.Length;
//We open a stream for writing the postvars
Stream PostData = WebReq.GetRequestStream();
//Now we write, and afterwards, we close. Closing is always important!
PostData.Write(buffer, 0, buffer.Length);
PostData.Close();
//Get the response handle, we have no true response yet!
System.Net.HttpWebResponse WebResp = (System.Net.HttpWebResponse)WebReq.GetResponse();
//Let's show some information about the response
Console.WriteLine(WebResp.StatusCode);
Console.WriteLine(WebResp.Server);
//Now, we read the response (the string), and output it.
Stream Answer = WebResp.GetResponseStream();
StreamReader _Answer = new StreamReader(Answer);
vystup = _Answer.ReadToEnd();
//Congratulations, you just requested your first POST page, you
//can now start logging into most login forms, with your application
//Or other examples.
}
catch (Exception ex)
{
throw ex;
}
return vystup.Trim() + "\n";
}
}
}
I'm having 403 forbidden error when I try to do a get request on http://qrng.physik.hu-berlin.de/download/sampledata-1MB.bin.
After debugging abit, I've realised that even though I've supplied a valid username and password, the response html that was sent after my POST request indicate that I was actually not logon to the system after my POST request.
Does anyone know why is this the case, and how may I work around it to call the service?
Bump. can anyone get this to work or is the site just a scam?

The site is surely not a scam. I developed the generator and I put my scientific reputation in it. The problem is that you are trying to use the service in a way that was not intended. The sample files were really only meant to be downloaded manually for basic test purposes. Automated access to fetch data into an application was meant to be implemented through the DLLs we provide.
On the other hand, I do not know of any explicit intent to prevent your implementation to work. I suppose if a web browser can log in and fetch data, some program should be able to do the same. Maybe only the login request is just a little more complicated. No idea. The server software was developed by someone else and I cannot bother him with this right now.
Mick

Actually, the generator can now also be purchased. See here:
http://www.picoquant.com/products/pqrng150/pqrng150.htm

Have you tried to change this
my_request.Credentials = System.Net.CredentialCache.DefaultCredentials
to
my_request.Credentials = new NetworkCredential(UserName,Password);
as described on MSDN page?