We Keep Coding

Downloading HTML page in C# and loading it in memory fastest way

I have written a following code which downloads a page from a given URL:
string html = string.Empty;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.AutomaticDecompression = DecompressionMethods.GZip;
request.Proxy = null;
request.ServicePoint.UseNagleAlgorithm = false;
request.ServicePoint.Expect100Continue = false;
request.Method = "GET";
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
using (Stream stream = response.GetResponseStream())
using (StreamReader reader = new StreamReader(stream))
{
html = reader.ReadToEnd();
}
But this takes about 5-8 seconds to download the HTML file which is quite quite slow. My question here is, is there any way to improve this code, or use some other piece of code/library that can perform the HTML download for a given URL faster than this one?
Can someone help me out ?

Why not use an httpclient then write the result to a file that way?
using (HttpClient client = new HttpClient())
{
using (HttpRequestMessage request = new HttpRequestMessage())
{
request.Method = HttpMethod.Get;
request.RequestUri = new Uri("http://www.google.com", UriKind.Absolute);
using (HttpResponseMessage response = await client.SendAsync(request))
{
if (response.IsSuccessStatusCode)
{
if (response.Content != null)
{
var result = await response.Content.ReadAsStringAsync();
// write result to file
}
}
}
}
}

HttpWebResponse timeout

I am getting timeout exception on code bellow. Only this site is problematic. What is the problem?
string ackoURL = "https://www.zomato.com/sk/brno/u-heligonky-z%C3%A1brdovice-brno-st%C5%99ed/denn%C3%A9-menu";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(ackoURL);
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
//this code is never executed
}
I tried modifying SecurityProtocol, KeepAlive and simmilar things. Without success.

it waiting these headers
..and it worked
Uri u = new Uri("https://www.zomato.com/sk/brno/u-heligonky-z%C3%A1brdovice-brno-st%C5%99ed/denn%C3%A9-menu");
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(u);
request.AutomaticDecompression = DecompressionMethods.GZip;
request.Headers.Add("Accept-Language", "en-gb,en;q=0.5");
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using(Stream stream = response.GetResponseStream())
{
using(StreamReader reader= new StreamReader(stream))
{
var result = reader.ReadToEnd();
}
}
}

How to properly make a http web GET request

i am still new on c# and i'm trying to create an application for this page that will tell me when i get a notification (answered, commented, etc..). But for now i'm just trying to make a simple call to the api which will get the user's data.
i'm using Visual studio express 2012 to build the C# application, where (for now) you enter your user id, so the application will make the request with the user id and show the stats of this user id.
here is the code where i'm trying to make the request:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
//Request library
using System.Net;
using System.IO;
namespace TestApplication
{
class Connect
{
public string id;
public string type;
protected string api = "https://api.stackexchange.com/2.2/";
protected string options = "?order=desc&sort=name&site=stackoverflow";
public string request()
{
string totalUrl = this.join(id);
return this.HttpGet(totalUrl);
}
protected string join(string s)
{
return api + type + "/" + s + options;
}
protected string get(string url)
{
try
{
string rt;
WebRequest request = WebRequest.Create(url);
WebResponse response = request.GetResponse();
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
rt = reader.ReadToEnd();
Console.WriteLine(rt);
reader.Close();
response.Close();
return rt;
}
catch(Exception ex)
{
return "Error: " + ex.Message;
}
}
public string HttpGet(string URI)
{
WebClient client = new WebClient();
// Add a user agent header in case the
// requested URI contains a query.
client.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)");
Stream data = client.OpenRead(URI);
StreamReader reader = new StreamReader(data);
string s = reader.ReadToEnd();
data.Close();
reader.Close();
return s;
}
}
}
the class is an object and its being accessed from the form by just parsing it the user id and make the request.
i have tried many of the examples i have looked on google, but not clue why i am getting on all ways this message "�".
i am new in this kind of algorithm, if anyone can share a book or tutorial that shows how to do this kind of stuff (explaining each step), i would appreciate it

If using .NET 6 or higher, please read the warning at the bottom of this answer.
Servers sometimes compress their responses to save on bandwidth, when this happens, you need to decompress the response before attempting to read it. Fortunately, the .NET framework can do this automatically, however, we have to turn the setting on.
Here's an example of how you could achieve that.
string html = string.Empty;
string url = #"https://api.stackexchange.com/2.2/answers?order=desc&sort=activity&site=stackoverflow";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.AutomaticDecompression = DecompressionMethods.GZip;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
using (Stream stream = response.GetResponseStream())
using (StreamReader reader = new StreamReader(stream))
{
html = reader.ReadToEnd();
}
Console.WriteLine(html);
GET
public string Get(string uri)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
using(HttpWebResponse response = (HttpWebResponse)request.GetResponse())
using(Stream stream = response.GetResponseStream())
using(StreamReader reader = new StreamReader(stream))
{
return reader.ReadToEnd();
}
}
GET async
public async Task<string> GetAsync(string uri)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
using(HttpWebResponse response = (HttpWebResponse)await request.GetResponseAsync())
using(Stream stream = response.GetResponseStream())
using(StreamReader reader = new StreamReader(stream))
{
return await reader.ReadToEndAsync();
}
}
POST
Contains the parameter method in the event you wish to use other HTTP methods such as PUT, DELETE, ETC
public string Post(string uri, string data, string contentType, string method = "POST")
{
byte[] dataBytes = Encoding.UTF8.GetBytes(data);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
request.ContentLength = dataBytes.Length;
request.ContentType = contentType;
request.Method = method;
using(Stream requestBody = request.GetRequestStream())
{
requestBody.Write(dataBytes, 0, dataBytes.Length);
}
using(HttpWebResponse response = (HttpWebResponse)request.GetResponse())
using(Stream stream = response.GetResponseStream())
using(StreamReader reader = new StreamReader(stream))
{
return reader.ReadToEnd();
}
}
POST async
Contains the parameter method in the event you wish to use other HTTP methods such as PUT, DELETE, ETC
public async Task<string> PostAsync(string uri, string data, string contentType, string method = "POST")
{
byte[] dataBytes = Encoding.UTF8.GetBytes(data);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
request.ContentLength = dataBytes.Length;
request.ContentType = contentType;
request.Method = method;
using(Stream requestBody = request.GetRequestStream())
{
await requestBody.WriteAsync(dataBytes, 0, dataBytes.Length);
}
using(HttpWebResponse response = (HttpWebResponse)await request.GetResponseAsync())
using(Stream stream = response.GetResponseStream())
using(StreamReader reader = new StreamReader(stream))
{
return await reader.ReadToEndAsync();
}
}
Warning notice: The methods of making a HTTP request outlined within this answer uses the HttpWebRequest class which is deprecated starting from .NET 6 and onwards. It's recommended to use HttpClient instead which this answer by DIG covers for environments that depends on .NET 6+.
Ref: https://learn.microsoft.com/en-us/dotnet/core/compatibility/networking/6.0/webrequest-deprecated

Another way is using 'HttpClient' like this:
using System;
using System.Net;
using System.Net.Http;
namespace Test
{
class Program
{
static void Main(string[] args)
{
Console.WriteLine("Making API Call...");
using (var client = new HttpClient(new HttpClientHandler { AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate }))
{
client.BaseAddress = new Uri("https://api.stackexchange.com/2.2/");
HttpResponseMessage response = client.GetAsync("answers?order=desc&sort=activity&site=stackoverflow").Result;
response.EnsureSuccessStatusCode();
string result = response.Content.ReadAsStringAsync().Result;
Console.WriteLine("Result: " + result);
}
Console.ReadLine();
}
}
}
Check HttpClient vs HttpWebRequest from stackoverflow and this from other.
Update June 22, 2020:
It's not recommended to use httpclient in a 'using' block as it might cause port exhaustion.
private static HttpClient client = null;
ContructorMethod()
{
if(client == null)
{
HttpClientHandler handler = new HttpClientHandler()
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
client = new HttpClient(handler);
}
client.BaseAddress = new Uri("https://api.stackexchange.com/2.2/");
HttpResponseMessage response = client.GetAsync("answers?order=desc&sort=activity&site=stackoverflow").Result;
response.EnsureSuccessStatusCode();
string result = response.Content.ReadAsStringAsync().Result;
Console.WriteLine("Result: " + result);
}
If using .Net Core 2.1+, consider using IHttpClientFactory and injecting like this in your startup code.
var timeout = Policy.TimeoutAsync<HttpResponseMessage>(
TimeSpan.FromSeconds(60));
services.AddHttpClient<XApiClient>().ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
}).AddPolicyHandler(request => timeout);

Simpliest way for my opinion
var web = new WebClient();
var url = $"{hostname}/LoadDataSync?systemID={systemId}";
var responseString = web.DownloadString(url);
OR
var bytes = web.DownloadData(url);

var request = (HttpWebRequest)WebRequest.Create("sendrequesturl");
var response = (HttpWebResponse)request.GetResponse();
string responseString;
using (var stream = response.GetResponseStream())
{
using (var reader = new StreamReader(stream))
{
responseString = reader.ReadToEnd();
}
}

Adding to the responses already given, this is a complete example hitting JSON PlaceHolder site.
using System;
using System.Net.Http;
using System.Threading.Tasks;
using Newtonsoft.Json;
namespace Publish
{
class Program
{
static async Task Main(string[] args)
{
// Get Reqeust
HttpClient req = new HttpClient();
var content = await req.GetAsync("https://jsonplaceholder.typicode.com/users");
Console.WriteLine(await content.Content.ReadAsStringAsync());
// Post Request
Post p = new Post("Some title", "Some body", "1");
HttpContent payload = new StringContent(JsonConvert.SerializeObject(p));
content = await req.PostAsync("https://jsonplaceholder.typicode.com/posts", payload);
Console.WriteLine("--------------------------");
Console.WriteLine(content.StatusCode);
Console.WriteLine(await content.Content.ReadAsStringAsync());
}
}
public struct Post {
public string Title {get; set;}
public string Body {get;set;}
public string UserID {get; set;}
public Post(string Title, string Body, string UserID){
this.Title = Title;
this.Body = Body;
this.UserID = UserID;
}
}
}

Get webpage page content and HTTP status code in C#

In a C# Windows Forms application I can get the contents of a webpage using:
string content = webClient.DownloadString(url);
And I can get the HTTP header using:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "GET";
string response = ((HttpWebResponse)request.GetResponse()).StatusCode.ToString();
Is there a way to get both the contents and the HTTP status code (if it fails) in one trip to the server instead of twice?
Thanks.

You can read the data from the Stream inside the HttpWebResponse object:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "GET";
using (var response = request.GetResponse())
using (var stream = response.GetResponseStream())
using (var reader = new StreamReader(stream))
{
HttpStatusCode statusCode = ((HttpWebResponse)response).StatusCode;
string contents = reader.ReadToEnd();
}
In this way you will have to detect the encoding by hand, or using a library to detect encoding. You can read the encoding as a string from the HttpWebResponse object as well, when one exists, it is inside the ContentType property. If the page is Html, then you will have to parse it for a possible encoding change in the top of the document or inside the head.
Read handling the encoding from ContentType header
var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "GET";
string content;
HttpStatusCode statusCode;
using (var response = request.GetResponse())
using (var stream = response.GetResponseStream())
{
var contentType = response.ContentType;
Encoding encoding = null;
if (contentType != null)
{
var match = Regex.Match(contentType, #"(?<=charset\=).*");
if (match.Success)
encoding = Encoding.GetEncoding(match.ToString());
}
encoding = encoding ?? Encoding.UTF8;
statusCode = ((HttpWebResponse)response).StatusCode;
using (var reader = new StreamReader(stream, encoding))
content = reader.ReadToEnd();
}

WebClient
I assume you use WebClient because its easy webrequest-to-string handling. Unfortunately, WebClient does not expose the HTTP response code. You can either assume the response was positive (2xx) unless you get an exception and read it:
try
{
string content = webClient.DownloadString(url);
}
catch (WebException e)
{
HttpWebResponse response = (System.Net.HttpWebResponse)we.Response;
var statusCode = response.StatusCode;
}
Or if you're really interested in the success code you can use reflection as explained here.
HttpClient
You can also use HttpClient if you're on .NET 4.5, which does expose the response code, as explained here:
using (HttpClient client = new HttpClient())
{
HttpResponseMessage response = await client.GetAsync(url);
string content = await response.Content.ReadAsStringAsync();
var statusCode = response.StatusCode;
}
HttpWebRequest
Alternatively, you can just use HttpWebRequest to get the status and response as explained here:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "GET";
var response = (HttpWebResponse)request.GetResponse();
using (Stream stream = response.GetResponseStream())
{
StreamReader reader = new StreamReader(stream);
string content = reader.ReadToEnd();
var statusCode = response.StatusCode;
}

I think, you have not realised, that in the second case you have access to the content as well (although it takes a little more effort to get as a string).
Look at the Microsoft documentation: http://msdn.microsoft.com/en-us/library/system.net.httpwebresponse.getresponsestream(v=vs.110).aspx which shows you how to ge a response stream from the web response, and then how to get the string data from that stream.

And I can get the HTTP header using:
request.Method = "GET";
Method GET returns HEAD and BODY sections in response.
HTTP also support a method HEAD - which returns HEAD section only.
You can get BODY from HttpWebResponse using GetResponseStream method.

Read text from response

HttpWebRequest request = WebRequest.Create("http://google.com") as HttpWebRequest;
request.Accept = "application/xrds+xml";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
WebHeaderCollection header = response.Headers;
Here google returns text. How to read it?

Your "application/xrds+xml" was giving me issues, I was receiving a Content-Length of 0 (no response).
After removing that, you can access the response using response.GetResponseStream().
HttpWebRequest request = WebRequest.Create("http://google.com") as HttpWebRequest;
//request.Accept = "application/xrds+xml";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
WebHeaderCollection header = response.Headers;
var encoding = ASCIIEncoding.ASCII;
using (var reader = new System.IO.StreamReader(response.GetResponseStream(), encoding))
{
string responseText = reader.ReadToEnd();
}

The accepted answer does not correctly dispose the WebResponse or decode the text. Also, there's a new way to do this in .NET 4.5.
To perform an HTTP GET and read the response text, do the following.
.NET 1.1 ‒ 4.0
public static string GetResponseText(string address)
{
var request = (HttpWebRequest)WebRequest.Create(address);
using (var response = (HttpWebResponse)request.GetResponse())
{
var encoding = Encoding.GetEncoding(response.CharacterSet);
using (var responseStream = response.GetResponseStream())
using (var reader = new StreamReader(responseStream, encoding))
return reader.ReadToEnd();
}
}
.NET 4.5
private static readonly HttpClient httpClient = new HttpClient();
public static async Task<string> GetResponseText(string address)
{
return await httpClient.GetStringAsync(address);
}

I've just tried that myself, and it gave me a 200 OK response, but no content - the content length was 0. Are you sure it's giving you content? Anyway, I'll assume that you've really got content.
Getting actual text back relies on knowing the encoding, which can be tricky. It should be in the Content-Type header, but then you've got to parse it etc.
However, if this is actually XML (e.g. from "http://google.com/xrds/xrds.xml"), it's a lot easier. Just load the XML into memory, e.g. via LINQ to XML. For example:
using System;
using System.IO;
using System.Net;
using System.Xml.Linq;
using System.Web;
class Test
{
static void Main()
{
string url = "http://google.com/xrds/xrds.xml";
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
XDocument doc;
using (WebResponse response = request.GetResponse())
{
using (Stream stream = response.GetResponseStream())
{
doc = XDocument.Load(stream);
}
}
// Now do whatever you want with doc here
Console.WriteLine(doc);
}
}
If the content is XML, getting the result into an XML object model (whether it's XDocument, XmlDocument or XmlReader) is likely to be more valuable than having the plain text.

This article gives a good overview of using the HttpWebResponse object:How to use HttpWebResponse
Relevant bits below:
HttpWebResponse webresponse;
webresponse = (HttpWebResponse)webrequest.GetResponse();
Encoding enc = System.Text.Encoding.GetEncoding(1252);
StreamReader loResponseStream = new StreamReader(webresponse.GetResponseStream(),enc);
string Response = loResponseStream.ReadToEnd();
loResponseStream.Close();
webresponse.Close();
return Response;

HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://www.google.com");
request.Method = "GET";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
string strResponse = reader.ReadToEnd();

response.GetResponseStream() should be used to return the response stream. And don't forget to close the Stream and Response objects.

If you http request is Post and request.Accept = "application/x-www-form-urlencoded";
then i think you can to get text of respone by code bellow:
var contentEncoding = response.Headers["content-encoding"];
if (contentEncoding != null && contentEncoding.Contains("gzip")) // cause httphandler only request gzip
{
// using gzip stream reader
using (var responseStreamReader = new StreamReader(new GZipStream(response.GetResponseStream(), CompressionMode.Decompress)))
{
strResponse = responseStreamReader.ReadToEnd();
}
}
else
{
// using ordinary stream reader
using (var responseStreamReader = new StreamReader(response.GetResponseStream()))
{
strResponse = responseStreamReader.ReadToEnd();
}
}