I can use both HttpWebRequest to send an HTTP request and get an HTTP response without a WebClient.
When should you use HttpWebRequest and when should you use WebClient?
WebClient can be used when you don't need any fine-tuning.
When using HttpWebRequest, you can control various options, including timeouts (very important). So basically - WebClient for toy projects / POCs, HttpWebRequest for actual business.
Personally I always use WebClient. The API seems more simple. It uses HttpWebRequest under the covers.
WebClient is ideal for downloads and uploads.
HttpWebRequest is ideal for web connections, including sending HTTP POST requests, as seen here: HTTP request with post
If you do not need access to the underlying stream but are just uploading or downloading "data", i.e. a file some bytes or a string, WebClient is a simplifying abstraction.
Related
I am building an application with C# and one of the requirements is that the headers have to contain very specific data. I have been working with the WebClient and HttpWebRequest classes in C# but both do not have the option to explicitly remove the Connection header. This is what I have tried:
WebClient
client.Headers.Remove("Connection");
client.Headers.Add("Connection", null); // if only...
HttpWebRequest
request.Connection = null;
request.Connection = "";
Some of what I have read is that you can only have this header set to Keep-Alive or Close. I need it to not even be on the web request.
All other required headers are adding just fine. It is this one that is causing a head ache. I'm beginning to believe that removing the Connection header is not possible. Is there any language where it is possible if this is the case?
Much appreciation for the time you guys take to look at this!
Thank you!
I'm not quite sure how to remove that specific header, but you could try using the HttpClient class instead of HttpWebRequest or WebClient.
HttpClient class
This class is more up-to-date way of executing HTTP requests in C#
I writting an application which is using WebClient class.
Adding something like that:
ExC.Headers.Add("Accept-Encoding: gzip, deflate");
where ExC is:
class ExWebClient1 : WebClient
{
protected override WebRequest GetWebRequest(Uri address)
{
HttpWebRequest request = (HttpWebRequest)base.GetWebRequest(address);
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
return request;
}
}
It will be a diffrence in speed when i will be using encoded response?
Short answer is usually Yes.
Long answer, It depends. on the following:
Is the Server configured to compress responses or not.
Whether the request is for a dynamic or static content. (some servers do not compress dynamic content)
Bandwidth and latency between server and client.
Size of the response being returned, on small responses it wont make big a difference.
Also note, that adding the "accept-encoding" on the client side, tells the server "I understand gzip/deflate" and does not force the server to compress response.
It depends, by adding this in header you are just letting server know that client app making request can accept zipped content. If server is capable of sending zipped response it will zip the data back after interpreting this in header. On performance If data to be fetched is huge zipping may help otherwise zipping will be small overhead which generally is negligible.
I'm doing research into web programming in ASP .NET and came across these two classes. I was wondering what might these be used for?
My first thought is that they could be used if you were setting up a proxy between the client and server, but I'm not sure if this is the main purpose or not.
Thanks
edit: classes not methods
They can indeed be used for that. This isn't specific to ASP.NET however.
You can create a HttpWebRequest object by doing:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://url.com");
And you invoke it to retrieve an HttpWebResponse:
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
There's a lot of customization you can do here, but hopefully this will give you a starting point for reading data off the web.
As their names imply, these are classes to either create a request (HttpWebRequest) or create a response (HttpWebResponse).
Using HttpWebRequest you define a request to a URI using the HTTP-protocol. Whereas the HttpWebResponse class delivers you an answer of an HTTP-server providing all the information like HTTP-headers and actual body of the request.
This is an example from MSDN
HttpWebRequest HttpWReq =
(HttpWebRequest)WebRequest.Create("http://www.contoso.com");
HttpWebResponse HttpWResp = (HttpWebResponse)HttpWReq.GetResponse();
// Insert code that uses the response object.
HttpWResp.Close();
Consider the casting from the base-class WebRequest.
See HttpWebRequest and HttpWebResponse.
They are used for communicating with another process using the HTTP protocol.
In the context of ASP.NET, your process could use them to talk to another service. Maybe your database uses the HTTP protocol, such as CouchDB. Perhaps you have a rest service that your ASP.NET application needs to talk to.
I'm trying to use YoutubeFisher library with ASP.NET. I make an HttpWebRequest to grab html content, process the contest to extract the video links and display links on the web page. I managed to make it work on localhost. I can retrieve video links and download the video on the locahost. But when I push it to the Server, it works only if I send the request from the same Server. If that page is accessed by a client browser, the client can see the links properly, but when link is clicked the client gets the HTTP Error 403, everytime the client clicks on the link even though the link is correct.
My analysis is that when the Server makes HttpWebRequest to grab HTML contet, it sends HTTP header as well. The HTML content (links to the video file) that is sent back from YouTube server, I think, will reponse to only the request that matches that HTTP header, that is sent from the Server. So, when client clicks on the link it sends request to YouTube server with different HTTP header.
So, I'm thinking of getting the HTTP header from the client, then modify the Server HTTP header to include HTTP header info of the client before making HttpWebRequest. I'm not quite sure if this will work. As far as I know, HTTP heaer cannot be modified.
Below is the code that makes HttpWebRequest from YouTubeFisher library,
public static YouTubeService Create(string youTubeVideoUrl)
{
YouTubeService service = new YouTubeService();
service.videoUrl = youTubeVideoUrl;
service.GetVideoPageHtmlSource();
service.GetVideoTitle();
service.GetDownloadUrl();
return service;
}
private void GetVideoPageHtmlSource()
{
HttpWebRequest req = HttpWebRequest.Create(videoUrl) as HttpWebRequest;
HttpWebResponse resp = req.GetResponse() as HttpWebResponse;
videoPageHtmlSource = new StreamReader(resp.GetResponseStream(), Encoding.UTF8).ReadToEnd();
resp.Close();
}
Client browses the page but the links are there but give HTTP 403:
Browse the page the from the Server itself, everything works as expected:
How do I make HttpWebRequest on the behalf of the client then? Is my analysis of this problem correct?
Thank you for your input.
Use an http monitor such as Charles, Fiddler or even Firebug to find out what additional headers are being sent from the brower in the success case. I suspect you'll need to duplicate one or more of accept, user-agent or referer.
In the past I've just assumed that youtube has those links encoded so that they only work for the original request IP. If that were the case it would be far more difficult. I have no clue if this is the case or not, try forwarding all the header elements you can before going down this route...
The only possibility that comes to mind is that you'd have to use a javascript request to download the page to the client's browser, then upload that to your server for processing, or do the processing in javascript.
Or you could have the client download the video stream via your server, so your server would pass through the data. This would obviously use a ton of your bandwidth.
I'm narrowing in on an underlying problem related to two prior questions.
Basically, I've got a URL that when I fetch it manually (paste it into browser) works just fine, but when I run through some code (using the HttpWebRequest) has a different result.
The URL (example):
http://208.106.250.207:8192/announce?info_hash=-%CA8%C1%C9rDb%ADL%ED%B4%2A%15i%80Z%B8%F%C&peer_id=01234567890123456789&port=6881&uploaded=0&downloaded=0&left=0&compact=0&no_peer_id=0&event=started
The code:
String uri = BuildURI(); //Returns the above URL
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(uri);
req.Proxy = new WebProxy();
WebResponse resp = req.GetResponse();
Stream stream = resp.GetResponseStream();
... Parse the result (which is an error message from the server claiming the url is incorrect) ...
So, how can I GET from a server given a URL? I'm obviously doing something wrong here, but can't tell what.
Either a fix for my code, or an alternative approach that actually works would be fine. I'm not wed at all to the HttpWebRequest method.
I recommend you use Fiddler to trace both the "paste in web browser" call and the HttpWebRequest call.
Once traced you will be able to see any differences between them, whether they are differences in the request url, in the form headers, etc, etc.
It may actually be worth pasting the raw requests from both (obtained from Fiddler) here, if you can't see anything obvious.
Well, the only they might differ is in the HTTP headers that get transmitted. In particular the User-Agent.
Also, why are you using a WebProxy? That is not really necessary and it most likely is not used by your browser.
The rest of your code is fine.. Just make sure you set up the HTTP headers correctly. Check this link out:
I would suggest that you get yourself a copy of WireShark and examine the communication that happens between your browser and the server that you are trying to access. Doing so will be rather trivial using WireShark and it will show you the exact HTTP message that is being sent from the browser.
Then take a look at the communication that goes on between your C# application and the server (again using WireShark) and then compare the two to find out what exactly is different.
If the communication is a pure HTTP GET method (i.e. there is no HTTP message body involved), and the URL is correct then the only two things I could think of are:
make sure that your are send the right protocol (i.e. HTTP/1.0 or HTTP/1.1 or whatever it is that you should be sending)
make sure that you are sending all required HTTP headers correctly, and obviously that you are not sending any HTTP headers that you shouldn't be sending.
There could be something wrong with the URL. Instead of using a string, it's usually better to use an instance of System.Uri:
String url = BuildURI(); //Returns the above URL
Uri uri = new Uri(url);
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(url);
req.Proxy = new WebProxy();
using (WebResponse resp = req.GetResponse()) {
using (Stream stream = resp.GetResponseStream()) {
// whatever
}
}
I think you need to see exactly what's flowing to your server in the HTTP request. Does sound likely that the headers are interestingly different.
You can introduce a some kind of debugging proxy between your request and the server (for example RAD has such a capability in the box).