I am building an application with C# and one of the requirements is that the headers have to contain very specific data. I have been working with the WebClient and HttpWebRequest classes in C# but both do not have the option to explicitly remove the Connection header. This is what I have tried:
WebClient
client.Headers.Remove("Connection");
client.Headers.Add("Connection", null); // if only...
HttpWebRequest
request.Connection = null;
request.Connection = "";
Some of what I have read is that you can only have this header set to Keep-Alive or Close. I need it to not even be on the web request.
All other required headers are adding just fine. It is this one that is causing a head ache. I'm beginning to believe that removing the Connection header is not possible. Is there any language where it is possible if this is the case?
Much appreciation for the time you guys take to look at this!
Thank you!
I'm not quite sure how to remove that specific header, but you could try using the HttpClient class instead of HttpWebRequest or WebClient.
HttpClient class
This class is more up-to-date way of executing HTTP requests in C#
Related
I'm doing research into web programming in ASP .NET and came across these two classes. I was wondering what might these be used for?
My first thought is that they could be used if you were setting up a proxy between the client and server, but I'm not sure if this is the main purpose or not.
Thanks
edit: classes not methods
They can indeed be used for that. This isn't specific to ASP.NET however.
You can create a HttpWebRequest object by doing:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://url.com");
And you invoke it to retrieve an HttpWebResponse:
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
There's a lot of customization you can do here, but hopefully this will give you a starting point for reading data off the web.
As their names imply, these are classes to either create a request (HttpWebRequest) or create a response (HttpWebResponse).
Using HttpWebRequest you define a request to a URI using the HTTP-protocol. Whereas the HttpWebResponse class delivers you an answer of an HTTP-server providing all the information like HTTP-headers and actual body of the request.
This is an example from MSDN
HttpWebRequest HttpWReq =
(HttpWebRequest)WebRequest.Create("http://www.contoso.com");
HttpWebResponse HttpWResp = (HttpWebResponse)HttpWReq.GetResponse();
// Insert code that uses the response object.
HttpWResp.Close();
Consider the casting from the base-class WebRequest.
See HttpWebRequest and HttpWebResponse.
They are used for communicating with another process using the HTTP protocol.
In the context of ASP.NET, your process could use them to talk to another service. Maybe your database uses the HTTP protocol, such as CouchDB. Perhaps you have a rest service that your ASP.NET application needs to talk to.
Basically what I've been trying to do is download a file off a server. The server sends a redirect automatically which is fine, but through packet sniffing a program that does successfully download the file I've found that the Headers (for the second request) are:
GET /path/to/file.txt
...
Host: server.com
Rather than the current response being generated (what I thought was standard):
GET www.server.com/path/to/file.txt
Using the normal HttpWebRequest method results in a 500 server error, and I get exceptions thrown when trying to use just the relative path as one would expect.
Using AllowAutoRedirect does not work for this scenario as the cookies are not handled properly, but even if I handle it manually the same error occurs.
How does one go about doing this (preferably without sockets :D)?
To be honest, I'm really not sure what you are asking, but you mentioned cookie troubles. As a total shot in the dark guess, are you setting the CookieContainer on your WebRequest?
request.CookieContainer = new CookieContainer();
request.AllowAutoRedirect = true;
I can use both HttpWebRequest to send an HTTP request and get an HTTP response without a WebClient.
When should you use HttpWebRequest and when should you use WebClient?
WebClient can be used when you don't need any fine-tuning.
When using HttpWebRequest, you can control various options, including timeouts (very important). So basically - WebClient for toy projects / POCs, HttpWebRequest for actual business.
Personally I always use WebClient. The API seems more simple. It uses HttpWebRequest under the covers.
WebClient is ideal for downloads and uploads.
HttpWebRequest is ideal for web connections, including sending HTTP POST requests, as seen here: HTTP request with post
If you do not need access to the underlying stream but are just uploading or downloading "data", i.e. a file some bytes or a string, WebClient is a simplifying abstraction.
I'm narrowing in on an underlying problem related to two prior questions.
Basically, I've got a URL that when I fetch it manually (paste it into browser) works just fine, but when I run through some code (using the HttpWebRequest) has a different result.
The URL (example):
http://208.106.250.207:8192/announce?info_hash=-%CA8%C1%C9rDb%ADL%ED%B4%2A%15i%80Z%B8%F%C&peer_id=01234567890123456789&port=6881&uploaded=0&downloaded=0&left=0&compact=0&no_peer_id=0&event=started
The code:
String uri = BuildURI(); //Returns the above URL
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(uri);
req.Proxy = new WebProxy();
WebResponse resp = req.GetResponse();
Stream stream = resp.GetResponseStream();
... Parse the result (which is an error message from the server claiming the url is incorrect) ...
So, how can I GET from a server given a URL? I'm obviously doing something wrong here, but can't tell what.
Either a fix for my code, or an alternative approach that actually works would be fine. I'm not wed at all to the HttpWebRequest method.
I recommend you use Fiddler to trace both the "paste in web browser" call and the HttpWebRequest call.
Once traced you will be able to see any differences between them, whether they are differences in the request url, in the form headers, etc, etc.
It may actually be worth pasting the raw requests from both (obtained from Fiddler) here, if you can't see anything obvious.
Well, the only they might differ is in the HTTP headers that get transmitted. In particular the User-Agent.
Also, why are you using a WebProxy? That is not really necessary and it most likely is not used by your browser.
The rest of your code is fine.. Just make sure you set up the HTTP headers correctly. Check this link out:
I would suggest that you get yourself a copy of WireShark and examine the communication that happens between your browser and the server that you are trying to access. Doing so will be rather trivial using WireShark and it will show you the exact HTTP message that is being sent from the browser.
Then take a look at the communication that goes on between your C# application and the server (again using WireShark) and then compare the two to find out what exactly is different.
If the communication is a pure HTTP GET method (i.e. there is no HTTP message body involved), and the URL is correct then the only two things I could think of are:
make sure that your are send the right protocol (i.e. HTTP/1.0 or HTTP/1.1 or whatever it is that you should be sending)
make sure that you are sending all required HTTP headers correctly, and obviously that you are not sending any HTTP headers that you shouldn't be sending.
There could be something wrong with the URL. Instead of using a string, it's usually better to use an instance of System.Uri:
String url = BuildURI(); //Returns the above URL
Uri uri = new Uri(url);
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(url);
req.Proxy = new WebProxy();
using (WebResponse resp = req.GetResponse()) {
using (Stream stream = resp.GetResponseStream()) {
// whatever
}
}
I think you need to see exactly what's flowing to your server in the HTTP request. Does sound likely that the headers are interestingly different.
You can introduce a some kind of debugging proxy between your request and the server (for example RAD has such a capability in the box).
I know about the WebRequest and the WebResponse objects. The problem is that I do not really want to get the source code of the webpage, I only want to check to see if the link exists or not. The thing is, if I use the GetResponse method, it goes an pull the entire source code of the site.
I am creating a broken link checker with many links. It takes quite a while to check them all. If there a way to to get MINIMAL information from a weblink? Only enough information to see if the link is valid or broken (not the entire source code).
An answer (BESIDES USING ASYNCHRONOUS TRANSFER) would be greatly appreciated!
WebRequest request = HttpWebRequest.Create("http://www.foo.com/");
request.Method = "HEAD"; // Just get the document headers, not the data.
HEAD is similar to GET, only that instead of getting the file contents, we get just the headers.
A standard way of checking the existence of a link is to use a HEAD request, which causes the remote server to send the headers for the requested object, but not the object itself. If you thus requested an object that is not on the server, the server gives you the normal 404 response, but if it does exist, you get a 200 response and no data after the headers. This way very little uninteresting data goes over the wire.