I writting an application which is using WebClient class.
Adding something like that:
ExC.Headers.Add("Accept-Encoding: gzip, deflate");
where ExC is:
class ExWebClient1 : WebClient
{
protected override WebRequest GetWebRequest(Uri address)
{
HttpWebRequest request = (HttpWebRequest)base.GetWebRequest(address);
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
return request;
}
}
It will be a diffrence in speed when i will be using encoded response?
Short answer is usually Yes.
Long answer, It depends. on the following:
Is the Server configured to compress responses or not.
Whether the request is for a dynamic or static content. (some servers do not compress dynamic content)
Bandwidth and latency between server and client.
Size of the response being returned, on small responses it wont make big a difference.
Also note, that adding the "accept-encoding" on the client side, tells the server "I understand gzip/deflate" and does not force the server to compress response.
It depends, by adding this in header you are just letting server know that client app making request can accept zipped content. If server is capable of sending zipped response it will zip the data back after interpreting this in header. On performance If data to be fetched is huge zipping may help otherwise zipping will be small overhead which generally is negligible.
Related
I have below code in a third party function, it posts file to a webserver. I want to post data in parts, what change should i do in the code. Below code is working and the "request" object contains everything.
private static HttpWebResponse GetRawResponse(HttpWebRequest request)
{
return (HttpWebResponse)request.GetResponse();
}
Also is there a way in which I can find out what is the full name (with path) of the file which is going to be uploaded from the httpwebrequest object.
Thanks.
HttpWebRequest does not spawn multiple HTTP requests. You cannot upload a file in chunks unless you actually create multiple HttpWebRequests. I don't think that would be useful unless you have a server side process to stitch them back together.
If you still want true chunked uploading you might want to look at raw TCP or some other mechanism.
Hope that helps. See this post as well.
I can use both HttpWebRequest to send an HTTP request and get an HTTP response without a WebClient.
When should you use HttpWebRequest and when should you use WebClient?
WebClient can be used when you don't need any fine-tuning.
When using HttpWebRequest, you can control various options, including timeouts (very important). So basically - WebClient for toy projects / POCs, HttpWebRequest for actual business.
Personally I always use WebClient. The API seems more simple. It uses HttpWebRequest under the covers.
WebClient is ideal for downloads and uploads.
HttpWebRequest is ideal for web connections, including sending HTTP POST requests, as seen here: HTTP request with post
If you do not need access to the underlying stream but are just uploading or downloading "data", i.e. a file some bytes or a string, WebClient is a simplifying abstraction.
I am using WebClient with C#
the following code works fine
wc = new WebClient();
wc.Headers.Add("Content-Type", "application/x-www-form-urlencoded");
NameValueCollection nvc = new NameValueCollection();
nvc.Add("webdata", JsonConvert.SerializeObject(webdata));
response = wc.UploadValues(#"http://localhost/api/results", "PUT", nvc);
The application is most likely to be used over a mobile data connection, so to minimize costs, i would like to make sure the data is compressed, as it is all txt. I have used json instead of xml to reduce the size (and could possibly alter the format to reduce overhead further)
Do i need to compress the data it manually prior to adding it to the WebClient
or is there some way i can tell WebClient that my webserver can handle compression
(or does compression on the webserver only work for downloads?)
i am running apache/php on the webserver
thanks in advance
Http compression is normally only used for responses. It is possible to compress requests, but not all web servers will accept these requests and decompress them.
Have you tried adding a header of type "Content-Encoding" and value "gzip" to your request?
You'll still have to compress the contents manually with a GZipStream and write the compressed bytes out to the request stream though.
don't forget to flush your writers to the streams, or not all data will be sent over the wire :)
I'm narrowing in on an underlying problem related to two prior questions.
Basically, I've got a URL that when I fetch it manually (paste it into browser) works just fine, but when I run through some code (using the HttpWebRequest) has a different result.
The URL (example):
http://208.106.250.207:8192/announce?info_hash=-%CA8%C1%C9rDb%ADL%ED%B4%2A%15i%80Z%B8%F%C&peer_id=01234567890123456789&port=6881&uploaded=0&downloaded=0&left=0&compact=0&no_peer_id=0&event=started
The code:
String uri = BuildURI(); //Returns the above URL
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(uri);
req.Proxy = new WebProxy();
WebResponse resp = req.GetResponse();
Stream stream = resp.GetResponseStream();
... Parse the result (which is an error message from the server claiming the url is incorrect) ...
So, how can I GET from a server given a URL? I'm obviously doing something wrong here, but can't tell what.
Either a fix for my code, or an alternative approach that actually works would be fine. I'm not wed at all to the HttpWebRequest method.
I recommend you use Fiddler to trace both the "paste in web browser" call and the HttpWebRequest call.
Once traced you will be able to see any differences between them, whether they are differences in the request url, in the form headers, etc, etc.
It may actually be worth pasting the raw requests from both (obtained from Fiddler) here, if you can't see anything obvious.
Well, the only they might differ is in the HTTP headers that get transmitted. In particular the User-Agent.
Also, why are you using a WebProxy? That is not really necessary and it most likely is not used by your browser.
The rest of your code is fine.. Just make sure you set up the HTTP headers correctly. Check this link out:
I would suggest that you get yourself a copy of WireShark and examine the communication that happens between your browser and the server that you are trying to access. Doing so will be rather trivial using WireShark and it will show you the exact HTTP message that is being sent from the browser.
Then take a look at the communication that goes on between your C# application and the server (again using WireShark) and then compare the two to find out what exactly is different.
If the communication is a pure HTTP GET method (i.e. there is no HTTP message body involved), and the URL is correct then the only two things I could think of are:
make sure that your are send the right protocol (i.e. HTTP/1.0 or HTTP/1.1 or whatever it is that you should be sending)
make sure that you are sending all required HTTP headers correctly, and obviously that you are not sending any HTTP headers that you shouldn't be sending.
There could be something wrong with the URL. Instead of using a string, it's usually better to use an instance of System.Uri:
String url = BuildURI(); //Returns the above URL
Uri uri = new Uri(url);
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(url);
req.Proxy = new WebProxy();
using (WebResponse resp = req.GetResponse()) {
using (Stream stream = resp.GetResponseStream()) {
// whatever
}
}
I think you need to see exactly what's flowing to your server in the HTTP request. Does sound likely that the headers are interestingly different.
You can introduce a some kind of debugging proxy between your request and the server (for example RAD has such a capability in the box).
I'm trying to come up with the most appropriate way to make a two way socket connection through a HTTP proxy - lets say it's a telnet style protocol. Unfortunately I also need to support NTLM authentication (with the proxy) as well as Basic and Digest, in addition to any other future authentication mechanisms that I can't forecast.
If it was just basic and digest I'd handle the connection myself, but I really don't want to get stuck in the mire that is NTLM. Looking at the underlying AuthenticationManager API it looks very tied to HttpWebRequest so I can't leverage that functionality if I'm using a socket/tcpclient/whatever or even writing a new WebRequest derivation.
Playing around with HttpWebResponse yields a stream that can't be written, using the RequestStream after the response stream has been retrieved gives a concurrent io exception.
Having run through all the possibilities I can think of, I've come up with some nasty code that gets out the NetworkStream associated with a HttpWebRequest which allows two way communication:
.....
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
Stream str = resp.GetResponseStream();
System.Type type = str.GetType();
PropertyInfo info = type.GetProperty("Connection", BindingFlags.NonPublic|BindingFlags.Instance| BindingFlags.Public);
object obj = info.GetValue(str, null);
type = obj.GetType();
info = type.GetProperty("NetworkStream", BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.Public);
object obj2 = info.GetValue(obj, null);
NetworkStream networkStream = obj2 as NetworkStream;
Which I'm fairly repulsed by (it won't work with Mono for a start), so I'm wondering if there's a better way using public APIs which will allow me to leverage the built in runtime functionality of proxy authentication.
HTTP is two way. Clients can send a dataless request using HTTP GET (although even then data can be put in the URL or headers), or they can send data using HTTP POST, and the server gets to send a response with headers and data.
If when you say "two way" you were thinking something more like a simple TCP socket where client and server read and write at will, then sorry, but that's not what HTTP does. The client sends a request and the server sense a response. That's all. Technically, if you didn't have a client side API getting in the way, enforcing the intended constraints of HTTP, and you could cook up your own non-standard server, you could have multiple client<->server exchanges within a single HTTP request, but at that point it wouldn't really be HTTP anymore, it'd be a TCP connection with an HTTP like handshake, and your proxy might not even allow it.
That said, it sounds like you don't really need to write to the response stream at all, either you're quite confused and you just need to do a POST (see GetRequestStream), or you're just a little confused and you can just send a new request after you've processed the response. You can even reuse the same HttpWebRequest instance once you've called the .Close method on the WebResponse you got. And all this will happen on the same TCP socket (if your server and proxy support it).
Okay, I hope that all made sense. If it didn't answer your question one way or the other, just provide a little more detail about what you're trying to accomplish with regard to "two-way" communication. I understand that you have the constraint of going through an HTTP proxy w/ HTTP authentication requirements which limits things a lot.