We Keep Coding

Making Http PUT request with query strings using C#

I tried below code as I wanted to send a http PUT request to a REST Api. The sample request looks like
http://localhost/SomeWebServices/some/something/setsomething?ID=12121212&Number=121212&Product=BUISCUITE&EffectiveDate=2015-10-22
WebClient request = new WebClient();
request.QueryString.Add("CustomerNumber", "bla");
request.QueryString.Add("AccountNumber", "bla");
request.UploadString(new Uri(BaseUrl + ServiceEndPoint), "PUT", String.Empty);
When I debug the code I am getting a 401 Unauthorized error, Am I doing the correct thing to make a HTTP PUT request ? Is there a way to intercept the request to see if its in correct format. My service is hosted in local IIS instance. Although I have used WebClient() you are more than welcome to share your elegant approach if there is a better & easy way. Thanks.

Inspect headers set by WebClient proxy

I'm executing request through some free proxy servers, and I would like to know what headers each proxy server sets. Right now I'm visiting a page that prints out the result in the html body.
using(WebClient client = new WebClient())
{
WebProxy wp = new WebProxy("proxy url");
client.Proxy = wp;
string str = client
.DownloadString("http://www.pagethatprintsrequestheaders.com");
}
The WebClient doesn't show the modified headers, but the page prints the correct ones. Is there any way to find out what headers that are being set by the proxy without visiting a page that prints them like in my example? Do I have to create my own http listener?

When the proxy server sets its own headers, it is essentially performing its own web request. It can even hide or override some of the headers that you set using your WebProxy.
Consequently, only the target page (pagethatprintsrequestheaders.com) can reliably see the headers being set by the proxy. There is no guarantee that the proxy server will send back the headers that it had sent to the target, back to you.
To put it another way, it really depends on the proxy server implementation. if the proxy server you are using is based on Apache's ProxyPass, you'd probably see the headers being set! If it's a custom implementation, then you may not see it.
You can first try inspecting the client.ResponseHeaders property of the WebClient after your response comes back. If this does not contain headers matching what (pagethatprintsrequestheaders.com) reports, then it's indeed a custom or modified implementation.
You could then create your own proxy servers, but this is more involved. You would probably spin up an EC2 instance, install Squid/TinyProxy/YourCustomProxy on it and use that in your WebProxy call.
You may also want to modify your question and explain why you want to read the headers. There may be solutions to your overall goal that don't require reading headers at all but could be done in some other way.

It looks like your sending a request from your WebClient, through the proxy and its received by the host at www.pagethatprintsrequestheaders.com.
If the proxy is adding headers to the request, your webclient will never see them on it's request.
webclient proxys request
request with headers added
client -----------> proxy ----------------------> destination host
The webclient can only see the state of the request between it and the proxy. The proxy will create a new request to send to the destination host, and its that request to which the headers are added. It also that request that is received by the destination host (which is why when it echoes back the headers it can see those added by the proxy)
When the response comes back, the headers are set by the host. It's possible that the proxy will add some headers to the response, but even if it did, they are not likely to be the same headers it adds to a request.
response response
(forwarded by proxy) (headers set by host)
client <------------------- proxy <------------------------- destination host
Using a host that echo the headers back as part of the response payload is one option.
Another would be to use something between the proxy and the destination host to inspect the request there (e.g a packet sniffer or another proxy like Fiddler that lets you see the request headers).
If the proxy is outside of you network, getting between the proxy and the destination host will be difficult (unless the host is under your control).

Access YouTube with ASP.NET and YoutubeFisher but gets HTTP Error 403

I'm trying to use YoutubeFisher library with ASP.NET. I make an HttpWebRequest to grab html content, process the contest to extract the video links and display links on the web page. I managed to make it work on localhost. I can retrieve video links and download the video on the locahost. But when I push it to the Server, it works only if I send the request from the same Server. If that page is accessed by a client browser, the client can see the links properly, but when link is clicked the client gets the HTTP Error 403, everytime the client clicks on the link even though the link is correct.
My analysis is that when the Server makes HttpWebRequest to grab HTML contet, it sends HTTP header as well. The HTML content (links to the video file) that is sent back from YouTube server, I think, will reponse to only the request that matches that HTTP header, that is sent from the Server. So, when client clicks on the link it sends request to YouTube server with different HTTP header.
So, I'm thinking of getting the HTTP header from the client, then modify the Server HTTP header to include HTTP header info of the client before making HttpWebRequest. I'm not quite sure if this will work. As far as I know, HTTP heaer cannot be modified.
Below is the code that makes HttpWebRequest from YouTubeFisher library,
public static YouTubeService Create(string youTubeVideoUrl)
{
YouTubeService service = new YouTubeService();
service.videoUrl = youTubeVideoUrl;
service.GetVideoPageHtmlSource();
service.GetVideoTitle();
service.GetDownloadUrl();
return service;
}
private void GetVideoPageHtmlSource()
{
HttpWebRequest req = HttpWebRequest.Create(videoUrl) as HttpWebRequest;
HttpWebResponse resp = req.GetResponse() as HttpWebResponse;
videoPageHtmlSource = new StreamReader(resp.GetResponseStream(), Encoding.UTF8).ReadToEnd();
resp.Close();
}
Client browses the page but the links are there but give HTTP 403:
Browse the page the from the Server itself, everything works as expected:
How do I make HttpWebRequest on the behalf of the client then? Is my analysis of this problem correct?
Thank you for your input.

Use an http monitor such as Charles, Fiddler or even Firebug to find out what additional headers are being sent from the brower in the success case. I suspect you'll need to duplicate one or more of accept, user-agent or referer.

In the past I've just assumed that youtube has those links encoded so that they only work for the original request IP. If that were the case it would be far more difficult. I have no clue if this is the case or not, try forwarding all the header elements you can before going down this route...
The only possibility that comes to mind is that you'd have to use a javascript request to download the page to the client's browser, then upload that to your server for processing, or do the processing in javascript.
Or you could have the client download the video stream via your server, so your server would pass through the data. This would obviously use a ton of your bandwidth.

C# Sending an HttpWebRequest without the servername

Basically what I've been trying to do is download a file off a server. The server sends a redirect automatically which is fine, but through packet sniffing a program that does successfully download the file I've found that the Headers (for the second request) are:
GET /path/to/file.txt
...
Host: server.com
Rather than the current response being generated (what I thought was standard):
GET www.server.com/path/to/file.txt
Using the normal HttpWebRequest method results in a 500 server error, and I get exceptions thrown when trying to use just the relative path as one would expect.
Using AllowAutoRedirect does not work for this scenario as the cookies are not handled properly, but even if I handle it manually the same error occurs.
How does one go about doing this (preferably without sockets :D)?

To be honest, I'm really not sure what you are asking, but you mentioned cookie troubles. As a total shot in the dark guess, are you setting the CookieContainer on your WebRequest?
request.CookieContainer = new CookieContainer();
request.AllowAutoRedirect = true;

How does HttpWebRequest differ (functional) from pasteing a URL into an address bar?

I'm narrowing in on an underlying problem related to two prior questions.
Basically, I've got a URL that when I fetch it manually (paste it into browser) works just fine, but when I run through some code (using the HttpWebRequest) has a different result.
The URL (example):
http://208.106.250.207:8192/announce?info_hash=-%CA8%C1%C9rDb%ADL%ED%B4%2A%15i%80Z%B8%F%C&peer_id=01234567890123456789&port=6881&uploaded=0&downloaded=0&left=0&compact=0&no_peer_id=0&event=started
The code:
String uri = BuildURI(); //Returns the above URL
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(uri);
req.Proxy = new WebProxy();
WebResponse resp = req.GetResponse();
Stream stream = resp.GetResponseStream();
... Parse the result (which is an error message from the server claiming the url is incorrect) ...
So, how can I GET from a server given a URL? I'm obviously doing something wrong here, but can't tell what.
Either a fix for my code, or an alternative approach that actually works would be fine. I'm not wed at all to the HttpWebRequest method.

I recommend you use Fiddler to trace both the "paste in web browser" call and the HttpWebRequest call.
Once traced you will be able to see any differences between them, whether they are differences in the request url, in the form headers, etc, etc.
It may actually be worth pasting the raw requests from both (obtained from Fiddler) here, if you can't see anything obvious.

Well, the only they might differ is in the HTTP headers that get transmitted. In particular the User-Agent.
Also, why are you using a WebProxy? That is not really necessary and it most likely is not used by your browser.
The rest of your code is fine.. Just make sure you set up the HTTP headers correctly. Check this link out:
I would suggest that you get yourself a copy of WireShark and examine the communication that happens between your browser and the server that you are trying to access. Doing so will be rather trivial using WireShark and it will show you the exact HTTP message that is being sent from the browser.
Then take a look at the communication that goes on between your C# application and the server (again using WireShark) and then compare the two to find out what exactly is different.
If the communication is a pure HTTP GET method (i.e. there is no HTTP message body involved), and the URL is correct then the only two things I could think of are:
make sure that your are send the right protocol (i.e. HTTP/1.0 or HTTP/1.1 or whatever it is that you should be sending)
make sure that you are sending all required HTTP headers correctly, and obviously that you are not sending any HTTP headers that you shouldn't be sending.

There could be something wrong with the URL. Instead of using a string, it's usually better to use an instance of System.Uri:
String url = BuildURI(); //Returns the above URL
Uri uri = new Uri(url);
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(url);
req.Proxy = new WebProxy();
using (WebResponse resp = req.GetResponse()) {
using (Stream stream = resp.GetResponseStream()) {
// whatever
}
}

I think you need to see exactly what's flowing to your server in the HTTP request. Does sound likely that the headers are interestingly different.
You can introduce a some kind of debugging proxy between your request and the server (for example RAD has such a capability in the box).