We Keep Coding

Does HttpWebRequest need to be closed? How

I have a small program that keep posting some data (4MB) to remote server. Each time when an internal 4MB buffer is full, I create a new HttpWebRequest, call BeginGetRequestStream/EndGetRequestStream, write data to stream, close it, and done. Because I don't care about response, so I didn't call GetResponse/BeginGetResponse/EndGetResponse.
My question is, since HttpWebRequest doesn't have a Close method, is it safe to leave current HttpWebRequest instance behind? If I didn't do anything to that HttpWebRequest, will the connection be automatically closed? If answer is no, how to explicitly close current HTTP connection? Thanks.

You shouldn't have to cleanup HttpWebRequest but you do need to close the stream coming back from GetRequestStream or the response from GetResponse. I know you said you aren't using any of the response methods, but you must be using one of them to actually make the request.
After you are finished with a WebResponse object, you must close it by calling the Close method. Alternatively, if you have gotten the response stream from the response object, you can close the stream by calling the Stream.Close method. If you do not close the response or the stream, your application can run out of connections to the server and become unable to process additional requests.[1]
[1] https://msdn.microsoft.com/en-us/library/debx8sh9.aspx

Reroute incoming request

How can I send a received request to another url, and send the response to the original sender?
I have an ashx generic handler and can get the sent request using Request.InputStream . However - that doesn't include everything (like headers). Is there a way of sending the whole request as is, and then sending the whole response as is?
Just to be completely clear: a.ashx gets an HttpContext from somewhere.com. I want it to send the response as if somewhere.com was communicating directly with b.ashx.

Have you considered using the Server.Transfer to redirect the query to the required url?
Simply use
Server.Transfer("Page2.aspx", true);

Access YouTube with ASP.NET and YoutubeFisher but gets HTTP Error 403

I'm trying to use YoutubeFisher library with ASP.NET. I make an HttpWebRequest to grab html content, process the contest to extract the video links and display links on the web page. I managed to make it work on localhost. I can retrieve video links and download the video on the locahost. But when I push it to the Server, it works only if I send the request from the same Server. If that page is accessed by a client browser, the client can see the links properly, but when link is clicked the client gets the HTTP Error 403, everytime the client clicks on the link even though the link is correct.
My analysis is that when the Server makes HttpWebRequest to grab HTML contet, it sends HTTP header as well. The HTML content (links to the video file) that is sent back from YouTube server, I think, will reponse to only the request that matches that HTTP header, that is sent from the Server. So, when client clicks on the link it sends request to YouTube server with different HTTP header.
So, I'm thinking of getting the HTTP header from the client, then modify the Server HTTP header to include HTTP header info of the client before making HttpWebRequest. I'm not quite sure if this will work. As far as I know, HTTP heaer cannot be modified.
Below is the code that makes HttpWebRequest from YouTubeFisher library,
public static YouTubeService Create(string youTubeVideoUrl)
{
YouTubeService service = new YouTubeService();
service.videoUrl = youTubeVideoUrl;
service.GetVideoPageHtmlSource();
service.GetVideoTitle();
service.GetDownloadUrl();
return service;
}
private void GetVideoPageHtmlSource()
{
HttpWebRequest req = HttpWebRequest.Create(videoUrl) as HttpWebRequest;
HttpWebResponse resp = req.GetResponse() as HttpWebResponse;
videoPageHtmlSource = new StreamReader(resp.GetResponseStream(), Encoding.UTF8).ReadToEnd();
resp.Close();
}
Client browses the page but the links are there but give HTTP 403:
Browse the page the from the Server itself, everything works as expected:
How do I make HttpWebRequest on the behalf of the client then? Is my analysis of this problem correct?
Thank you for your input.

Use an http monitor such as Charles, Fiddler or even Firebug to find out what additional headers are being sent from the brower in the success case. I suspect you'll need to duplicate one or more of accept, user-agent or referer.

In the past I've just assumed that youtube has those links encoded so that they only work for the original request IP. If that were the case it would be far more difficult. I have no clue if this is the case or not, try forwarding all the header elements you can before going down this route...
The only possibility that comes to mind is that you'd have to use a javascript request to download the page to the client's browser, then upload that to your server for processing, or do the processing in javascript.
Or you could have the client download the video stream via your server, so your server would pass through the data. This would obviously use a ton of your bandwidth.

c# httpwebrequest credential problem

I am trying to login into www.diary.com using a httpwebrequest object. However, it always fail to login, and kept giving me back the login page. Can anyone enlighten me on what is/are wrong?
My code is as follows:
// prepare the web page we will be asking for
HttpWebRequest request = (HttpWebRequest)
WebRequest.Create(#"http://diary.com/events/agenda");
request.ContentType = "text/html";
request.Credentials = new NetworkCredential(#"user#hotmail.com", "password");
request.AllowAutoRedirect = true;
request.Referer = #"http://diary.com/";
// execute the request
HttpWebResponse response = (HttpWebResponse)
request.GetResponse();
// we will read data via the response stream
Stream resStream = response.GetResponseStream();
// set the WebBrowser object documentStream to the response stream
myWB.DocumentStream = resStream;
// simply tell me the title of the webpage
MessageBox.Show(myWB.Document.Title);

You have two problems here:
You are providing credentials at the protocol level, which is not how most websites (including this one) work. The protocol is totally anonymous, and the site uses Forms Authentication to log you in. Your code needs to actually create a POST request that mimics submitting the login form. The response that comes back from the server will include a cookie that has your auth token, which leads into...
You need to persist cookies across requests. After you submit the login request and get the cookie, you'll need to hang on to it and send it along in the request headers of each subsequent request. The easiest way to do this is to use a WebClient for spanning multiple requests, and a CookieContainer to track the cookies for you.
If you're ever unsure about how to mimic the traffic that goes between your browser and a website, a great tool to use is Fiddler. It captures the raw request/response for you to observe.

How does HttpWebRequest differ (functional) from pasteing a URL into an address bar?

I'm narrowing in on an underlying problem related to two prior questions.
Basically, I've got a URL that when I fetch it manually (paste it into browser) works just fine, but when I run through some code (using the HttpWebRequest) has a different result.
The URL (example):
http://208.106.250.207:8192/announce?info_hash=-%CA8%C1%C9rDb%ADL%ED%B4%2A%15i%80Z%B8%F%C&peer_id=01234567890123456789&port=6881&uploaded=0&downloaded=0&left=0&compact=0&no_peer_id=0&event=started
The code:
String uri = BuildURI(); //Returns the above URL
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(uri);
req.Proxy = new WebProxy();
WebResponse resp = req.GetResponse();
Stream stream = resp.GetResponseStream();
... Parse the result (which is an error message from the server claiming the url is incorrect) ...
So, how can I GET from a server given a URL? I'm obviously doing something wrong here, but can't tell what.
Either a fix for my code, or an alternative approach that actually works would be fine. I'm not wed at all to the HttpWebRequest method.

I recommend you use Fiddler to trace both the "paste in web browser" call and the HttpWebRequest call.
Once traced you will be able to see any differences between them, whether they are differences in the request url, in the form headers, etc, etc.
It may actually be worth pasting the raw requests from both (obtained from Fiddler) here, if you can't see anything obvious.

Well, the only they might differ is in the HTTP headers that get transmitted. In particular the User-Agent.
Also, why are you using a WebProxy? That is not really necessary and it most likely is not used by your browser.
The rest of your code is fine.. Just make sure you set up the HTTP headers correctly. Check this link out:
I would suggest that you get yourself a copy of WireShark and examine the communication that happens between your browser and the server that you are trying to access. Doing so will be rather trivial using WireShark and it will show you the exact HTTP message that is being sent from the browser.
Then take a look at the communication that goes on between your C# application and the server (again using WireShark) and then compare the two to find out what exactly is different.
If the communication is a pure HTTP GET method (i.e. there is no HTTP message body involved), and the URL is correct then the only two things I could think of are:
make sure that your are send the right protocol (i.e. HTTP/1.0 or HTTP/1.1 or whatever it is that you should be sending)
make sure that you are sending all required HTTP headers correctly, and obviously that you are not sending any HTTP headers that you shouldn't be sending.

There could be something wrong with the URL. Instead of using a string, it's usually better to use an instance of System.Uri:
String url = BuildURI(); //Returns the above URL
Uri uri = new Uri(url);
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(url);
req.Proxy = new WebProxy();
using (WebResponse resp = req.GetResponse()) {
using (Stream stream = resp.GetResponseStream()) {
// whatever
}
}

I think you need to see exactly what's flowing to your server in the HTTP request. Does sound likely that the headers are interestingly different.
You can introduce a some kind of debugging proxy between your request and the server (for example RAD has such a capability in the box).