We Keep Coding

Is there a method to send HttpWebRequest to download all url's and handles all exceptions?

I am working on a MultiThreadingDownloader over Http. So I have to work using HttpWebRequest to get partial requests. The application is working good for almost all URLs. But sometimes, when trying to get response it throws exception or works wrong (e.g ContentLength returns -1, TSL/SSL secure exception, cookie-required links)
I have not enough knowledge about client-server relationship, so I can't handle all exceptions.
I am using currently this method:
public static HttpWebRequest SendRequest(string url)
{
HttpWebRequest req = WebRequest.Create(url) as HttpWebRequest;
req.AllowAutoRedirect = true;
req.Accept = "*/*";
req.Method = "GET";
req.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)";
req.ServicePoint.ConnectionLimit = 8;
req.ServicePoint.Expect100Continue = true;
req.ProtocolVersion = HttpVersion.Version10;
ServicePointManager.ServerCertificateValidationCallback += (sender, cert, chain, sslPolicyErrors) => true;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12 | SecurityProtocolType.Ssl3;
return req;
}
What should an HttpWebRequest contain to get the HttpWebResponse successfully for all URL sources? My method is not working for all as I said. A detailed documentation or code block for the method can help me.

The server is at liberty to not provide a content length. This is useful for streaming data. Here, you can't do much about it. You can't support segmented downloads for such URLs. There is no way to discover this situation other than trying.
It's the same thing with HTTPS and cookies. The server is free to reply with anything that it wants and require any input that it likes.
You will need to handle all these cases specially. They are allowed by the HTTP protocol. HttpWebRequest does not have much built-in to help you. The only helpful features in this regard are following redirects and decompression. This is default-on so you probably did not even notice it's there.

suppress save/dialog in web browser and automate the download

I want to automate the download of an exe prompted from a link from the client side. I can get the first redirected link from http://go.microsoft.com/fwlink/?LinkID=149156 to http://www.microsoft.com/getsilverlight/handlers/getsilverlight.ashx. Please click and check how it works. fwlink -> .ashx - >.exe ...i want to get the direct link to the .exe.
But the response returns 404 when requesting the Web handler through the code but if you try on Browser it actually downloads.
Can anyone suggest how to automate the download form the above link? The code i am using to get the link redirected is this one.
public static string GetLink(string url)
{
HttpWebRequest httpWebRequest = WebRequest.Create(url) as HttpWebRequest;
httpWebRequest.Method = "HEAD";
httpWebRequest.AllowAutoRedirect = false;
// httpWebRequest.ContentType = "application/octet-stream";
//httpWebRequest.Headers.Add("content-disposition", "attachment; filename=Silverlight.exe");
HttpWebResponse httpWebResponse = httpWebRequest.GetResponse() as HttpWebResponse;
if (httpWebResponse.StatusCode == HttpStatusCode.Redirect)
{
return httpWebResponse.GetResponseHeader("Location");
}
else
{
return null;
}
}

Just tested this out and it will download the file.
WebClient client = new WebClient();
client.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
client.DownloadFile(url, "Filename.exe");
You just needed to add the user-agent as the particular silverlight download depends on what browser you are running on, hence if it can't detect one then it will fail.
Change the user-agent to something that will trigger the appropriate download you want.

Detecting 302 Redirect

I'm trying to check the redirect location of a url but am always getting the wrong results. For example, for the url http://www.yellowpages.com.eg/Mjg3NF9VUkxfMTEwX2h0dHA6Ly93d3cubG90dXMtYWlyLmNvbV8=/Lotus-Air/profile.html, it redirects to http://www.lotus-air.com with a type of redirect 302 Found (you can test it on the this service http://www.internetofficer.com/seo-tool/redirect-check/), however am getting "http://mobile.yellowpages.com.eg/" as the webResp.GetResponseHeader("Location") . My Code is as follows:
string url = #"http://www.yellowpages.com.eg/Mjg3NF9VUkxfMTEwX2h0dHA6Ly93d3cubG90dXMtYWlyLmNvbV8=/Lotus-Air/profile.html";
HttpWebRequest webReq = WebRequest.Create(url) as HttpWebRequest;
webReq.Method = "HEAD";
webReq.AllowAutoRedirect = false;
HttpWebResponse webResp = webReq.GetResponse() as HttpWebResponse;
txtOutput.Text += webResp.StatusCode.ToString() + "\r\n" ;
txtOutput.Text += webResp.GetResponseHeader("Location") + "\r\n";
txtOutput.Text += webResp.ResponseUri.ToString();
webResp.Close();
Thanks.
Yehia

They are probably sending different redirects based on the user agent, so you get one result in a browser and another in your code.

You could use a HTTP debugging proxy to get an understanding of the headers moving back and forth and enables to you to change your user-agent to help test Ben's theory (I +1'd that).
A good one is Fiddler - Web Debugging Proxy free and easy to use/
The screenshot below shows me changing the useragent to an old IEMobile one "Mozilla/4.0 (compatible; MSIE 6.0; Windows CE; IEMobile 6.12; en-US; KIN.Two 1.0)", which redirects me to mobile.yellowpages.com.eg
n.b. changing to an ipad useragent takes you to iphone.yellowpages.com.eg

As Ben pointed out, it redirects based on user agent. Just add some user agent (this one is for chrome):
webReq.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13";
For me it redirects to http://www.lotus-air.com.

C# webclient cannot getting response from https protocol

When i trying to load html from server by https, it returning an error code 500: but when i open same link in browser it works fine: is there any way to do this? I'm using Webclient and also sending a useragent information to the server:
HttpWebRequest req1 = (HttpWebRequest)WebRequest.Create("mobile.unibet.com/";);
req1.UserAgent = #"Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5";
var response1 = req1.GetResponse();
var responsestream1 = response1.GetResponseStream();

David is correct, this generally happens when the server is expecting some headers that is not passed through, in your case Accept
this code works now
string requestUrl = "https://mobile.unibet.com/unibet_index.t";
var request = (HttpWebRequest)WebRequest.Create(requestUrl);
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.UserAgent = "//Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)";
using (var response = request.GetResponse() as HttpWebResponse)
{
using (var sr = new StreamReader(response.GetResponseStream()))
{
var responsestring = sr.ReadToEnd();
if (!string.IsNullOrEmpty(responsestring))
{
Console.WriteLine(responsestring);
}
}
}

This should probably be a comment but there's not enough room in the comment for all the questions... I don't think the question has enough information to answer with any level of confidence.
A 500 error means a problem at the server. The short answer is that the browser is sending some content that the WebClient is not.
The WebClient may not be sending headers that are expected by the server. Does the server require authentication? Is this a page on a company that you've contracted with that perhaps provided you with credentials or an API key that was Do you need to add HTTP Authorization?
If this is something you're doing with a company that you've got a partnership with, you should be able to ask them to help trace why you're getting a 500 error. Otherwise, you may need to provide us with a code sample and more details so we can offer more suggestions.

Website not returning cookies

I am trying to log into a website to send SMS via a windows phone 7 app. I have 2 providers working but when I try Vodafone I am running into an error.
From what I gather it seems that the response does not contain cookies, or they are not being read. The request logs in ok and the response I get back is the correct page but it contains no cookies.
The Url:
RequestUrl = String.Format("https://www.vodafone.ie/myv/services/login/Login.shtml?username={0}&password={1}", userSettings.Username, userSettings.Password),
The Request:
Request = (HttpWebRequest)WebRequest.Create((requestCollection.CurrentRequest().RequestUrl));
if (Request.CookieContainer == null)
{
Request.CookieContainer = cookieJar.CookieContainer;
Request.AllowAutoRedirect = true;
Request.AllowReadStreamBuffering = true;
Request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.6) Gecko/20060728 Firefox/1.5.0.6";
}
Where the code errors as the response cookies could not be evaluated:
public void AddCookiesToContainer(HttpWebResponse response)
{
CookieCollection.Add(response.Cookies);
CookieContainer.Add(response.ResponseUri, CookieCollection);
}
And below is the debugger showing no cookies :(

Which line of the code has the error?
Have you verified that the service does return cookies? (i.e. If you make the same request from a PC)
Edit:
The remote host is returning cookies in it's redirection to the index page but in that page there are no cookies in the response. This would explain why there are no cookies in the collection when you try and use it.
Verify this behaviour against a PC client, inspect the body of the response from index.jsp ans this may contain information to help debug and check the documentation on how the process is supposed to work.