We Keep Coding

C# webrequests->getresponse casts exception instead of returning status code

I'm attempting to make an image scraper, now for some pages the image was not specified so i wanted to parse my output based on the status code i receive when accessing the page but when i attempt to pare my status code i get an exception instead of the status code if the page was not found, any idea why this happends?
if (gameinfo != null)
if (!string.IsNullOrEmpty(gameinfo.image_uri))
try
{
using (System.Net.WebClient client = new System.Net.WebClient())
{
// Build Uri and attempt to fetch a response.
UriBuilder uribuild = new UriBuilder(gameinfo.image_uri);
WebRequest request = WebRequest.Create(uribuild.Uri);
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
switch (response.StatusCode)
{
// Page found and valid entry.
case HttpStatusCode.OK:
using (Stream stream = client.OpenRead(uribuild.Uri))
{
Console.WriteLine(String.Format("Downloading {0}", uribuild.Uri));
System.Drawing.Bitmap bitmap = new System.Drawing.Bitmap(stream);
bitmap.Save(System.IO.Path.Combine(rom_root, String.Format("{0}.jpg", file_name.Substring(2).Split('.').First())));
}
break;
// Unspecified status codes.
default:
Console.WriteLine("Unspecified status code found, aborting...");
break;
}
}
} catch(System.Net.WebException ex)
{
// Should be moved to switch with HttpStatusCode.NotFound ^
Console.WriteLine("Image page not found.");
}

That's just how the GetResponse() implementation is. If the response is anything other than a success, a WebException is thrown.
I agree, I find it a bit odd - it would be nice if it was optional behaviour at least.
Thankfully, you can read the status code from the WebException which is being thrown:
....
catch (WebException e)
{
using (WebResponse response = e.Response)
{
HttpWebResponse httpResponse = (HttpWebResponse) response;
var statusCode = httpResponse.StatusCode;
// Do stuff with the statusCode
}
}

How to read status code from response body of c# webclient request [duplicate]

I am using the WebClient class to post some data to a web form. I would like to get the response status code of the form submission. So far I've found out how to get the status code if there is a exception
Catch wex As WebException
If TypeOf wex.Response Is HttpWebResponse Then
msgbox(DirectCast(wex.Response, HttpWebResponse).StatusCode)
End If
However if the form is submitted successfully and no exception is thrown then I won't know the status code(200,301,302,...)
Is there some way to get the status code when there is no exceptions thrown?
PS: I prefer not to use httpwebrequest/httpwebresponse

You can check if the error is of type WebException and then inspect the response code;
if (e.Error.GetType().Name == "WebException")
{
WebException we = (WebException)e.Error;
HttpWebResponse response = (System.Net.HttpWebResponse)we.Response;
if (response.StatusCode==HttpStatusCode.NotFound)
System.Diagnostics.Debug.WriteLine("Not found!");
}
or
try
{
// send request
}
catch (WebException e)
{
// check e.Status as above etc..
}

There is a way to do it using reflection. It works with .NET 4.0. It accesses a private field and may not work in other versions of .NET without modifications.
I have no idea why Microsoft did not expose this field with a property.
private static int GetStatusCode(WebClient client, out string statusDescription)
{
FieldInfo responseField = client.GetType().GetField("m_WebResponse", BindingFlags.Instance | BindingFlags.NonPublic);
if (responseField != null)
{
HttpWebResponse response = responseField.GetValue(client) as HttpWebResponse;
if (response != null)
{
statusDescription = response.StatusDescription;
return (int)response.StatusCode;
}
}
statusDescription = null;
return 0;
}

If you are using .Net 4.0 (or less):
class BetterWebClient : WebClient
{
private WebRequest _Request = null;
protected override WebRequest GetWebRequest(Uri address)
{
this._Request = base.GetWebRequest(address);
if (this._Request is HttpWebRequest)
{
((HttpWebRequest)this._Request).AllowAutoRedirect = false;
}
return this._Request;
}
public HttpStatusCode StatusCode()
{
HttpStatusCode result;
if (this._Request == null)
{
throw (new InvalidOperationException("Unable to retrieve the status
code, maybe you haven't made a request yet."));
}
HttpWebResponse response = base.GetWebResponse(this._Request)
as HttpWebResponse;
if (response != null)
{
result = response.StatusCode;
}
else
{
throw (new InvalidOperationException("Unable to retrieve the status
code, maybe you haven't made a request yet."));
}
return result;
}
}
If you are using .Net 4.5.X or newer, switch to HttpClient:
var response = await client.GetAsync("http://www.contoso.com/");
var statusCode = response.StatusCode;

Tried it out. ResponseHeaders do not include status code.
If I'm not mistaken, WebClient is capable of abstracting away multiple distinct requests in a single method call (e.g. correctly handling 100 Continue responses, redirects, and the like). I suspect that without using HttpWebRequest and HttpWebResponse, a distinct status code may not be available.
It occurs to me that, if you are not interested in intermediate status codes, you can safely assume the final status code is in the 2xx (successful) range, otherwise, the call would not be successful.
The status code unfortunately isn't present in the ResponseHeaders dictionary.

Erik's answer doesn't work on Windows Phone as is. The following does:
class WebClientEx : WebClient
{
private WebResponse m_Resp = null;
protected override WebResponse GetWebResponse(WebRequest Req, IAsyncResult ar)
{
try
{
this.m_Resp = base.GetWebResponse(request);
}
catch (WebException ex)
{
if (this.m_Resp == null)
this.m_Resp = ex.Response;
}
return this.m_Resp;
}
public HttpStatusCode StatusCode
{
get
{
if (m_Resp != null && m_Resp is HttpWebResponse)
return (m_Resp as HttpWebResponse).StatusCode;
else
return HttpStatusCode.OK;
}
}
}
At least it does when using OpenReadAsync; for other xxxAsync methods, careful testing would be highly recommended. The framework calls GetWebResponse somewhere along the code path; all one needs to do is capture and cache the response object.
The fallback code is 200 in this snippet because genuine HTTP errors - 500, 404, etc - are reported as exceptions anyway. The purpose of this trick is to capture non-error codes, in my specific case 304 (Not modified). So the fallback assumes that if the status code is somehow unavailable, at least it's a non-erroneous one.

You should use
if (e.Status == WebExceptionStatus.ProtocolError)
{
HttpWebResponse response = (HttpWebResponse)ex.Response;
if (response.StatusCode == HttpStatusCode.NotFound)
System.Diagnostics.Debug.WriteLine("Not found!");
}

This is what I use for expanding WebClient functionality. StatusCode and StatusDescription will always contain the most recent response code/description.
/// <summary>
/// An expanded web client that allows certificate auth and
/// the retrieval of status' for successful requests
/// </summary>
public class WebClientCert : WebClient
{
private X509Certificate2 _cert;
public WebClientCert(X509Certificate2 cert) : base() { _cert = cert; }
protected override WebRequest GetWebRequest(Uri address)
{
HttpWebRequest request = (HttpWebRequest)base.GetWebRequest(address);
if (_cert != null) { request.ClientCertificates.Add(_cert); }
return request;
}
protected override WebResponse GetWebResponse(WebRequest request)
{
WebResponse response = null;
response = base.GetWebResponse(request);
HttpWebResponse baseResponse = response as HttpWebResponse;
StatusCode = baseResponse.StatusCode;
StatusDescription = baseResponse.StatusDescription;
return response;
}
/// <summary>
/// The most recent response statusCode
/// </summary>
public HttpStatusCode StatusCode { get; set; }
/// <summary>
/// The most recent response statusDescription
/// </summary>
public string StatusDescription { get; set; }
}
Thus you can do a post and get result via:
byte[] response = null;
using (WebClientCert client = new WebClientCert())
{
response = client.UploadValues(postUri, PostFields);
HttpStatusCode code = client.StatusCode;
string description = client.StatusDescription;
//Use this information
}

Just in case someone else needs an F# version of the above described hack.
open System
open System.IO
open System.Net
type WebClientEx() =
inherit WebClient ()
[<DefaultValue>] val mutable m_Resp : WebResponse
override x.GetWebResponse (req: WebRequest ) =
x.m_Resp <- base.GetWebResponse(req)
(req :?> HttpWebRequest).AllowAutoRedirect <- false;
x.m_Resp
override x.GetWebResponse (req: WebRequest , ar: IAsyncResult ) =
x.m_Resp <- base.GetWebResponse(req, ar)
(req :?> HttpWebRequest).AllowAutoRedirect <- false;
x.m_Resp
member x.StatusCode with get() : HttpStatusCode =
if not (obj.ReferenceEquals (x.m_Resp, null)) && x.m_Resp.GetType() = typeof<HttpWebResponse> then
(x.m_Resp :?> HttpWebResponse).StatusCode
else
HttpStatusCode.OK
let wc = new WebClientEx()
let st = wc.OpenRead("http://www.stackoverflow.com")
let sr = new StreamReader(st)
let res = sr.ReadToEnd()
wc.StatusCode
sr.Close()
st.Close()

You should be able to use the "client.ResponseHeaders[..]" call, see this link for examples of getting stuff back from the response

You can try this code to get HTTP status code from WebException or from OpenReadCompletedEventArgs.Error. It works in Silverlight too because SL does not have WebExceptionStatus.ProtocolError defined.
HttpStatusCode GetHttpStatusCode(System.Exception err)
{
if (err is WebException)
{
WebException we = (WebException)err;
if (we.Response is HttpWebResponse)
{
HttpWebResponse response = (HttpWebResponse)we.Response;
return response.StatusCode;
}
}
return 0;
}

HttpWebResponse get stuck while running in a loop

I build this method (c#) in order to receive the HTTP response status code from an URL.
whene I run this method ones it's works fine, but when I run it in a loop, the third time its stuck. any clue??
public static string isAlive(string url)
{
Console.WriteLine("start: Is Alive Test");
WebRequest request = WebRequest.Create(url);
try
{
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
return Convert.ToString((int)response.StatusCode);
}
catch(WebException ex)
{
HttpWebResponse res = (HttpWebResponse)ex.Response;
return Convert.ToString((int)res.StatusCode);
}
}
the loop
for (int i = 0; i < 5; i++)
{
string a = isAlive("https://www.yahoo.com/");
Console.WriteLine(a);
}

You're not calling Dispose on the HttpWebResponse object, which means that the connection is still lying around. If you change your code to the following:
public static string isAlive(string url)
{
Console.WriteLine("start: Is Alive Test");
WebRequest request = WebRequest.Create(url);
try
{
using(HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
return Convert.ToString((int)response.StatusCode);
}
}
catch(WebException ex)
{
using(HttpWebResponse res = (HttpWebResponse)ex.Response)
{
return Convert.ToString((int)res.StatusCode);
}
}
}
the using statement will implicitly call Dispose for you, which will close the connection.
The reason your code is halting after the second iteration is because .Net has a built in maximum number of connections it will open to a website, which is by default 2. This is controlled by System.Net.ServicePointManager.DefaultConnectionLimit which you can increase should you need to.

You need to wrap HttpWebResponse var into using statement because it's disposable
Before checking ex.Response.StatusCode make sure that ex.Status is a ProtocolError
And also consider making your method asynchronous for performance considerations
Since your method is returning a status code, there might be a better name for it than isAlive
Sample:
public static async Task<string> GetStatusCode(string url)
{
var request = (HttpWebRequest)WebRequest.Create(url);
try
{
using (var response = (HttpWebResponse)await request.GetResponseAsync())
{
return response.StatusCode.ToString();
}
}
catch (WebException ex)
{
return ex.Status == WebExceptionStatus.ProtocolError ?
((HttpWebResponse)e.Response).StatusCode.ToString() : null;
}
}

It might have to do with you not closing the HttpWebResponse.
Add a finally to that try catch which closes the response.
Also close the WebException response within the catch.

Use "using" and it will work well.
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
return Convert.ToString((int)response.StatusCode);
}

WebRequest and protocol agnostic URL

In a web application I have to first check if an image exists and then display this image or a dummy image.
I use the following code and it works for URLS like:
"http://www.somedomain.com/niceimage.png"
"https://www.somedomain.com/niceimage.png".
public virtual bool WebResourceExists(string url)
{
WebHeaderCollection headers = null;
WebResponse response = null;
try
{
WebRequest request = WebRequest.Create(url);
request.Method = "HEAD";
response = request.GetResponse();
headers = response.Headers;
bool result = int.Parse(headers["Content-Length"]) > 0;
return result;
}
catch (System.Net.WebException)
{
return false;
}
catch (Exception e)
{
_log.Error(e);
return false;
}
finally
{
if (response != null)
{
response.Close();
}
}
}
In some places the the method is called with protocol agnostic urls like "//www.somedomain.com/niceimage.png".
There is an exception thrown for such urls:
System.InvalidCastException: Unable to cast object of type
'System.Net.FileWebRequest' to type 'System.Net.HttpWebRequest'
Is there a way to use protocol agnostic urls other then just prepending "http:" to the url?

Protocol-agnostic URLs are resolved by the browser using the current protocol, and are used to avoid making HTTP requests from an HTTPS page.
Code executing on the server doesn't really have a concept of a "current protocol". Whilst ASP.NET can determine whether the current request was issued over HTTP or HTTPS, the WebRequest classes are not restricted to ASP.NET applications, so they cannot rely on this.
You will need to specify the protocol. Whether you use HTTP or HTTPS will depend on whether you're concerned about third-parties eavesdropping on the connection between your server and "www.somedomain.com".

What about a two step process, check for the http version, if it doesn't exist check for https. I've quickly hacked together a basic example of how this could work but am not able to properly test and check it so it might need some tidying up/refactoring!
public virtual bool WebResourceExists(string url)
{
WebHeaderCollection headers = null;
WebResponse response = null;
try
{
if (url.StartsWith(#"//") {
url = "http:";
}
WebRequest request = WebRequest.Create(url);
request.Method = "HEAD";
response = request.GetResponse();
headers = response.Headers;
bool result = int.Parse(headers["Content-Length"]) > 0;
return result;
}
catch (System.Net.WebException)
{
if (url.StartsWith(#"http://") {
url = url.Replace("http://","https://");
} else {
return false;
}
try {
WebRequest request = WebRequest.Create(url);
request.Method = "HEAD";
response = request.GetResponse();
headers = response.Headers;
bool result = int.Parse(headers["Content-Length"]) > 0;
return result;
}
catch (System.Net.WebException)
{
return false;
}
}
catch (Exception e)
{
_log.Error(e);
return false;
}
finally
{
if (response != null)
{
response.Close();
}
}
}

can I check if a file exists at a URL?

I know I can locally, on my filesystem, check if a file exists:
if(File.Exists(path))
Can I check at a particular remote URL?

If you're attempting to verify the existence of a web resource, I would recommend using the HttpWebRequest class. This will allow you to send a HEAD request to the URL in question. Only the response headers will be returned, even if the resource exists.
var url = "http://www.domain.com/image.png";
HttpWebResponse response = null;
var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "HEAD";
try
{
response = (HttpWebResponse)request.GetResponse();
}
catch (WebException ex)
{
/* A WebException will be thrown if the status of the response is not `200 OK` */
}
finally
{
// Don't forget to close your response.
if (response != null)
{
response.Close();
}
}
Of course, if you want to download the resource if it exists it would most likely be more efficient to send a GET request instead (by not setting the Method property to "HEAD", or by using the WebClient class).

If you want to just copy & paste Justin's code and get a method to use, here's how I've implemented it:
using System.Net;
public class MyClass {
static public bool URLExists (string url) {
bool result = false;
WebRequest webRequest = WebRequest.Create(url);
webRequest.Timeout = 1200; // miliseconds
webRequest.Method = "HEAD";
HttpWebResponse response = null;
try {
response = (HttpWebResponse)webRequest.GetResponse();
result = true;
} catch (WebException webException) {
Debug.Log(url +" doesn't exist: "+ webException.Message);
} finally {
if (response != null) {
response.Close();
}
}
return result;
}
}
I'll keep his observation:
If you want to download the resource, and it exists, it would be more efficient to send a GET request instead by not setting the Method property to "HEAD" or by using the WebClient class.

Below is a simplified version of the code:
public bool URLExists(string url)
{
bool result = true;
WebRequest webRequest = WebRequest.Create(url);
webRequest.Timeout = 1200; // miliseconds
webRequest.Method = "HEAD";
try
{
webRequest.GetResponse();
}
catch
{
result = false;
}
return result;
}

If you are using a unc path or a mapped drive, this will work fine.
If you are using a web address (http, ftp etc) you are better off using WebClient - you will get a WebException if it doesn't exist.

public static bool UrlExists(string file)
{
bool exists = false;
HttpWebResponse response = null;
var request = (HttpWebRequest)WebRequest.Create(file);
request.Method = "HEAD";
request.Timeout = 5000; // milliseconds
request.AllowAutoRedirect = false;
try
{
response = (HttpWebResponse)request.GetResponse();
exists = response.StatusCode == HttpStatusCode.OK;
}
catch
{
exists = false;
}
finally
{
// close your response.
if (response != null)
response.Close();
}
return exists;
}

I had the same problem to solve in asp.net core, I've solved with HttpClient
private async Task<bool> isFileExist(string url)
{
using (HttpClient client = new HttpClient())
{
var restponse = await client.GetAsync(url);
return restponse.StatusCode == System.Net.HttpStatusCode.OK;
}
}

My version:
public bool IsUrlExist(string url, int timeOutMs = 1000)
{
WebRequest webRequest = WebRequest.Create(url);
webRequest.Method = "HEAD";
webRequest.Timeout = timeOutMs;
try
{
var response = webRequest.GetResponse();
/* response is `200 OK` */
response.Close();
}
catch
{
/* Any other response */
return false;
}
return true;
}

WebRequest will waiting long time(ignore the timeout user set) because not set proxy, so I change to use RestSharp to do this.
var client = new RestClient(url);
var request = new RestRequest(Method.HEAD);
request.Timeout = 5000;
var response = client.Execute(request);
result = response.StatusCode == HttpStatusCode.OK;

Thanks for all answers.
And I would like to add my implementation which includes default state when we get errors, for specific cases like mine.
private bool HTTP_URLExists(String vstrURL, bool vResErrorDefault = false, int vTimeOut = 1200)
{
bool vResult = false;
WebRequest webRequest = WebRequest.Create(vstrURL);
webRequest.Timeout = vTimeOut; // miliseconds
webRequest.Method = "HEAD";
HttpWebResponse response = null;
try
{
response = (HttpWebResponse)webRequest.GetResponse();
if (response.StatusCode == HttpStatusCode.OK) vResult = true;
else if (response.StatusCode == HttpStatusCode.NotFound) vResult = false;
else vResult = vResErrorDefault;
}
catch (WebException ex)
{
if (ex.Status == WebExceptionStatus.ProtocolError && ex.Response != null)
{
var resp01 = (HttpWebResponse)ex.Response;
if (resp01.StatusCode == HttpStatusCode.NotFound)
{
vResult = false;
}
else
{
vResult = vResErrorDefault;
}
}
else
{
vResult = vResErrorDefault;
}
}
finally
{
// Don't forget to close your response.
if (response != null)
{
response.Close();
}
}
return vResult;
}

Anoter version with define timeout :
public bool URLExists(string url,int timeout = 5000)
{
...
webRequest.Timeout = timeout; // miliseconds
...
}

This works for me:
bool HaveFile(string url)
{
try
{
using (WebClient webClient = new WebClient())
{
webClient.DownloadString(url);
}
return true;
}
catch (Exception)
{
return false;
}
}