We Keep Coding

C# webrequests->getresponse casts exception instead of returning status code

I'm attempting to make an image scraper, now for some pages the image was not specified so i wanted to parse my output based on the status code i receive when accessing the page but when i attempt to pare my status code i get an exception instead of the status code if the page was not found, any idea why this happends?
if (gameinfo != null)
if (!string.IsNullOrEmpty(gameinfo.image_uri))
try
{
using (System.Net.WebClient client = new System.Net.WebClient())
{
// Build Uri and attempt to fetch a response.
UriBuilder uribuild = new UriBuilder(gameinfo.image_uri);
WebRequest request = WebRequest.Create(uribuild.Uri);
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
switch (response.StatusCode)
{
// Page found and valid entry.
case HttpStatusCode.OK:
using (Stream stream = client.OpenRead(uribuild.Uri))
{
Console.WriteLine(String.Format("Downloading {0}", uribuild.Uri));
System.Drawing.Bitmap bitmap = new System.Drawing.Bitmap(stream);
bitmap.Save(System.IO.Path.Combine(rom_root, String.Format("{0}.jpg", file_name.Substring(2).Split('.').First())));
}
break;
// Unspecified status codes.
default:
Console.WriteLine("Unspecified status code found, aborting...");
break;
}
}
} catch(System.Net.WebException ex)
{
// Should be moved to switch with HttpStatusCode.NotFound ^
Console.WriteLine("Image page not found.");
}

That's just how the GetResponse() implementation is. If the response is anything other than a success, a WebException is thrown.
I agree, I find it a bit odd - it would be nice if it was optional behaviour at least.
Thankfully, you can read the status code from the WebException which is being thrown:
....
catch (WebException e)
{
using (WebResponse response = e.Response)
{
HttpWebResponse httpResponse = (HttpWebResponse) response;
var statusCode = httpResponse.StatusCode;
// Do stuff with the statusCode
}
}

C# GetResponse not coming back

I have an strange problem. My .NET DataController send a response to an Java server (REST). All works fine, but when I send a response very often, suddenly the GetResponse method not coming back just hangs.
I don't know what's the problem is.
Here is my code
ServicePointManager.DefaultConnectionLimit = 20;
public string HttpGet(string url, string requestAccept)
{
if(string.IsNullOrEmpty(url))
throw new Exception("HttpGet: no REST service URL provided");
if(string.IsNullOrEmpty(requestAccept))
requestAccept = defaultRequestAccept;
if(!url.Equals("pc/alive"))
{
if(string.IsNullOrEmpty(SessionId))
{
if(string.IsNullOrEmpty(Authorization()))
throw new WebException("HttpGet: no login");
}
}
int tries = RestAccessData.MaxReconnect;
string result = string.Empty;
do
{
try
{
var request = NewWebRequest(url, "GET", false, true, requestAccept);
using(var response = request.GetResponse() as HttpWebResponse)
{
UpdateSessionId(response);
HttpStatusCode statusCode = response.StatusCode;
LOG.Debug("...StatusCode: {0} ({1})", (int) response.StatusCode, response.StatusDescription);
if((int) response.StatusCode < 500)
lastContactWithServer = DateTime.Now.Ticks;
switch(statusCode)
{
// Informational 1xx
case HttpStatusCode.Continue: // 100
case HttpStatusCode.SwitchingProtocols: // 101
throw new HttpResponseException(response);
// Successful 2xx
case HttpStatusCode.OK: // 200
result = ReadContent(response);
response.Close();
return (result);
case HttpStatusCode.NoContent: // 204 The server has fulfilled the request but does not need to return an entity-body, and might want to return updated metainformation.
response.Close();
return (string.Empty);
case HttpStatusCode.Created: // 201
case HttpStatusCode.NonAuthoritativeInformation: // 203
result = ReadContent(response);
throw new HttpResponseException(response, result);
case HttpStatusCode.Accepted: // 202 The request has been accepted for processing, but the processing has not been completed.
case HttpStatusCode.ResetContent: // 205 The server has fulfilled the request and the user agent SHOULD reset the document view which caused the request to be sent.
case HttpStatusCode.PartialContent: // 206 The server has fulfilled the partial GET request for the resource.
throw new HttpResponseException(response);
case HttpStatusCode.Unauthorized:
throw new HttpResponseException(response);
default:
throw new HttpResponseException(response);
}
}
}
catch(WebException ex)
{
HandleWebException(ex);
}
catch(HttpResponseException ex)
{
throw ex;
}
catch(SystemException ex)
{
LOG.Error(ex, "caused a(n) {0}", ex.GetType().Name);
}
catch(Exception ex)
{
LOG.Warn(ex, "HttpGet: An error occured while trying to contact the server. Reason: {0}", ex.Message);
throw new UserException("An error occured while trying to contact the server.", ex);
}
}
while(0 < tries--);
return (string.Empty);
}
private HttpWebRequest NewWebRequest(string urlAsString, string requestType, bool withContentType, bool applicationJson, string requestAccept)
{
urlAsString = string.Format("{0}{1}", RestAccessData.GetUrl(), EncodeUrl(urlAsString));
Uri url;
if(!Uri.TryCreate(urlAsString, UriKind.Absolute, out url))
throw new NotSupportedException("url is not compatible");
LOG.Info("RESTUrl {0}", urlAsString);
try
{
var request = HttpWebRequest.Create(url) as HttpWebRequest;
if(!string.IsNullOrEmpty(SessionId))
{
CookieContainer cookies = new CookieContainer();
Cookie cookie = new Cookie
{
Name = "JSESSIONID",
Value = SessionId,
Domain = url.Host
};
cookies.Add(cookie);
request.CookieContainer = cookies;
}
request.Timeout = RestAccessData.Timeout;
request.Method = requestType;
request.AllowAutoRedirect = true;
request.AllowWriteStreamBuffering = true;
request.KeepAlive = false;
request.ContentLength = 0;
request.Headers["Accept-Charset"] = "utf-8";
request.Accept = requestAccept;
request.UserAgent = LvsClient;
if(withContentType)
request.ContentType = applicationJson ? defaultApplicationJson : defaultApplicationWwwForm;
if(!string.IsNullOrEmpty(credential.UserName) && !string.IsNullOrEmpty(credential.Password))
request.Headers["Authorization"] = string.Format("Basic {0}", Convert.ToBase64String(Encoding.UTF8.GetBytes(string.Format("{0}:{1}", credential.UserName, credential.Password))));
return (request);
}
catch(Exception ex)
{
throw new UserException(TH.Translate("NewWebRequest caused an error"), ex);
}
}
Any ideas where the problem is?

This line is making me uncomfortable:
using(var response = request.GetResponse() as HttpWebResponse)
If the response is not an HttpWebResponse then it will never be disposed - the as operator will give you `null.

I found a solution. Very useful is this MSDN article about Understanding MaxServicePointIdleTime and DefaultConnectionLimit. After I play with MaxServicePointIdleTime property and read this comment
change the connection limit at any time
I found a good balance between connection alive and connection close. My client works fine after playing with MaxServicePointIdleTime and DefaultConnectionLimit.

Is there another solution to WebResponse 403 error?

There are many posts on WebResponse 403 error but my situation is a little different. I have created a console application that will run as a task on my server. The console application passes user emails in WebRequest and waits for WebResponse to receive the uri with the returning parameters. The code below worked perfectly a few days ago but one of the other programmers added a new parameter for a return web address. I know for a fact that this is causing the 403 error because if I paste the uri in IE with new parameter it works. But since I have a console application a return web address is something I cannot do, at least I don't think so.
Unfortunately the programmer said that he cannot change it back and said that there is a way to receive the uri or the entire page content and I can process it that way. I still have no clue what he was talking about because StreamReader requires a WebResponse and pretty much all other solutions I could think of.
Even though I get a 403 error the response still has the uri with the parameters I need because I can see it in IE in the web address. So all I need is the response uri. I would appreciate any help you have to offer. Below is the method giving me problems.
String employeeInfo = "";
try
{
HttpWebRequest request = (HttpWebRequest)WebRequest
.Create("http://example.com/subsub.aspx?instprod=xxx&vabid=emailaddress");
using (HttpWebResponse webResponse =
(HttpWebResponse)request.GetResponse()) //Error occurs here. 403 Forbidden
{
Uri myUri = new Uri(webResponse.ResponseUri.ToString());
String queryParamerter = myUri.Query;
employeeInfo = HttpUtility.ParseQueryString(queryParamerter).Get("vres");
if (employeeInfo != "N/A")
{
return employeeInfo;
}
else
{
employeeInfo = "0";
return employeeInfo;
}
}
}
catch (WebException)
{
employeeInfo = "0";
return employeeInfo;
}

Let's follow Jim Mischel's idea. We'll handle the WebException and use the Response property of the exception.
String employeeInfo = "";
try
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://example.com/subsub.aspx?instprod=xxx&vabid=emailaddress");
using (HttpWebResponse webResponse = (HttpWebResponse)request.GetResponse()) //Error occurs here. 403 Forbidden
{
Uri myUri = new Uri(webResponse.ResponseUri.ToString());
String queryParamerter = myUri.Query;
employeeInfo = HttpUtility.ParseQueryString(queryParamerter).Get("vres");
if (employeeInfo != "N/A")
{
return employeeInfo;
}
else
{
employeeInfo = "0";
return employeeInfo;
}
}
}
catch (WebException ex)
{
HttpWebResponse response = ex.Response as HttpWebResponse;
if(response.StatusCode != HttpStatusCode.Forbidden)
{
throw;
}
Uri myUri = new Uri(response.ResponseUri.ToString());
String queryParamerter = myUri.Query;
employeeInfo = HttpUtility.ParseQueryString(queryParamerter).Get("vres");
if (employeeInfo != "N/A")
{
return employeeInfo;
}
else
{
employeeInfo = "0";
return employeeInfo;
}
}

WebRequest and protocol agnostic URL

In a web application I have to first check if an image exists and then display this image or a dummy image.
I use the following code and it works for URLS like:
"http://www.somedomain.com/niceimage.png"
"https://www.somedomain.com/niceimage.png".
public virtual bool WebResourceExists(string url)
{
WebHeaderCollection headers = null;
WebResponse response = null;
try
{
WebRequest request = WebRequest.Create(url);
request.Method = "HEAD";
response = request.GetResponse();
headers = response.Headers;
bool result = int.Parse(headers["Content-Length"]) > 0;
return result;
}
catch (System.Net.WebException)
{
return false;
}
catch (Exception e)
{
_log.Error(e);
return false;
}
finally
{
if (response != null)
{
response.Close();
}
}
}
In some places the the method is called with protocol agnostic urls like "//www.somedomain.com/niceimage.png".
There is an exception thrown for such urls:
System.InvalidCastException: Unable to cast object of type
'System.Net.FileWebRequest' to type 'System.Net.HttpWebRequest'
Is there a way to use protocol agnostic urls other then just prepending "http:" to the url?

Protocol-agnostic URLs are resolved by the browser using the current protocol, and are used to avoid making HTTP requests from an HTTPS page.
Code executing on the server doesn't really have a concept of a "current protocol". Whilst ASP.NET can determine whether the current request was issued over HTTP or HTTPS, the WebRequest classes are not restricted to ASP.NET applications, so they cannot rely on this.
You will need to specify the protocol. Whether you use HTTP or HTTPS will depend on whether you're concerned about third-parties eavesdropping on the connection between your server and "www.somedomain.com".

What about a two step process, check for the http version, if it doesn't exist check for https. I've quickly hacked together a basic example of how this could work but am not able to properly test and check it so it might need some tidying up/refactoring!
public virtual bool WebResourceExists(string url)
{
WebHeaderCollection headers = null;
WebResponse response = null;
try
{
if (url.StartsWith(#"//") {
url = "http:";
}
WebRequest request = WebRequest.Create(url);
request.Method = "HEAD";
response = request.GetResponse();
headers = response.Headers;
bool result = int.Parse(headers["Content-Length"]) > 0;
return result;
}
catch (System.Net.WebException)
{
if (url.StartsWith(#"http://") {
url = url.Replace("http://","https://");
} else {
return false;
}
try {
WebRequest request = WebRequest.Create(url);
request.Method = "HEAD";
response = request.GetResponse();
headers = response.Headers;
bool result = int.Parse(headers["Content-Length"]) > 0;
return result;
}
catch (System.Net.WebException)
{
return false;
}
}
catch (Exception e)
{
_log.Error(e);
return false;
}
finally
{
if (response != null)
{
response.Close();
}
}
}

can I check if a file exists at a URL?

I know I can locally, on my filesystem, check if a file exists:
if(File.Exists(path))
Can I check at a particular remote URL?

If you're attempting to verify the existence of a web resource, I would recommend using the HttpWebRequest class. This will allow you to send a HEAD request to the URL in question. Only the response headers will be returned, even if the resource exists.
var url = "http://www.domain.com/image.png";
HttpWebResponse response = null;
var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "HEAD";
try
{
response = (HttpWebResponse)request.GetResponse();
}
catch (WebException ex)
{
/* A WebException will be thrown if the status of the response is not `200 OK` */
}
finally
{
// Don't forget to close your response.
if (response != null)
{
response.Close();
}
}
Of course, if you want to download the resource if it exists it would most likely be more efficient to send a GET request instead (by not setting the Method property to "HEAD", or by using the WebClient class).

If you want to just copy & paste Justin's code and get a method to use, here's how I've implemented it:
using System.Net;
public class MyClass {
static public bool URLExists (string url) {
bool result = false;
WebRequest webRequest = WebRequest.Create(url);
webRequest.Timeout = 1200; // miliseconds
webRequest.Method = "HEAD";
HttpWebResponse response = null;
try {
response = (HttpWebResponse)webRequest.GetResponse();
result = true;
} catch (WebException webException) {
Debug.Log(url +" doesn't exist: "+ webException.Message);
} finally {
if (response != null) {
response.Close();
}
}
return result;
}
}
I'll keep his observation:
If you want to download the resource, and it exists, it would be more efficient to send a GET request instead by not setting the Method property to "HEAD" or by using the WebClient class.

Below is a simplified version of the code:
public bool URLExists(string url)
{
bool result = true;
WebRequest webRequest = WebRequest.Create(url);
webRequest.Timeout = 1200; // miliseconds
webRequest.Method = "HEAD";
try
{
webRequest.GetResponse();
}
catch
{
result = false;
}
return result;
}

If you are using a unc path or a mapped drive, this will work fine.
If you are using a web address (http, ftp etc) you are better off using WebClient - you will get a WebException if it doesn't exist.

public static bool UrlExists(string file)
{
bool exists = false;
HttpWebResponse response = null;
var request = (HttpWebRequest)WebRequest.Create(file);
request.Method = "HEAD";
request.Timeout = 5000; // milliseconds
request.AllowAutoRedirect = false;
try
{
response = (HttpWebResponse)request.GetResponse();
exists = response.StatusCode == HttpStatusCode.OK;
}
catch
{
exists = false;
}
finally
{
// close your response.
if (response != null)
response.Close();
}
return exists;
}

I had the same problem to solve in asp.net core, I've solved with HttpClient
private async Task<bool> isFileExist(string url)
{
using (HttpClient client = new HttpClient())
{
var restponse = await client.GetAsync(url);
return restponse.StatusCode == System.Net.HttpStatusCode.OK;
}
}

My version:
public bool IsUrlExist(string url, int timeOutMs = 1000)
{
WebRequest webRequest = WebRequest.Create(url);
webRequest.Method = "HEAD";
webRequest.Timeout = timeOutMs;
try
{
var response = webRequest.GetResponse();
/* response is `200 OK` */
response.Close();
}
catch
{
/* Any other response */
return false;
}
return true;
}

WebRequest will waiting long time(ignore the timeout user set) because not set proxy, so I change to use RestSharp to do this.
var client = new RestClient(url);
var request = new RestRequest(Method.HEAD);
request.Timeout = 5000;
var response = client.Execute(request);
result = response.StatusCode == HttpStatusCode.OK;

Thanks for all answers.
And I would like to add my implementation which includes default state when we get errors, for specific cases like mine.
private bool HTTP_URLExists(String vstrURL, bool vResErrorDefault = false, int vTimeOut = 1200)
{
bool vResult = false;
WebRequest webRequest = WebRequest.Create(vstrURL);
webRequest.Timeout = vTimeOut; // miliseconds
webRequest.Method = "HEAD";
HttpWebResponse response = null;
try
{
response = (HttpWebResponse)webRequest.GetResponse();
if (response.StatusCode == HttpStatusCode.OK) vResult = true;
else if (response.StatusCode == HttpStatusCode.NotFound) vResult = false;
else vResult = vResErrorDefault;
}
catch (WebException ex)
{
if (ex.Status == WebExceptionStatus.ProtocolError && ex.Response != null)
{
var resp01 = (HttpWebResponse)ex.Response;
if (resp01.StatusCode == HttpStatusCode.NotFound)
{
vResult = false;
}
else
{
vResult = vResErrorDefault;
}
}
else
{
vResult = vResErrorDefault;
}
}
finally
{
// Don't forget to close your response.
if (response != null)
{
response.Close();
}
}
return vResult;
}

Anoter version with define timeout :
public bool URLExists(string url,int timeout = 5000)
{
...
webRequest.Timeout = timeout; // miliseconds
...
}

This works for me:
bool HaveFile(string url)
{
try
{
using (WebClient webClient = new WebClient())
{
webClient.DownloadString(url);
}
return true;
}
catch (Exception)
{
return false;
}
}