I have a WCF service that is running frequent (1000+) outbound connections per minute to external APIs.
My code throws the following exceptions frequently, but not always showing that is is a WebException with the WebException status property being ReceiveFailure
The code that is making the outbound request is the following:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(paramBuilder.ToString());
request.ServicePoint.ConnectionLeaseTimeout = 0;
request.Method = "GET";
request.Timeout = 33000; //33 Second Timeout Is By Design
Stream stream = default(Stream);
HttpWebResponse response = default(HttpWebResponse);
try
{
response = (HttpWebResponse) request.GetResponse();
stream = response.GetResponseStream();
reader = new StreamReader(stream,Encoding.UTF8);
string str = reader.ReadToEnd();
return str;
}
catch (WebException exception)
{
//Handle WebException
}
catch (Exception exception)
{
//Handle Exception
}
finally
{
if (reader != null)
reader.Dispose();
if (response != null)
response.Close();
if (stream != null)
stream.Dispose();
}
The exception stack trace shows that the exception is caused from GetResponse().
What could be causing this to happen that I receive an occasional WebException -ReceiveFailure.
I have already reference the MSDN documentation for this status, but that doesn't help me.
Shooting in the dark here...
There is a special condition, while waiting for response: if the system clock is being set automatically by the Windows Time service, or manually, you may experience some unpredictable results.
If you're sending your requests over HTTPS, maybe you're facing a regular timeout that was wrongly thrown as a ReceiveFailure.
Check this article for more information: http://support.microsoft.com/kb/2007873
I have a related problem and I realise a few things while I was searching for a solution.
WebExceptionStatus enum is not equivalent to http status code that the API you call returned. Instead it is a enum of possible error that may occour during a http call.
The WebExceptionStatus error code that will be returned when you receive an error (400 to 599) from your API is WebExceptionStatus.ProtocolError aka number 7 as int.
When you need to get the response body or the real http status code returned from the api, first you need to check if WebException.Status is WebExceptionStatus.ProtocolError. Then you can get the real response from WebExceptionStatus.Response and read its content.
Sometimes the timeout is handled by the caller (aka your code) so you do not have a response in that case. So you can look if WebException.Status is WebExceptionStatus.Timeout
This is an example:
try
{
...
}
catch (WebException webException)
{
if (webException.Status == WebExceptionStatus.ProtocolError)
{
var httpResponse = (HttpWebResponse)webException.Response;
var responseText = "";
using (var content = new StreamReader(httpResponse.GetResponseStream()))
{
responseText = content.ReadToEnd(); // Get response body as text
}
int statusCode = (int)httpResponse.StatusCode; // Get the status code
}
else if (webException.Status == WebExceptionStatus.ProtocolError)
{
// Timeout handled by your code. You do not have a response here.
}
// Handle other webException.Status errors. You do not have a response here.
}
I'm using a .Net MVC application as a simplified web service.
I've got an async method that I call:
public void RunQueue()
{
QueueDelegate queue = new QueueDelegate(Queue);
AsyncCallback completedCallback = new AsyncCallback(QueueCompleteCallback);
lock (_sync)
{
if (!queueIsRunning)
{
AsyncOperation async = AsyncOperationManager.CreateOperation(null);
queue.BeginInvoke(QueueCompleteCallback, async);
queueIsRunning = true;
}
}
}
and the hope is that when I call it, it starts the queue and then lets the user continue on with their day (they'll get an email when the queue's done).
As it stands right now, everything works fine except that instead of letting the user continue on, the webpage calling the "web service" just hangs and the request eventually times out.
How do I build an HttpWebResponse and send it back to the other server so that the user can continue on?
I've tried having it return things other than "void" but that doesn't do much.
Here's the Controller that's calling it.
public ActionResult StartQueue()
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://localhost:2394/Home/RunQueue/");
HttpWebResponse response;
string r = "";
try
{
response = (HttpWebResponse)request.GetResponse();
Stream stream = response.GetResponseStream();
StreamReader reader = new StreamReader(stream);
r = reader.ReadToEnd();
}
catch (WebException ex) // A WebException is not fatal. Record the status code.
{
response = (HttpWebResponse)ex.Response;
if (response != null) // timeout
{
r = response.StatusCode.ToString();
}
}
ViewData["message"] = r;
return View();
}
What is the QueueDelegate class? My guess would be that the request is waiting for the other thread to complete (maybe something like Thread.Join()?). I don't think sending a response is the solution you want - I'd suggest either finding a way to spawn a thread that is disconnected from the current request so that it ends naturally, or move the logic out completely into something like a Windows Service.
Hosting your long-running processes in the web context will be complicated at best :(
I use this code snippet that verifies if the file specified in the URL exists and keep trying it every few seconds for every user. Sometimes (mostly when there are large number of users using the site) the code doesn't work.
[WebMethod()]
public static string GetStatus(string URL)
{
bool completed = false;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(URL);
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
try
{
if (response.StatusCode == HttpStatusCode.OK)
{
completed = true;
}
}
catch (Exception)
{
//Just don't do anything. Retry after few seconds
}
}
return completed.ToString();
}
When I look at the Windows Event logs there are several errors:
Unable to read data from the transport connection. An existing connection was forcibly closed
The Operation has timed out
The remote host closed the connection. The error code is 0x800703E3
When I restart the IIS, things work fine until the next time this happens.
You are putting the try/catch inside the using statement while it's the request.GetResponse method that might throw:
bool completed = false;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(URL);
try
{
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
if (response.StatusCode == HttpStatusCode.OK)
{
completed = true;
}
}
}
catch (Exception)
{
//Just don't do anything. Retry after few seconds
}
return completed.ToString();
I wrote this method to check if a page exists or not:
protected bool PageExists(string url)
{
try
{
Uri u = new Uri(url);
WebRequest w = WebRequest.Create(u);
w.Method = WebRequestMethods.Http.Head;
using (StreamReader s = new StreamReader(w.GetResponse().GetResponseStream()))
{
return (s.ReadToEnd().Length >= 0);
}
}
catch
{
return false;
}
}
I am using it to check a set of pages (iterates from AAAA-AAAZ), and it takes between 3 and 7 seconds to run the entire loop. Is there a faster or more efficient way to do this?
I think your approach is rather good, but would change it into only downloading the headers by adding w.Method = WebRequestMethods.Http.Head; before calling GetResponse.
This could do it:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.example.com");
request.Method = WebRequestMethods.Http.Head;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
bool pageExists = response.StatusCode == HttpStatusCode.OK;
You may probably want to check for other status codes as well.
static bool GetCheck(string address)
{
try
{
HttpWebRequest request = WebRequest.Create(address) as HttpWebRequest;
request.Method = "GET";
request.CachePolicy = new RequestCachePolicy(RequestCacheLevel.NoCacheNoStore);
var response = request.GetResponse();
return (response.Headers.Count > 0);
}
catch
{
return false;
}
}
static bool HeadCheck(string address)
{
try
{
HttpWebRequest request = WebRequest.Create(address) as HttpWebRequest;
request.Method = "HEAD";
request.CachePolicy = new RequestCachePolicy(RequestCacheLevel.NoCacheNoStore);
var response = request.GetResponse();
return (response.Headers.Count > 0);
}
catch
{
return false;
}
}
Beware, certain pages (eg. WCF .svc files) may not return anything from a head request. I know because I'm working around this right now.
EDIT - I know there are better ways to check the return data than counting headers, but this is a copy/paste from stuff where this is important to us.
One obvious speedup is to run several requests in parallel - most of the time will be spent on IO, so spawning 10 threads to each check a page will complete the whole iteration around 10 times faster.
You could do it using asynchronous way, because now you are waiting for results after each request. For few pages, you could just throw your function in ThreadPool, and wait for all requests to finish. For more requests, you could use asynchronous methods for your ResponseStream() (BeginRead etc.).
The other thing that can help you (help me for sure) is to clear .Proxy property:
w.Proxy = null;
Without this, at least 1st request is much slower, at least on my machine.
3. You can not download whole page, but download only header, by setting .Method to "HEAD".
I simply used Fredrik Mörk answer above but placed it within a method:
private bool checkURL(string url)
{
bool pageExists = false;
try
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = WebRequestMethods.Http.Head;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
pageExists = response.StatusCode == HttpStatusCode.OK;
}
catch (Exception e)
{
//Do what ever you want when its no working...
//Response.Write( e.ToString());
}
return pageExists;
}
I am making a simple program in visual c# 2005 that looks up a stock symbol on Yahoo! Finance, downloads the historical data, and then plots the price history for the specified ticker symbol.
I know the exact URL that I need to acquire the data, and if the user inputs an existing ticker symbol (or at least one with data on Yahoo! Finance) it works perfectly fine. However, I have a run-time error if the user makes up a ticker symbol, as the program tries to pull data from a non-existent web page.
I am using the WebClient class, and using the DownloadString function. I looked through all the other member functions of the WebClient class, but didn't see anything I could use to test a URL.
How can I do this?
Here is another implementation of this solution:
using System.Net;
///
/// Checks the file exists or not.
///
/// The URL of the remote file.
/// True : If the file exits, False if file not exists
private bool RemoteFileExists(string url)
{
try
{
//Creating the HttpWebRequest
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
//Setting the Request method HEAD, you can also use GET too.
request.Method = "HEAD";
//Getting the Web Response.
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
//Returns TRUE if the Status code == 200
response.Close();
return (response.StatusCode == HttpStatusCode.OK);
}
catch
{
//Any exception will returns false.
return false;
}
}
From: http://www.dotnetthoughts.net/2009/10/14/how-to-check-remote-file-exists-using-c/
You could issue a "HEAD" request rather than a "GET"?
So to test a URL without the cost of downloading the content:
// using MyClient from linked post
using(var client = new MyClient()) {
client.HeadOnly = true;
// fine, no content downloaded
string s1 = client.DownloadString("http://google.com");
// throws 404
string s2 = client.DownloadString("http://google.com/silly");
}
You would try/catch around the DownloadString to check for errors; no error? It exists...
With C# 2.0 (VS2005):
private bool headOnly;
public bool HeadOnly {
get {return headOnly;}
set {headOnly = value;}
}
and
using(WebClient client = new MyClient())
{
// code as before
}
These solutions are pretty good, but they are forgetting that there may be other status codes than 200 OK. This is a solution that I've used on production environments for status monitoring and such.
If there is a url redirect or some other condition on the target page, the return will be true using this method. Also, GetResponse() will throw an exception and hence you will not get a StatusCode for it. You need to trap the exception and check for a ProtocolError.
Any 400 or 500 status code will return false. All others return true.
This code is easily modified to suit your needs for specific status codes.
/// <summary>
/// This method will check a url to see that it does not return server or protocol errors
/// </summary>
/// <param name="url">The path to check</param>
/// <returns></returns>
public bool UrlIsValid(string url)
{
try
{
HttpWebRequest request = HttpWebRequest.Create(url) as HttpWebRequest;
request.Timeout = 5000; //set the timeout to 5 seconds to keep the user from waiting too long for the page to load
request.Method = "HEAD"; //Get only the header information -- no need to download any content
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
int statusCode = (int)response.StatusCode;
if (statusCode >= 100 && statusCode < 400) //Good requests
{
return true;
}
else if (statusCode >= 500 && statusCode <= 510) //Server Errors
{
//log.Warn(String.Format("The remote server has thrown an internal error. Url is not valid: {0}", url));
Debug.WriteLine(String.Format("The remote server has thrown an internal error. Url is not valid: {0}", url));
return false;
}
}
}
catch (WebException ex)
{
if (ex.Status == WebExceptionStatus.ProtocolError) //400 errors
{
return false;
}
else
{
log.Warn(String.Format("Unhandled status [{0}] returned for url: {1}", ex.Status, url), ex);
}
}
catch (Exception ex)
{
log.Error(String.Format("Could not test url {0}.", url), ex);
}
return false;
}
If I understand your question correctly, you could use a small method like this to give you the results of your URL test:
WebRequest webRequest = WebRequest.Create(url);
WebResponse webResponse;
try
{
webResponse = webRequest.GetResponse();
}
catch //If exception thrown then couldn't get response from address
{
return 0;
}
return 1;
You could wrap the above code in a method and use it to perform validation. I hope this answers the question you were asking.
Try this (Make sure you use System.Net):
public bool checkWebsite(string URL) {
try {
WebClient wc = new WebClient();
string HTMLSource = wc.DownloadString(URL);
return true;
}
catch (Exception) {
return false;
}
}
When the checkWebsite() function gets called, it tries to get the source code of
the URL passed into it. If it gets the source code, it returns true. If not,
it returns false.
Code Example:
//The checkWebsite command will return true:
bool websiteExists = this.checkWebsite("https://www.google.com");
//The checkWebsite command will return false:
bool websiteExists = this.checkWebsite("https://www.thisisnotarealwebsite.com/fakepage.html");
I have always found Exceptions are much slower to be handled.
Perhaps a less intensive way would yeild a better, faster, result?
public bool IsValidUri(Uri uri)
{
using (HttpClient Client = new HttpClient())
{
HttpResponseMessage result = Client.GetAsync(uri).Result;
HttpStatusCode StatusCode = result.StatusCode;
switch (StatusCode)
{
case HttpStatusCode.Accepted:
return true;
case HttpStatusCode.OK:
return true;
default:
return false;
}
}
}
Then just use:
IsValidUri(new Uri("http://www.google.com/censorship_algorithm"));
A lot of the answers are older than HttpClient (I think it was introduced in Visual Studio 2013) or without async/await functionality, so I decided to post my own solution:
private static async Task<bool> DoesUrlExists(String url)
{
try
{
using (HttpClient client = new HttpClient())
{
//Do only Head request to avoid download full file
var response = await client.SendAsync(new HttpRequestMessage(HttpMethod.Head, url));
if (response.IsSuccessStatusCode) {
//Url is available is we have a SuccessStatusCode
return true;
}
return false;
}
} catch {
return false;
}
}
I use HttpClient.SendAsync with HttpMethod.Head to make only a head request, and not downlaod the whole file. Like David and Marc already say there is not only http 200 for ok, so I use IsSuccessStatusCode to allow all Sucess Status codes.
WebRequest request = WebRequest.Create("http://www.google.com");
try
{
request.GetResponse();
}
catch //If exception thrown then couldn't get response from address
{
MessageBox.Show("The URL is incorrect");`
}
This solution seems easy to follow:
public static bool isValidURL(string url) {
WebRequest webRequest = WebRequest.Create(url);
WebResponse webResponse;
try
{
webResponse = webRequest.GetResponse();
}
catch //If exception thrown then couldn't get response from address
{
return false ;
}
return true ;
}
Here is another option
public static bool UrlIsValid(string url)
{
bool br = false;
try {
IPHostEntry ipHost = Dns.Resolve(url);
br = true;
}
catch (SocketException se) {
br = false;
}
return br;
}
A lot of other answers are using WebRequest which is now obsolete.
Here is a method that has minimal code and uses currently up-to-date classes and methods.
I have also tested the other most up-voted functions which can produce false positives.
I tested with these URLs, which points to the Visual Studio Community Installer, found on this page.
//Valid URL
https://aka.ms/vs/17/release/vs_community.exe
//Invalid URL, redirects. Produces false positive on other methods.
https://aka.ms/vs/14/release/vs_community.exe
using System.Net;
using System.Net.Http;
//HttpClient is not meant to be created and disposed frequently.
//Declare it staticly in the class to be reused.
static HttpClient client = new HttpClient();
/// <summary>
/// Checks if a remote file at the <paramref name="url"/> exists, and if access is not restricted.
/// </summary>
/// <param name="url">URL to a remote file.</param>
/// <returns>True if the file at the <paramref name="url"/> is able to be downloaded, false if the file does not exist, or if the file is restricted.</returns>
public static bool IsRemoteFileAvailable(string url)
{
//Checking if URI is well formed is optional
Uri uri = new Uri(url);
if (!uri.IsWellFormedOriginalString())
return false;
try
{
using (HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Head, uri))
using (HttpResponseMessage response = client.Send(request))
{
return response.IsSuccessStatusCode && response.Content.Headers.ContentLength > 0;
}
}
catch
{
return false;
}
}
Just note that this will not work with .NET Framework, as HttpClient.Send does not exist.
To get it working on .NET Framework you will need to change client.Send(request) to client.SendAsync(request).Result.
Web servers respond with a HTTP status code indicating the outcome of the request e.g. 200 (sometimes 202) means success, 404 - not found etc (see here). Assuming the server address part of the URL is correct and you are not getting a socket timeout, the exception is most likely telling you the HTTP status code was other than 200. I would suggest checking the class of the exception and seeing if the exception carries the HTTP status code.
IIRC - The call in question throws a WebException or a descendant. Check the class name to see which one and wrap the call in a try block to trap the condition.
i have a more simple way to determine weather a url is valid.
if (Uri.IsWellFormedUriString(uriString, UriKind.RelativeOrAbsolute))
{
//...
}
Following on from the examples already given, I'd say, it's best practice to also wrap the response in a using like this
public bool IsValidUrl(string url)
{
try
{
var request = WebRequest.Create(url);
request.Timeout = 5000;
request.Method = "HEAD";
using (var response = (HttpWebResponse)request.GetResponse())
{
response.Close();
return response.StatusCode == HttpStatusCode.OK;
}
}
catch (Exception exception)
{
return false;
}
}