We Keep Coding

How to get the favicon from a 403 page

I am writing a tool that allows the user to input a URL, to which the program responds by attempting to show that website's favicon. I have this working for many sites but one site that is giving me trouble is my self-hosted Trac site. It seems that Trac's normal behaviour, until the end user is autenticated, is to show a custom 403 page (Forbidden), inviting the user to log in. Accessing Trac from a web browser, the favicon displays in the browser's tab, even though I'm not logged in (and Firebug, for instance, shows a 403 for the page content). If I view source from the browser, the favicon's location is right there in the source. However, from my application, requesting the Trac website with request.GetResponse() throws a WebException containing a 403, giving me no opportunity to read the response stream that contains the vital information required to find the favicon.
I already have code to download a website's HTML and extract the location of its favicon. What I am stuck with is downloading a site's HTML even when it responds with a 403.
I played with various UserAgent, Accept and AcceptLanguage properties of the HttpWebRequest object but it didn't help. I also tried following any redirects myself as I read somewhere that .NET doesn't do them well. Still no luck.
Here's what I have:
public static MemoryStream DownloadHtml(
string urlParam,
int timeoutMs = DefaultHttpRequestTimeoutMs,
string userAgent = "",
bool silent = false
)
{
MemoryStream result = null;
HttpWebRequest request = null;
HttpWebResponse response = null;
try
{
Func<string, HttpWebRequest> createRequest = (urlForFunc) =>
{
var requestForAction = (HttpWebRequest)HttpWebRequest.Create(urlForFunc);
// This step is now required by Wikipedia (and others?) to prevent periodic or
// even constant 403's (Forbidden).
requestForAction.UserAgent = userAgent;
requestForAction.Accept = "text/html";
requestForAction.AllowAutoRedirect = false;
requestForAction.Timeout = timeoutMs;
return requestForAction;
};
string urlFromResponse = "";
string urlForRequest = "";
do
{
if(response == null)
{
urlForRequest = urlParam;
}
else
{
urlForRequest = urlFromResponse;
response.Close();
}
request = createRequest(urlForRequest);
response = (HttpWebResponse)request.GetResponse();
urlFromResponse = response.Headers[HttpResponseHeader.Location];
}
while(urlFromResponse != null
&& urlFromResponse.Length > 0
&& urlFromResponse != urlForRequest);
using(var stream = response.GetResponseStream())
{
result = new MemoryStream();
stream.CopyTo(result);
}
}
catch(WebException ex)
{
// Things like 404 and, well, all other web-type exceptions.
Debug.WriteLine(ex.Message);
if(ex.InnerException != null) Debug.WriteLine(ex.InnerException.Message);
}
catch(System.Threading.ThreadAbortException)
{
// Let ac.Thread handle some cleanup.
throw;
}
catch(Exception)
{
if(!silent) throw;
}
finally
{
if(response != null) response.Close();
}
return result;
}

The stream content is stored in Exception object.
var resp = new StreamReader(ex.Response.GetResponseStream()).ReadToEnd();

HttpWebRequest - GetResponse() - WebException ReceiveFailure

I have a WCF service that is running frequent (1000+) outbound connections per minute to external APIs.
My code throws the following exceptions frequently, but not always showing that is is a WebException with the WebException status property being ReceiveFailure
The code that is making the outbound request is the following:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(paramBuilder.ToString());
request.ServicePoint.ConnectionLeaseTimeout = 0;
request.Method = "GET";
request.Timeout = 33000; //33 Second Timeout Is By Design
Stream stream = default(Stream);
HttpWebResponse response = default(HttpWebResponse);
try
{
response = (HttpWebResponse) request.GetResponse();
stream = response.GetResponseStream();
reader = new StreamReader(stream,Encoding.UTF8);
string str = reader.ReadToEnd();
return str;
}
catch (WebException exception)
{
//Handle WebException
}
catch (Exception exception)
{
//Handle Exception
}
finally
{
if (reader != null)
reader.Dispose();
if (response != null)
response.Close();
if (stream != null)
stream.Dispose();
}
The exception stack trace shows that the exception is caused from GetResponse().
What could be causing this to happen that I receive an occasional WebException -ReceiveFailure.
I have already reference the MSDN documentation for this status, but that doesn't help me.

Shooting in the dark here...
There is a special condition, while waiting for response: if the system clock is being set automatically by the Windows Time service, or manually, you may experience some unpredictable results.
If you're sending your requests over HTTPS, maybe you're facing a regular timeout that was wrongly thrown as a ReceiveFailure.
Check this article for more information: http://support.microsoft.com/kb/2007873

I have a related problem and I realise a few things while I was searching for a solution.
WebExceptionStatus enum is not equivalent to http status code that the API you call returned. Instead it is a enum of possible error that may occour during a http call.
The WebExceptionStatus error code that will be returned when you receive an error (400 to 599) from your API is WebExceptionStatus.ProtocolError aka number 7 as int.
When you need to get the response body or the real http status code returned from the api, first you need to check if WebException.Status is WebExceptionStatus.ProtocolError. Then you can get the real response from WebExceptionStatus.Response and read its content.
Sometimes the timeout is handled by the caller (aka your code) so you do not have a response in that case. So you can look if WebException.Status is WebExceptionStatus.Timeout
This is an example:
try
{
...
}
catch (WebException webException)
{
if (webException.Status == WebExceptionStatus.ProtocolError)
{
var httpResponse = (HttpWebResponse)webException.Response;
var responseText = "";
using (var content = new StreamReader(httpResponse.GetResponseStream()))
{
responseText = content.ReadToEnd(); // Get response body as text
}
int statusCode = (int)httpResponse.StatusCode; // Get the status code
}
else if (webException.Status == WebExceptionStatus.ProtocolError)
{
// Timeout handled by your code. You do not have a response here.
}
// Handle other webException.Status errors. You do not have a response here.
}

Xml response file: Received in browser, not via C#

I'm trying to access the last.fm APIs via C#. As a first test I'm querying similar artists if that matters.
I get an XML response when I pass a correct artist name, i.e. "Nirvana". My problem is when I deliver an invalid name (i.e. "Nirvana23") I don't receive XML but an error code (403 or 400) and a WebException.
Interesting thing: If I enter the URL inside a browser (tested with Firefox and Chrome) I receive the XML file I want (containing a lastfm specific error message).
I tried both XmlReader and XDocument:
XDocument doc = XDocument.Load(requestUrl);
and HttpWebRequest:
string httpResponse = "";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestUrl);
HttpWebResponse response = null;
StreamReader reader = null;
try
{
response = (HttpWebResponse)request.GetResponse();
reader = new StreamReader(response.GetResponseStream());
httpResponse = reader.ReadToEnd();
}
The URL is something like "http://ws.audioscrobbler.com/2.0/?method=artist.getsimilar&artist=Nirvana23" (and a specific key given by lastfm, but even without it - it should return XML). A link to give it a try: link (this is the error file I cannot access via C#).
What I also tried (without success): comparing the request by both the browser and my program with the help of WireShark. Then I added some headers to the request, but that didn't help either.

In .NET the WebRequest is converting HTTP error codes into exceptions, while your browser is just ignoring them since the response is not empty. If you catch the exception then the GetResponseStream method should still return the error XML that you are expecting.
Edit:
Try this:
string httpResponse = "";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestUrl);
WebResponse response = null;
StreamReader reader = null;
try
{
response = request.GetResponse();
}
catch (WebException ex)
{
response = ex.Response;
}
reader = new StreamReader(response.GetResponseStream());
httpResponse = reader.ReadToEnd();

Why don't you catch the exception and then process that accordingly. If you want to display any custom error, you can do that also in your catch block.

WebRequest Exception in .NET

I use this code snippet that verifies if the file specified in the URL exists and keep trying it every few seconds for every user. Sometimes (mostly when there are large number of users using the site) the code doesn't work.
[WebMethod()]
public static string GetStatus(string URL)
{
bool completed = false;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(URL);
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
try
{
if (response.StatusCode == HttpStatusCode.OK)
{
completed = true;
}
}
catch (Exception)
{
//Just don't do anything. Retry after few seconds
}
}
return completed.ToString();
}
When I look at the Windows Event logs there are several errors:
Unable to read data from the transport connection. An existing connection was forcibly closed
The Operation has timed out
The remote host closed the connection. The error code is 0x800703E3
When I restart the IIS, things work fine until the next time this happens.

You are putting the try/catch inside the using statement while it's the request.GetResponse method that might throw:
bool completed = false;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(URL);
try
{
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
if (response.StatusCode == HttpStatusCode.OK)
{
completed = true;
}
}
}
catch (Exception)
{
//Just don't do anything. Retry after few seconds
}
return completed.ToString();

Getting connection closed error when trying to read response from FitNesse REST URI

Have you run into this problem? I run code remarkably similar to that from a this previous question, When in nUnitTest mode and the URI includes "/?test&format=xml" the nUnit test fails with and IOException, "Unable to read data from the transport connection: The connection is closed."
However the Fiddler trace that was running at the time shows the very xml I expected.
I've recreated the request headers exactly (almost) as they are sent when sent through the browser.
Finally, if I leave off the "/?test&format=xml" from the URI, I get the html I would have otherwise expected.
SOURCE CODE:
public virtual bool Run()
{
var request = CreateRequest();
var response = GetResponse(request);
var responseString = ReadResponse(response);
this.SetResults(responseString);
return this.IsSuccessful;
}
protected internal virtual HttpWebRequest CreateRequest()
{
var address = TestConfig.Address;
var request = (HttpWebRequest)WebRequest.Create(address);
request.Accept = "*/*";
request.UseDefaultCredentials = true;
request.CachePolicy = new RequestCachePolicy(RequestCacheLevel.NoCacheNoStore);
return request;
}
protected internal virtual HttpWebResponse GetResponse(HttpWebRequest request)
{
var response = (HttpWebResponse) request.GetResponse();
return response;
}
protected internal virtual string ReadResponse(HttpWebResponse response)
{
var stream = response.GetResponseStream();
var responseString = ReadResponse(stream);
stream.Close();
response.Close();
return responseString;
}
protected internal virtual string ReadResponse(Stream stream)
{
var reader = new StreamReader(stream);
var responseString = reader.ReadToEnd();
return responseString;
}

The error message "Unable to read data from the transport connection: The connection is closed." doesn't really tie up with the fact you're seeing Fiddler getting a html response body back.
Check the StatusCode of the HttpWebResponse (should be 200 if ok), also wrap the request in try/catch block (example from http://msdn.microsoft.com/en-us/library/system.net.httpwebresponse.statuscode(v=vs.80).aspx)
try
{
// Creates an HttpWebRequest for the specified URL.
HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create(url);
// Sends the HttpWebRequest and waits for a response.
HttpWebResponse myHttpWebResponse = (HttpWebResponse)myHttpWebRequest.GetResponse();
if (myHttpWebResponse.StatusCode == HttpStatusCode.OK)
Console.WriteLine("\r\nResponse Status Code is OK and StatusDescription is: {0}",
myHttpWebResponse.StatusDescription);
// Releases the resources of the response.
myHttpWebResponse.Close();
}
catch(WebException e)
{
Console.WriteLine("\r\nWebException Raised. The following error occured : {0}",e.Status);
}
catch(Exception e)
{
Console.WriteLine("\nThe following Exception was raised : {0}",e.Message);
}
If you're creating and disposing of the HttpWebRequest object quickly you might be getting the socket going into a time_wait state as it is shutting down, then you can't re-open it again until it has completely closed. If this is the case, look into using another port or changing the time the connection lives for.