We Keep Coding

GetResponse keeps throwing page not found exception

Can you please help me out here? I am trying to get content of any web page. But GetResponse keeps throwing exception page not found. I appreciate your help. Following is my code.
try
{
System.Net.ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.smallchiptechnologies.com/");
request.Method = "POST";
request.ContentType = "text/plain";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
}
catch (WebException ex)
{
}

First, it looks like it should be GET response, not POST (since you just trying to get data from server and not posting any form data or something similar) so change request.Method = "POST"; to request.Method = "GET";
Second, you're not reading anything from response stream. Add something like this to your code to get page content:
string text;
using (var reader = new StreamReader(response.GetResponseStream(), Encoding.UTF8))
{
text = reader.ReadToEnd();
}

Have you discarded potential proxy issues? For example, if you are running that code behind a corporate webproxy, you'll need to modify slightly your code in order to support the connection thru that proxy.
Something like this...
webrequest.Proxy = new WebProxy(ProxyServer,ProxyPort);
More details here: https://msdn.microsoft.com/en-us/library/czdt10d3(v=vs.110).aspx

Status code 301 not showing correctly in C#

I am able to get numbers with enum as suggested by dtb in Getting Http Status code number (200, 301, 404, etc.) from HttpWebRequest and HttpWebResponse. However, for moved permanently site also i am getting 200 (OK). What I want to see is 301 instead. Please help. My code is below. What could be wrong/needs to be corrected?
public int GetHeaders(string url)
{
//HttpStatusCode result = default(HttpStatusCode);
int result = 0;
var request = HttpWebRequest.Create(url);
request.Method = "HEAD";
try
{
using (var response = request.GetResponse() as HttpWebResponse)
{
if (response != null)
{
result = (int)response.StatusCode; // response.StatusCode;
response.Close();
}
}
}
catch (WebException we)
{
if (we.Response != null)
{
result = (int)((HttpWebResponse)we.Response).StatusCode;
}
}
return result;
}
The tool where i am using this code is capable of showing 404, not existing domains but it is ignoring the redirects and shows the details about the redirected url. e.g if i put my older domain easytipsandtricks.com in the text field, it shows the results for tipscow.com (if you check easytipsandtricks.com in any checker tool online, you will notice that it is giving the correct redirect message - 301 Moved). Please help.

You need to set HttpWebRequest.AllowAutoRedirect to false (default is true) for it to not automatically follow redirects (30x responses).
If AllowAutoRedirect is set to false, all responses with an HTTP status code from 300 to 399 is returned to the application.
Sample:
var request = (HttpWebRequest)HttpWebRequest.Create(url);
request.Method = "HEAD";
request.AllowAutoRedirect = false;

HttpWebRequest.GetResponse() keeps getting timed out

i wrote a simple C# function to retrieve trade history from MtGox with following API call:
https://data.mtgox.com/api/1/BTCUSD/trades?since=<trade_id>
documented here: https://en.bitcoin.it/wiki/MtGox/API/HTTP/v1#Multi_currency_trades
here's the function:
string GetTradesOnline(Int64 tid)
{
Thread.Sleep(30000);
// communicate
string url = "https://data.mtgox.com/api/1/BTCUSD/trades?since=" + tid.ToString();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string json = reader.ReadToEnd();
reader.Close();
reader.Dispose();
response.Close();
return json;
}
i'm starting at tid=0 (trade id) to get the data (from the very beginning). for each request, i receive a response containing 1000 trade details. i always send the trade id from the previous response for the next request. it works fine for exactly 4 requests & responses. but after that, the following line throws a "System.Net.WebException", saying that "The operation has timed out":
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
here are the facts:
catching the exception and retying keeps causing the same exception
the default HttpWebRequest .Timeout and .ReadWriteTimeout are already high enough (over a minute)
changing HttpWebRequest.KeepAlive to false didn't solve anything either
it seems to always work in the browser even while the function is failing
it has no problems retrieveing the response from https://www.google.com
the amount of successful responses before the exceptions varies from day to day (but browser always works)
starting at the trade id that failed last time causes the exception immediately
calling this function from the main thread instead still caused the exception
running it on a different machine didn't work
running it from a different IP didn't work
increasing Thread.Sleep inbetween requests does not help
any ideas of what could be wrong?

I had the very same issue.
For me the fix was as simple as wrapping the HttpWebResponse code in using block.
using (HttpWebResponse response = (HttpWebResponse) request.GetResponse())
{
// Do your processings here....
}
Details: This issue usually happens when several requests are made to the same host, and WebResponse is not disposed properly. That is where using block will properly dispose the WebResponse object properly and thus solving the issue.

There are two kind of timeouts. Client timeout and server timeout. Have you tried doing something like this:
request.Timeout = Timeout.Infinite;
request.KeepAlive = true;
Try something like this...

I just had similar troubles calling a REST Service on a LINUX Server thru ssl. After trying many different configuration scenarios I found out that I had to send a UserAgent in the http head.
Here is my final method for calling the REST API.
private static string RunWebRequest(string url, string json)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
// Header
request.ContentType = "application/json";
request.Method = "POST";
request.AllowAutoRedirect = false;
request.KeepAlive = false;
request.Timeout = 30000;
request.ReadWriteTimeout = 30000;
request.UserAgent = "test.net";
request.Accept = "application/json";
request.ProtocolVersion = HttpVersion.Version11;
request.Headers.Add("Accept-Language","de_DE");
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls;
ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
byte[] bytes = Encoding.UTF8.GetBytes(json);
request.ContentLength = bytes.Length;
using (var writer = request.GetRequestStream())
{
writer.Write(bytes, 0, bytes.Length);
writer.Flush();
writer.Close();
}
var httpResponse = (HttpWebResponse)request.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var jsonReturn = streamReader.ReadToEnd();
return jsonReturn;
}
}

This is not a solution, but just an alternative:
These days i almost only use WebClient instead of HttpWebRequest. Especially WebClient.UploadString for POST and PUT and WebClient.DownloadString. These simply take and return strings. This way i don't have to deal with streams objects, except when i get a WebException. i can also set the content type with WebClient.Headers["Content-type"] if necessary. The using statement also makes life easier by calling Dispose for me.
Rarely for performance, i set System.Net.ServicePointManager.DefaultConnectionLimit high and instead use HttpClient with it's Async methods for simultaneous calls.
This is how i would do it now
string GetTradesOnline(Int64 tid)
{
using (var wc = new WebClient())
{
return wc.DownloadString("https://data.mtgox.com/api/1/BTCUSD/trades?since=" + tid.ToString());
}
}
2 more POST examples
// POST
string SubmitData(string data)
{
string response;
using (var wc = new WebClient())
{
wc.Headers["Content-type"] = "text/plain";
response = wc.UploadString("https://data.mtgox.com/api/1/BTCUSD/trades", "POST", data);
}
return response;
}
// POST: easily url encode multiple parameters
string SubmitForm(string project, string subject, string sender, string message)
{
// url encoded query
NameValueCollection query = HttpUtility.ParseQueryString(string.Empty);
query.Add("project", project);
query.Add("subject", subject);
// url encoded data
NameValueCollection data = HttpUtility.ParseQueryString(string.Empty);
data.Add("sender", sender);
data.Add("message", message);
string response;
using (var wc = new WebClient())
{
wc.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
response = wc.UploadString( "https://data.mtgox.com/api/1/BTCUSD/trades?"+query.ToString()
, WebRequestMethods.Http.Post
, data.ToString()
);
}
return response;
}
Error handling
try
{
Console.WriteLine(GetTradesOnline(0));
string data = File.ReadAllText(#"C:\mydata.txt");
Console.WriteLine(SubmitData(data));
Console.WriteLine(SubmitForm("The Big Project", "Progress", "John Smith", "almost done"));
}
catch (WebException ex)
{
string msg;
if (ex.Response != null)
{
// read response HTTP body
using (var sr = new StreamReader(ex.Response.GetResponseStream())) msg = sr.ReadToEnd();
}
else
{
msg = ex.Message;
}
Log(msg);
}

For what it's worth, I was experiencing the same issues with timeouts every time I used it, even though calls went through to the server I was calling. The problem in my case was that I had Expect set to application/json, when that wasn't what the server was returning.

"Request timed out" when makinng two HttpWebRequests to the same Java server from C# over .NET Compact

I have a client application running on C# on a .NETCF 3.5 device POSTing to a Java servlet located remotely. I am receiving a "Request Timed out" during my third HTTP POST to the same servlet. For example, if the servlet manages login to our Java server, the first two login attempts from the client would go through (same client device) and when I attempt the third one, it will return in a "Request timed out" exception from the server. I have noticed this happen always and I can't figure out the problem. I read that C# by default sends the Request 100 continue in the HTTP headers so I used the ServicePointManager to set the request 100 to false to no avail.
Here is the code that is throwing this error:
serverUrl = url;
string responseFromServer = "";
try
{
System.Net.ServicePointManager.Expect100Continue = false;
int tmp = ServicePointManager.DefaultConnectionLimit;
// Create a request using a URL that can receive a post.
request = (HttpWebRequest)WebRequest.Create(url);
// Set the Method property of the request to POST.
request.Method = "POST";
// Create POST data and convert it to a byte array.
byte[] byteArray = System.Text.Encoding.UTF8.GetBytes(url);
// Set the ContentType property of the WebRequest.
request.ContentType = "application/x-www-form-urlencoded";
// Set the ContentLength property of the WebRequest.
request.ContentLength = byteArray.Length;
request.Timeout = (50 * 100);
request.Proxy = System.Net.GlobalProxySelection.GetEmptyWebProxy();
// Get the request stream.
Stream dataStream = request.GetRequestStream();
// Write the data to the request stream.
dataStream.Write(byteArray, 0, byteArray.Length);
// Close the Stream object.
dataStream.Close();
// Get the response.
response = (HttpWebResponse)request.GetResponse();
// Get the stream containing content returned by the server.
dataStream = response.GetResponseStream();
// Open the stream using a StreamReader for easy access.
StreamReader reader = new StreamReader(dataStream);
// Read the content.
responseFromServer = reader.ReadToEnd();
// Clean up the streams.
reader.Close();
dataStream.Close();
response.Close();
return responseFromServer;
}
catch (Exception WebExp)
{
Logging.Instance.Log(Logging.Levels.Error, "Error in DoPost while retrieving : "+url+ " " + WebExp.Message.ToString());
Logging.Instance.Log(Logging.Levels.Error, WebExp.StackTrace.ToString());
throw WebExp;
}
Any help would be much appreciated. Thanks!

This behaviour is due to wrong Exception handling regarding to the WebResponse. You have always to handle the response and close it. Otherwise the third attempt of HTTP Webrequest will fail with timeout which is limited by WinCE.
Following source code will be safe:
HttpWebResponse response = null;
try
{
//...
response = (HttpWebResponse)request.GetResponse();
//...
}
catch(Exception e)
{
// logging, etc.
throw e;
}
finally
{
if(response!=null)
{
response.Close();
}
}

Getting connection closed error when trying to read response from FitNesse REST URI

Have you run into this problem? I run code remarkably similar to that from a this previous question, When in nUnitTest mode and the URI includes "/?test&format=xml" the nUnit test fails with and IOException, "Unable to read data from the transport connection: The connection is closed."
However the Fiddler trace that was running at the time shows the very xml I expected.
I've recreated the request headers exactly (almost) as they are sent when sent through the browser.
Finally, if I leave off the "/?test&format=xml" from the URI, I get the html I would have otherwise expected.
SOURCE CODE:
public virtual bool Run()
{
var request = CreateRequest();
var response = GetResponse(request);
var responseString = ReadResponse(response);
this.SetResults(responseString);
return this.IsSuccessful;
}
protected internal virtual HttpWebRequest CreateRequest()
{
var address = TestConfig.Address;
var request = (HttpWebRequest)WebRequest.Create(address);
request.Accept = "*/*";
request.UseDefaultCredentials = true;
request.CachePolicy = new RequestCachePolicy(RequestCacheLevel.NoCacheNoStore);
return request;
}
protected internal virtual HttpWebResponse GetResponse(HttpWebRequest request)
{
var response = (HttpWebResponse) request.GetResponse();
return response;
}
protected internal virtual string ReadResponse(HttpWebResponse response)
{
var stream = response.GetResponseStream();
var responseString = ReadResponse(stream);
stream.Close();
response.Close();
return responseString;
}
protected internal virtual string ReadResponse(Stream stream)
{
var reader = new StreamReader(stream);
var responseString = reader.ReadToEnd();
return responseString;
}

The error message "Unable to read data from the transport connection: The connection is closed." doesn't really tie up with the fact you're seeing Fiddler getting a html response body back.
Check the StatusCode of the HttpWebResponse (should be 200 if ok), also wrap the request in try/catch block (example from http://msdn.microsoft.com/en-us/library/system.net.httpwebresponse.statuscode(v=vs.80).aspx)
try
{
// Creates an HttpWebRequest for the specified URL.
HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create(url);
// Sends the HttpWebRequest and waits for a response.
HttpWebResponse myHttpWebResponse = (HttpWebResponse)myHttpWebRequest.GetResponse();
if (myHttpWebResponse.StatusCode == HttpStatusCode.OK)
Console.WriteLine("\r\nResponse Status Code is OK and StatusDescription is: {0}",
myHttpWebResponse.StatusDescription);
// Releases the resources of the response.
myHttpWebResponse.Close();
}
catch(WebException e)
{
Console.WriteLine("\r\nWebException Raised. The following error occured : {0}",e.Status);
}
catch(Exception e)
{
Console.WriteLine("\nThe following Exception was raised : {0}",e.Message);
}
If you're creating and disposing of the HttpWebRequest object quickly you might be getting the socket going into a time_wait state as it is shutting down, then you can't re-open it again until it has completely closed. If this is the case, look into using another port or changing the time the connection lives for.