We Keep Coding

GetResponse keeps throwing page not found exception

Can you please help me out here? I am trying to get content of any web page. But GetResponse keeps throwing exception page not found. I appreciate your help. Following is my code.
try
{
System.Net.ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.smallchiptechnologies.com/");
request.Method = "POST";
request.ContentType = "text/plain";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
}
catch (WebException ex)
{
}

First, it looks like it should be GET response, not POST (since you just trying to get data from server and not posting any form data or something similar) so change request.Method = "POST"; to request.Method = "GET";
Second, you're not reading anything from response stream. Add something like this to your code to get page content:
string text;
using (var reader = new StreamReader(response.GetResponseStream(), Encoding.UTF8))
{
text = reader.ReadToEnd();
}

Have you discarded potential proxy issues? For example, if you are running that code behind a corporate webproxy, you'll need to modify slightly your code in order to support the connection thru that proxy.
Something like this...
webrequest.Proxy = new WebProxy(ProxyServer,ProxyPort);
More details here: https://msdn.microsoft.com/en-us/library/czdt10d3(v=vs.110).aspx

HttpWebRequest.GetResponse() keeps getting timed out

i wrote a simple C# function to retrieve trade history from MtGox with following API call:
https://data.mtgox.com/api/1/BTCUSD/trades?since=<trade_id>
documented here: https://en.bitcoin.it/wiki/MtGox/API/HTTP/v1#Multi_currency_trades
here's the function:
string GetTradesOnline(Int64 tid)
{
Thread.Sleep(30000);
// communicate
string url = "https://data.mtgox.com/api/1/BTCUSD/trades?since=" + tid.ToString();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string json = reader.ReadToEnd();
reader.Close();
reader.Dispose();
response.Close();
return json;
}
i'm starting at tid=0 (trade id) to get the data (from the very beginning). for each request, i receive a response containing 1000 trade details. i always send the trade id from the previous response for the next request. it works fine for exactly 4 requests & responses. but after that, the following line throws a "System.Net.WebException", saying that "The operation has timed out":
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
here are the facts:
catching the exception and retying keeps causing the same exception
the default HttpWebRequest .Timeout and .ReadWriteTimeout are already high enough (over a minute)
changing HttpWebRequest.KeepAlive to false didn't solve anything either
it seems to always work in the browser even while the function is failing
it has no problems retrieveing the response from https://www.google.com
the amount of successful responses before the exceptions varies from day to day (but browser always works)
starting at the trade id that failed last time causes the exception immediately
calling this function from the main thread instead still caused the exception
running it on a different machine didn't work
running it from a different IP didn't work
increasing Thread.Sleep inbetween requests does not help
any ideas of what could be wrong?

I had the very same issue.
For me the fix was as simple as wrapping the HttpWebResponse code in using block.
using (HttpWebResponse response = (HttpWebResponse) request.GetResponse())
{
// Do your processings here....
}
Details: This issue usually happens when several requests are made to the same host, and WebResponse is not disposed properly. That is where using block will properly dispose the WebResponse object properly and thus solving the issue.

There are two kind of timeouts. Client timeout and server timeout. Have you tried doing something like this:
request.Timeout = Timeout.Infinite;
request.KeepAlive = true;
Try something like this...

I just had similar troubles calling a REST Service on a LINUX Server thru ssl. After trying many different configuration scenarios I found out that I had to send a UserAgent in the http head.
Here is my final method for calling the REST API.
private static string RunWebRequest(string url, string json)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
// Header
request.ContentType = "application/json";
request.Method = "POST";
request.AllowAutoRedirect = false;
request.KeepAlive = false;
request.Timeout = 30000;
request.ReadWriteTimeout = 30000;
request.UserAgent = "test.net";
request.Accept = "application/json";
request.ProtocolVersion = HttpVersion.Version11;
request.Headers.Add("Accept-Language","de_DE");
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls;
ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
byte[] bytes = Encoding.UTF8.GetBytes(json);
request.ContentLength = bytes.Length;
using (var writer = request.GetRequestStream())
{
writer.Write(bytes, 0, bytes.Length);
writer.Flush();
writer.Close();
}
var httpResponse = (HttpWebResponse)request.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var jsonReturn = streamReader.ReadToEnd();
return jsonReturn;
}
}

This is not a solution, but just an alternative:
These days i almost only use WebClient instead of HttpWebRequest. Especially WebClient.UploadString for POST and PUT and WebClient.DownloadString. These simply take and return strings. This way i don't have to deal with streams objects, except when i get a WebException. i can also set the content type with WebClient.Headers["Content-type"] if necessary. The using statement also makes life easier by calling Dispose for me.
Rarely for performance, i set System.Net.ServicePointManager.DefaultConnectionLimit high and instead use HttpClient with it's Async methods for simultaneous calls.
This is how i would do it now
string GetTradesOnline(Int64 tid)
{
using (var wc = new WebClient())
{
return wc.DownloadString("https://data.mtgox.com/api/1/BTCUSD/trades?since=" + tid.ToString());
}
}
2 more POST examples
// POST
string SubmitData(string data)
{
string response;
using (var wc = new WebClient())
{
wc.Headers["Content-type"] = "text/plain";
response = wc.UploadString("https://data.mtgox.com/api/1/BTCUSD/trades", "POST", data);
}
return response;
}
// POST: easily url encode multiple parameters
string SubmitForm(string project, string subject, string sender, string message)
{
// url encoded query
NameValueCollection query = HttpUtility.ParseQueryString(string.Empty);
query.Add("project", project);
query.Add("subject", subject);
// url encoded data
NameValueCollection data = HttpUtility.ParseQueryString(string.Empty);
data.Add("sender", sender);
data.Add("message", message);
string response;
using (var wc = new WebClient())
{
wc.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
response = wc.UploadString( "https://data.mtgox.com/api/1/BTCUSD/trades?"+query.ToString()
, WebRequestMethods.Http.Post
, data.ToString()
);
}
return response;
}
Error handling
try
{
Console.WriteLine(GetTradesOnline(0));
string data = File.ReadAllText(#"C:\mydata.txt");
Console.WriteLine(SubmitData(data));
Console.WriteLine(SubmitForm("The Big Project", "Progress", "John Smith", "almost done"));
}
catch (WebException ex)
{
string msg;
if (ex.Response != null)
{
// read response HTTP body
using (var sr = new StreamReader(ex.Response.GetResponseStream())) msg = sr.ReadToEnd();
}
else
{
msg = ex.Message;
}
Log(msg);
}

For what it's worth, I was experiencing the same issues with timeouts every time I used it, even though calls went through to the server I was calling. The problem in my case was that I had Expect set to application/json, when that wasn't what the server was returning.

Unable to read data from the transport connection: The connection was closed error in console application

I have this code in console application and it runs in a loop
try
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(search);
request.Headers.Add("Accept-Language", "de-DE");
request.Method = "GET";
request.Accept = "text/html";
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (StreamReader reader = new StreamReader(response.GetResponseStream(),
Encoding.ASCII))
{
string html = reader.ReadToEnd();
FindForMatch(html, url);
}
}
}
catch (Exception ex)
{
throw new Exception(ex.Message);
}
after few loops it gives
Unable to read data from the transport connection: The connection was closed
error. any idea why this happen? thanx..

After adding
request.KeepAlive = false;
request.ProtocolVersion = HttpVersion.Version10;
it works fine..
I found it form this blog post
WebRequest and Unable to read data from the transport connection Error

I just tried the code, looping 10 times to load google.com and it worked for me. Is there something special about search - perhaps try replacing it with another uri. I did not include findForMatch - I assume it is not doing anything that would cause the exception.

Try disposing the reader in the finally block of your try catch

HttpWebRequest method HEAD returns body

My web request with the method "Head" keeps returning the body of my webpage (on localhost).
Here is how it is basically created:
HttpWebRequest webrequest = WebRequest.Create(url.ToString()) as HttpWebRequest;
webrequest.Method = "HEAD";
WebResponse response = webrequest.GetResponse();
As I put a breakpoint in my aspx.cs page, I step into the OnInit() method and also the Page_Load() method where I believe I'm not supposed to step in with a Head method request (am I wrong?).
In my Page_Load() I execute some code that I do not want to be executed when I call with the Head method, but later when I call with the Get method (once I got the headers).
Am I missing something? (not too familiar with Http requests and responses yet... :/)
Thank you for your help!

Try this sample code approach ....
for (int i = 0; i < ParsedLinks.Count; i++)
{
Thread.Sleep(500);
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(ParsedLinks[i]);
req.Method = "HEAD";
req.Credentials = CredentialCache.DefaultCredentials;
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
bool b_Result = int.TryParse(resp.Headers.Get("Content-Length"), out i_ContentLength);
int i_Size = (int)(i_ContentLength / 1024);
req.Abort();
resp.Close();
}
hope it helps
http://forums.asp.net/t/1412824.aspx/1

The difference is, if you try to read the response you get nothing in case of HEAD. Where as you can see the response content in case of GET
var response = webrequest.GetResponse().GetResponseStream();
Encoding encode = System.Text.Encoding.GetEncoding("utf-8");
StreamReader readStream = new StreamReader(response, encode);
Console.WriteLine(readStream.ReadToEnd().Length) // you should see 0
readStream.Close();

500 internal server error at GetResponse()

I have a heavy traffic aspx page calling a web service upon every user`s request as follows.
string uri = "Path.asmx";
string soap = "soap xml string";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.Headers.Add("SOAPAction", "\"http://xxxxxx"");
request.ContentType = "text/xml;charset=\"utf-8\"";
request.Accept = "text/xml";
request.Method = "POST";
using (Stream stm = request.GetRequestStream())
{
using (StreamWriter stmw = new StreamWriter(stm))
{
stmw.Write(soap);
}
}
WebResponse response = request.GetResponse();
response.close();
Everything is working fine but sometimes I am getting the following error.
The remote server returned an error: (500) Internal Server Error.
at System.Net.HttpWebRequest.GetResponse()
Does anybody have any idea about this error or can anybody tell me if I am doing wrong.

For me this error occurred because I had 2 web API actions that had the exact same signatures and both had the same verbs, HttpPost, what I did was change one of the verbs (the one used for updating) to PUT and the error was removed. The following in my catch statement helped in getting to the root of the problem:
catch (WebException webex)
{
WebResponse errResp = webex.Response;
using (Stream respStream = errResp.GetResponseStream())
{
StreamReader reader = new StreamReader(respStream);
string text = reader.ReadToEnd();
}
}

From that error, I would say that your code is fine, at least the part that calls the webservice. The error seems to be in the actual web service.
To get the error from the web server, add a try catch and catch a WebException. A WebException has a property called Response which is a HttpResponse. you can then log anything that is returned, and upload you code. Check back later in the logs and see what is actually being returned.

Finally I get rid of internal server error message with the following code. Not sure if there is another way to achieve it.
string uri = "Path.asmx";
string soap = "soap xml string";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.Headers.Add("SOAPAction", "\"http://xxxxxx"");
request.ContentType = "text/xml;charset=\"utf-8\"";
request.Accept = "text/xml";
request.Method = "POST";
using (Stream stm = request.GetRequestStream())
{
using (StreamWriter stmw = new StreamWriter(stm))
{
stmw.Write(soap);
}
}
using (WebResponse webResponse = request.GetResponse())
{
}

Have you tried to specify UserAgent for your request? For example:
request.UserAgent = "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)";

looking at your error message first of all I would suggest you to recompile your whole application, make sure all the required dlls are there in bin folder when you recompile it.

In my case my request object inherited from base object. Without knowingly I added a property with int? in my request object and my base object also has same property ( same name ) with int datatype. I noticed this and deleted the property which I added in request object and after that it worked fine.

For me the error was misleading. I discovered the true error by testing the errant web service with SoapUI.

In my case I just remove the SoapAction instruction from the HttpWebRequest object. So, I don't define .Headers.Add("SOAPAction","someurl") in HttpWebRequest definitions and my code works fine.
ResultXML is an XDocument.
ResultString is a string.
try
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(Url);
//req.Headers.Add("SOAPAction", "http://tempuri.org/IWebService/GetMessage");
req.ProtocolVersion = HttpVersion.Version11;
req.ContentType = "text/xml;charset=\"utf-8\"";
req.Accept = "text/xml";
req.KeepAlive = true;
req.Method = "POST";
using (Stream stm = req.GetRequestStream())
{
using (StreamWriter stmw = new StreamWriter(stm))
stmw.Write(soapStr);
}
using (StreamReader responseReader = new StreamReader(req.GetResponse().GetResponseStream()))
{
string result = responseReader.ReadToEnd();
ResultXML = XDocument.Parse(result);
ResultString = result;
}
}