We Keep Coding

GetResponse keeps throwing page not found exception

Can you please help me out here? I am trying to get content of any web page. But GetResponse keeps throwing exception page not found. I appreciate your help. Following is my code.
try
{
System.Net.ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.smallchiptechnologies.com/");
request.Method = "POST";
request.ContentType = "text/plain";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
}
catch (WebException ex)
{
}

First, it looks like it should be GET response, not POST (since you just trying to get data from server and not posting any form data or something similar) so change request.Method = "POST"; to request.Method = "GET";
Second, you're not reading anything from response stream. Add something like this to your code to get page content:
string text;
using (var reader = new StreamReader(response.GetResponseStream(), Encoding.UTF8))
{
text = reader.ReadToEnd();
}

Have you discarded potential proxy issues? For example, if you are running that code behind a corporate webproxy, you'll need to modify slightly your code in order to support the connection thru that proxy.
Something like this...
webrequest.Proxy = new WebProxy(ProxyServer,ProxyPort);
More details here: https://msdn.microsoft.com/en-us/library/czdt10d3(v=vs.110).aspx

Receiving final status code 200 but their are 301-> 302-> 200

I am using
HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse myHttpWebResponse = (HttpWebResponse)myHttpWebRequest.GetResponse();
//myHttpWebResponse.StatusCode
for this I'm receiving 200 but actual status codes are 301->302->200
http://brita.com/
Please let me know. Is there any mistake in code.
Thanks in advance.

That's because the HttpWebRequest automatically follows redirects and only gives you the last status code. If you don't want that you can set the AllowAutoRedirect property to false:
HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create(url);
myHttpWebRequest.AllowAutoRedirect = false;
HttpWebResponse myHttpWebResponse = (HttpWebResponse)myHttpWebRequest.GetResponse();
//myHttpWebResponse.StatusCode
Note that it will stop after the first request though, you'll have to create a new request manually to follow up the redirects.

Getting multiple malformed cookies via c#

After a long day in searching and attempts I could not find a solution for my problem - getting multiple "Set-Cookie" headers.
I was trying to get them via Headers["Set-Cookie"] but it contains the only the first cookie.
Then my next try was to get them via Cookies (which is always with Count 0 and the purpose because I think they are malformed).
Here is the code to the last sample:
var request = WebRequest.Create(resourceUrl) as HttpWebRequest;
request.Method = "GET";
var response = request.GetResponse() as HttpWebResponse;
var cookiesCount = response.Cookies.Count;
And these are the cookies:
Set-Cookie:vjfmrii=67ea0de93a423ab17d168ee8327617b0
Set-Cookie:alpocjengi=dcf10w329x5d7e503ffb9f28123c7492f1c2deb4
Set-Cookie:vjfmrii=4abf7b9e97fff1a61fbcf5e11899ce71
Before I began, I thought that this would be one of the easiest parts, but unfortunately I was wrong. These cookies which have the same names and have several... have left me devastated.
I appreciate any help. Thanks in advance.

You need to add a CookieContainer to your request when you make it or the cookies will not be added to the response. See http://msdn.microsoft.com/en-us/library/dd920298(v=vs.95).aspx.
var request = WebRequest.Create(resourceUrl) as HttpWebRequest;
request.Method = "GET";
request.CookieContainer = new CookieContainer(); // <-- Add this
var response = request.GetResponse() as HttpWebResponse;
var cookiesCount = response.Cookies.Count;

HttpWebRequest.GetResponse() keeps getting timed out

i wrote a simple C# function to retrieve trade history from MtGox with following API call:
https://data.mtgox.com/api/1/BTCUSD/trades?since=<trade_id>
documented here: https://en.bitcoin.it/wiki/MtGox/API/HTTP/v1#Multi_currency_trades
here's the function:
string GetTradesOnline(Int64 tid)
{
Thread.Sleep(30000);
// communicate
string url = "https://data.mtgox.com/api/1/BTCUSD/trades?since=" + tid.ToString();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string json = reader.ReadToEnd();
reader.Close();
reader.Dispose();
response.Close();
return json;
}
i'm starting at tid=0 (trade id) to get the data (from the very beginning). for each request, i receive a response containing 1000 trade details. i always send the trade id from the previous response for the next request. it works fine for exactly 4 requests & responses. but after that, the following line throws a "System.Net.WebException", saying that "The operation has timed out":
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
here are the facts:
catching the exception and retying keeps causing the same exception
the default HttpWebRequest .Timeout and .ReadWriteTimeout are already high enough (over a minute)
changing HttpWebRequest.KeepAlive to false didn't solve anything either
it seems to always work in the browser even while the function is failing
it has no problems retrieveing the response from https://www.google.com
the amount of successful responses before the exceptions varies from day to day (but browser always works)
starting at the trade id that failed last time causes the exception immediately
calling this function from the main thread instead still caused the exception
running it on a different machine didn't work
running it from a different IP didn't work
increasing Thread.Sleep inbetween requests does not help
any ideas of what could be wrong?

I had the very same issue.
For me the fix was as simple as wrapping the HttpWebResponse code in using block.
using (HttpWebResponse response = (HttpWebResponse) request.GetResponse())
{
// Do your processings here....
}
Details: This issue usually happens when several requests are made to the same host, and WebResponse is not disposed properly. That is where using block will properly dispose the WebResponse object properly and thus solving the issue.

There are two kind of timeouts. Client timeout and server timeout. Have you tried doing something like this:
request.Timeout = Timeout.Infinite;
request.KeepAlive = true;
Try something like this...

I just had similar troubles calling a REST Service on a LINUX Server thru ssl. After trying many different configuration scenarios I found out that I had to send a UserAgent in the http head.
Here is my final method for calling the REST API.
private static string RunWebRequest(string url, string json)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
// Header
request.ContentType = "application/json";
request.Method = "POST";
request.AllowAutoRedirect = false;
request.KeepAlive = false;
request.Timeout = 30000;
request.ReadWriteTimeout = 30000;
request.UserAgent = "test.net";
request.Accept = "application/json";
request.ProtocolVersion = HttpVersion.Version11;
request.Headers.Add("Accept-Language","de_DE");
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls;
ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
byte[] bytes = Encoding.UTF8.GetBytes(json);
request.ContentLength = bytes.Length;
using (var writer = request.GetRequestStream())
{
writer.Write(bytes, 0, bytes.Length);
writer.Flush();
writer.Close();
}
var httpResponse = (HttpWebResponse)request.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var jsonReturn = streamReader.ReadToEnd();
return jsonReturn;
}
}

This is not a solution, but just an alternative:
These days i almost only use WebClient instead of HttpWebRequest. Especially WebClient.UploadString for POST and PUT and WebClient.DownloadString. These simply take and return strings. This way i don't have to deal with streams objects, except when i get a WebException. i can also set the content type with WebClient.Headers["Content-type"] if necessary. The using statement also makes life easier by calling Dispose for me.
Rarely for performance, i set System.Net.ServicePointManager.DefaultConnectionLimit high and instead use HttpClient with it's Async methods for simultaneous calls.
This is how i would do it now
string GetTradesOnline(Int64 tid)
{
using (var wc = new WebClient())
{
return wc.DownloadString("https://data.mtgox.com/api/1/BTCUSD/trades?since=" + tid.ToString());
}
}
2 more POST examples
// POST
string SubmitData(string data)
{
string response;
using (var wc = new WebClient())
{
wc.Headers["Content-type"] = "text/plain";
response = wc.UploadString("https://data.mtgox.com/api/1/BTCUSD/trades", "POST", data);
}
return response;
}
// POST: easily url encode multiple parameters
string SubmitForm(string project, string subject, string sender, string message)
{
// url encoded query
NameValueCollection query = HttpUtility.ParseQueryString(string.Empty);
query.Add("project", project);
query.Add("subject", subject);
// url encoded data
NameValueCollection data = HttpUtility.ParseQueryString(string.Empty);
data.Add("sender", sender);
data.Add("message", message);
string response;
using (var wc = new WebClient())
{
wc.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
response = wc.UploadString( "https://data.mtgox.com/api/1/BTCUSD/trades?"+query.ToString()
, WebRequestMethods.Http.Post
, data.ToString()
);
}
return response;
}
Error handling
try
{
Console.WriteLine(GetTradesOnline(0));
string data = File.ReadAllText(#"C:\mydata.txt");
Console.WriteLine(SubmitData(data));
Console.WriteLine(SubmitForm("The Big Project", "Progress", "John Smith", "almost done"));
}
catch (WebException ex)
{
string msg;
if (ex.Response != null)
{
// read response HTTP body
using (var sr = new StreamReader(ex.Response.GetResponseStream())) msg = sr.ReadToEnd();
}
else
{
msg = ex.Message;
}
Log(msg);
}

For what it's worth, I was experiencing the same issues with timeouts every time I used it, even though calls went through to the server I was calling. The problem in my case was that I had Expect set to application/json, when that wasn't what the server was returning.

500 internal server error at GetResponse()

I have a heavy traffic aspx page calling a web service upon every user`s request as follows.
string uri = "Path.asmx";
string soap = "soap xml string";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.Headers.Add("SOAPAction", "\"http://xxxxxx"");
request.ContentType = "text/xml;charset=\"utf-8\"";
request.Accept = "text/xml";
request.Method = "POST";
using (Stream stm = request.GetRequestStream())
{
using (StreamWriter stmw = new StreamWriter(stm))
{
stmw.Write(soap);
}
}
WebResponse response = request.GetResponse();
response.close();
Everything is working fine but sometimes I am getting the following error.
The remote server returned an error: (500) Internal Server Error.
at System.Net.HttpWebRequest.GetResponse()
Does anybody have any idea about this error or can anybody tell me if I am doing wrong.

For me this error occurred because I had 2 web API actions that had the exact same signatures and both had the same verbs, HttpPost, what I did was change one of the verbs (the one used for updating) to PUT and the error was removed. The following in my catch statement helped in getting to the root of the problem:
catch (WebException webex)
{
WebResponse errResp = webex.Response;
using (Stream respStream = errResp.GetResponseStream())
{
StreamReader reader = new StreamReader(respStream);
string text = reader.ReadToEnd();
}
}

From that error, I would say that your code is fine, at least the part that calls the webservice. The error seems to be in the actual web service.
To get the error from the web server, add a try catch and catch a WebException. A WebException has a property called Response which is a HttpResponse. you can then log anything that is returned, and upload you code. Check back later in the logs and see what is actually being returned.

Finally I get rid of internal server error message with the following code. Not sure if there is another way to achieve it.
string uri = "Path.asmx";
string soap = "soap xml string";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.Headers.Add("SOAPAction", "\"http://xxxxxx"");
request.ContentType = "text/xml;charset=\"utf-8\"";
request.Accept = "text/xml";
request.Method = "POST";
using (Stream stm = request.GetRequestStream())
{
using (StreamWriter stmw = new StreamWriter(stm))
{
stmw.Write(soap);
}
}
using (WebResponse webResponse = request.GetResponse())
{
}

Have you tried to specify UserAgent for your request? For example:
request.UserAgent = "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)";

looking at your error message first of all I would suggest you to recompile your whole application, make sure all the required dlls are there in bin folder when you recompile it.

In my case my request object inherited from base object. Without knowingly I added a property with int? in my request object and my base object also has same property ( same name ) with int datatype. I noticed this and deleted the property which I added in request object and after that it worked fine.

For me the error was misleading. I discovered the true error by testing the errant web service with SoapUI.

In my case I just remove the SoapAction instruction from the HttpWebRequest object. So, I don't define .Headers.Add("SOAPAction","someurl") in HttpWebRequest definitions and my code works fine.
ResultXML is an XDocument.
ResultString is a string.
try
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(Url);
//req.Headers.Add("SOAPAction", "http://tempuri.org/IWebService/GetMessage");
req.ProtocolVersion = HttpVersion.Version11;
req.ContentType = "text/xml;charset=\"utf-8\"";
req.Accept = "text/xml";
req.KeepAlive = true;
req.Method = "POST";
using (Stream stm = req.GetRequestStream())
{
using (StreamWriter stmw = new StreamWriter(stm))
stmw.Write(soapStr);
}
using (StreamReader responseReader = new StreamReader(req.GetResponse().GetResponseStream()))
{
string result = responseReader.ReadToEnd();
ResultXML = XDocument.Parse(result);
ResultString = result;
}
}