We Keep Coding

HTTP 500 error when calling REST service using C#

I'm Getting HTTP 500 error when calling following REST via c#, but I was able to do the POST by using chrome REST Console
Request body sent ( Got from REST console)
Request Url: http://dev-6666.amazonaws.com/CourseEnrollment
Request Method: POST
Status Code: 201
Params: {
"MESSAGE-TYPE": "UserEnrollmentCreate",
"PAYLOAD": "<userstd version=\"\"\"\"><user password=\"\"PSEUDO-5621b55a\"\" lastname=\"\"Wara\"\" emailaddress=\"\"student3#bloody.com\"\" loginid=\"\"BloodyStudent3\"\" firstname=\"\"Buddihi\"\" userid=\"\"19658121\"\" clientid=\"\"117112\"\" createdate=\"\"2014-04-29T03:55:54.873\"\" lastupdate=\"\"2014-05-29T03:55:54.873\"\"><userproperty propertyname=\"\"\"\" propertyvalue=\"\"\"\" createdate=\"\"\"\" lastupdate=\"\"\"\"></userproperty></user></userstd>"
}
C# method used (This is the one getting 500 error)
private void TestCreateUser()
{
string requestData = "{\"MESSAGE-TYPE\":UserEnrollmentCreate,\"PAYLOAD\":\"<userstd version=\"\"\"\"><user password=\"\"PSEUDO-5621b55a\"\" lastname=\"\"Wara\"\" emailaddress=\"\"student3#bloody.com\"\" loginid=\"\"BloodyStudent3\"\" firstname=\"\"Buddihi\"\" userid=\"\"19658121\"\" clientid=\"\"117112\"\" createdate=\"\"2014-04-29T03:55:54.873\"\" lastupdate=\"\"2014-05-29T03:55:54.873\"\"><userproperty propertyname=\"\"\"\" propertyvalue=\"\"\"\" createdate=\"\"\"\" lastupdate=\"\"\"\"></userproperty></user></userstd>}";
byte[] data = Encoding.UTF8.GetBytes(requestData);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://dev-6666.amazonaws.com/CourseEnrollment");
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
Stream dataStream = request.GetRequestStream();
dataStream.Write(data, 0, data.Length);
dataStream.Close();
WebResponse response = request.GetResponse();
string result = new StreamReader(response.GetResponseStream()).ReadToEnd();
}

form-url-encoded data should be in form (variablename)=(data)&(variablename)=(data)....
The console representation is a bit messy, so the real data string should be
string requestData = "MESSAGE-TYPE=UserEnrollmentCreate&PAYLOAD=<userstd version=\"\"\"\"><user password=\"\"PSEUDO-5621b55a\"\" lastname=\"\"Wara\"\" emailaddress=\"\"student3#bloody.com\"\" loginid=\"\"BloodyStudent3\"\" firstname=\"\"Buddihi\"\" userid=\"\"19658121\"\" clientid=\"\"117112\"\" createdate=\"\"2014-04-29T03:55:54.873\"\" lastupdate=\"\"2014-05-29T03:55:54.873\"\"><userproperty propertyname=\"\"\"\" propertyvalue=\"\"\"\" createdate=\"\"\"\" lastupdate=\"\"\"\"></userproperty></user></userstd>";

The response should have been 400 in a working implementation as you specify application/x-www-form-urlencoded while you post application/json as content.
WebApi probably returns 500 as it tried to decode the content as a form while the content actually is JSON, hence it get's an exception and therefore return 500 as status code.

HttpWebRequest.GetResponse() keeps getting timed out

i wrote a simple C# function to retrieve trade history from MtGox with following API call:
https://data.mtgox.com/api/1/BTCUSD/trades?since=<trade_id>
documented here: https://en.bitcoin.it/wiki/MtGox/API/HTTP/v1#Multi_currency_trades
here's the function:
string GetTradesOnline(Int64 tid)
{
Thread.Sleep(30000);
// communicate
string url = "https://data.mtgox.com/api/1/BTCUSD/trades?since=" + tid.ToString();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string json = reader.ReadToEnd();
reader.Close();
reader.Dispose();
response.Close();
return json;
}
i'm starting at tid=0 (trade id) to get the data (from the very beginning). for each request, i receive a response containing 1000 trade details. i always send the trade id from the previous response for the next request. it works fine for exactly 4 requests & responses. but after that, the following line throws a "System.Net.WebException", saying that "The operation has timed out":
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
here are the facts:
catching the exception and retying keeps causing the same exception
the default HttpWebRequest .Timeout and .ReadWriteTimeout are already high enough (over a minute)
changing HttpWebRequest.KeepAlive to false didn't solve anything either
it seems to always work in the browser even while the function is failing
it has no problems retrieveing the response from https://www.google.com
the amount of successful responses before the exceptions varies from day to day (but browser always works)
starting at the trade id that failed last time causes the exception immediately
calling this function from the main thread instead still caused the exception
running it on a different machine didn't work
running it from a different IP didn't work
increasing Thread.Sleep inbetween requests does not help
any ideas of what could be wrong?

I had the very same issue.
For me the fix was as simple as wrapping the HttpWebResponse code in using block.
using (HttpWebResponse response = (HttpWebResponse) request.GetResponse())
{
// Do your processings here....
}
Details: This issue usually happens when several requests are made to the same host, and WebResponse is not disposed properly. That is where using block will properly dispose the WebResponse object properly and thus solving the issue.

There are two kind of timeouts. Client timeout and server timeout. Have you tried doing something like this:
request.Timeout = Timeout.Infinite;
request.KeepAlive = true;
Try something like this...

I just had similar troubles calling a REST Service on a LINUX Server thru ssl. After trying many different configuration scenarios I found out that I had to send a UserAgent in the http head.
Here is my final method for calling the REST API.
private static string RunWebRequest(string url, string json)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
// Header
request.ContentType = "application/json";
request.Method = "POST";
request.AllowAutoRedirect = false;
request.KeepAlive = false;
request.Timeout = 30000;
request.ReadWriteTimeout = 30000;
request.UserAgent = "test.net";
request.Accept = "application/json";
request.ProtocolVersion = HttpVersion.Version11;
request.Headers.Add("Accept-Language","de_DE");
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls;
ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
byte[] bytes = Encoding.UTF8.GetBytes(json);
request.ContentLength = bytes.Length;
using (var writer = request.GetRequestStream())
{
writer.Write(bytes, 0, bytes.Length);
writer.Flush();
writer.Close();
}
var httpResponse = (HttpWebResponse)request.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var jsonReturn = streamReader.ReadToEnd();
return jsonReturn;
}
}

This is not a solution, but just an alternative:
These days i almost only use WebClient instead of HttpWebRequest. Especially WebClient.UploadString for POST and PUT and WebClient.DownloadString. These simply take and return strings. This way i don't have to deal with streams objects, except when i get a WebException. i can also set the content type with WebClient.Headers["Content-type"] if necessary. The using statement also makes life easier by calling Dispose for me.
Rarely for performance, i set System.Net.ServicePointManager.DefaultConnectionLimit high and instead use HttpClient with it's Async methods for simultaneous calls.
This is how i would do it now
string GetTradesOnline(Int64 tid)
{
using (var wc = new WebClient())
{
return wc.DownloadString("https://data.mtgox.com/api/1/BTCUSD/trades?since=" + tid.ToString());
}
}
2 more POST examples
// POST
string SubmitData(string data)
{
string response;
using (var wc = new WebClient())
{
wc.Headers["Content-type"] = "text/plain";
response = wc.UploadString("https://data.mtgox.com/api/1/BTCUSD/trades", "POST", data);
}
return response;
}
// POST: easily url encode multiple parameters
string SubmitForm(string project, string subject, string sender, string message)
{
// url encoded query
NameValueCollection query = HttpUtility.ParseQueryString(string.Empty);
query.Add("project", project);
query.Add("subject", subject);
// url encoded data
NameValueCollection data = HttpUtility.ParseQueryString(string.Empty);
data.Add("sender", sender);
data.Add("message", message);
string response;
using (var wc = new WebClient())
{
wc.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
response = wc.UploadString( "https://data.mtgox.com/api/1/BTCUSD/trades?"+query.ToString()
, WebRequestMethods.Http.Post
, data.ToString()
);
}
return response;
}
Error handling
try
{
Console.WriteLine(GetTradesOnline(0));
string data = File.ReadAllText(#"C:\mydata.txt");
Console.WriteLine(SubmitData(data));
Console.WriteLine(SubmitForm("The Big Project", "Progress", "John Smith", "almost done"));
}
catch (WebException ex)
{
string msg;
if (ex.Response != null)
{
// read response HTTP body
using (var sr = new StreamReader(ex.Response.GetResponseStream())) msg = sr.ReadToEnd();
}
else
{
msg = ex.Message;
}
Log(msg);
}

For what it's worth, I was experiencing the same issues with timeouts every time I used it, even though calls went through to the server I was calling. The problem in my case was that I had Expect set to application/json, when that wasn't what the server was returning.

.NET service responds 500 internal error and "missing parameter" to HttpWebRequest POSTS but test form works fine

I am using a simple .NET service (asmx) that works fine when invoking via the test form (POST). When invoking via a HttpWebRequest object, I get a WebException "System.Net.WebException: The remote server returned an error: (500) Internal Server Error." Digging deeper, reading the WebException.Response.GetResponseStream() I get the message: "Missing parameter: serviceType." but I've clearly included this parameter.
I'm at a loss here, and its worse that I don't have access to debug the service itself.
Here is the code being used to make the request:
string postData = String.Format("serviceType={0}&SaleID={1}&Zip={2}", request.service, request.saleId, request.postalCode);
byte[] data = (new ASCIIEncoding()).GetBytes(postData);
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
httpWebRequest.Timeout = 60000;
httpWebRequest.Method = "POST";
httpWebRequest.ContentType = "application/x-www-form-urlencoded";
httpWebRequest.ContentLength = data.Length;
using (Stream newStream = httpWebRequest.GetRequestStream())
{
newStream.Write(data, 0, data.Length);
}
try
{
using (response = (HttpWebResponse)httpWebRequest.GetResponse())
{
if (response.StatusCode != HttpStatusCode.OK)
throw new Exception("There was an error with the shipping freight service.");
string responseData;
using (StreamReader responseStream = new StreamReader(httpWebRequest.GetResponse().GetResponseStream(), System.Text.Encoding.GetEncoding("iso-8859-1")))
{
responseData = responseStream.ReadToEnd();
responseStream.Close();
}
if (string.IsNullOrEmpty(responseData))
throw new Exception("There was an error with the shipping freight service. Request went through but response is empty.");
XmlDocument providerResponse = new XmlDocument();
providerResponse.LoadXml(responseData);
return providerResponse;
}
}
catch (WebException webExp)
{
string exMessage = webExp.Message;
if (webExp.Response != null)
{
using (StreamReader responseReader = new StreamReader(webExp.Response.GetResponseStream()))
{
exMessage = responseReader.ReadToEnd();
}
}
throw new Exception(exMessage);
}
Anyone have an idea what could be happening?
Thanks.
UPDATE
Stepping through the debugger, I see the parameters are correct. I also see the parameters are correct in fiddler.
Examining fiddler, I get 2 requests each time this code executes. The first request is a post that sends the parameters. It gets a 301 response code with a "Document Moved Object Moved This document may be found here" message. The second request is a GET to the same URL with no body. It gets a 500 server error with "Missing parameter: serviceType." message.

It seems like you found your problem when you looked at the requests in Fiddler. Taking an excerpt from http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html:
10.3.2 301 Moved Permanently
The requested resource has been assigned a new permanent URI and any future references to this resource SHOULD use one of the returned URIs. Clients with link editing capabilities ought to automatically re-link references to the Request-URI to one or more of the new references returned by the server, where possible.
.....
Note: When automatically redirecting a POST request after
receiving a 301 status code, some existing HTTP/1.0 user agents
will erroneously change it into a GET request.
Here's a couple options that you can take:
Hard-code your program to use the new Url that you see in the 301 response in Fiddler
Adjust your code to retrieve the 301 response, parse out the new Url from the response, and build a new response with the new Url.
The latter option would be ideal if you're dealing with user-based input on the Url (like a web browser), since you don't know where the user is going to want your program to go.

"Request timed out" when makinng two HttpWebRequests to the same Java server from C# over .NET Compact

I have a client application running on C# on a .NETCF 3.5 device POSTing to a Java servlet located remotely. I am receiving a "Request Timed out" during my third HTTP POST to the same servlet. For example, if the servlet manages login to our Java server, the first two login attempts from the client would go through (same client device) and when I attempt the third one, it will return in a "Request timed out" exception from the server. I have noticed this happen always and I can't figure out the problem. I read that C# by default sends the Request 100 continue in the HTTP headers so I used the ServicePointManager to set the request 100 to false to no avail.
Here is the code that is throwing this error:
serverUrl = url;
string responseFromServer = "";
try
{
System.Net.ServicePointManager.Expect100Continue = false;
int tmp = ServicePointManager.DefaultConnectionLimit;
// Create a request using a URL that can receive a post.
request = (HttpWebRequest)WebRequest.Create(url);
// Set the Method property of the request to POST.
request.Method = "POST";
// Create POST data and convert it to a byte array.
byte[] byteArray = System.Text.Encoding.UTF8.GetBytes(url);
// Set the ContentType property of the WebRequest.
request.ContentType = "application/x-www-form-urlencoded";
// Set the ContentLength property of the WebRequest.
request.ContentLength = byteArray.Length;
request.Timeout = (50 * 100);
request.Proxy = System.Net.GlobalProxySelection.GetEmptyWebProxy();
// Get the request stream.
Stream dataStream = request.GetRequestStream();
// Write the data to the request stream.
dataStream.Write(byteArray, 0, byteArray.Length);
// Close the Stream object.
dataStream.Close();
// Get the response.
response = (HttpWebResponse)request.GetResponse();
// Get the stream containing content returned by the server.
dataStream = response.GetResponseStream();
// Open the stream using a StreamReader for easy access.
StreamReader reader = new StreamReader(dataStream);
// Read the content.
responseFromServer = reader.ReadToEnd();
// Clean up the streams.
reader.Close();
dataStream.Close();
response.Close();
return responseFromServer;
}
catch (Exception WebExp)
{
Logging.Instance.Log(Logging.Levels.Error, "Error in DoPost while retrieving : "+url+ " " + WebExp.Message.ToString());
Logging.Instance.Log(Logging.Levels.Error, WebExp.StackTrace.ToString());
throw WebExp;
}
Any help would be much appreciated. Thanks!

This behaviour is due to wrong Exception handling regarding to the WebResponse. You have always to handle the response and close it. Otherwise the third attempt of HTTP Webrequest will fail with timeout which is limited by WinCE.
Following source code will be safe:
HttpWebResponse response = null;
try
{
//...
response = (HttpWebResponse)request.GetResponse();
//...
}
catch(Exception e)
{
// logging, etc.
throw e;
}
finally
{
if(response!=null)
{
response.Close();
}
}

getting this error: "The remote server returned an error: (422) Unprocessable Entity." when doing post from C# to RoR

This code is for an outlook plugin. We're trying to POST to a page and are getting this error:
The remote server returned an error: (422) Unprocessable Entity.
The C# code is here:
webClient.Headers.Add("Content-Type", "application/x-www-form-urlencoded");
ASCIIEncoding asciiEncoding = new System.Text.ASCIIEncoding();
Byte[] postData = asciiEncoding.GetBytes("email=e2#email.com&password=hunter2");
char[] resultHTML = asciiEncoding.GetChars(webClient.UploadData("http://url", "POST", postData));
string convertedResultHTML = new string(resultHTML);
Any idea what could be causing this?

POST data must be encoded prior to be sent out on the wire as ASCII, if you are sending character not in the ASCII range. You should try something like:
Byte[] postData = asciiEncoding.GetBytes(HttpUtility.UrlEncode("email=e2#email.com&password=hunter2"));

Because of its limited functionality, I avoid using WebClient and use WebRequest instead. The code below:
does not expect an HTTP 100 status code to be returned,
creates a CookieContainer to store any cookies we pick up,
sets the Content Length header, and
UrlEncodes each value in the post data.
Give the following a try and see if it works for you.
System.Net.ServicePointManager.Expect100Continue = false;
System.Net.CookieContainer cookies = new System.Net.CookieContainer();
// this first request just ensures we have a session cookie, if one exists
System.Net.WebRequest req = System.Net.WebRequest.Create("http://localhost/test.aspx");
((System.Net.HttpWebRequest)req).CookieContainer = cookies;
req.GetResponse().Close();
// this request submits the data to the server
req = System.Net.WebRequest.Create("http://localhost/test.aspx");
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
((System.Net.HttpWebRequest)req).CookieContainer = cookies;
string parms = string.Format("email={0}&password={1}",
System.Web.HttpUtility.UrlEncode("e2#email.com"),
System.Web.HttpUtility.UrlEncode("hunter2"));
byte[] bytes = System.Text.Encoding.ASCII.GetBytes(parms);
req.ContentLength = bytes.Length;
// perform the POST
using (System.IO.Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
// read the response
string response;
using (System.Net.WebResponse resp = req.GetResponse())
{
if (resp == null) return;
using (System.IO.StreamReader sr = new System.IO.StreamReader(resp.GetResponseStream()))
{
response = sr.ReadToEnd().Trim();
}
}
// the variable response holds the results of the request...
Credits: Hanselman, Simon (SO Question)

This is the RoR application telling you that you have not formed a request that it can handle; the destination script exists (otherwise you'd see a 404), the request is being handled (otherwise you'd get a 400 error) and it's been encoded correctly (or you'd get a 415 error) but the actual instruction can't be carried out.
Looking at it, you seem to be loading some email information. The RoR application could be telling you that the username and password is wrong, or that the user doesn't exist, or something else. It's up to the RoR application itself.
I think the code itself is good; it's just that the app at the other end isn't happy about doing what you ask it. Are you missing something else in the request information, like a command? (eg command=getnetemails&email=e2#email.com&password=hunter2) Are you sure the email/password combination you are passing is good?
see here for more on the 422 error.

Add the below line above your code.
System.Net.ServicePointManager.Expect100Continue = false;
Are you trying to access an authentication required page?

it was solved by returning xml instead of just unstructured text on the RoR side