We Keep Coding

Debugging an HttpWebResponse

This is being done on a Windows Forms App. I've spent a ton of time stepping through this code with the debugger. What I've found are the following things and they all seem to be at this line:
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
1. If I include
request.SendChunked = true;
I get this error at the response line previously stated:
'System.Net.WebException: The remote server returned an error: (415) Unsupported Media Type.
2. If I comment out the code in #1, I receive this error at that main response line that I mentioned in the beginning:
'System.Net.WebException: The underlying connection was closed: The connection was closed unexpectedly.
3. If I go with route #1, the "Connection" of the request remains as "KeepAlive" all the way through. But if I go with route #2, the "Connection" of the request changes to "null" at the response line that I mentioned in the beginning.
private void HttpPost()
{
HttpWebRequest request = null;
Uri uri = new Uri("https://post.craigslist.org/bulk-rss/post");
request = (HttpWebRequest)WebRequest.Create(uri);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
XmlDocument doc = new XmlDocument();
doc.Load("XMLFile1.xml");
//request.ContentLength = doc.InnerXml.Length;
request.SendChunked = true;
using (Stream writeStream = request.GetRequestStream())
{
UTF8Encoding encoding = new UTF8Encoding();
byte[] bytes = encoding.GetBytes(doc.InnerXml);
//request.ContentLength = bytes.Length;
writeStream.Write(bytes, 0, bytes.Length);
}
string result = string.Empty;
request.ProtocolVersion = System.Net.HttpVersion.Version11;
request.KeepAlive = false;
try
{
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream responseStream = response.GetResponseStream())
{
using (System.IO.StreamReader readStream = new System.IO.StreamReader(responseStream, Encoding.UTF8))
{
result = readStream.ReadToEnd();
}
}
}
}
catch (Exception e)
{
string innerException = String.Format("Inner exception: '{0}'", e.Data);
string exceptionCause = String.Format("An error occurred: '{0}'", e);
System.IO.File.WriteAllText(#"C:\Users\Nathan\Documents\DebugOutputFile\exception.txt", exceptionCause);
System.IO.File.WriteAllText(#"C:\Users\Nathan\Documents\DebugOutputFile\innerException.txt", innerException);
}
}
I feel like these things are adding up towards a solution, but I could really use some guidance.

Option 1: Change your content type to match the body encoding
request.ContentType = "application/xml";
Option 2: Change your body encoding to match the specified content-type
If your server expects only "application/x-www-form-urlencoded", then you need to change your body encoding to suit it, for example, like this:
using (Stream writeStream = request.GetRequestStream())
{
UTF8Encoding encoding = new UTF8Encoding();
string response = String.Concat("arg=", HttpUtility.UrlEncode(doc.InnerXml))
byte[] bytes = encoding.GetBytes(doc.InnerXml);
//request.ContentLength = bytes.Length;
writeStream.Write(bytes, 0, bytes.Length);
}
You need to know the parameter name (above was set to "arg") and add a reference to System.Web, if you haven't.
See following XML...
<?xml version="1.0" encoding="UTF-8"?><test></test>
and encoded string for reference (your request body should look similar to this):
arg=%3C%3Fxml%20version%3D%221.0%22%20encoding%3D%22UTF-8%22%3F%3E%3Ctest%3E%3C%2Ftest%3E
Explanation
If you look at the response you are getting with the first approach: 415 - Unsupported Media Type, you can notice that the content type you are specifying ("application/x-www-form-urlencoded") doesn't match what you are sending in the body (an XML document). Chunk encoding should be enabled when sending files.
Note
When you are having trouble with a request done in source code, try to test the request alone with a web debugging tool, like Fiddler. There you would compose and issue the request until you get the response you want. Then you can compare that with what you are sending from source code (again you should use the same tool for inspecting your request).

How to handle and get the body from POST using .NET C#?

So, I know how to get form data out by using Request.Form["abc"] however how would I go by getting the body out?
I've used this snippet in the below link:
https://gist.github.com/leggetter/769688
But, I'm not sure what to pass in as the Response.
In PHP to do this: file_get_contents('php://input'); and it's as simple as that.
Notes: The content type of the POST is application/json and the body contains the json string.
Thanks in advance.

If I understood your question correctly, here's what you could do when posting to a resource for which you expect a JSON response:
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create("http://foo.com/bar/");
httpWebRequest.Method = WebRequestMethods.Http.Post;
httpWebRequest.Accept = "application/json";
HttpWebResponse response = (HttpWebResponse)httpWebRequest.GetResponse();
Stream receiveStream = response.GetResponseStream();
StreamReader readStream = new StreamReader (receiveStream, Encoding.UTF8);
//store json in a variable for later use
string json = readStream.ReadToEnd();
//make sure to close
response.Close();
readStream.Close();
Of course, this approach is synchronous; but, depending on your requirements, this might be just fine.
Since your question didn't specify whether you need to know how to parse the JSON itself, so I've left out an example of JSON parsing.

You'll need code similar to this to read the raw request body into a string variable.
using (StreamReader reader = new StreamReader(HttpContext.Current.Request.InputStream))
{
string text = reader.ReadToEnd();
}

Xml response file: Received in browser, not via C#

I'm trying to access the last.fm APIs via C#. As a first test I'm querying similar artists if that matters.
I get an XML response when I pass a correct artist name, i.e. "Nirvana". My problem is when I deliver an invalid name (i.e. "Nirvana23") I don't receive XML but an error code (403 or 400) and a WebException.
Interesting thing: If I enter the URL inside a browser (tested with Firefox and Chrome) I receive the XML file I want (containing a lastfm specific error message).
I tried both XmlReader and XDocument:
XDocument doc = XDocument.Load(requestUrl);
and HttpWebRequest:
string httpResponse = "";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestUrl);
HttpWebResponse response = null;
StreamReader reader = null;
try
{
response = (HttpWebResponse)request.GetResponse();
reader = new StreamReader(response.GetResponseStream());
httpResponse = reader.ReadToEnd();
}
The URL is something like "http://ws.audioscrobbler.com/2.0/?method=artist.getsimilar&artist=Nirvana23" (and a specific key given by lastfm, but even without it - it should return XML). A link to give it a try: link (this is the error file I cannot access via C#).
What I also tried (without success): comparing the request by both the browser and my program with the help of WireShark. Then I added some headers to the request, but that didn't help either.

In .NET the WebRequest is converting HTTP error codes into exceptions, while your browser is just ignoring them since the response is not empty. If you catch the exception then the GetResponseStream method should still return the error XML that you are expecting.
Edit:
Try this:
string httpResponse = "";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestUrl);
WebResponse response = null;
StreamReader reader = null;
try
{
response = request.GetResponse();
}
catch (WebException ex)
{
response = ex.Response;
}
reader = new StreamReader(response.GetResponseStream());
httpResponse = reader.ReadToEnd();

Why don't you catch the exception and then process that accordingly. If you want to display any custom error, you can do that also in your catch block.

Deserialize problem for timestamp in ASP.NET

We have webpage in our project where website post the XML data to.
string data = string.Concat("XMLParameter=", SampleXML, "&", "AccessCode=", "XYZ");
if (uri.Scheme == Uri.UriSchemeHttp)
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(uri);
request.Method = WebRequestMethods.Http.Post;
request.ContentLength = data.Length;
request.ContentType = "application/x-www-form-urlencoded";
StreamWriter writer = new StreamWriter(request.GetRequestStream());
writer.Write(data);
writer.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream()); string tmp = reader.ReadToEnd();
response.Close();
Response.Write(tmp);
}
When we pass following XML with post,
<notification timestamp="2009-09-11T11:51:07+02:00">
<reservation creation_date="2010-09-10T12:03:13">
</reservation>
</notification>
On the receiving end in our web page, timestamp we receive do not contain +. Instead of +, it gives us Space charactor. So, while deserializing it, we got error.
We get the data in page using Request.Form["XMLParameter"]
Any solution ???

You have a funny service where the XML data is posted as if it were coming from an HTML form with a text field containing the XML data.
As a consequence, the web server expects your data to be URL encoded. You even tell in the content type that it's encoded. But it isn't. You don't encode it anywhere in your code.
So if you want to stick with your funny service, then you need to run your XML data through HttpUtility.UrlEncode first.

WebRequest to connect to the Wikipedia API

This may be a pathetically simple problem, but I cannot seem to format the post webrequest/response to get data from the Wikipedia API. I have posted my code below if anyone can help me see my problem.
string pgTitle = txtPageTitle.Text;
Uri address = new Uri("http://en.wikipedia.org/w/api.php");
HttpWebRequest request = WebRequest.Create(address) as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
string action = "query";
string query = pgTitle;
StringBuilder data = new StringBuilder();
data.Append("action=" + HttpUtility.UrlEncode(action));
data.Append("&query=" + HttpUtility.UrlEncode(query));
byte[] byteData = UTF8Encoding.UTF8.GetBytes(data.ToString());
request.ContentLength = byteData.Length;
using (Stream postStream = request.GetRequestStream())
{
postStream.Write(byteData, 0, byteData.Length);
}
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
// Get the response stream.
StreamReader reader = new StreamReader(response.GetResponseStream());
divWikiData.InnerText = reader.ReadToEnd();
}

You might want to try a GET request first because it's a little simpler (you will only need to POST for wikipedia login). For example, try to simulate this request:
http://en.wikipedia.org/w/api.php?action=query&prop=images&titles=Main%20Page
Here's the code:
HttpWebRequest myRequest =
(HttpWebRequest)WebRequest.Create("http://en.wikipedia.org/w/api.php?action=query&prop=images&titles=Main%20Page");
using (HttpWebResponse response = (HttpWebResponse)myRequest.GetResponse())
{
string ResponseText;
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
ResponseText = reader.ReadToEnd();
}
}
Edit: The other problem he was experiencing on the POST request was, The exception is : The remote server returned an error: (417) Expectation failed. It can be solved by setting:
System.Net.ServicePointManager.Expect100Continue = false;
(This is from: HTTP POST Returns Error: 417 "Expectation Failed.")

I'm currently in the final stages of implementing an C# MediaWiki API which allows the easy scripting of most MediaWiki viewing and editing actions.
The main API is here: http://o2platform.googlecode.com/svn/trunk/O2%20-%20All%20Active%20Projects/O2_XRules_Database/_Rules/APIs/OwaspAPI.cs and here is an example of the API in use:
var wiki = new O2MediaWikiAPI("http://www.o2platform.com/api.php");
wiki.login(userName, password);
var page = "Test"; // "Main_Page";
wiki.editPage(page,"Test content2");
var rawWikiText = wiki.raw(page);
var htmlText = wiki.html(page);
return rawWikiText.line().line() + htmlText;

You seem to be pushing the input data on HTTP POST, but it seems you should use HTTP GET.
From the MediaWiki API docs:
The API takes its input through
parameters in the query string. Every
module (and every action=query
submodule) has its own set of
parameters, which is listed in the
documentation and in action=help, and
can be retrieved through
action=paraminfo.
http://www.mediawiki.org/wiki/API:Data_formats