We Keep Coding

Http request ending as bad request due to Encoding and Escape character issues

Note: Code is in F# but tagging C# because it's a general encoding and escape character issue across .net while converting from object to json, please look at the output at the bottom of the page.
Following request is ending up a bad request, please have a look at any discrepancies especially in the Encoding.Default.GetString and then encoding back into System.Text.Encoding.ASCII.GetBytes especially.
Context: An object is serialized and written to the body of the PUT request and is ended up as bad request 400. JSON is serialized correctly and I can see the string with Console.WriteLine
I took the JSON encoding example from F# snippets
let update (schema: Article) : Article =
let url = "http://example.com"
use memoryStream = new MemoryStream()
(new DataContractJsonSerializer(typeof<Article>)).WriteObject(memoryStream, schema)
let d = Encoding.Default.GetString(memoryStream.ToArray()) // I suspect problem may be here
let data : byte[] = System.Text.Encoding.ASCII.GetBytes(d); // or here
let request = WebRequest.Create(url) :?> HttpWebRequest
request.Method <- "PUT"
request.ContentType <- "application/json;charset=UTF-8"
request.Accept <- "application/json;charset=UTF-8"
request.ContentLength <- (int64)data.Length
use requestStream = request.GetRequestStream()
requestStream.Write(data, 0, data.Length)
requestStream.Flush()
requestStream.Close()
let response = request.GetResponse() :?> HttpWebResponse
use reader = new StreamReader(response.GetResponseStream())
use memoryStream = new MemoryStream(ASCIIEncoding.Default.GetBytes(reader.ReadToEnd()))
let result = (new DataContractJsonSerializer(typeof<Article>)).ReadObject(memoryStream) :?> Article
reader.Close()
response.Close()
request.Abort()
result
Further Discoveries
I was able to get Charles Proxy up and I saw that forward slashes have been escaped in the output serialized json to the server.
Actual Output: with escaped slashes \ /
\/publication\/d40a4e4c-d6a3-45ae-98b3-924b31d8712a\/altasset\/c42d0df7-a563-4bb5-b7fa-313e6a98032f\/
Expected output: They are supposed to have forward slashes only, no escaping of the characters.
/publication/d40a4e4c-d6a3-45ae-98b3-924b31d8712a....
I guess something causing the character to be escaped in Encoding process, is there a way to fix it?
Edit
I also noticed that memoryStream.ToArray() returns a byte[] so instead of going through encodings I also tried the following but it didn't make any difference.
let data : byte[] = memoryStream.ToArray()
requestStream.Write(data, 0, data.Length)
in Charles proxy, I'm seeing Broken pipe message.

Add a reference to the Newtonsoft.Json NuGet package and add this to the top of your F# file:
open Newtonsoft.Json
Then try this as the body of your update method (it uses UTF-8 encoding explicitly rather than ASCII or the platform default):
let url = "http://example.com"
let serialized = JsonConvert.SerializeObject(schema)
let data : byte[] = Encoding.UTF8.GetBytes(serialized)
let request = WebRequest.Create(url) :?> HttpWebRequest
request.Method <- "PUT"
request.ContentType <- "application/json;charset=UTF-8"
request.Accept <- "application/json;charset=UTF-8"
request.ContentLength <- (int64)data.Length
use requestStream = request.GetRequestStream()
requestStream.Write(data, 0, data.Length)
requestStream.Flush()
requestStream.Close()
let response = request.GetResponse() :?> HttpWebResponse
use reader = new StreamReader(response.GetResponseStream(), Encoding.UTF8)
let result = JsonConvert.DeserializeObject<Article>(reader.ReadToEnd())
reader.Close()
response.Close()
request.Abort()
result
Unless the server you're using was also using DataContractJsonSerializer (which escapes "/" as "\/") then you're probably better off with Newtonsoft. And it doesn't hurt to fix the UTF-8 encoding issues before they arise (it didn't look like you were sending/receiving any non-ASCII text but that would inevitably happen in the real world).

HttpWebResponse text does not appear to be JSON

I am making a rest call in which I get a HttpWebResponse that contains data. It seems the data is serialized, and I am trying to get the plain text of the request. I have been using the chrome extension Advanced Rest client, which when calling the same request it is able to display the text version of the json response.
From what I have read on here, you are required to deserialize into the expected object. However, it is pretty clear that chrome plugin has no idea about the object type and can still print out plain text.
Is it possible to do the same in c#?
HttpWebRequest request = WebRequest.Create(address) as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/json";
// [code removed for setting json data via stream writer
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
// This is where I am trying to figure out how to get plain readable text out of response.GetResponseStream()
}
Edit: If I simply use a StreamReader to get the text from the response stream, I get a bunch of binary data and not the plain json text.
Edit: realized the problem had to do with compression. This can be closed.

I'm not sure if got it right, but you can get the response as a string doing this:
using (var sr = new StreamReader(response.GetResponseStream()))
{
text = sr.ReadToEnd();
}

Turned out my problem was due to compression. I realized the header contained "Content-Encoding: gzip" so I searched on how to unzip with gzip compression and then the text was proper json. Thanks all

C# StreamReader fail, some garbage symbol showed

Encoding encode = System.Text.Encoding.GetEncoding("utf-8");
Stream receiveStream = response.GetResponseStream();
StreamReader readStream = new StreamReader(response.GetResponseStream(), encode);
string ResponseJSON = readStream.ReadToEnd();
Here is part of my code.
I want create a https request then get the response string.
Actually my code work well when I didnt use proxy.
But when I use a https proxy, the program fail.
Then I checked in the debug mode, I found that the response string was something garbage symbol and characters. And this case did not happen when I did not use proxy.
It is a problem related to encoding ?? But I had used same encoding all the time , why only when I using a proxy the code not work?
Sorry for poor english,if anything unclear, please ask me to clarify.
Thanks alot.

The encoding of an HTTP response is stored in the charset attribute of the Content-Type header. You can access it through the HttpWebResponse.CharacterSet property.
It seems your proxy is changing the character set of the response to something else. You should check both Charset and the ContentType property to find the encoding it uses. Perhaps the proxy changed the character set to something else or it may even have compressed it.
You should check Charset before reading the stream eg:
if (response.CharacterSet!="utf-8")
{
....
}
UPDATE
It seems the response is GZIP-compressed. To automatically decompress the stream you should set the HttpWebRequest.AutomaticDecompression property. This will add the proper headers to the request and automatically decompress the response, eg:
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
var response=request.GetResponse();
var stream=response.GetResponseStream();
...

Communicating with an ASP website

I have a know a website that contains an open database of the result of an academic test.
http://nts.org.pk/NTSWeb/PPL_30Sep2012_Result/search.asp
I am expert with C# but newbie to web development.
Usually, using web browser, we can enter and roll number and server sends back the result. E.G. Use my Roll Num: 3912125
What I need to do is, use a C# application to communicate this roll null number and get anything, of my result. (any string is excepted, I will parse out my result from that string.)
How do I send query? when I don't know a list of possible query strings.
I tried this code:
string queryString = "RollNo=3912125";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(#"http://nts.org.pk/NTSWeb/PPL_30Sep2012_Result/search.asp");
request.UseDefaultCredentials = true;
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
byte[] requestBytes = Encoding.UTF8.GetBytes(queryString);
request.ContentLength = requestBytes.Length;
using (var requestStream = request.GetRequestStream())
{
requestStream.Write(requestBytes, 0, requestBytes.Length);
requestStream.Close();
}
WebResponse response = request.GetResponse();
textBox1.AppendText(((HttpWebResponse)response).StatusDescription);
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
string responseFromServer = reader.ReadToEnd();
textBox1.AppendText(responseFromServer);
reader.Close();
dataStream.Close();
response.Close();

You have to append the querystring to the url like this:
string queryString = "RollNo=3912125";
string url = String.Format(#"http://foo/search.asp?{0}", queryString);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);

You should take a look at the code in my answer to C# https login and download file. It gives a good demonstration of how to perform a POST request. As far as knowing what's valid to use for the query-formatted string in your POST, it's simply a matter of looking for appropriate input elements in the page content. It looks like what you have (RollNo) is correct. You may, however, need to also add the submit button value to your request as well depending on how the server behaves, giving you something like. RollNo=3912125&submit=submit.

You're most of the way there. Your queryString should look like RollNo=3912125&Submit=+Search+. When you are calling WebRequest.Create, the Url should in fact be http://nts.org.pk/NTSWeb/PPL_30Sep2012_Result/result.asp.
The rest of your code should work, though the answer #JamieSee recommended to you has some very good advice about wrapping things in using blocks correctly

HTTPWebRequest Body Formatting

This is a stupidly trivial question, but I can't seem to find a proper example anywhere with more than one property being set. Basically, I'm trying to send a POST request with C#'s HTTPWebRequest library while specifying two different fields in the body of the request.
So far, I have this:
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
request.Method = "POST";
ASCIIEncoding encoding = new ASCIIEncoding();
byte[] byt = encoding.GetBytes("recipient=12345ABC");
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = byt.Length;
Stream newStream = request.GetRequestStream();
newStream.Write(byt, 0, byt.Length);
Followed by the usual GetResponse() stuff. This works fine, everything's dandy, but I can't figure out how to specify multiple body elements, such as both of these:
recipient=12345ABC
body="testmessage"
I've tried separating them with a semicolon, an ampersand, and a comma, but the server keeps returning Error 400: Bad Request. Perhaps I'm just misunderstanding how this process works?

API docs were sloppily done. Actual parameter names were "recipients" and "text" - code worked fine after this change.
URL encoding in the body works fine, "recipients=12345ABC&text=This+is+URL+escaped+text" worked like a charm in either the URL itself or the POST body.