HttpWebResponse not getting full response if Content-Length is 67722 - c#

In my code, I am doing below
HttpWebResponse response = (HttpWebResponse)httpWebRequest.GetResponse();
Stream dataStream = response.GetResponseStream();
string responseFromServer = reader.ReadToEnd();
reader.Close();
dataStream.Close();
response.Close();
httpWebRequest is a POST request whose response size in Fiddler after one redirection is 67722. response.ContentLength field returned in my case is -1 and total size of responseFromServer is approximately 32695. It simply cuts the remaining data from response. I have seen somewhere to use maxReceivedMessageSize but not able to figure our how to do it.
responseFromServer.Length 32695
response.ContentLength -1
response.GetResponseHeader("transfer-encoding") ""
UPDATE: FINAL
Response was coming right. Issue was with my code where parameters being sent into POST query were not encoded properly through UrlEncoding. If we are dealing with ASP.net contentpanel control, point to remember is that it sends the POST query with control names having special character '$'. We need to encode that using HttpUtility.UrlEncoding method

As mentioned by spender, check the response headers..the value for 'ContentLength' is supplied in the response headers, and may NOT be the actual length of data; if transfer-encoding is 'chunked', it is "normal" to mark contentlength as -1.
I'm not familiar with fiddler..but perhaps it is fetching things that the webresponse is not, like external scripts? Try loading the page in a normal webbrowser, and 'View Source' --> use some tool (A textbox..) to count the characters in the source: is it actually 67k or closer to 32? Similarly, check the last part of data in 'responseFromServer'..is it the same as when 'Viewing page source'? ie. both are probably (hopefully?) < / h t m l >

Related

The request body did not contain the specified number of bytes

I am calling an API from by C# Windows service. In some cases the following error is being raised.
The request body did not contain the specified number of bytes. Got 101,379, expected 102,044
In the RAW Request captured using fiddler content length as specified.
Content-Length: 102044
In the response from the API I am receiving the following message.
The request body did not contain the specified number of bytes. Got 101,379, expected 102,044
The strange thing for me is that it does not happen for each and every request, it is generated randomly and different points. Code which I am using to get the content length is specified below.
var data = Encoding.ASCII.GetBytes(requestBody); // requestBody is the JSON String
webReqeust.ContentLength = data.Length;
Is it mandatory to provide content length in REST API calls ?
Edit 1:
This is what my sample code looks like for web request
webReqeust = (HttpWebRequest)WebRequest.Create(string.Format("{0}{1}", requestURI, queryString));
webReqeust.Method = RequestMethod.ToString();
webReqeust.Headers.Add("Authorization", string.Format("{0} {1}", token_type, access_token));
webReqeust.Method = RequestMethod.ToString();
webReqeust.ContentType = "application/json";
var data = Encoding.ASCII.GetBytes(requestBody);
webReqeust.ContentLength = data.Length;
using (var streamWriter = new StreamWriter(webReqeust.GetRequestStream()))
{
streamWriter.Write(requestBody);
streamWriter.Flush();
streamWriter.Close();
}
I would suggest maybe instead try using HttpClient as done in the linked post from mjwills here. You don't have to use content length, but it sounds like that is being enforced by the API and ultimately you are trying to post too much.
Otherwise the way I see it is that something is making the request body too large. Is it serialized input data which gets encoded into a byte array? If that is what is happening then perhaps the correct length requirements are not being enforced on the data that composes the request body, and I would suggest inspecting what goes on in the composition of the request body object itself.

HttpWebResponse text does not appear to be JSON

I am making a rest call in which I get a HttpWebResponse that contains data. It seems the data is serialized, and I am trying to get the plain text of the request. I have been using the chrome extension Advanced Rest client, which when calling the same request it is able to display the text version of the json response.
From what I have read on here, you are required to deserialize into the expected object. However, it is pretty clear that chrome plugin has no idea about the object type and can still print out plain text.
Is it possible to do the same in c#?
HttpWebRequest request = WebRequest.Create(address) as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/json";
// [code removed for setting json data via stream writer
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
// This is where I am trying to figure out how to get plain readable text out of response.GetResponseStream()
}
Edit: If I simply use a StreamReader to get the text from the response stream, I get a bunch of binary data and not the plain json text.
Edit: realized the problem had to do with compression. This can be closed.
I'm not sure if got it right, but you can get the response as a string doing this:
using (var sr = new StreamReader(response.GetResponseStream()))
{
text = sr.ReadToEnd();
}
Turned out my problem was due to compression. I realized the header contained "Content-Encoding: gzip" so I searched on how to unzip with gzip compression and then the text was proper json. Thanks all

How do I check for binary vs. text in an HttpWebRequest in c#?

Is there a way to determine if the response from an HttpWebRequest in C# contains binary data vs. text? Or is there another class or function I should be using to do this?
Here's some sample code. I'd like to know before reading the StreamReader if the content is not text.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.someurl.com");
request.Method = WebRequestMethods.Http.Get;
using (WebResponse response = request.GetResponse())
{
// check somewhere in here if the response is binary data and ignore it
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
string responseDetails = reader.ReadToEnd().Trim();
}
}
In general, web sites will tell you in the Content-Type header what kind of data they're returning. You can determine that by getting the ContentType property from the response.
But sites have been known to lie. Or not say anything. I've seen both. If there is no Content-Type header or you don't want to trust it, then the only way you can tell what kind of data is there, is by reading it.
But then, if you don't trust the site, why are you reading data from it?

Communicating with an ASP website

I have a know a website that contains an open database of the result of an academic test.
http://nts.org.pk/NTSWeb/PPL_30Sep2012_Result/search.asp
I am expert with C# but newbie to web development.
Usually, using web browser, we can enter and roll number and server sends back the result. E.G. Use my Roll Num: 3912125
What I need to do is, use a C# application to communicate this roll null number and get anything, of my result. (any string is excepted, I will parse out my result from that string.)
How do I send query? when I don't know a list of possible query strings.
I tried this code:
string queryString = "RollNo=3912125";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(#"http://nts.org.pk/NTSWeb/PPL_30Sep2012_Result/search.asp");
request.UseDefaultCredentials = true;
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
byte[] requestBytes = Encoding.UTF8.GetBytes(queryString);
request.ContentLength = requestBytes.Length;
using (var requestStream = request.GetRequestStream())
{
requestStream.Write(requestBytes, 0, requestBytes.Length);
requestStream.Close();
}
WebResponse response = request.GetResponse();
textBox1.AppendText(((HttpWebResponse)response).StatusDescription);
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
string responseFromServer = reader.ReadToEnd();
textBox1.AppendText(responseFromServer);
reader.Close();
dataStream.Close();
response.Close();
You have to append the querystring to the url like this:
string queryString = "RollNo=3912125";
string url = String.Format(#"http://foo/search.asp?{0}", queryString);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
You should take a look at the code in my answer to C# https login and download file. It gives a good demonstration of how to perform a POST request. As far as knowing what's valid to use for the query-formatted string in your POST, it's simply a matter of looking for appropriate input elements in the page content. It looks like what you have (RollNo) is correct. You may, however, need to also add the submit button value to your request as well depending on how the server behaves, giving you something like. RollNo=3912125&submit=submit.
You're most of the way there. Your queryString should look like RollNo=3912125&Submit=+Search+. When you are calling WebRequest.Create, the Url should in fact be http://nts.org.pk/NTSWeb/PPL_30Sep2012_Result/result.asp.
The rest of your code should work, though the answer #JamieSee recommended to you has some very good advice about wrapping things in using blocks correctly

Webservice and encoding

I connect to the web service as follows:
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(
"http://mywebserviceaddress.com/attributes=someatt");
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
using (StreamReader stIn = new StreamReader(
req.GetResponse().GetResponseStream(),Encoding.UTF8))
{
strResponse = stIn.ReadToEnd();
return strResponse;
}
However I get the response with (probably) bad encoding so as a result, on my page i get the following issue:
Am I doing something wrong or is it a third-party web service issue? How can I get the respone without this silly issues?
Here's the screenshot from debugger:
The page is probably not UTF8. It looks like special characters use the upper half of an ASCII character, so it's some kind of old charset. Reading it as UTF8 causes errors because the reader doesn't expect single-byte special characters.
Store the result of GetResponse() into a variable and output the contents of ContentTypeCharacterSet. If the server acts correctly, it shows the used charset in this property. Then, you can use the correct charset in your StreamReader.

Categories