Encoding encode = System.Text.Encoding.GetEncoding("utf-8");
Stream receiveStream = response.GetResponseStream();
StreamReader readStream = new StreamReader(response.GetResponseStream(), encode);
string ResponseJSON = readStream.ReadToEnd();
Here is part of my code.
I want create a https request then get the response string.
Actually my code work well when I didnt use proxy.
But when I use a https proxy, the program fail.
Then I checked in the debug mode, I found that the response string was something garbage symbol and characters. And this case did not happen when I did not use proxy.
It is a problem related to encoding ?? But I had used same encoding all the time , why only when I using a proxy the code not work?
Sorry for poor english,if anything unclear, please ask me to clarify.
Thanks alot.
The encoding of an HTTP response is stored in the charset attribute of the Content-Type header. You can access it through the HttpWebResponse.CharacterSet property.
It seems your proxy is changing the character set of the response to something else. You should check both Charset and the ContentType property to find the encoding it uses. Perhaps the proxy changed the character set to something else or it may even have compressed it.
You should check Charset before reading the stream eg:
if (response.CharacterSet!="utf-8")
{
....
}
UPDATE
It seems the response is GZIP-compressed. To automatically decompress the stream you should set the HttpWebRequest.AutomaticDecompression property. This will add the proper headers to the request and automatically decompress the response, eg:
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
var response=request.GetResponse();
var stream=response.GetResponseStream();
...
Related
I am making a rest call in which I get a HttpWebResponse that contains data. It seems the data is serialized, and I am trying to get the plain text of the request. I have been using the chrome extension Advanced Rest client, which when calling the same request it is able to display the text version of the json response.
From what I have read on here, you are required to deserialize into the expected object. However, it is pretty clear that chrome plugin has no idea about the object type and can still print out plain text.
Is it possible to do the same in c#?
HttpWebRequest request = WebRequest.Create(address) as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/json";
// [code removed for setting json data via stream writer
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
// This is where I am trying to figure out how to get plain readable text out of response.GetResponseStream()
}
Edit: If I simply use a StreamReader to get the text from the response stream, I get a bunch of binary data and not the plain json text.
Edit: realized the problem had to do with compression. This can be closed.
I'm not sure if got it right, but you can get the response as a string doing this:
using (var sr = new StreamReader(response.GetResponseStream()))
{
text = sr.ReadToEnd();
}
Turned out my problem was due to compression. I realized the header contained "Content-Encoding: gzip" so I searched on how to unzip with gzip compression and then the text was proper json. Thanks all
In my code, I am doing below
HttpWebResponse response = (HttpWebResponse)httpWebRequest.GetResponse();
Stream dataStream = response.GetResponseStream();
string responseFromServer = reader.ReadToEnd();
reader.Close();
dataStream.Close();
response.Close();
httpWebRequest is a POST request whose response size in Fiddler after one redirection is 67722. response.ContentLength field returned in my case is -1 and total size of responseFromServer is approximately 32695. It simply cuts the remaining data from response. I have seen somewhere to use maxReceivedMessageSize but not able to figure our how to do it.
responseFromServer.Length 32695
response.ContentLength -1
response.GetResponseHeader("transfer-encoding") ""
UPDATE: FINAL
Response was coming right. Issue was with my code where parameters being sent into POST query were not encoded properly through UrlEncoding. If we are dealing with ASP.net contentpanel control, point to remember is that it sends the POST query with control names having special character '$'. We need to encode that using HttpUtility.UrlEncoding method
As mentioned by spender, check the response headers..the value for 'ContentLength' is supplied in the response headers, and may NOT be the actual length of data; if transfer-encoding is 'chunked', it is "normal" to mark contentlength as -1.
I'm not familiar with fiddler..but perhaps it is fetching things that the webresponse is not, like external scripts? Try loading the page in a normal webbrowser, and 'View Source' --> use some tool (A textbox..) to count the characters in the source: is it actually 67k or closer to 32? Similarly, check the last part of data in 'responseFromServer'..is it the same as when 'Viewing page source'? ie. both are probably (hopefully?) < / h t m l >
Is there a way to determine if the response from an HttpWebRequest in C# contains binary data vs. text? Or is there another class or function I should be using to do this?
Here's some sample code. I'd like to know before reading the StreamReader if the content is not text.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.someurl.com");
request.Method = WebRequestMethods.Http.Get;
using (WebResponse response = request.GetResponse())
{
// check somewhere in here if the response is binary data and ignore it
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
string responseDetails = reader.ReadToEnd().Trim();
}
}
In general, web sites will tell you in the Content-Type header what kind of data they're returning. You can determine that by getting the ContentType property from the response.
But sites have been known to lie. Or not say anything. I've seen both. If there is no Content-Type header or you don't want to trust it, then the only way you can tell what kind of data is there, is by reading it.
But then, if you don't trust the site, why are you reading data from it?
I connect to the web service as follows:
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(
"http://mywebserviceaddress.com/attributes=someatt");
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
using (StreamReader stIn = new StreamReader(
req.GetResponse().GetResponseStream(),Encoding.UTF8))
{
strResponse = stIn.ReadToEnd();
return strResponse;
}
However I get the response with (probably) bad encoding so as a result, on my page i get the following issue:
Am I doing something wrong or is it a third-party web service issue? How can I get the respone without this silly issues?
Here's the screenshot from debugger:
The page is probably not UTF8. It looks like special characters use the upper half of an ASCII character, so it's some kind of old charset. Reading it as UTF8 causes errors because the reader doesn't expect single-byte special characters.
Store the result of GetResponse() into a variable and output the contents of ContentTypeCharacterSet. If the server acts correctly, it shows the used charset in this property. Then, you can use the correct charset in your StreamReader.
When I try to fetch the content of this URL
http://www.yellowpages.com.au/qld/gatton/a-12000029-listing.html
using System.Net;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(link);
request.AllowAutoRedirect = true;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream resStream = response.GetResponseStream();
StreamReader objSR;
objSR = new StreamReader(resStream, System.Text.Encoding.GetEncoding("utf-8"));
string sResponse = objSR.ReadToEnd();
I don't get any response from the server. Please help me find out why this happens.
Thanks in advance!
It may well be looking at the user agent and refusing to serve content to a client that doesn't identify itself. Try setting the UserAgent property on your request object.
Looks to me like that site is checking the referrer url and may be serving up empty content if an invalid referrer is specified.
Try setting request.Referer = "http://www.google.com";. Experiment with the referrer to see if that changes the response. I'd also try the UserAgent property as Matthew suggested.
I had the same problem and the cause was that I previously had set the method to HEAD and in later revisions had the need to parse the body.