We Keep Coding

WebRequest loading HTMLDocument coming back with all special characters for SSL site

Pretty standard implementation of HttpWebRequest, whenever I pass a certain URL to get the html it comes back with nothing but special characters. An example of what comes back is below.
Now this site is SSL so I'm wondering if that has something to do with it but I've never had this problem before and I've used this with other SSL sites.
�
ServicePointManager.ServerCertificateValidationCallback = new System.Net.Security.RemoteCertificateValidationCallback(AcceptAllCertifications);
var request = (HttpWebRequest)WebRequest.Create(url);
using (var response = (HttpWebResponse)request.GetResponse())
{
Stream data = response.GetResponseStream();
HtmlDocument hDoc = new HtmlDocument();
using (StreamReader readURLContent = new StreamReader(data))
{
html = readURLContent.ReadToEnd();
hDoc.LoadHtml(html);
}
}
I can't really find anything for this specific issue so I'm kind of lost if anybody could point me in the right direction that would be awesome.
Edit: here's an image of what it looks like since I can't copy paste it

My guess is that the response is compressed. If you use a WebDebugger like Charles or Fiddler. You can see how the requests and structured and what data they contain - it makes it a lot easier to replicate the http requests later on when programming them. Try the following code.
try
{
string webAddr = url;
var httpWebRequest = (HttpWebRequest)WebRequest.Create(webAddr);
httpWebRequest.ContentType = "text/html; charset=utf-8";
httpWebRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0";
httpWebRequest.AllowAutoRedirect = true;
httpWebRequest.Method = "GET";
httpWebRequest.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream(), Encoding.UTF8))
{
var responseText = streamReader.ReadToEnd();
doc.LoadHtml(responseText);
}
}
catch (WebException ex)
{
Console.WriteLine(ex.Message);
}
The code sets the encoding on the requsts. You an also set the encoding at the streamreader when reading the response. And automatic decompression is enabled.

Why python request working but C# request not working?

First, i try to post the script in PostMan tool.
{"AO":"ECHO"}
It working fine. Then i'm writing this request in C# but it not working.
And more i wrote the request again in Python, and it working well.
But my project is in Microsoft C#. I dont want to run script Python in C# at all.
==== Python =========
import httplib
import json
import sys
data = '{"AO":"ECHO"}'
headers = {"Content-Type": "application/json", "Connection": "Keep-Alive" }
conn = httplib.HTTPConnection("http://10.10.10.1",1040)
conn.request("POST", "/guardian", data, headers)
response = conn.getresponse()
print response.status, response.reason
print response.msg
==== C# ============
var httpWebRequest = (HttpWebRequest)WebRequest.Create("http://10.10.10.1:1040/guardian");
httpWebRequest.Method = "POST";
httpWebRequest.ContentType = "application/json";
using (var streamWriter = new StreamWriter(httpWebRequest.GetRequestStream()))
{
string json = "{\"AO\":\"ECHO\"}";
streamWriter.Write(json);
streamWriter.Flush();
streamWriter.Close();
var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var result = streamReader.ReadToEnd();
Console.WriteLine(result);
}
}
I try to put "ContentLength" but it still timeout exception.
And i try to using RestSharp, it's not timeout but return null.
Any one please help...
var client = new RestClient("http://10.10.10.1:1040/guardian");
var request = new RestRequest();
request.Method = Method.POST;
request.AddHeader("Content-Type", "application/json");
request.Parameters.Clear();
request.RequestFormat = DataFormat.Json;
request.AddBody(new { AO = "ECHO" });
var response = client.Execute(request);
var content = response.Content;
Please help me,
I dont understand why it working fine in python.
But why it not working in C#.
I try to find many request in C# but it got error exception with timeout.

Python will automatically add Content-Length http header.
https://docs.python.org/2/library/httplib.html#httpconnection-objects
I think you might have to set this header manually in C#.
httpWebRequest.ContentLength = json.length;
Depending on the server, you may have to set UserAgent as well.
httpWebRequest.UserAgent=".NET Framework Test Client";

web request using API

I have done something similar to this before however I'm not sure how to do this with a bigger project.
I'm trying to return the titles of all the stuff on the front page of reddit.
From this site:
http://www.reddit.com/r/all.json
I pasted the data into
http://json2csharp.com/#
to find out the class I need.
From here though, I'm not too sure on how to proceed. If I wanted to return an array of all this data so I can easily get information, how could I do it.
Sorry for the vagueness of this question but I'm just at a loss and don't know what to do.

Use
using (var webClient = new System.Net.WebClient()) {
var json = webClient.DownloadString("http://www.reddit.com/r/all.json");
}
For old .Net:
var request = WebRequest.Create(url);
string text;
request.ContentType = "application/json; charset=utf-8";
var response = (HttpWebResponse) request.GetResponse();
using (var sr = new StreamReader(response.GetResponseStream()))
{
text = sr.ReadToEnd();
}

HttpWebRequest.GetResponse() keeps getting timed out

i wrote a simple C# function to retrieve trade history from MtGox with following API call:
https://data.mtgox.com/api/1/BTCUSD/trades?since=<trade_id>
documented here: https://en.bitcoin.it/wiki/MtGox/API/HTTP/v1#Multi_currency_trades
here's the function:
string GetTradesOnline(Int64 tid)
{
Thread.Sleep(30000);
// communicate
string url = "https://data.mtgox.com/api/1/BTCUSD/trades?since=" + tid.ToString();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string json = reader.ReadToEnd();
reader.Close();
reader.Dispose();
response.Close();
return json;
}
i'm starting at tid=0 (trade id) to get the data (from the very beginning). for each request, i receive a response containing 1000 trade details. i always send the trade id from the previous response for the next request. it works fine for exactly 4 requests & responses. but after that, the following line throws a "System.Net.WebException", saying that "The operation has timed out":
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
here are the facts:
catching the exception and retying keeps causing the same exception
the default HttpWebRequest .Timeout and .ReadWriteTimeout are already high enough (over a minute)
changing HttpWebRequest.KeepAlive to false didn't solve anything either
it seems to always work in the browser even while the function is failing
it has no problems retrieveing the response from https://www.google.com
the amount of successful responses before the exceptions varies from day to day (but browser always works)
starting at the trade id that failed last time causes the exception immediately
calling this function from the main thread instead still caused the exception
running it on a different machine didn't work
running it from a different IP didn't work
increasing Thread.Sleep inbetween requests does not help
any ideas of what could be wrong?

I had the very same issue.
For me the fix was as simple as wrapping the HttpWebResponse code in using block.
using (HttpWebResponse response = (HttpWebResponse) request.GetResponse())
{
// Do your processings here....
}
Details: This issue usually happens when several requests are made to the same host, and WebResponse is not disposed properly. That is where using block will properly dispose the WebResponse object properly and thus solving the issue.

There are two kind of timeouts. Client timeout and server timeout. Have you tried doing something like this:
request.Timeout = Timeout.Infinite;
request.KeepAlive = true;
Try something like this...

I just had similar troubles calling a REST Service on a LINUX Server thru ssl. After trying many different configuration scenarios I found out that I had to send a UserAgent in the http head.
Here is my final method for calling the REST API.
private static string RunWebRequest(string url, string json)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
// Header
request.ContentType = "application/json";
request.Method = "POST";
request.AllowAutoRedirect = false;
request.KeepAlive = false;
request.Timeout = 30000;
request.ReadWriteTimeout = 30000;
request.UserAgent = "test.net";
request.Accept = "application/json";
request.ProtocolVersion = HttpVersion.Version11;
request.Headers.Add("Accept-Language","de_DE");
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls;
ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
byte[] bytes = Encoding.UTF8.GetBytes(json);
request.ContentLength = bytes.Length;
using (var writer = request.GetRequestStream())
{
writer.Write(bytes, 0, bytes.Length);
writer.Flush();
writer.Close();
}
var httpResponse = (HttpWebResponse)request.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var jsonReturn = streamReader.ReadToEnd();
return jsonReturn;
}
}

This is not a solution, but just an alternative:
These days i almost only use WebClient instead of HttpWebRequest. Especially WebClient.UploadString for POST and PUT and WebClient.DownloadString. These simply take and return strings. This way i don't have to deal with streams objects, except when i get a WebException. i can also set the content type with WebClient.Headers["Content-type"] if necessary. The using statement also makes life easier by calling Dispose for me.
Rarely for performance, i set System.Net.ServicePointManager.DefaultConnectionLimit high and instead use HttpClient with it's Async methods for simultaneous calls.
This is how i would do it now
string GetTradesOnline(Int64 tid)
{
using (var wc = new WebClient())
{
return wc.DownloadString("https://data.mtgox.com/api/1/BTCUSD/trades?since=" + tid.ToString());
}
}
2 more POST examples
// POST
string SubmitData(string data)
{
string response;
using (var wc = new WebClient())
{
wc.Headers["Content-type"] = "text/plain";
response = wc.UploadString("https://data.mtgox.com/api/1/BTCUSD/trades", "POST", data);
}
return response;
}
// POST: easily url encode multiple parameters
string SubmitForm(string project, string subject, string sender, string message)
{
// url encoded query
NameValueCollection query = HttpUtility.ParseQueryString(string.Empty);
query.Add("project", project);
query.Add("subject", subject);
// url encoded data
NameValueCollection data = HttpUtility.ParseQueryString(string.Empty);
data.Add("sender", sender);
data.Add("message", message);
string response;
using (var wc = new WebClient())
{
wc.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
response = wc.UploadString( "https://data.mtgox.com/api/1/BTCUSD/trades?"+query.ToString()
, WebRequestMethods.Http.Post
, data.ToString()
);
}
return response;
}
Error handling
try
{
Console.WriteLine(GetTradesOnline(0));
string data = File.ReadAllText(#"C:\mydata.txt");
Console.WriteLine(SubmitData(data));
Console.WriteLine(SubmitForm("The Big Project", "Progress", "John Smith", "almost done"));
}
catch (WebException ex)
{
string msg;
if (ex.Response != null)
{
// read response HTTP body
using (var sr = new StreamReader(ex.Response.GetResponseStream())) msg = sr.ReadToEnd();
}
else
{
msg = ex.Message;
}
Log(msg);
}

For what it's worth, I was experiencing the same issues with timeouts every time I used it, even though calls went through to the server I was calling. The problem in my case was that I had Expect set to application/json, when that wasn't what the server was returning.

WebRequest extremely slow

My method looks like this:
public string Request(string action, NameValueCollection parameters, uint? timeoutInSeconds = null)
{
parameters = parameters ?? new NameValueCollection();
ProvideCredentialsFor(ref parameters);
var data = parameters.ToUrlParams(); // my extension method converts the collection to a string, works well
byte[] dataStream = Encoding.UTF8.GetBytes(data);
string request = ServiceUrl + action;
var webRequest = (HttpWebRequest)WebRequest.Create(request);
webRequest.AllowAutoRedirect = false;
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.ContentLength = dataStream.Length;
webRequest.Timeout = (int)(timeoutInSeconds == null ? DefaultTimeoutMs : timeoutInSeconds * 1000);
webRequest.Proxy = null; // should make it faster...
using (var newStream = webRequest.GetRequestStream())
{
newStream.Write(dataStream, 0, dataStream.Length);
}
var webResponse = (HttpWebResponse)webRequest.GetResponse();
string uri = webResponse.Headers["Location"];
string result;
using (var sr = new StreamReader(webResponse.GetResponseStream()))
{
result = sr.ReadToEnd();
}
return result;
}
The server sends JSON in response. It works fine for small JSON, but when I request a large one - something goes wrong. By large one I mean something that takes 1-2 minutes to appear in a browser (google chrome, including server side generation time). It's actually 412KB of text. When I try to ask for the same JSON with the method above I get a web exception (timeout). I changed the timeout to 10 minutes (at least 5 times longer than chrome). Still the same.
Any ideas?
EDIT
This seems to have something to do with MS technologies. On IE this JSON also won't load.

Make sure you close your request. Otherwise, once you hit the maximum number of allowed connections (as low as four, for me, on one occasion), you have to wait for the earlier ones to time out. This is best done using
using (var response = webRequest.GetResponse()) {...