I try to read a web page that always throws a http 500 internal server error. When accessing it with firefox or internet explorer I got the full page (~200KB). When I read it with my program, I only got a part of the page (~64KB). This is my code:
try
{
HttpWebRequest request = null;
request = (HttpWebRequest)WebRequest.Create("http:// ... ");
request.Timeout = 30000;
request.Method = "GET";
responseContent = null;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
StreamReader responseReader = new StreamReader(responseStream);
responseContent = responseReader.ReadToEnd();
}
catch (WebException ex)
{
Stream responseStream = ex.Response.GetResponseStream();
StreamReader responseReader = new StreamReader(responseStream);
responseContent = responseReader.ReadToEnd();
}
When I check the traffic with fiddler, I see that Http 500 is returned when accessing the page with the browser and with my program. But in all three cases I see the full page in fiddler.
What can I do, to get the full page? The web page I'm try to read is not under my control.
The solutions in "HTTPWebResponse Response string is truncated" do not change anything.
Related
I've got a web service that returns an http 500 with some diagnostic information in the body of the response.
I'm doing something like
Stream responseStream = null;
WebResponse _Response = null;
Stream responseStream = null;
HttpWebRequest _Request = null;
try
{
_Response = _Request.GetResponse();
responseStream = _Response.GetResponseStream();
}
catch {
//try to view the Request.GetResponse() body here.
}
Since _Request.GetResponse() is returning an http 500 there doesn't seem to be a way to view the response body. According to HTTP 500 Response with Body? this was a known issue in Java 9 years ago. I'm wondering if there's a way to do it in .NET today.
The microsoft docs give a good run down of what HttpWebRequest.GetResponse returns if it fails, you can check it out here https://learn.microsoft.com/en-us/dotnet/api/system.net.httpwebrequest.getresponse?view=netframework-4.8
In your example I believe you need to check for WebException and handle it.
Stream responseStream = null;
WebResponse _Response = null;
Stream responseStream = null;
HttpWebRequest _Request = null;
try
{
_Response = _Request.GetResponse();
responseStream = _Response.GetResponseStream();
}
catch (WebException w)
{
//here you can check the reason for the web exception
WebResponse res = w.Response;
using (Stream s = res.GetResponseStream())
{
StreamReader r= new StreamReader(s);
string exceptionMessage = r.ReadToEnd(); //here is your error info
}
}
catch {
//any other exception
}
the catch is supposed to give me a 504 but for someone reason, I get a null on:
response = (HttpWebResponse)e.Response;
Below is my code:
var url = "http://www.go435345ogle.com";
HttpWebResponse response = null;
HttpStatusCode statusCode;
get http response
get status
try
{
// Creates an HttpWebRequest for the specified URL.
HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create(url);
//HttpWebRequest myHttpWebRequest = (HttpWebRequest)WebRequest.Create(keys.Value.Substring(0, keys.Key.Length - 1));
// Sends the HttpWebRequest and waits for a response.
response = (HttpWebResponse)myHttpWebRequest.GetResponse();
}
catch (WebException e)
{
Console.WriteLine("\r\nWebException Raised. The following error occured : {0}", e.Status);
response = (HttpWebResponse)e.Response;
}
statusCode = response.StatusCode;
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
var sResponse = reader.ReadToEnd();
// Console.WriteLine(sResponse);
Console.WriteLine("Response Code: " + (int)statusCode + " - " + statusCode.ToString());
}
This is expected behaviour. Since domain www.go435345ogle.com doesn't exist, there is no server you could send request to, and therefore no response to receive. So WebException.Response simply returns null. Microsoft's docs clearly states, that WebException.Response returns:
If a response is available from the Internet resource, a WebResponse
instance that contains the error response from an Internet resource;
otherwise, null.
i'm having a problem with this simple code which sends a request using provided url and reads html from responce. Looks like it's something with encoding of cyrillic symbols after ?q=, but i can't see why (url was actually obtained from browser address bar, not generated or anything else).
url =
"http://www.avito.ru/nizhniy_novgorod/kvartiry/sdam/na_dlitelnyy_srok?q=%D0%93%D0%B5%D0%BD%D0%B5%D1%80%D0%B0%D0%BB%D0%B0+%D0%98%D0%B2%D0%BB%D0%B8%D0%B5%D0%B2%D0%B0+10%D0%BA1";
string html = "";
try
{
Uri uri = new Uri(url);
WebRequest request = WebRequest.Create(uri);
request.Timeout = 100000;
using (WebResponse responce = request.GetResponse())
{
Stream stream = responce.GetResponseStream();
StreamReader reader = new StreamReader(stream);
html = reader.ReadToEnd();
}
}
catch (Exception e)
{
System.Console.WriteLine(e.Message);
}
Error occures in GetResponce() method. The message is:
The request was aborted: The connection was closed unexpectedly.
You should cast your request and response to HttpWebRequest and HttpWebResponse, respectively.
var url = "http://www.avito.ru/nizhniy_novgorod/kvartiry/sdam/na_dlitelnyy_srok?q=%D0%93%D0%B5%D0%BD%D0%B5%D1%80%D0%B0%D0%BB%D0%B0+%D0%98%D0%B2%D0%BB%D0%B8%D0%B5%D0%B2%D0%B0+10%D0%BA1";
string html = "";
try
{
Uri uri = new Uri(uri);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Timeout = 100000;
using (HttpWebResponse responce = (HttpWebResponse)request.GetResponse())
{
Stream stream = responce.GetResponseStream();
StreamReader reader = new StreamReader(stream);
html = reader.ReadToEnd();
Console.WriteLine(html);
}
}
catch (Exception e)
{
System.Console.WriteLine(e.Message);
}
Also, It seems that the url http://www.avito.ru/nizhniy_novgorod/kvartiry/sdam/na_dlitelnyy_srok?q=%D0%93%D0%B5%D0%BD%D0%B5%D1%80%D0%B0%D0%BB%D0%B0+%D0%98%D0%B2%D0%BB%D0%B8%D0%B5%D0%B2%D0%B0+10%D0%BA1 is invalid.
Using Fiddler, the url returns a 404 error.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(videoInfoUrl);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
StreamReader reader = new StreamReader(responseStream, Encoding.GetEncoding("utf-8"));
string videoInfo = HttpUtility.HtmlDecode(reader.ReadToEnd());
NameValueCollection videoParams = HttpUtility.ParseQueryString(videoInfo);
My code:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(videoInforUrl);
// HttpWebResponse response = await (HttpWebResponse)request.GetResponseAsync();
//Stream responseStream = response.GetResponseStream();
//StreamReader reader = new StreamReader(responseStream, Encoding.GetEncoding("utf-8"));
//string videoInfor=HttpUtility
I got issues from HttpWebResponse response = await (HttpWebResponse)request.GetResponseAsync(); show cannot convert httpwebresponse -> getresponseAsynce.
On universal app not support in HttpUtility
Don't force your typing. Allow it to return what it wants as long as you're able to do what you want with it.
Get rid of the explicit types like this, and you should be fine:
var request = WebRequest.Create(videoInfoUrl);
var response = await request.GetResponseAsync();
The error was occurring because you probably had the (HttpWebResponse) in the wrong location. Where you have it would be trying to convert the Task that GetResponseAsync to the response, which is why you're getting that error.
How may I detect that was returned 404 page if server answer code 200? Now I checking content length, but it takes too long. Is there any other ways?
Current code:
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
// Sends the HttpWebRequest and waits for a response.
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
Stream s = resp.GetResponseStream();
StreamReader rs = new StreamReader(s);
string st = rs.ReadToEnd();
if (resp.StatusCode == HttpStatusCode.OK & st.Length > 18795)
{
handling answer here
}