I´m making repeated requests to a web server using HttpWebRequest, but I randomly get a 'broken' response stream in return. e.g it doesn´t contain the tags that I KNOW is supposed to be there. If I request the same page multiple times in a row it turns up 'broken' ~3/5.
The request always returns a 200 response so I first thought there was a null value inserted in the response that made the StreamReader think it reached the end.
I´ve tried:
1) reading everything into a byte array and cleaning it
2) inserting a random Thread.Sleep after each request
Is there any potentially bad practice with my code below or can anyone tell me why I´m randomly getting an incomplete response stream? As far as I can see I´m closing all unmanaged resources so that shouldn´t be a problem, right?
public string ReturnHtmlResponse(string url)
{
string result;
var request = (HttpWebRequest)WebRequest.Create(url);
{
using(var response = (HttpWebResponse)request.GetResponse())
{
Console.WriteLine((int)response.StatusCode);
var encoding = Encoding.GetEncoding(response.CharacterSet);
using(var stream = response.GetResponseStream())
{
using(var sr = new StreamReader(stream,encoding))
{
result = sr.ReadToEnd();
}
}
}
}
return result;
}
I do not see any direct flaws in you're code. What could be is that one of the 'Parent' using statements is done before the nested one. Try changing the using to a Dispose() and Close() method.
public string ReturnHtmlResponse(string url)
{
string result;
var request = (HttpWebRequest)WebRequest.Create(url);
var response = (HttpWebResponse)request.GetResponse();
Console.WriteLine((int)response.StatusCode);
var encoding = Encoding.GetEncoding(response.CharacterSet);
var stream = response.GetResponseStream();
var sr = new StreamReader(stream,encoding);
result = sr.ReadToEnd();
sr.Close();
stream.Close();
response.Close();
sr.Dispose();
stream.Dispose();
response.Dispose();
return result;
}
Related
I'm trying to get weather data from online as json and then deserialize the json into an object that I can use. Here's my code:
public static RootObject7 GetWeather7(int zip)
{
var url = "http://api.weatherunlocked.com/api/forecast/us." + zip.ToString() + "?app_id=xxxxxxx&app_key=xxxxxxxxxxxxxxxxxxxxxxx";
var weather = new wunlocked();
string json = weather.getJson(url);
JavaScriptSerializer serializer = new JavaScriptSerializer();
var data = (RootObject7)serializer.Deserialize<RootObject7>(json);
return data;
}
private string getJson(string url)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
try
{
WebResponse response = request.GetResponse();
using (Stream responseStream = response.GetResponseStream())
{
StreamReader reader = new StreamReader(responseStream, Encoding.UTF8);
return reader.ReadToEnd();
}
}
catch (WebException ex)
{
WebResponse errorResponse = ex.Response;
using (Stream responseStream = errorResponse.GetResponseStream())
{
StreamReader reader = new StreamReader(responseStream, Encoding.GetEncoding("utf-8"));
String errorText = reader.ReadToEnd();
}
throw;
}
}
I'm debugging, and what's happening is my RootObject7 data object is being created, and inside it has a "Forecast" object, which is supposed to contain a list of other information but instead it's null. I've already defined all of the classes (they're long, so if it's important, I'll post them but otherwise I don't think I need to). I've never done anything like this before so most of this came from other code examples on here that I've found, but obviously I didn't put them together correctly, since my object is always null but when I go to the url, there's valid xml there. I'm not sure if I need to be somehow converting the xml to json in my code, or if that is being done somehow? Like I said, I really don't know what I'm doing but if anyone has suggestions, that'd be great.
Try
dynamic data = serializer.Deserialize(json);
and then inspect the data object in the debugger - you may not need to deserialise to a fixed interface to get out the data you need. Using dynamic may also be a more robust solution to deal with upgrades to the service that may make a set interface/object more brittle.
App is talking to a REST service.
Fiddler shows full good XML response coming in as the Apps response
The App is in French Polynesia and an identical copy in NZ works so the prime suspect seemed encoding but we have checked that out and came up empty handed.
Looking at the output string (UTF8 encoding) from the stream reader you can see where it has been truncated. It is in an innocuous piece of xml. The downstream error on an XmlDocument object claims to have an encountered an unexpected end of file while loading the string into an XML Document object which is fair enough.
The truncation point is
ns6:sts-krn>1&
which is part of
ns6:sts-krn>1</ns6:sts-krn><
Is there any size limitation on the response string or some other parameter that we should check.
I am all out of ideas. Code Supplied as requested.
Stream streamResponse = response.GetResponseStream();
StringBuilder sb = new StringBuilder();
Encoding encode = Encoding.GetEncoding("utf-8");
if (streamResponse != null)
{
StreamReader readStream = new StreamReader(streamResponse, encode);
while (readStream.Peek() >= 0)
{
sb.Append((char)readStream.Read());
}
streamResponse.Close();
}
You need to be using using blocks:
using (WebResponse response = request.GetResponse())
{
using (Stream streamResponse = response.GetResponseStream())
{
StringBuilder sb = new StringBuilder();
if (streamResponse != null)
{
using (StreamReader readStream = new StreamReader(streamResponse, Encoding.UTF8))
{
sb.Append(readStream.ReadToEnd());
}
}
}
}
This will ensure that your WebResponse, Stream, and StreamReader all get cleaned up, regardless of whether there are any exceptions.
The reasoning that led me to think about using blocks was:
Some operation was not completed
There were no try/catch blocks hiding exceptions, so if the operation wasn't completed due to exceptions, we would know about it.
There were objects which implement IDisposable which were not in using blocks
Conclusion: try implementing the using blocks to see if disposing the objects will cause the operation to complete.
I added this because the reasoning is actually quite general. The same reasoning works for "my mail message doesn't get sent for two minutes". In that case, the operation which isn't completed is "send email" and the instances are the SmtpClient and MailMessage objects.
An easier way to do this would be:
string responseString;
using (StreamReader readStream = new StreamReader(streamResponse, encode))
{
responseString = readStream.ReadToEnd();
}
For debugging, I would suggest writing that response stream to a file so that you can see exactly what was read. In addition, you might consider using a single byte encoding (like ISO-8859-1) to read the data and write to the file.
You should check the response.ContentType property to see if some other text encoding is used.
Okay so let's say I have a program which makes a WebRequest and gets a WebResponse, and uses that WebResponse for a StreamReader in a using statement, which obviously gets disposed of after, but what about the WebResponse stream? For example:
WebResponse response = request.GetResponse();
using(StreamReader reader = new StreamReader(response.GetResponseStream()))
{
x = reader.ReadToEnd();
}
So obviously the StreamReader will get disposed after, and all resources devoted to it, but what about the Response Stream from response? Does that? Or should I do it like this:
WebResponse response = request.GetResponse();
using(Stream stream = response.GetResponseStream())
{
using(StreamReader reader = new StreamReader(stream))
{
x = reader.ReadToEnd();
}
}
Thanks
StreamReader assumes ownership of the Stream that you pass to its constructor. In other words, when you dispose the StreamReader object it will automatically also dispose the Stream object that you passed. That's very convenient and exactly what you want, simplifying the code to:
using (var response = request.GetResponse())
using (var reader = new StreamReader(response.GetResponseStream()))
{
x = reader.ReadToEnd();
}
When I use Visual Studio 2013's Code Analysis, it claims the use of two 'usings' here will cause Dispose() to be called twice on 'response' and will generate a hidden 'System.ObjectDisposedException'?
warning CA2202: Microsoft.Usage : Object 'response' can be disposed
more than once in method 'xxx'. To avoid generating a
System.ObjectDisposedException you should not call Dispose more than
one time on an object.
Has anyone else seen this?
I need to check if a text file exists on a site on a different domain. The URL could be:
http://sub.somedomain.com/blah/atextfile.txt
I need to do this from code behind. I am trying to use the HttpWebRequest object, but not sure how to do it.
EDIT: I am looking for a light weight way of doing this as I'll be executing this logic every few seconds
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(
"http://sub.somedomain.com/blah/atextfile.txt");
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode == HttpStatusCode.OK)
{
// FILE EXISTS!
}
response.Close();
You could probably use the method used here:
http://www.eggheadcafe.com/tutorials/aspnet/2c13cafc-be1c-4dd8-9129-f82f59991517/the-lowly-http-head-reque.aspx
Something like this might work for you:
using (WebClient webClient = new WebClient())
{
try
{
using (Stream stream = webClient.OpenRead("http://does.not.exist.com/textfile.txt"))
{
}
}
catch (WebException)
{
throw;
}
}
In my application I use the WebClient class to download files from a Webserver by simply calling the DownloadFile method. Now I need to check whether a certain file exists prior to downloading it (or in case I just want to make sure that it exists). I've got two questions with that:
What is the best way to check whether a file exists on a server without transfering to much data across the wire? (It's quite a huge number of files I need to check)
Is there a way to get the size of a given remote file without downloading it?
Thanks in advance!
WebClient is fairly limited; if you switch to using WebRequest, then you gain the ability to send an HTTP HEAD request. When you issue the request, you should either get an error (if the file is missing), or a WebResponse with a valid ContentLength property.
Edit: Example code:
WebRequest request = WebRequest.Create(new Uri("http://www.example.com/"));
request.Method = "HEAD";
using(WebResponse response = request.GetResponse()) {
Console.WriteLine("{0} {1}", response.ContentLength, response.ContentType);
}
When you request file using the WebClient Class, the 404 Error (File Not Found) will lead to an exception. Best way is to handle that exception and use a flag which can be set to see if the file exists or not.
The example code goes as follows:
System.Net.HttpWebRequest request = null;
System.Net.HttpWebResponse response = null;
request = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create("www.example.com/somepath");
request.Timeout = 30000;
try
{
response = (System.Net.HttpWebResponse)request.GetResponse();
flag = 1;
}
catch
{
flag = -1;
}
if (flag==1)
{
Console.WriteLine("File Found!!!");
}
else
{
Console.WriteLine("File Not Found!!!");
}
You can put your code in respective if blocks.
Hope it helps!
What is the best way to check whether a file exists on a server
without transfering to much data across the wire?
You can test with WebClient.OpenRead to open the file stream without reading all the file bytes:
using (var client = new WebClient())
{
Stream stream = client.OpenRead(url);
// ^ throws System.Net.WebException: 'Could not find file...' if file is not present
stream.Close();
}
This will indicate if the file exists at the remote location or not.
To fully read the file stream, you would do:
using (var client = new WebClient())
{
Stream stream = client.OpenRead(url);
StreamReader sr = new StreamReader(stream);
Console.WriteLine(sr.ReadToEnd());
stream.Close();
}
In case anyone stuck with ssl certificate issue
ServicePointManager.ServerCertificateValidationCallback = new RemoteCertificateValidationCallback
(
delegate { return true; }
);
WebRequest request = WebRequest.Create(new Uri("http://.com/flower.zip"));
request.Method = "HEAD";
using (WebResponse response = request.GetResponse())
{
Console.WriteLine("{0} {1}", response.ContentLength, response.ContentType);
}