We Keep Coding

HttpWebResponse returns an 'incomplete' stream

I´m making repeated requests to a web server using HttpWebRequest, but I randomly get a 'broken' response stream in return. e.g it doesn´t contain the tags that I KNOW is supposed to be there. If I request the same page multiple times in a row it turns up 'broken' ~3/5.
The request always returns a 200 response so I first thought there was a null value inserted in the response that made the StreamReader think it reached the end.
I´ve tried:
1) reading everything into a byte array and cleaning it
2) inserting a random Thread.Sleep after each request
Is there any potentially bad practice with my code below or can anyone tell me why I´m randomly getting an incomplete response stream? As far as I can see I´m closing all unmanaged resources so that shouldn´t be a problem, right?
public string ReturnHtmlResponse(string url)
{
string result;
var request = (HttpWebRequest)WebRequest.Create(url);
{
using(var response = (HttpWebResponse)request.GetResponse())
{
Console.WriteLine((int)response.StatusCode);
var encoding = Encoding.GetEncoding(response.CharacterSet);
using(var stream = response.GetResponseStream())
{
using(var sr = new StreamReader(stream,encoding))
{
result = sr.ReadToEnd();
}
}
}
}
return result;
}

I do not see any direct flaws in you're code. What could be is that one of the 'Parent' using statements is done before the nested one. Try changing the using to a Dispose() and Close() method.
public string ReturnHtmlResponse(string url)
{
string result;
var request = (HttpWebRequest)WebRequest.Create(url);
var response = (HttpWebResponse)request.GetResponse();
Console.WriteLine((int)response.StatusCode);
var encoding = Encoding.GetEncoding(response.CharacterSet);
var stream = response.GetResponseStream();
var sr = new StreamReader(stream,encoding);
result = sr.ReadToEnd();
sr.Close();
stream.Close();
response.Close();
sr.Dispose();
stream.Dispose();
response.Dispose();
return result;
}

Why is the stream left open in this code?

I have a question from a piece of code from the book:
Programming in C#
Exam 70-483
Here's the code:
WebRequest request = WebRequest.Create(“http://www.microsoft.com”);
WebResponse response = request.GetResponse();
StreamReader responseStream = new StreamReader(response.GetResponseStream());
string responseText = responseStream.ReadToEnd();
Console.WriteLine(responseText); // Displays the HTML of the website
response.Close();
My question is that why is the responseStream not closed in this example? Why is just the response object closed?

Calling WebResponse.Close implicity closes the response stream.
Taken from here - http://msdn.microsoft.com/en-us/library/system.net.webresponse.close(v=vs.110).aspx
The Close method cleans up the resources used by a WebResponse and closes the underlying stream by calling the Stream.Close method.

As nsgocev said, the docs say that calling Close on the WebResponse closes the underlying stream. So there's no point trying to close it again.
But I couldn't help responding and pointing out that the code in the example does nothing to protect itself from resource leaks. If an exception occurs, whether in your code or in the communication with the remote web server, the response won't get closed and the underlying stream won't be released, which means the TCP socket probably won't be released, etc.
At the very least,the code should take advantage of the "using" syntactical sugar (which the compiler expands into try/finally) to protect itself and guarantee that the resources are released:
using( WebRequest request = WebRequest.Create( “http://www.microsoft.com” ) )
{
using( WebResponse response = request.GetResponse() )
{
StreamReader responseStream = new StreamReader( response.GetResponseStream() );
string responseText = responseStream.ReadToEnd();
Console.WriteLine( responseText ); // Displays the HTML of the website
}
}

HTTPWebResponse Response string is truncated

App is talking to a REST service.
Fiddler shows full good XML response coming in as the Apps response
The App is in French Polynesia and an identical copy in NZ works so the prime suspect seemed encoding but we have checked that out and came up empty handed.
Looking at the output string (UTF8 encoding) from the stream reader you can see where it has been truncated. It is in an innocuous piece of xml. The downstream error on an XmlDocument object claims to have an encountered an unexpected end of file while loading the string into an XML Document object which is fair enough.
The truncation point is
ns6:sts-krn>1&
which is part of
ns6:sts-krn>1</ns6:sts-krn><
Is there any size limitation on the response string or some other parameter that we should check.
I am all out of ideas. Code Supplied as requested.
Stream streamResponse = response.GetResponseStream();
StringBuilder sb = new StringBuilder();
Encoding encode = Encoding.GetEncoding("utf-8");
if (streamResponse != null)
{
StreamReader readStream = new StreamReader(streamResponse, encode);
while (readStream.Peek() >= 0)
{
sb.Append((char)readStream.Read());
}
streamResponse.Close();
}

You need to be using using blocks:
using (WebResponse response = request.GetResponse())
{
using (Stream streamResponse = response.GetResponseStream())
{
StringBuilder sb = new StringBuilder();
if (streamResponse != null)
{
using (StreamReader readStream = new StreamReader(streamResponse, Encoding.UTF8))
{
sb.Append(readStream.ReadToEnd());
}
}
}
}
This will ensure that your WebResponse, Stream, and StreamReader all get cleaned up, regardless of whether there are any exceptions.
The reasoning that led me to think about using blocks was:
Some operation was not completed
There were no try/catch blocks hiding exceptions, so if the operation wasn't completed due to exceptions, we would know about it.
There were objects which implement IDisposable which were not in using blocks
Conclusion: try implementing the using blocks to see if disposing the objects will cause the operation to complete.
I added this because the reasoning is actually quite general. The same reasoning works for "my mail message doesn't get sent for two minutes". In that case, the operation which isn't completed is "send email" and the instances are the SmtpClient and MailMessage objects.

An easier way to do this would be:
string responseString;
using (StreamReader readStream = new StreamReader(streamResponse, encode))
{
responseString = readStream.ReadToEnd();
}
For debugging, I would suggest writing that response stream to a file so that you can see exactly what was read. In addition, you might consider using a single byte encoding (like ISO-8859-1) to read the data and write to the file.
You should check the response.ContentType property to see if some other text encoding is used.

Convert Stream data to custom object in C#.net

I want to convert stream data from response stream into a custom object.
I want to convert respose stream into custom object,I am following these steps.
My code is as follows.
myMethod()
{
state s = new state();
Stream receiveStream;
StreamReader readStream;
HttpWebRequest request;
HttpWebResponse response;
try
{
request = (HttpWebRequest)WebRequest.Create (url);
request.Method = "GET";
request.ContentType = "application/json";
response = (HttpWebResponse)request.GetResponse ();
receiveStream = response.GetResponseStream();
readStream = new StreamReader(receiveStream);
Console.WriteLine (readStream.ReadToEnd());
serializer = new DataContractJsonSerializer(typeof(state));
s = serializer.ReadObject(readStream.BaseStream)as state;
Console.Write(s.name+"\n");
response.Close ();
readStream.Close ();
}
catch (Exception ex)
{
}
}
Object s returning nothing.
Can anyone help me?

The trouble is that you're trying to deserialize an object when you've already read all the data from it just beforehand:
readStream = new StreamReader(receiveStream);
Console.WriteLine (readStream.ReadToEnd());
After those line, the stream will be empty, so there's nothing to deserialize. Get rid of those lines (then use receiveStream below), and you may well find it just works.
Additionally, a few suggestions:
Rather than closing streams explicitly, use using statements
Add a using statement for the response itself, as that implements IDisposable
Keep the scope of each variable as small as it can be, assigning a value at the point of declaration
It's rarely a good idea to catch Exception, and it's almost never a good idea to just swallow exceptions in the way that you are doing, with no logging etc
Follow .NET naming conventions, where state would be State (and possibly make the name a bit more descriptive anyway)
Use a cast rather than as - see my blog post on the topic for reasons

Update CSV header Using CsvReader

I have a CSV file that I grab & convert it into Stream. I need a way to modify header without converting stream to string. Is there a way around?
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream resStream = response.GetResponseStream();
//need to update stream here & continue with the CSV reader
using (CsvReader csv = new CsvReader(new StreamReader(resStream), true))
{
//other code
}

Derive your own class from StreamReader and override the function you use for reading it, inserting the proper behavior.