We Keep Coding

C# Large JSON to string causes out of memory exception

I'm trying to download very large JSON file. However, I keep getting an error message:
"An unhandled exception of type 'System.OutOfMemoryException' occurred in mscorlib.dll"
{The function evaluation was disabled because of an out of memory exception.}
Any tips how I can download this large JSON filet? I have tried to use string and StringBuilder but no luck.
Here is my code:
public static string DownloadJSON(string url)
{
try
{
String json = new WebClient().DownloadString(url); // This part fails!
return json;
}
catch (Exception)
{
throw;
}
}
I have created console application. I have tried this code with smaller JSON file and it worked.
My idea is later to split this larger JSON file and put it to database. However I need to encode it before I can put it to database. I have not write yet database part or anything else, because downloading this big JSON causes problems. I don't need it as a stream, but that was my example way how I made encoding. I need to encode it because data have special characters like å.
I tried also this but same problem:
var http = (HttpWebRequest)WebRequest.Create(url);
var response = http.GetResponse();
var stream = response.GetResponseStream();
var sr = new StreamReader(stream);
var content = sr.ReadToEnd();

I assume that you have very very large response. It is better to process stream. Now comes to point that cause outofmemoryexcetion.
In .net max size of any object 2GB. This is even for 64 bit machine. If your machine is 32 bit then this limit is very low.
In your case above rules get break so it will not work but if you have file size less than that then try to build your code against 64 bit and it will give your result.

Download it to a file:
using (WebClient webClient = new WebClient())
{
webClient.DownloadFile(
url,
path);
}
Of course it could be not handled by JSON libraries if it's too big to open in memory, but you can try to process it somehow if you don't have a choice and file must be so big.

Find the reason for FTP error?

How do I find the exact reason for the FTP 500 error - System.Net.WebException: The remote server returned an error: (500) Syntax error, command unrecognized
This happened while doing FTP transfers. All files below 300KB are transferred without error. But only one is transferred with error. How do I find out the cause ? Also, how do
i get the number associated with an FtpStatusCode ? Is the enumeration name for error 500 = CommandSyntaxError ???
This code (in catch section) did not help -
catch (WebException webex)
{
FtpWebResponse ftpWebResponse = (FtpWebResponse) webex.Response;
if(ftpWebResponse != null){
Stream stream = ftpWebResponse.GetResponseStream();
StreamReader streamReader = new StreamReader(stream, true);
string error = ftpWebResponse.ToString();
string respStream = "";
try
{
respStream = streamReader.ReadToEnd();
}
finally
{
streamReader.Close();
}
MessageBox.Show("to string " + error + Environment.NewLine +
"stream " + streamReader);
}
}

500 is the code from the FTP server. You can see details on FTP return codes here:
http://en.wikipedia.org/wiki/List_of_FTP_server_return_codes
"500 Series: Syntax error, command unrecognized and the requested action did not take place. This may include errors such as command line too long."
Things to check:
Is there a unicode or "weird" character in the file name?
Not all FTP servers support unicode. The easiest option in this case- if you're allowed to do it- is remove or replace any non-ASCII characters. Practically speaking, dealing with unicode is very hard with FTP, so if that solution is okay, it's the simplest.
Alternatively, you can check if your FTP server supports unicode using the raw FTP command "FEAT UTF8". Even so, you then need to send the raw command "OPTS UTF8 ON" to enable it. (My guess is that FTPWebRequest sends "OPTS UTF8 ON" automatically.)
Is the file path extremely long?
This could result in the server's path exceeding the character length limit (ex: imaging if the server's homedir is a really long path).

Yet another thread on The underlying connection was closed. A connection that was expected to be kept alive was closed by the server

I have seen lots of threads on this topic... but solutions on google are not working for me.
I am doing a POST operation using HttpWebRequest object and when I try to post a lot of data I get an error
The underlying connection was closed. A connection that was expected to be kept alive was closed by the server
Now I googled and I found three solutions
Set KeepAlive=False and set ProtocolVersion = HttpVersion10.
when I do this, there is no error... but somehow the data which I am posing doesn't reach the server. (so somehow it fails silently... without any error).
If I remove KeepAlive=false and Set ProtocolVersion = HttpVersion10. Then I can see that for small requests everything works fine.... but for large requests I get error of Underlying connection was closed.
I also found that some people solved the problem by moving to HttpClient instead of HttpWebRequest... but i think its only for .NET 4.5 but I must compile code for .NET 3.5 at a minimum.
Some people solved the problem by
ServicePoint sp = ServicePointManager.FindServicePoint(request.RequestUri);
sp.Expect100Continue = false;
once again, this threw no errors. but data was no committed.
So for me these solutions are not working.
Do you have any ideas?
Here is my code
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(completeUrl);
request.CookieContainer = Utility.GetSSOCookie(completeUrl);
request.Method = httpMethod;
request.Timeout = int.MaxValue;
Stream reqStream = null;
string output = null;
try {
if (String.IsNullOrEmpty(input) == false) {
byte[] buffer = Encoding.GetEncoding("UTF-8").GetBytes(
input
);
request.ContentLength = buffer.Length;
reqStream = request.GetRequestStream();
reqStream.Write(buffer, 0, buffer.Length);
}
using (WebResponse response = request.GetResponse()) {
using (StreamReader reader = new StreamReader(response.GetResponseStream())) {
output = reader.ReadToEnd();
}
}
}

This Exception can be missleading.
I got the same Exception in a scenario where a c# httpclient talks to an asp-mvc (v5) app on iis (v7). After i while it found out that the server runs in a recrusion-loop without throwing a stackoverflow or any log. The result was that the server response the disconection to the client. Maybe this could help someone running into this kind of error.

"An error occurred while parsing EntityName" while Loading an XmlDocument

I have written some code to parse RSS feeds for a ASP.NET C# application and it works fine for all RSS feeds that I have tried, until I tried Facebook.
My code fails at the last line below...
WebRequest request = WebRequest.Create(url);
WebResponse response = request.GetResponse();
Stream rss = response.GetResponseStream();
XmlDocument xml = new XmlDocument();
xml.Load(rss);
...with the error "An error occurred while parsing EntityName. Line 12, position 53."
It is hard to work out what is at thhat position of the XML file as the entire file is all in one line, but it is straight from Facebook and all characters appear to be encoded properly except possibly one character (♥).
I don't particularly want to rewrite my RSS parser to use a different method. Any suggestions for how to bypass this error? Is there a way of turning off checking of the file?

Look at the downloaded stream. It doesn't contain the RSS feed, but a HTML page with message about incompatible browser. That's because when downloading the URL like this, the user agent header is not set. If you do that, your code should work:
var request = (HttpWebRequest)WebRequest.Create(url);
request.UserAgent = "MyApplication";
var xml = new XmlDocument();
using (var response = request.GetResponse())
using (var rss = response.GetResponseStream())
{
xml.Load(rss);
}

Proper way of reading from a HttpWebResponse and .NET 4.0 Framework

We are currently experiencing a problem reading a ResponseStream that we have had no issues with in the past. Since adding .NET 4.0 Framwework to our server last night and assigning IIS to use the new framework we are experiencing a few different exceptions when trying to read the responseStream using the following statement (responseStream = httpResponse.GetResponseStream();). Everything up to that point works completely fine. So, I am looking for changes/improvements on how to read from the response. I have pasted the code below that we are using and the exceptions we are experiencing.
Our environment is:
.NET Framework 4.0
Windows Server 2003
THE CODE:
HttpWebResponse httpResponse;
Stream responseStream;
//Accept All Certificate Policy
ServicePointManager.ServerCertificateValidationCallback += delegate { return true; };
HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create(new Uri(theUrl));
httpRequest.Method = "POST";
httpRequest.KeepAlive = false;
httpRequest.Timeout = timeOut;
try
{
httpResponse = (HttpWebResponse)httpRequest.GetResponse();
responseStream = httpResponse.GetResponseStream();
}
THE EXCEPTIONS:
'httpResponse.GetResponseStream().Length' threw an exception of type 'System.NotSupportedException' long {System.NotSupportedException}
'httpResponse.GetResponseStream().Position' threw an exception of type 'System.NotSupportedException' long {System.NotSupportedException}
{"This stream does not support seek operations."} System.SystemException {System.NotSupportedException}
Regards,
Mike

You haven't shown the bit of code which tries to use the length - but basically, you aren't guaranteed that you'll know the length of the stream. The server may not have given a content-length header.
If you need to mess with the stream, it's probably best to just copy all the data into a MemoryStream. With .NET 4, that's really easy:
MemoryStream ms = new MemoryStream();
responseStream.CopyTo(ms);
If you don't actually need the length for anything (e.g. you just want to load the results as an XML document) then just read from the stream without calling Length.
Note that it's generally good practice to declare variables at the point of initialization if possible - and you should use using statements for the web response and the response stream.