We Keep Coding

C# Large JSON to string causes out of memory exception

I'm trying to download very large JSON file. However, I keep getting an error message:
"An unhandled exception of type 'System.OutOfMemoryException' occurred in mscorlib.dll"
{The function evaluation was disabled because of an out of memory exception.}
Any tips how I can download this large JSON filet? I have tried to use string and StringBuilder but no luck.
Here is my code:
public static string DownloadJSON(string url)
{
try
{
String json = new WebClient().DownloadString(url); // This part fails!
return json;
}
catch (Exception)
{
throw;
}
}
I have created console application. I have tried this code with smaller JSON file and it worked.
My idea is later to split this larger JSON file and put it to database. However I need to encode it before I can put it to database. I have not write yet database part or anything else, because downloading this big JSON causes problems. I don't need it as a stream, but that was my example way how I made encoding. I need to encode it because data have special characters like å.
I tried also this but same problem:
var http = (HttpWebRequest)WebRequest.Create(url);
var response = http.GetResponse();
var stream = response.GetResponseStream();
var sr = new StreamReader(stream);
var content = sr.ReadToEnd();

I assume that you have very very large response. It is better to process stream. Now comes to point that cause outofmemoryexcetion.
In .net max size of any object 2GB. This is even for 64 bit machine. If your machine is 32 bit then this limit is very low.
In your case above rules get break so it will not work but if you have file size less than that then try to build your code against 64 bit and it will give your result.

Download it to a file:
using (WebClient webClient = new WebClient())
{
webClient.DownloadFile(
url,
path);
}
Of course it could be not handled by JSON libraries if it's too big to open in memory, but you can try to process it somehow if you don't have a choice and file must be so big.

Can't connect to FTP: (553) File name not allowed

I need to FTP a file to a directory. In .Net I have to use a file on the destination folder to create a connection so I manually put Blank.dat on the server using FTP. I checked the access (ls -l) and it is -rw-r--r--. But when I attempt to connect to the FTP folder I get: "The remote server returned an error: (553) File name not allowed" back from the server. The research I have done says that this may arrise from a permissions issue but as I have said I have permissions to view the file and can run ls from the folder. What other reasons could cause this issue and is there a way to connect to the folder without having to specify a file?
byte[] buffer;
Stream reqStream;
FileStream stream;
FtpWebResponse response;
FtpWebRequest request = (FtpWebRequest)FtpWebRequest.Create(new Uri(string.Format("ftp://{0}/{1}", SRV, DIR)));
request.Method = WebRequestMethods.Ftp.UploadFile;
request.Credentials = new NetworkCredential(UID, PASS);
request.UseBinary = true;
request.Timeout = 60000 * 2;
for (int fl = 0; fl < files.Length; fl++)
{
request.KeepAlive = (files.Length != fl);
stream = File.OpenRead(Path.Combine(dir, files[fl]));
reqStream = request.GetRequestStream();
buffer = new byte[4096 * 2];
int nRead = 0;
while ((nRead = stream.Read(buffer, 0, buffer.Length)) != 0)
{
reqStream.Write(buffer, 0, nRead);
}
stream.Close();
reqStream.Close();
response = (FtpWebResponse)request.GetResponse();
response.Close();
}

Although replying to an old post just thought it might help someone.
When you create your ftp url make sure you are not including the default directory for that login.
for example this was the path which I was specifying and i was getting the exception 553 FileName not allowed exception
ftp://myftpip/gold/central_p2/inbound/article_list/jobs/abc.txt
The login which i used had the default directory gold/central_p2.so the supplied url became invalid as it was trying to locate the whole path in the default directory.I amended my url string accordingly and was able to get rid of the exception.
my amended url looked like
ftp://myftpip/inbound/article_list/jobs/abc.txt
Thanks,
Sab

This may help for Linux FTP server.
So, Linux FTP servers unlike IIS don't have common FTP root directory. Instead, when you log on to FTP server under some user's credentials, this user's root directory is used. So FTP directory hierarchy starts from /root/ for root user and from /home/username for others.
So, if you need to query a file not relative to user account home directory, but relative to file system root, add an extra / after server name. Resulting URL will look like:
ftp://servername.net//var/lalala

You must be careful with names and paths:
string FTP_Server = #"ftp://ftp.computersoft.com//JohnSmith/";
string myFile="text1.txt";
string myDir=#"D:/Texts/Temp/";
if you are sending to ftp.computersoft.com/JohnSmith a file caled text1.txt located at d:/texts/temp
then
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(FTP_Server+myFile);
request.Method = WebRequestMethods.Ftp.UploadFile;
request.Credentials = new NetworkCredential(FTP_User, FTP_Password);
StreamReader sourceStream = new StreamReader(TempDir+myFile);
byte[] fileContents = Encoding.UTF8.GetBytes(sourceStream.ReadToEnd());
sourceStream.Close();
Stream requestStream = request.GetRequestStream();
requestStream.Write(fileContents, 0, fileContents.Length);
requestStream.Close();
notice that at one moment you use as destination
ftp://ftp.computersoft.com//JohnSmith/text1.txt
which contains not only directory but the new file name at FTP server as well (which in general can be different than the name of file on you hard drive)
and at other place you use as source
D:/Texts/Temp/text1.txt

your directory has access limit.
delete your directory and then create again with this code:
//create folder
FtpWebRequest request = (FtpWebRequest)FtpWebRequest.Create("ftp://mpy1.vvs.ir/Subs/sub2");
request.Method = WebRequestMethods.Ftp.MakeDirectory;
request.Credentials = new NetworkCredential(username, password);
request.UsePassive = true;
request.UseBinary = true;
request.KeepAlive = true ;
using (var resp = (FtpWebResponse)request.GetResponse())
{
}

I saw something similar to this a while back, it turned out to be the fact that I was trying to connect to an internal iis ftp server that was secured using Active Directory.
In my network credentials I was using new NetworkCredential(#"domain\user", "password"); and that was failing. Switching to new NetworkCredential("user", "password", "domain"); worked for me.

I hope this will be helpful for someone
if you are using LINUX server, replace your request path from
FtpWebRequest req= (FtpWebRequest)WebRequest.Create(#"ftp://yourdomain.com//yourpath/" + filename);
to
FtpWebRequest req= (FtpWebRequest)WebRequest.Create(#"ftp://yourdomain.com//public_html/folderpath/" + filename);
the folder path is how you see in the server(ex: cpanel)

Although it's an older post I thought it would be good to share my experience. I came faced the same problem however I solved it myself. The main 2 reasons of this problem is Error in path (if your permissions are correct) happens when your ftp path is wrong. Without seeing your path it is impossible to say what's wrong but one must remember the things
a. Unlike browser FTP doesn't accept some special characters like ~
b. If you have several user accounts under same IP, do not include username or the word "home" in path
c. Don't forget to include "public_html" in the path (normally you need to access the contents of public_html only) otherwise you may end up in a bottomless pit

Another reason for this error could be that the FTP server is case sensitive. That took me good while to figure out.

I had this problem when I tried to write to the FTP site's root directory pro grammatically. When I repeated the operation manually, the FTP automatically rerouted me to a sub-directory and completed the write operation. I then changed my code to prefix the target filename with the sub-directory and the operation was successful.

Mine was as simple as a file name collision. A previous file hadn't been sent to an archive folder so we tried to send it again. Got the 553 because it wouldn't overwrite the existing file.

Check disk space on the remote server first.
I had the same issue and found out it was because the remote server i was attempting to upload files to had run out of disk space on the partition or filessytem.

To whom it concerns...
I've been stuck for a long time with this problem.
I tried to upload a file onto my web server like that:
Say that my domain is www.mydomain.com.
I wanted to upload to a subdomain which is order.mydomain.com, so I've used:
FtpWebRequest ftpReq = (FtpWebRequest)WebRequest.Create($#"ftp://order.mydomain.com//uploads/{FileName}");
After many tries and getting this 553 error, I found out that I must make the FTP request refer to the main domain not the sub domain and to include the subdomain as a subfolder (which is normally created when creating subdomains).
As I've created my subdomain subfolder out of the public_html (at the root), so I've changed the FTP Request to:
FtpWebRequest ftpReq = (FtpWebRequest)WebRequest.Create($#"ftp://www.mydomain.com//order.mydomain.com//uploads/{FileName}");
And it finally worked.

C#: HttpListener Error Serving Content

I have implemented something similar to this
only real difference is
string filename = context.Request.RawUrl.Replace("/", "\\").Remove(0,1);
string path = Uri.UnescapeDataString(Path.Combine(_baseFolder, filename));
so that I can traverse to subdirectories. This works great for webpages and other text file types but when trying to serve up media content I get the exception
HttpListenerException: The I/O
operation has been aborted because of
either a thread exit or an application
request
Followed by
InvalidOperationException: Cannot close stream until all bytes are written.
In the using statement.
Any suggestions on how to handle this or stop these exceptions?
Thanks

I should mention that I am using Google Chrome for my browser (Google Chrome doesn't seem to care about the MIME types, when it sees audio it will try to use it like it's in a HTML5 player), but this is also applicable if you are trying to host media content in a page.
Anyways, I was inspecting my headers with fiddler and noticed that Chrome passes 3 requests to the server. I started playing with other browsers and noticed they did not do this, but depending on the browser and what I had hard coded as the MIME type I would either get a page of crazy text, or a download of the file.
On further inspection I noticed that chrome would first request the file. Then request the file again with a few different headers most notably the range header. The first one with byte=0- then the next with a different size depending on how large the file was (more than 3 requests can be made depending how large the file is).
So there was the problem. Chrome will first ask for the file. Once seeing the type it would send another request which seems to me looking for how large the file is (byte=0-) then another one asking for the second half of the file or something similar to allow for a sort of streaming experienced when using HTML5. I coded something quickly up to handle MIME types and threw a HTML5 page together with the audio component and found that other browsers also do this (except IE)
So here is a quick solution and I no longer get these errors
string range = context.Request.Headers["Range"];
int rangeBegin = 0;
int rangeEnd = msg.Length;
if (range != null)
{
string[] byteRange = range.Replace("bytes=", "").Split('-');
Int32.TryParse(byteRange[0], out rangeBegin);
if (byteRange.Length > 1 && !string.IsNullOrEmpty(byteRange[1]))
{
Int32.TryParse(byteRange[1], out rangeEnd);
}
}
context.Response.ContentLength64 = rangeEnd - rangeBegin;
using (Stream s = context.Response.OutputStream)
{
s.Write(msg, rangeBegin, rangeEnd - rangeBegin);
}

Try:
using (Stream s = context.Response.OutputStream)
{
s.Write(msg, 0, msg.Length);
s.Flush()
}

Retrieving creation date of file (FTP)

I'm using the System.Net.FtpWebRequest class and my code is as follows:
FtpWebRequest request = (FtpWebRequest)WebRequest.Create("ftp://example.com/folder");
request.Method = WebRequestMethods.Ftp.ListDirectory;
request.Credentials = new NetworkCredential("username", "password");
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);
string names = reader.ReadToEnd();
reader.Close();
response.Close();
This is based off of the examples provided on MSDN but I couldn't find anything more detailed.
I'm storing all the filenames in the folder in names but how can I now iterate through each of those and retrieve their dates? I want to retrieve the dates so I can find the newest files. Thanks.

This seems to work just fine
http://msdn.microsoft.com/en-us/library/system.net.ftpwebresponse.lastmodified(v=VS.90).aspx
FtpWebRequest request = (FtpWebRequest)WebRequest.Create (serverUri);
request.Method = WebRequestMethods.Ftp.GetDateTimestamp;
FtpWebResponse response = (FtpWebResponse)request.GetResponse ();
Console.WriteLine ("{0} {1}",serverUri,response.LastModified);

WebRequestMethods.Ftp.ListDirectory returns a "short listing" of all the files in an FTP directory. This type of listing is only going to provide file names - not additional details on the file (like permissions or last modified date).
Use WebRequestMethods.Ftp.ListDirectoryDetails instead. This method will return a long listing of files on the FTP server. Once you've retrieved this list into the names variable, you can split the names variable into an array based on an end of line character. This will result in each array element being a file (or directory) name listing that includes the permissions, last modified date owner, etc...
At this point, you can iterate over this array, examine the last modified date for each file, and decide whether to download the file.
I hope this helps!!

Unfortunately, there's no really reliable and efficient way to retrieve timestamps using features offered by .NET framework, as it does not support the FTP MLSD command. The MLSD command provides a listing of remote directory in a standardized machine-readable format. The command and the format is standardized by RFC 3659.
Alternatives you can use, that are supported by .NET framework:
ListDirectoryDetails method (the FTP LIST command) to retrieve details of all files in a directory and then you deal with FTP server specific format of the details (*nix format similar to the ls *nix command is the most common, a drawback is that the format may change over time, as for newer files "May 8 17:48" format is used and for older files "Oct 18 2009" format is used).
DOS/Windows format: C# class to parse WebRequestMethods.Ftp.ListDirectoryDetails FTP response
*nix format: Parsing FtpWebRequest ListDirectoryDetails line
GetDateTimestamp method (the FTP MDTM command) to individually retrieve timestamps for each file. An advantage is that the response is standardized by RFC 3659 to YYYYMMDDHHMMSS[.sss]. A disadvantage is that you have to send a separate request for each file, what can be quite inefficient.
const string uri = "ftp://example.com/remote/path/file.txt";
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(uri);
request.Method = WebRequestMethods.Ftp.GetDateTimestamp;
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
Console.WriteLine("{0} {1}", uri, response.LastModified);
Alternatively you can use a 3rd party FTP client implementation that supports the modern MLSD command.
For example WinSCP .NET assembly supports that.
There's even an example for your specific task: Downloading the most recent file.
The example is for PowerShell and the SFTP, but translates to C# and the FTP easily:
// Setup session options
SessionOptions sessionOptions = new SessionOptions
{
Protocol = Protocol.Ftp,
HostName = "example.com",
UserName = "username",
Password = "password",
};
using (Session session = new Session())
{
// Connect
session.Open(sessionOptions);
// Get list of files in the directory
string remotePath = "/remote/path/";
RemoteDirectoryInfo directoryInfo = session.ListDirectory(remotePath);
// Select the most recent file
RemoteFileInfo latest =
directoryInfo.Files
.OrderByDescending(file => file.LastWriteTime)
.First();
// Download the selected file
string localPath = #"C:\local\path\";
string sourcePath = RemotePath.EscapeFileMask(remotePath + latest.Name);
session.GetFiles(sourcePath, localPath).Check();
}
(I'm the author of WinSCP)

First you will need to break apart the names using String.Split on the filename delimiter. Then iterate through all of the strings and navigate the directories

Is there any issue with below code?

I'm creating a Windows application which reads XML file from given server. This application has installed in about 30 clients. Maybe they will call this function at the same time.
My question:
Will any problem occur if several user call this method at same time?
public string GetXmlInnerText()
{
FtpWebRequest tmpReq = null;
System.Net.WebResponse tmpRes = null;
try
{
if (Settings.Default.Internal)
tmpReq = (FtpWebRequest)FtpWebRequest.Create("ftp://<IPhere>/XMLData.xml");
else
tmpReq = (FtpWebRequest)FtpWebRequest.Create("ftp://<IPhere>/XMLData.xml");
tmpReq.Credentials = new System.Net.NetworkCredential("userName", "password");
tmpReq.KeepAlive = false;
tmpRes = tmpReq.GetResponse();
}
catch (Exception ex)
{
//------
}
string fileContents = null;
using (System.IO.Stream tmpStream = tmpRes.GetResponseStream())
{
using (System.IO.TextReader tmpReader = new System.IO.StreamReader(tmpStream))
{
fileContents = tmpReader.ReadToEnd();
}
}
return fileContents;
}
thanks

One problem - you're not disposing of the WebResponse. It implements IDisposable, so you should use a using statement. With your current structuring, that's not terribly easy to do - you should consider restructuring your try/catch blocks appropriately.
Further, StreamReader uses UTF-8 by default - if your XML documents aren't encoded in UTF-8, you could have problems. If it's an XML document, why not load it via XmlReader.Create(Stream) or something similar? That will handle the encoding for you.

I don't see a problem as you are only reading from a file. The only problem could be the server prohibiting concurrent access to a user account and restricts how many connections are allowed at the same time. In that case, you might be better of with a webservice or script (eg. php) delivering the xml via HTTP, not FTP.

If you're wondering about multiple clients accessing the FTP server at once, it will depend on how the FTP server is set up.
Some will be set up to only allow 2 or 3 clients at once, whereas some will allow (almost) as many as you could ever need.
If the FTP server causes troubles, you could serve it through a HTTP server instead.