We Keep Coding

C# Download File from FTP AS400

we currently have a *.BAT file that contains some FTP commands to download a file from our AS400 and save into a TEXT file. The BAT works fine and the text file will show the records inside the downloaded file one under the other.
Now, we wanted to get rid of this *.BAT file and use C# to download the file for us and save into a text file. The problem now is that the file we get contains all the records in ONE single line of string! they are no longer listed under each other.
here is the code we are using:
tpWebRequest request = default(FtpWebRequest);
FtpWebResponse response = default(FtpWebResponse);
StreamWriter writer = default(StreamWriter);
request = WebRequest.Create("*******URL******") as FtpWebRequest;
request.Credentials = new NetworkCredential("user", "pass");
request.Method = WebRequestMethods.Ftp.DownloadFile;
request.UseBinary = true;
response = request.GetResponse() as FtpWebResponse;
writer = new StreamWriter(Server.MapPath("/filename.txt"));
using (StreamReader reader = new StreamReader(response.GetResponseStream(), Encoding.GetEncoding(37))) //37 for IBM encoding
{
writer.WriteLine(reader.ReadToEnd());
}
writer.Close();
response.Close();
Any idea why we are getting this? and why the simple DOS FTP command work better than our code?
Thanks a lot! :)

ASCII mode will add record delimiters when downloading a physical file. It is the default transfer mode of most ftp clients.
request.UseBinary = false;
Specifying false causes the FtpWebRequest to send a "Type A" command to the server.
Data Transfer Methods
Transferring QSYS.LIB files

The problem might be simple: you read the whole document at once. You need to read every line seperately:
using(StreamReader sr = new StreamReader(fs))
{
while(!sr.EndOfStream)
{
Console.WriteLine(sr.ReadLine());
}
}

How to Determine if FTP Directory Exists

I wrote the following code to detect if a directory exists. The DirectoryExists method accepts either a fully qualified or relative path.
public bool DirectoryExists(string directory)
{
try
{
FtpWebRequest request = GetRequest(directory);
request.Method = WebRequestMethods.Ftp.ListDirectory;
using (FtpWebResponse response = request.GetResponse() as FtpWebResponse)
using (StreamReader sr = new StreamReader(response.GetResponseStream(), System.Text.Encoding.ASCII))
{
sr.ReadToEnd();
}
return true;
}
catch { }
return false;
}
protected FtpWebRequest GetRequest(string filename = "")
{
FtpWebRequest request = WebRequest.Create(_host.GetUrl(filename)) as FtpWebRequest;
request.Credentials = new NetworkCredential(Username, Password);
request.Proxy = null;
request.KeepAlive = false;
return request;
}
Note: _host.GetUrl(filename) returns the fully qualified path of the specified directory or filename. In my case, this is ftp://www.mydomain.com/Articles/controls/precisely-defining-kilobytes-megabytes-and-gigabytes.
This code has worked for many months. But all of a sudden it stopped working. Now, there is no exception raised in DirectoryExists when the directory does not exist. sr.ReadToEnd() simply returns an empty string.
I posted a similar question and it was suggested that I should always append / to the end of my path. I tried that, and thought I got an exception once, but it's not raising an exception now.
I don't know if the behavior changed on the FTP server I'm communicating with or what. Again, this worked fine when I wrote it and now it doesn't.
How can I determine whether or not an FTP directory exists?
EDIT:
Inspecting the response after calling ReadToEnd(), I see:
BannerMessage="220 Microsoft FTP Service\r\n"
ContentLength=-1
ExitMessage="221 Goodbye.\r\n"
StatusCode=ClosingData
StatusDescription="226 Transfer complete.\r\n"
WelcomeMessage="230 User logged in.\r\n"
UPDATE:
Ultimately, I see that most people have been recommending variations of what I was doing originally. This has lent weight to Hans Passant's suggestion that the issue lies with the server. I am attempting to get them to look at this but they seem a little baffled by the entire discussion. I know they are using a Microsoft server, and I'm skeptical I will be able to get a resolution from them.
If all else fails, I think the answer is to do a listing of the parent directory, which does require some extra work to handle cases such as when the directory in question is the root, and also cases when the parent doesn't exist.

FTP servers reply with a 550 message when you ask for a non existing directory/file. This 550 is translated to an exception in .NET.
Regarding the code you presented I don't see the reason to use a StreamReader in production. A simple modification is the following:
public bool DirectoryExists(string directory)
{
try
{
FtpWebRequest request = GetRequest(directory);
request.Method = WebRequestMethods.Ftp.ListDirectory;
return request.GetResponse() != null;
}
catch
{
return false;
}
}
I kept the exception handling as is (missing) but I suggest you to work on it as there are many and different cases to catch that are not related to ftp responses.
request.GetResponse() will generate an exception if the directory does not exist.
I tried it with the following paths for true returns:
ftp://ftp.mozilla.org/
ftp://ftp.mozilla.org/pub/
ftp://ftp.mozilla.org/pub/data/
ftp://ftp.mozilla.org/pub/data/bloat-reports/
The last one is an existing but empty directory
ftp://ftp.mozilla.org/pub/data/bloat-reports2/
returns false
There is a big but regarding trailing / in paths.
mozilla.org is a zero byte length file under pub directory
ftp://ftp.mozilla.org/pub/mozilla.org returns true
ftp://ftp.mozilla.org/pub/mozilla.org/ returned false and then true ?????
This behavior looks like with what you described. You can easily reproduce it on a browser. I suggest you to do the same with your own paths and check the contents of the directories. In case you do have empty files, remove them or replace their contents with a space, and finally check your code to ensure that you will not create new files or recreate them with zero byte length.
I hope that this helps.
UPDATE
No news, bad news! I assume that nothing of the above helped you. Maybe by going one level up in your path will solve the problem. The idea is to get a list of the parent directory and check it for the child name. If the parent path is valid it should always return non empty string. The code is the following:
static bool DirectoryExists(string directory)
{
try
{
directory = GetRequest(directory);
string
parent = directory.Substring(0, directory.TrimEnd('/').LastIndexOf('/') + 1),
child = directory.Substring(directory.TrimEnd('/').LastIndexOf('/') + 1).TrimEnd('/');
FtpWebRequest request = GetRequest(parent);
request.Method = WebRequestMethods.Ftp.ListDirectory;
using (FtpWebResponse response = request.GetResponse() as FtpWebResponse)
{
if (response == null)
return false;
string data = new StreamReader(response.GetResponseStream(), true).ReadToEnd();
return data.IndexOf(child, StringComparison.InvariantCultureIgnoreCase) >= 0;
}
}
catch
{
return false;
}
}
As you can see there is no need to worry about trailing slashes.
Warning: The code will not cover the case of a subdirectory and a filename with same names under the same path.

This might happen because FTP server software installed on your server.
Otherwise, you may want to take a look at response.StatusCode to see if there is any hints.
http://msdn.microsoft.com/en-us/library/system.net.ftpstatuscode(v=vs.110).aspx
Finally, if all of that doesn't work, try to write a dumb text file on that un-existed directory.

I had similar code that would deploy items via FTP to an IIS 6/Windows 2003 Server. This worked fine until I pointed it at an IIS7/Windows 2008 server. I started seeing the exact same behavior you did. I debugged it and ultimately changed my code to the following. This is a direct snippet from my code. I can include the source from the methods it calls if you like, but I think you understand those operations.
try
{
//Get a recursive FTP Listing
items = GetFtpListing(target, true);
}
catch(WebException e)
{
if (string.Equals(e.Message, "The remote server returned an error: (550) File unavailable (e.g., file not found, no access)."))
{
CreateRemoteDirectory(target);
items = GetFtpListing(target, true);
}
else
{
throw e;
}
}
relevant section of GetFtpListing
List<string> ftpRawList = new List<string>();
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(path);
request.Timeout = 60000;
request.ReadWriteTimeout = 60000;
request.Credentials = this.credentials;
request.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
using (FtpWebResponse response = (FtpWebResponse)request.GetResponse())
{
Stream data = response.GetResponseStream();
using (StreamReader reader = new StreamReader(data))
{
string responseString = reader.ReadToEnd();
ftpRawList.AddRange(responseString.Split(new char[2] { '\r', '\n' }, StringSplitOptions.RemoveEmptyEntries));
}

I use the below to check if a ftp connection credentials are valid. You may use the same to check if directory exists. Returns true if url, username, and password are correct.
URL: you specify your directory path in here.
user: ftp username
password: ftp password
public static bool isValidConnection(string url, string user, string password, ref string errorMessage = "")
{
try {
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(url);
request.Method = WebRequestMethods.Ftp.ListDirectory;
request.Credentials = new NetworkCredential(user, password);
request.KeepAlive = false;
request.UsePassive = true;
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
response.Close();
} catch (WebException ex) {
errorMessage = ex.Message;
return false;
}
return true;
}

One way is to list the directory content as present in many solutions, but in my case i also have to read data content so the server checks the directory. If the directory is not present, an webException is throw that should be interpreted to filter from unexpected errors.
Here is my solution:
bool directoryExists(string path)
{
bool? exists = null;
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(path);
request.Method = WebRequestMethods.Ftp.ListDirectory;
request.Credentials = new NetworkCredential("username", "*****");
FtpWebResponse response = null;
try
{
response = (FtpWebResponse)request.GetResponse();
using (StreamReader sr = new StreamReader(response.GetResponseStream()))
{
string str = sr.ReadLine(); //Just needs to read one line to fire check on server
sr.Close();
return true;
}
}
catch (WebException ex)
{
var r = (FtpWebResponse)ex.Response;
if (r.StatusCode ==
FtpStatusCode.ActionNotTakenFileUnavailable)
{
//Does not exist
return false;
}
throw; //if error is not related to directory existence then the exception needs to be treated above.
}
finally
{
if (response != null)
response.Close();
}
return false;
}
The FTP server i am targeting i know to be a windows server but i don't have access to the version information.

Reversing a tsv/csv file or reading only last line using asp.net

I am pretty much stuck on a problem from last few days. I have a file while is located on a remote server can be access by using userId and password. Well no problem in accessing.
Problem is I have around 150 of them. and each of them is of variable size minimum is 2 MB and max is 3 MB.
I have to read them one by one and read last row/line data from them. I am doing it in my current code.
The main problem is it is taking too much time since it is reading files from top to bottom.
public bool TEst(string ControlId, string FileName, long offset)
{
// The serverUri parameter should use the ftp:// scheme.
// It identifies the server file that is to be downloaded
// Example: ftp://contoso.com/someFile.txt.
// The fileName parameter identifies the local file.
//The serverUri parameter identifies the remote file.
// The offset parameter specifies where in the server file to start reading data.
Uri serverUri;
String ftpserver = "ftp://xxx.xxx.xx.xxx/"+FileName;
serverUri = new Uri(ftpserver);
if (serverUri.Scheme != Uri.UriSchemeFtp)
{
return false;
}
// Get the object used to communicate with the server.
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(serverUri);
request.Credentials = new NetworkCredential("test", "test");
request.Method = WebRequestMethods.Ftp.DownloadFile;
//request.Method = WebRequestMethods.Ftp.DownloadFile;
request.ContentOffset = offset;
FtpWebResponse response = null;
try
{
response = (FtpWebResponse)request.GetResponse();
// long Size = response.ContentLength;
}
catch (WebException e)
{
Console.WriteLine(e.Status);
Console.WriteLine(e.Message);
return false;
}
// Get the data stream from the response.
Stream newFile = response.GetResponseStream();
// Use a StreamReader to simplify reading the response data.
StreamReader reader = new StreamReader(newFile);
string newFileData = reader.ReadToEnd();
// Append the response data to the local file
// using a StreamWriter.
string[] parser = newFileData.Split('\t');
string strID = parser[parser.Length - 5];
string strName = parser[parser.Length - 3];
string strStatus = parser[parser.Length-1];
if (strStatus.Trim().ToLower() != "suspect")
{
HtmlTableCell control = (HtmlTableCell)this.FindControl(ControlId);
control.InnerHtml = strName.Split('.')[0];
}
else
{
HtmlTableCell control = (HtmlTableCell)this.FindControl(ControlId);
control.InnerHtml = "S";
}
// Display the status description.
// Cleanup.
reader.Close();
response.Close();
//Console.WriteLine("Download restart - status: {0}", response.StatusDescription);
return true;
}
Threading:
protected void Page_Load(object sender, EventArgs e)
{
new Task(()=>this.TEst("controlid1", "file1.tsv", 261454)).Start();
new Task(()=>this.TEst1("controlid2", "file2.tsv", 261454)).Start();
}

FTP is not capable of seeking a file to read only the last few lines. Reference: FTP Commands You'll have to coordinate with the developers and owners of the remote ftp server and ask them make an additional file containing the data you need.
Example Ask owners of remote ftp server to create for each of the files a [filename]_lastrow file that contains the last row of the files. Your program would then operate on the [filename]_lastrow files. You'll probably be pleasantly surprised with an accommodating answer of "Ok we can do that for you"
If the ftp server can't be changed ask for a database connection.

You can also download all your files in parallel and start popping them into a queue for parsing when they are done rather than doing this process synchronously. If the ftp server can handle more connections, use as many as would be reasonable for the scenario. Parsing can be done in parallel too.
More reading: System.Threading.Tasks
It's kinda buried, but I placed a comment in your original answer. This SO question leads to this blog post which has some awesome code you can draw from.

Rather than your while loop you can skip directly to the end of the Stream by using Seek. You then want to work your way backwards though the stream until you find the first new line variable. This post should give you everything your need to know.
Get last 10 lines of very large text file > 10GB

FtpWebRequest includes the ContentOffset property. Find/choose a way to keep the offset of the last line (locally or remotely - ie by uploading a 4 byte file to ftp). This is the fastest way to do it and the most optimal for network traffic.
More information about FtpWebRequest can be found at MSDN

Checking if multiple files exists on FTP server (C#)

I need a way to check if multiple files exist in a given FTP path. I will have a list, which lists all the file names to check, and I will need to check if all these exist on the server, and return errors for ones that don't. How easy is this?
Thanks

The safest approach would be to retrieve a list of files/directories per directory and parse that list.
// Get the object used to communicate with the server.
var request = WebRequest.Create(url);
request.Credentials = new NetworkCredential(username, password);
request.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
try
{
using(var response = request.GetResponse())
{
using(var stream = response.GetResponseStream())
{
using(var reader = new StreamReader(stream))
{
while(reader.Peek() >= 0)
{
var line = reader.ReadLine();
// check if this is a file or directory, filter list etc..
}
}
}
}
}
catch
{
}
Another- easier - option would be to try to retrieve the files DateTimestamp and catch the exception if the file doesn't exist. You should check the exception since one could be thrown for a different reason.
var request = WebRequest.Create(url);
request.Credentials = new NetworkCredential(username, password);
request.Method = WebRequestMethods.Ftp.GetDateTimestamp;
try
{
using(var response = (FtpWebResponse)request.GetResponse())
{
// file exists
}
}
catch(WebException e)
{
// file probably doesn't exits
}

Well, if you have access to the server, you could write your script there and then just request that script and thus only have to make one server request. Otherwise, you'll just need to check each file, one by one.

How to check if a file exists on a server using c# and the WebClient class

In my application I use the WebClient class to download files from a Webserver by simply calling the DownloadFile method. Now I need to check whether a certain file exists prior to downloading it (or in case I just want to make sure that it exists). I've got two questions with that:
What is the best way to check whether a file exists on a server without transfering to much data across the wire? (It's quite a huge number of files I need to check)
Is there a way to get the size of a given remote file without downloading it?
Thanks in advance!

WebClient is fairly limited; if you switch to using WebRequest, then you gain the ability to send an HTTP HEAD request. When you issue the request, you should either get an error (if the file is missing), or a WebResponse with a valid ContentLength property.
Edit: Example code:
WebRequest request = WebRequest.Create(new Uri("http://www.example.com/"));
request.Method = "HEAD";
using(WebResponse response = request.GetResponse()) {
Console.WriteLine("{0} {1}", response.ContentLength, response.ContentType);
}

When you request file using the WebClient Class, the 404 Error (File Not Found) will lead to an exception. Best way is to handle that exception and use a flag which can be set to see if the file exists or not.
The example code goes as follows:
System.Net.HttpWebRequest request = null;
System.Net.HttpWebResponse response = null;
request = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create("www.example.com/somepath");
request.Timeout = 30000;
try
{
response = (System.Net.HttpWebResponse)request.GetResponse();
flag = 1;
}
catch
{
flag = -1;
}
if (flag==1)
{
Console.WriteLine("File Found!!!");
}
else
{
Console.WriteLine("File Not Found!!!");
}
You can put your code in respective if blocks.
Hope it helps!

What is the best way to check whether a file exists on a server
without transfering to much data across the wire?
You can test with WebClient.OpenRead to open the file stream without reading all the file bytes:
using (var client = new WebClient())
{
Stream stream = client.OpenRead(url);
// ^ throws System.Net.WebException: 'Could not find file...' if file is not present
stream.Close();
}
This will indicate if the file exists at the remote location or not.
To fully read the file stream, you would do:
using (var client = new WebClient())
{
Stream stream = client.OpenRead(url);
StreamReader sr = new StreamReader(stream);
Console.WriteLine(sr.ReadToEnd());
stream.Close();
}

In case anyone stuck with ssl certificate issue
ServicePointManager.ServerCertificateValidationCallback = new RemoteCertificateValidationCallback
(
delegate { return true; }
);
WebRequest request = WebRequest.Create(new Uri("http://.com/flower.zip"));
request.Method = "HEAD";
using (WebResponse response = request.GetResponse())
{
Console.WriteLine("{0} {1}", response.ContentLength, response.ContentType);
}