We Keep Coding

Bulk download by date modified - 505 File unavailable

I'm trying to create a script that download multiple text documents from an FTP server by date modified that are not more than a day old:
if (_datemodified > DateTime.Now.AddDays(-2));
But when I'm running my code it manages to download 200-300 files then times out with
" System.Net.WebException: The remote server returned an error: (550)
File unavailable (e.g., file not found, no access)."
Here's a snippet of the code that the error points to:
static void Checkdatemodified(string url, string savePath)
{
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(url);
request.Method = WebRequestMethods.Ftp.GetDateTimestamp;
request.Credentials = new NetworkCredential("user", "******");
request.UseBinary = true;
DateTime temp;
using (FtpWebResponse response = (FtpWebResponse)request.GetResponse())
{
temp = response.LastModified;
}
FtpWebRequest request2 = (FtpWebRequest)WebRequest.Create(url);
request2.Credentials = new NetworkCredential("user", "******");
request2.Method = WebRequestMethods.Ftp.DownloadFile;
request2.UseBinary = true;
if (temp > DateTime.Now.AddDays(-2))
{
using (FtpWebResponse response2 = (FtpWebResponse)request2.GetResponse())
{
// DateTime temp = response.LastModified;
using (Stream rs = response2.GetResponseStream())
{
using (FileStream ws = new FileStream(savePath, FileMode.Create))
{
byte[] buffer = new byte[2048];
int bytesRead = rs.Read(buffer, 0, buffer.Length);
while (bytesRead > 0)
{
ws.Write(buffer, 0, bytesRead);
bytesRead = rs.Read(buffer, 0, buffer.Length);
}
}
// DateTime temp = Checkdatemodified(url);
System.IO.File.SetLastWriteTime(savePath, temp);
}
}
}
And here's the code that calls the function
//Create new FTPClient in memory
using (var ftpClient = new FtpClient())
{
// Sets the location of the FTP folder we will use in the request
FtpWebRequest request = (FtpWebRequest)WebRequest.Create("ftp://(ftp folder link)");
//Set the method it will use to list the items in the directory
request.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
// Set the credentials used to get onto the FTP
request.Credentials = new NetworkCredential("user", "******");
//Processes request
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
//Stream reader reads each character of a line
Stream responseStream = response.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);
//Creates new list of strings and a new string variable
List<string> lines = new List<string>();
List<string> lines2 = new List<string>();
string line;
//For each line the streamreader reads
//Split the line into an array and take the last entry and add this to the list
while ((line = reader.ReadLine()) != null)
{
string[] temp = line.Split(null);
string temp2 = temp[temp.Length - 1];
lines.Add(temp2);
}
//Close everything
responseStream.Close();
reader.Close();
//For each each line in the list starting at number 2(the first 2 are just the directory names)
for (int i = 2; i < lines.Count - 1; i++)
{
//Set temp strings. 1 for the file path + file name and the other for the location to download to
string temp = "ftp://(ftp folder)" + lines[i];
string temp2 = #"\\(server folder location)" + lines[i];
//If the date modified in the FTP between now and 2 days old then download the fil
if (System.IO.File.Exists(temp2))
{
System.IO.File.Delete(temp2);
}
Checkdatemodified(temp, temp2);
//File.AppendAllText(#"C:\TestFolder\error.txt", value);
}
// DownloadFtpFile(temp, temp2);
}
}
I feel the issue is that I'm using 3 different FtpWebRequest per download for 300+ files which are
1) Get a list of directory
2) Find date modified
3) Download the files
Or it could be an issue with the list that holds the file names.
Does anyone know why this isn't working? Or if there's a better way of bulk downloading ftp files by datemodified
P.S - Code is a bit messy and needs to be cleaned up but I'm just trying to make it work first before hand

How to read files one by one from ftp server using c#?

I am getting files from ftp folder like this.
request = (FtpWebRequest)FtpWebRequest.Create(new Uri("ftp://ipaddress/foldername"));
request.UseBinary = true;
request.Credentials = new NetworkCredential("username", "password");
request.Method = WebRequestMethods.Ftp.ListDirectory;
request.UseBinary = true;
WebResponse response = request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string line = reader.ReadLine();
while (line != null)
{
result.Append(line);
result.Append("\n");
line = reader.ReadLine();
}
// to remove the trailing '\n'
result.Remove(result.ToString().LastIndexOf('\n'), 1);
reader.Close();
response.Close();
string[] results = result.ToString().Split('\n');
foreach (string filename in results)
{
connetionString = "Data Source=testservername;Initial Catalog=testdb;User ID=sa;Password=sa";
connection = new SqlConnection(connetionString);
FtpWebRequest request1 = (FtpWebRequest)FtpWebRequest.Create(new Uri("ftp://servername/" + filename));
request1.Method = WebRequestMethods.Ftp.DownloadFile;
request1.UseBinary = true;
request1.Credentials = new NetworkCredential("username", "password");
responses = (FtpWebResponse)request1.GetResponse();
StreamReader reader1 = new StreamReader(responses.GetResponseStream());
string xml = reader1.ReadToEnd();
ds.ReadXml(new XmlTextReader(new StringReader(xml)));
int i = 0;
connection.Open();
for (i = 0; i <= ds.Tables[1].Rows.Count - 1 ; i++)
{
sql = "insert into Details values('" + ds.Tables[1].Rows[i].ItemArray[0]
+ "')";
command = new SqlCommand(sql, connection);
adpter.InsertCommand = command;
adpter.InsertCommand.ExecuteNonQuery();
}
FtpWebRequest requestDelete = (FtpWebRequest)WebRequest.Create("ftp://sername/" + filename);
requestDelete.Method = WebRequestMethods.Ftp.DeleteFile;
requestDelete.UseBinary = true;
requestDelete.Credentials = new NetworkCredential("username", "password");
responseDelete = (FtpWebResponse)requestDelete.GetResponse();
try
{
string res = responseDelete.StatusDescription;
if (res == "250 Delete operation successful.")
{
MessageBox.Show("Done......");
}
}
catch (Exception ex)
{
System.Windows.Forms.MessageBox.Show(ex.Message);
}
}
responseDelete.Close();
connection.Close();
responses.Close();
}
catch (Exception ex)
{
System.Windows.Forms.MessageBox.Show(ex.Message);
}
}
Hi need to get files from ftp folder and read that xml files one by one and save it as records in my Databse after that need to delete that file ftp folder.
if for testing i placed two files in ftp folder first files reading successfully .but getting error for reading second file like "Object Reference is not set instance of object "in the below line.
ds.ReadXml(new XmlTextReader(new StringReader(xml)));
please tell me any thing wrong in my code.i am sending ftp request multiple times it is good or suggest me by seeing above lines of code how to do that?

How to ZIP files in FTP Server Directory

In my Application I need to download some files from the FTP server..I coded to download one by one via for loop and it works fine.My Problem is it is very slow process coz for each file it will gives credential. So if I can zip those files in FTP server it can be Download quickly..If there is a way pls guide me..
here My Code:
Blockquote
private void button1_Click(object sender, EventArgs e)
{
string[] list = GetFileList();
DownloadFile(list, "e:\\sample\\");
}
private String[] GetFileList()
{
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(new Uri(_remoteHost));
request.Credentials = new NetworkCredential(_remoteUser, _remotePass);
request.Method = WebRequestMethods.Ftp.ListDirectory;
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);
string FileNames = reader.ReadToEnd();
string[] Files = Regex.Split(FileNames, "\r\n");
return Files;
}
private void DownloadFile(string[] fileList, string destination)
{
for (int i = 2; i <= fileList.Length - 1; i++)
{
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(_remoteHost + fileList[i]);
request.Method = WebRequestMethods.Ftp.DownloadFile;
request.Credentials = new NetworkCredential(_remoteUser, _remotePass);
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);
StreamWriter writer = new StreamWriter(destination + fileList[i]);
writer.Write(reader.ReadToEnd());
writer.Close();
reader.Close();
response.Close();
}
}
Blockquote

Why dont you initiate your FtpWebRequest once with the Username and Password and then use it in a foreach Loop for all files instead of creating a new one each round?
Another way would be to use a 3rd party tool (like: http://www.rebex.net/ftp-ssl.net/) which may be faster, but its something you will have to test, take a looka t this: https://stackoverflow.com/a/2343689/395659
A 3rd way could be to let a job running on the server which creates the ZIP file once in a period and you download it as a Zip file as mentioned in your Question.

You can re-use the credentials, which might speed you up...
private void DownloadFile(string[] fileList, string destination)
{
var myCred = new NetworkCredential(_remoteUser, _remotePass);
for (int i = 2; i <= fileList.Length - 1; i++)
{
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(_remoteHost + fileList[i]);
request.Method = WebRequestMethods.Ftp.DownloadFile;
request.Credentials = myCred;
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);
StreamWriter writer = new StreamWriter(destination + fileList[i]);
writer.Write(reader.ReadToEnd());
writer.Close();
reader.Close();
response.Close();
}
}

If you are using .NET 3.0 or above you can make use of ZipPackage Class present in the System.IO.Packaging name space. I haven't yet tried to zip files present on remote machines but you can give it a try else you may have to write a small job to zip the files and deploy it on your remote(ftp) server.

How to upload a file in a non-default directory using FTPWEBREQUEST?

I face an issue while I try to upload a file in a non-default directory of the server.
When I use "ftp://server01/autofile/test.zip", the file gets uploaded without any issues since autofile is the default directory.
But when I use the below code, I get an exception which says "The remote server returned an error: (550) File unavailable (e.g., file not found, no access).". Shown below is the piece of code I use.
string inputfilepath = "E:\\Test\\test.ZIP";
string ftpfullpath = "ftp://server01/../bcp/ftp/ftpsftiu/test.ZIP";
FtpWebRequest ftp = (FtpWebRequest)FtpWebRequest.Create(ftpfullpath);
ftp.Method = WebRequestMethods.Ftp.UploadFile;
ftp.Credentials = new NetworkCredential("Username", "password");
ftp.UsePassive = true;
ftp.KeepAlive = true;
ftp.UseBinary = true;
FileStream fs = File.OpenRead(inputfilepath);
byte[] buffer = new byte[fs.Length];
fs.Read(buffer, 0, buffer.Length);
fs.Close();
Stream reqStream = ftp.GetRequestStream();
But I can open the above non-default directory path(ftp://server01/../bcp/ftp/ftpsftiu/test.ZIP) through the Run Command of windows.
How do I upload my file to this non-default directory of the server through C# code?
please help me to fix this.
Thanks in advance!!!
With regards,
Shadu

It worked for me if I put another slash after the server (//server//).
I did (also) fully qualify the output path name.
The static GetCread() is a method call into a home-grown class.
Example:
static void Main(string[] args)
{
string strSystem = "MAPS";
FtpWebRequest req = (FtpWebRequest)WebRequest.Create("ftp://" + strSystem +"//export/home/hinest/science/trans/fred.txt");
req.Credentials = CGetCred.GetCred(strSystem);
req.Method = WebRequestMethods.Ftp.UploadFile;
FtpWebResponse resp = (FtpWebResponse)req.GetResponse();
StreamWriter fileOut = new StreamWriter(req.GetRequestStream());
StreamReader fileIn = new StreamReader(#"c:\science\"+strSystem+".txt");
while (!fileIn.EndOfStream)
{
fileOut.WriteLine(fileIn.ReadLine());
}
fileIn.Close();
fileOut.Close();
resp.Close();
}

How to recursively delete a folder with the files within using FtpWebRequest?

I want to delete a folder in FTP and it's files recursively.
Any example code do I can implement?

First you have to list all your files in your directory :
public static List<string> DirectoryListing(string Path, string ServerAdress, string Login, string Password)
{
FtpWebRequest request = (FtpWebRequest)WebRequest.Create("ftp://" + ServerAdress + Path);
request.Credentials = new NetworkCredential(Login, Password);
request.Method = WebRequestMethods.Ftp.ListDirectory;
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);
List<string> result = new List<string>();
while (!reader.EndOfStream)
{
result.Add(reader.ReadLine());
}
reader.Close();
response.Close();
return result;
}
Then you need a method to delete a single file (because you can delete a folder only if it's empty) :
public static void DeleteFTPFile(string Path, string ServerAdress, string Login, string Password)
{
FtpWebRequest clsRequest = (System.Net.FtpWebRequest)WebRequest.Create("ftp://" + ServerAdress + Path);
clsRequest.Credentials = new System.Net.NetworkCredential(Login, Password);
clsRequest.Method = WebRequestMethods.Ftp.DeleteFile;
string result = string.Empty;
FtpWebResponse response = (FtpWebResponse)clsRequest.GetResponse();
long size = response.ContentLength;
Stream datastream = response.GetResponseStream();
StreamReader sr = new StreamReader(datastream);
result = sr.ReadToEnd();
sr.Close();
datastream.Close();
response.Close();
}
And finally :
public static void DeleteFTPDirectory(string Path, string ServerAdress, string Login, string Password)
{
FtpWebRequest clsRequest = (System.Net.FtpWebRequest)WebRequest.Create("ftp://" + ServerAdress + Path);
clsRequest.Credentials = new System.Net.NetworkCredential(Login, Password);
List<string> filesList = DirectoryListing(Path, ServerAdress, Login, Password);
foreach (string file in filesList)
{
DeleteFTPFile(Path + file, ServerAdress, Login, Password);
}
clsRequest.Method = WebRequestMethods.Ftp.RemoveDirectory;
string result = string.Empty;
FtpWebResponse response = (FtpWebResponse)clsRequest.GetResponse();
long size = response.ContentLength;
Stream datastream = response.GetResponseStream();
StreamReader sr = new StreamReader(datastream);
result = sr.ReadToEnd();
sr.Close();
datastream.Close();
response.Close();
}
You can easily call this like that (for me those methods are in a class called "Ftp") :
Ftp.DeleteFTPDirectory(the_path_of_your_folder_in_ftp,your_server_address,your_ftp_login,your_ftp_password);
Of course, you'll need to customize those lines, but it worked perfectly for me :)

There's no support for recursive operations in the FtpWebRequest class (or any other FTP implementation in the .NET framework). You have to implement the recursion yourself:
List the remote directory
Iterate the entries, deleting files and recursing into subdirectories (listing them again, etc.)
Tricky part is to identify files from subdirectories. There's no way to do that in a portable way with the FtpWebRequest. The FtpWebRequest unfortunately does not support the MLSD command, which is the only portable way to retrieve directory listing with file attributes in FTP protocol. See also Checking if object on FTP server is file or directory.
Your options are:
Do an operation on a file name that is certain to fail for a file and to succeed for directory (or vice versa). I.e. you can try to download the "name". If that succeeds, it's a file, if that fails, it's a directory. But that can become a performance problem, when you have a large number of entries.
You may be lucky and in your specific case, you can tell a file from a directory by a file name (i.e. all your files have an extension, while subdirectories do not)
You use a long directory listing (LIST command = ListDirectoryDetails method) and try to parse a server-specific listing. Many FTP servers use *nix-style listing, where you identify a directory by the d at the very beginning of the entry. But many servers use a different format. The following example uses this approach (assuming the *nix format).
In this specific case, you can just try to delete the entry as a file. If deleting fails, try to list the entry as a directory. If the listing succeeds, you assume it's a folder and proceed accordingly. Unfortunately some servers do not error, when you try to list a file. They will just return a listing with a single entry for the file.
static void DeleteFtpDirectory(string url, NetworkCredential credentials)
{
FtpWebRequest listRequest = (FtpWebRequest)WebRequest.Create(url);
listRequest.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
listRequest.Credentials = credentials;
List<string> lines = new List<string>();
using (FtpWebResponse listResponse = (FtpWebResponse)listRequest.GetResponse())
using (Stream listStream = listResponse.GetResponseStream())
using (StreamReader listReader = new StreamReader(listStream))
{
while (!listReader.EndOfStream)
{
lines.Add(listReader.ReadLine());
}
}
foreach (string line in lines)
{
string[] tokens =
line.Split(new[] { ' ' }, 9, StringSplitOptions.RemoveEmptyEntries);
string name = tokens[8];
string permissions = tokens[0];
string fileUrl = url + name;
if (permissions[0] == 'd')
{
DeleteFtpDirectory(fileUrl + "/", credentials);
}
else
{
FtpWebRequest deleteRequest = (FtpWebRequest)WebRequest.Create(fileUrl);
deleteRequest.Method = WebRequestMethods.Ftp.DeleteFile;
deleteRequest.Credentials = credentials;
deleteRequest.GetResponse();
}
}
FtpWebRequest removeRequest = (FtpWebRequest)WebRequest.Create(url);
removeRequest.Method = WebRequestMethods.Ftp.RemoveDirectory;
removeRequest.Credentials = credentials;
removeRequest.GetResponse();
}
The url should be like ftp://example.com/directory/to/delete/
Or use a 3rd party library that supports recursive operations.
For example with WinSCP .NET assembly you can delete whole directory with a single call to Session.RemoveFiles:
// Setup session options
SessionOptions sessionOptions = new SessionOptions
{
Protocol = Protocol.Ftp,
HostName = "example.com",
UserName = "user",
Password = "mypassword",
};
using (Session session = new Session())
{
// Connect
session.Open(sessionOptions);
// Delete folder
session.RemoveFiles("/home/user/foldertoremove").Check();
}
Internally, WinSCP uses the MLSD command, if supported by the server. If not, it uses the LIST command and supports dozens of different listing formats.
(I'm the author of WinSCP)

Nice Little example you can find here:
http://msdn.microsoft.com/en-us/library/system.net.ftpwebrequest.aspx
In the example they used WebRequestMethods.Ftp.UploadFile class to direct what kind of operation they want to do.
Use the WebRequestMethods.Ftp.RemoveDirectory method once you have a handle on the parent directory you want to delete:
http://msdn.microsoft.com/en-us/library/system.net.webrequestmethods.ftp.aspx

None of the solutions really worked on different types of servers except using
System.Net.FtpClient
using System.Net.FtpClient;
static void DeleteFtpDirectoryAndContent(string host, string path, NetworkCredential credentials, string dontDeleteFileUrl)
{
using (FtpClient conn = new FtpClient())
{
conn.Host = host;
conn.Credentials = credentials;
foreach (FtpListItem item in conn.GetListing(path, FtpListOption.AllFiles | FtpListOption.ForceList))
{
switch (item.Type)
{
case FtpFileSystemObjectType.Directory:
conn.DeleteDirectory(item.FullName, true, FtpListOption.AllFiles | FtpListOption.ForceList);
break;
case FtpFileSystemObjectType.File:
if (!dontDeleteFileUrl.EndsWith(item.FullName, StringComparison.InvariantCultureIgnoreCase))
conn.DeleteFile(item.FullName);
break;
}
}
}
}