I'm trying to create a script that download multiple text documents from an FTP server by date modified that are not more than a day old:
if (_datemodified > DateTime.Now.AddDays(-2));
But when I'm running my code it manages to download 200-300 files then times out with
" System.Net.WebException: The remote server returned an error: (550)
File unavailable (e.g., file not found, no access)."
Here's a snippet of the code that the error points to:
static void Checkdatemodified(string url, string savePath)
{
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(url);
request.Method = WebRequestMethods.Ftp.GetDateTimestamp;
request.Credentials = new NetworkCredential("user", "******");
request.UseBinary = true;
DateTime temp;
using (FtpWebResponse response = (FtpWebResponse)request.GetResponse())
{
temp = response.LastModified;
}
FtpWebRequest request2 = (FtpWebRequest)WebRequest.Create(url);
request2.Credentials = new NetworkCredential("user", "******");
request2.Method = WebRequestMethods.Ftp.DownloadFile;
request2.UseBinary = true;
if (temp > DateTime.Now.AddDays(-2))
{
using (FtpWebResponse response2 = (FtpWebResponse)request2.GetResponse())
{
// DateTime temp = response.LastModified;
using (Stream rs = response2.GetResponseStream())
{
using (FileStream ws = new FileStream(savePath, FileMode.Create))
{
byte[] buffer = new byte[2048];
int bytesRead = rs.Read(buffer, 0, buffer.Length);
while (bytesRead > 0)
{
ws.Write(buffer, 0, bytesRead);
bytesRead = rs.Read(buffer, 0, buffer.Length);
}
}
// DateTime temp = Checkdatemodified(url);
System.IO.File.SetLastWriteTime(savePath, temp);
}
}
}
And here's the code that calls the function
//Create new FTPClient in memory
using (var ftpClient = new FtpClient())
{
// Sets the location of the FTP folder we will use in the request
FtpWebRequest request = (FtpWebRequest)WebRequest.Create("ftp://(ftp folder link)");
//Set the method it will use to list the items in the directory
request.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
// Set the credentials used to get onto the FTP
request.Credentials = new NetworkCredential("user", "******");
//Processes request
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
//Stream reader reads each character of a line
Stream responseStream = response.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);
//Creates new list of strings and a new string variable
List<string> lines = new List<string>();
List<string> lines2 = new List<string>();
string line;
//For each line the streamreader reads
//Split the line into an array and take the last entry and add this to the list
while ((line = reader.ReadLine()) != null)
{
string[] temp = line.Split(null);
string temp2 = temp[temp.Length - 1];
lines.Add(temp2);
}
//Close everything
responseStream.Close();
reader.Close();
//For each each line in the list starting at number 2(the first 2 are just the directory names)
for (int i = 2; i < lines.Count - 1; i++)
{
//Set temp strings. 1 for the file path + file name and the other for the location to download to
string temp = "ftp://(ftp folder)" + lines[i];
string temp2 = #"\\(server folder location)" + lines[i];
//If the date modified in the FTP between now and 2 days old then download the fil
if (System.IO.File.Exists(temp2))
{
System.IO.File.Delete(temp2);
}
Checkdatemodified(temp, temp2);
//File.AppendAllText(#"C:\TestFolder\error.txt", value);
}
// DownloadFtpFile(temp, temp2);
}
}
I feel the issue is that I'm using 3 different FtpWebRequest per download for 300+ files which are
1) Get a list of directory
2) Find date modified
3) Download the files
Or it could be an issue with the list that holds the file names.
Does anyone know why this isn't working? Or if there's a better way of bulk downloading ftp files by datemodified
P.S - Code is a bit messy and needs to be cleaned up but I'm just trying to make it work first before hand
Related
I have a remote folder with n files and I need to copy content in another remote file. I guess it can be done through streams, and this is what I tried:
WebRequest destRequest = WebRequest.Create(destFile);
destRequest.Method = "PUT";
destRequest.Headers.Add("x-ms-blob-type", "BlockBlob"); //just an example with Azure blob, doesn't matter
using (Stream destStream = destRequest.GetRequestStream())
{
string sourceName = "mysourcefolder";
int blockSize = 8388608; //all the files have the same lenght, except one (sometimes)
for (int i = 0; i < n; i++)
{
string source = sourceName + i;
WebRequest sourceRequest = WebRequest.Create(source);
destRequest.Method = "GET";
HttpWebResponse destResp = (HttpWebResponse)destRequest.GetResponse();
using (Stream sourceStream = destResp.GetResponseStream())
{
sourceStream.CopyTo(destStream, blockSize);
}
}
Console.Write("ok");
}
}
catch (Exception e)
{
Console.Write("nope !");
}
There are multiple issues in my code:
1) I have to specify the lenght in my PUT request. Probably it is blockSize*n since I have no exceptions about this;
2) If that is the case, I have still the exception Cannot close stream until all bytes are written. What does it means?
There has been confusion in resource and dest requests.
I have added comments to the changing lines.
WebRequest destRequest = WebRequest.Create(destFile);
destRequest.Method = "PUT";
destRequest.Headers.Add("x-ms-blob-type", "BlockBlob"); //just an example with Azure blob, doesn't matter
using (Stream destStream = destRequest.GetRequestStream())
{
string sourceName = "mysourcefolder";
//int blockSize = 8388608; //all the files have the same lenght, except one (sometimes) //all the files have the same lenght, except one (sometimes)
for (int i = 0; i < n; i++)
{
string source = sourceName + i;
WebRequest sourceRequest = WebRequest.Create(source);
destRequest.Method = "GET";
//HttpWebResponse destResp = (HttpWebResponse)destRequest.GetResponse();
//using (Stream sourceStream = destResp.GetResponseStream())
// you need source response
HttpWebResponse sourceResp = (HttpWebResponse)sourceRequest.GetResponse();
using (Stream sourceStream = sourceResp.GetResponseStream())
{
sourceStream.CopyTo(destStream);
}
}
// The request is made here
var destinationResponse = (HttpWebResponse) destRequest.GetResponse();
//Console.Write("ok");
Console.Write(destinationResponse.StatusCode.ToString());
}
I ve got the following code which
foreach (string dir in dirs) { //dirs all files in directory
try{
// Get an instance of WebClient
WebClient client = new System.Net.WebClient();
// parse the ftp host and file into a uri path for the upload
Uri uri = new Uri(m_FtpHost + new FileInfo(dir).Name);
// set the username and password for the FTP server
client.Credentials = new System.Net.NetworkCredential(m_FtpUsername, m_FtpPassword);
// upload the file asynchronously, non-blocking.
client.UploadFileAsync(uri, "STOR",dir);
}
catch(Exception e){
print(e.Message);
}
}
Can I retrieve back the progress of the upload? I have in the dirs 4-5 files. I want exact the progress (not the files uploaded/(total files))
EDIT: Thus the right approach is the following:
public int percentage;
try{
// Get an instance of WebClient
WebClient client = new System.Net.WebClient();
// parse the ftp host and file into a uri path for the upload
Uri uri = new Uri(m_FtpHost + new FileInfo(dir).Name);
// set the username and password for the FTP server
client.Credentials = new System.Net.NetworkCredential(m_FtpUsername, m_FtpPassword);
// upload the file asynchronously, non-blocking.
client.UploadProgressChanged += WebClientUploadProgressChanged;
client.UploadFileCompleted += WebClientUploadCompleted;
client.UploadFileAsync(uri, "STOR",dir);
}
void WebClientUploadProgressChanged(object sender, UploadProgressChangedEventArgs e)
{
percentage = e.ProgressPercentage;
}
void WebClientUploadCompleted(object sender, UploadFileCompletedEventArgs e)
{
print( "Upload is finished. ");
}
I add this implementation to my code, however it seems that it doenst print anything in the console.
WebClient contains a dedicated event for this
public event UploadProgressChangedEventHandler UploadProgressChanged
https://msdn.microsoft.com/en-us/library/system.net.webclient.uploadprogresschanged(v=vs.110).aspx
EDIT : HttpWebRequest approach based on a google result :
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.Method = "POST";
request.ContentType = "text/plain";
request.Timeout = -1; //Infinite wait for the response.
// Get the file information object.
FileInfo fileInfo = new FileInfo("C:\\Test\\uploadFile.dat");
// Set the file content length.
request.ContentLength = fileInfo.Length;
// Get the number of segments the file will be uploaded to the stream.
int segments = Convert.ToInt32(fileInfo.Length / (1024 * 4));
// Get the source file stream.
using (FileStream fileStream = fileInfo.OpenRead())
{
// Create 4KB buffer which is file page size.
byte[] tempBuffer = new byte[1024 * 4];
int bytesRead = 0;
// Write the source data to the network stream.
using (Stream requestStream = request.GetRequestStream())
{
// Loop till the file content is read completely.
while ((bytesRead = fileStream.Read(tempBuffer, 0, tempBuffer.Length)) > 0)
{
// Write the 4 KB data in the buffer to the network stream.
requestStream.Write(tempBuffer, 0, bytesRead);
// Update your progress bar here using segment count.
}
}
}
// Post the request and Get the response from the server.
using (WebResponse response = request.GetResponse())
{
// Request is successfully posted to the server.
}
I am trying to get full list of files, directories and sub-directories on tree view using StreamReader. The problem is it took too long and throws *"Operation time out exception" and shows only one level.
Here is my code
public void getServerSubfolder(TreeNode tv, string parentNode) {
string ptNode;
List<string> files = new List<string> ();
try {
FtpWebRequest request = (FtpWebRequest) WebRequest.
Create(parentNode);
request.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
request.Credentials = new NetworkCredential(this.userName, this.Password);
request.UseBinary = true;
request.UsePassive = true;
request.Timeout = 10000;
request.ReadWriteTimeout = 10000;
request.KeepAlive = false;
FtpWebResponse response = (FtpWebResponse) request.GetResponse();
Stream responseStream = response.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);
string fileList;
string[] fileName;
//MessageBox.Show(reader.ReadToEnd().ToString());
while (!reader.EndOfStream) {
fileList = reader.ReadLine();
fileName = fileList.Split(' ');
if (fileName[0] == "drwxr-xr-x") {
// if it is directory
TreeNode tnS = new TreeNode(fileName[fileName.Length - 1]);
tv.Nodes.Add(tnS);
ptNode = parentNode + "/" + fileName[fileName.Length - 1] + "/";
getServerSubfolder(tnS, ptNode);
} else files.Add(fileName[fileName.Length - 1]);
}
reader.Close();
response.Close();
} catch (Exception ex) {
MessageBox.Show("Sub--here " + ex.Message + "----" + ex.StackTrace);
}
}
You have to read (and cache) whole listing before recursing into subdirectories, otherwise the top-level request will timeout before you complete subdirectories listing.
You can keep using ReadLine, no need to use ReadToEnd and separate the lines yourself.
void ListFtpDirectory(
string url, string rootPath, NetworkCredential credentials, List<string> list)
{
FtpWebRequest listRequest = (FtpWebRequest)WebRequest.Create(url + rootPath);
listRequest.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
listRequest.Credentials = credentials;
List<string> lines = new List<string>();
using (FtpWebResponse listResponse = (FtpWebResponse)listRequest.GetResponse())
using (Stream listStream = listResponse.GetResponseStream())
using (StreamReader listReader = new StreamReader(listStream))
{
while (!listReader.EndOfStream)
{
lines.Add(listReader.ReadLine());
}
}
foreach (string line in lines)
{
string[] tokens =
line.Split(new[] { ' ' }, 9, StringSplitOptions.RemoveEmptyEntries);
string name = tokens[8];
string permissions = tokens[0];
string filePath = rootPath + name;
if (permissions[0] == 'd')
{
ListFtpDirectory(url, filePath + "/", credentials, list);
}
else
{
list.Add(filePath);
}
}
}
Use the function like:
List<string> list = new List<string>();
NetworkCredential credentials = new NetworkCredential("user", "mypassword");
string url = "ftp://ftp.example.com/";
ListFtpDirectory(url, "", credentials, list);
A disadvantage of the above approach is that it has to parse server-specific listing to retrieve information about files and folders. The code above expects a common *nix-style listing. But many servers use a different format.
The FtpWebRequest unfortunately does not support the MLSD command, which is the only portable way to retrieve directory listing with file attributes in FTP protocol.
If you want to avoid troubles with parsing the server-specific directory listing formats, use a 3rd party library that supports the MLSD command and/or parsing various LIST listing formats; and recursive downloads.
For example with WinSCP .NET assembly you can list whole directory with a single call to the Session.EnumerateRemoteFiles:
// Setup session options
SessionOptions sessionOptions = new SessionOptions
{
Protocol = Protocol.Ftp,
HostName = "ftp.example.com",
UserName = "user",
Password = "mypassword",
};
using (Session session = new Session())
{
// Connect
session.Open(sessionOptions);
// List files
IEnumerable<string> list =
session.EnumerateRemoteFiles("/", null, EnumerationOptions.AllDirectories).
Select(fileInfo => fileInfo.FullName);
}
Internally, WinSCP uses the MLSD command, if supported by the server. If not, it uses the LIST command and supports dozens of different listing formats.
(I'm the author of WinSCP)
I'm doing similar things, but instead of using StreamReader.ReadLine() for each one, I get it all at once with StreamReader.ReadToEnd(). You don't need ReadLine() to just get a list of directories. Below is my entire code (Entire howto is explained in this tutorial):
FtpWebRequest request=(FtpWebRequest)FtpWebRequest.Create(path);
request.Method=WebRequestMethods.Ftp.ListDirectoryDetails;
List<ftpinfo> files=new List<ftpinfo>();
//request.Proxy = System.Net.WebProxy.GetDefaultProxy();
//request.Proxy.Credentials = CredentialCache.DefaultNetworkCredentials;
request.Credentials = new NetworkCredential(_username, _password);
Stream rs=(Stream)request.GetResponse().GetResponseStream();
OnStatusChange("CONNECTED: " + path, 0, 0);
StreamReader sr = new StreamReader(rs);
string strList = sr.ReadToEnd();
string[] lines=null;
if (strList.Contains("\r\n"))
{
lines=strList.Split(new string[] {"\r\n"},StringSplitOptions.None);
}
else if (strList.Contains("\n"))
{
lines=strList.Split(new string[] {"\n"},StringSplitOptions.None);
}
//now decode this string array
if (lines==null || lines.Length == 0)
return null;
foreach(string line in lines)
{
if (line.Length==0)
continue;
//parse line
Match m= GetMatchingRegex(line);
if (m==null) {
//failed
throw new ApplicationException("Unable to parse line: " + line);
}
ftpinfo item=new ftpinfo();
item.filename = m.Groups["name"].Value.Trim('\r');
item.path = path;
item.size = Convert.ToInt64(m.Groups["size"].Value);
item.permission = m.Groups["permission"].Value;
string _dir = m.Groups["dir"].Value;
if(_dir.Length>0 && _dir != "-")
{
item.fileType = directoryEntryTypes.directory;
}
else
{
item.fileType = directoryEntryTypes.file;
}
try
{
item.fileDateTime = DateTime.Parse(m.Groups["timestamp"].Value);
}
catch
{
item.fileDateTime = DateTime.MinValue; //null;
}
files.Add(item);
}
return files;
I am trying to get a list of files on FTP folder.
The code was working when I ran it locally, but on deploying it I started receiving html instead of file name
ArrayList fName = new ArrayList();
try
{
StringBuilder result = new StringBuilder();
//create the directory
FtpWebRequest requestDir =
(FtpWebRequest) FtpWebRequest.Create(new Uri(directory));
requestDir.Method = WebRequestMethods.Ftp.ListDirectory;
requestDir.Credentials = new NetworkCredential(FTP_USER_NAME, FTP_PASSWORD);
requestDir.UsePassive = true;
requestDir.UseBinary = true;
requestDir.KeepAlive = false;
requestDir.Proxy = null;
FtpWebResponse response = (FtpWebResponse)requestDir.GetResponse();
Stream ftpStream = response.GetResponseStream();
StreamReader reader = new StreamReader(ftpStream, Encoding.ASCII);
while (!reader.EndOfStream)
{
fName.Add(reader.ReadLine().ToString());
}
response.Close();
ftpStream.Close();
reader.Close();
You can try with GetFileName
Uri uri = new Uri(hreflink);
string filename = Path.GetFileName(uri.LocalPath);
I was able to access the file names in a list format(not as html) by using the ip address of the ftp server and not the name
i.e.
ftp://xxx.x.x.xxx/folder_name/
instead of
ftp://abc.some_company.com/folder_name/
I will edit this answer with more details.
In my Application I need to download some files from the FTP server..I coded to download one by one via for loop and it works fine.My Problem is it is very slow process coz for each file it will gives credential. So if I can zip those files in FTP server it can be Download quickly..If there is a way pls guide me..
here My Code:
Blockquote
private void button1_Click(object sender, EventArgs e)
{
string[] list = GetFileList();
DownloadFile(list, "e:\\sample\\");
}
private String[] GetFileList()
{
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(new Uri(_remoteHost));
request.Credentials = new NetworkCredential(_remoteUser, _remotePass);
request.Method = WebRequestMethods.Ftp.ListDirectory;
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);
string FileNames = reader.ReadToEnd();
string[] Files = Regex.Split(FileNames, "\r\n");
return Files;
}
private void DownloadFile(string[] fileList, string destination)
{
for (int i = 2; i <= fileList.Length - 1; i++)
{
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(_remoteHost + fileList[i]);
request.Method = WebRequestMethods.Ftp.DownloadFile;
request.Credentials = new NetworkCredential(_remoteUser, _remotePass);
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);
StreamWriter writer = new StreamWriter(destination + fileList[i]);
writer.Write(reader.ReadToEnd());
writer.Close();
reader.Close();
response.Close();
}
}
Blockquote
Why dont you initiate your FtpWebRequest once with the Username and Password and then use it in a foreach Loop for all files instead of creating a new one each round?
Another way would be to use a 3rd party tool (like: http://www.rebex.net/ftp-ssl.net/) which may be faster, but its something you will have to test, take a looka t this: https://stackoverflow.com/a/2343689/395659
A 3rd way could be to let a job running on the server which creates the ZIP file once in a period and you download it as a Zip file as mentioned in your Question.
You can re-use the credentials, which might speed you up...
private void DownloadFile(string[] fileList, string destination)
{
var myCred = new NetworkCredential(_remoteUser, _remotePass);
for (int i = 2; i <= fileList.Length - 1; i++)
{
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(_remoteHost + fileList[i]);
request.Method = WebRequestMethods.Ftp.DownloadFile;
request.Credentials = myCred;
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);
StreamWriter writer = new StreamWriter(destination + fileList[i]);
writer.Write(reader.ReadToEnd());
writer.Close();
reader.Close();
response.Close();
}
}
If you are using .NET 3.0 or above you can make use of ZipPackage Class present in the System.IO.Packaging name space. I haven't yet tried to zip files present on remote machines but you can give it a try else you may have to write a small job to zip the files and deploy it on your remote(ftp) server.