We Keep Coding

Monitor FTP directory in ASP.NET/C#

I have FileSystem watcher for a local directory. It's working fine. I want same to implement for FTP. Is there any way I can achieve it? I have checked many solutions but it's not clear.
Logic: Want to get files from FTP later than some timestamp.
Problem faced: Getting all files from FTP and then filtering the result is hitting the performance (used FtpWebRequest).
Is there any right way to do this? (WinSCP is on hold. Cant use it now.)
FileSystemWatcher oFsWatcher = new FileSystemWatcher();
OFSWatchers.Add(oFsWatcher);
oFsWatcher.Path = sFilePath;
oFsWatcher.Filter = string.IsNullOrWhiteSpace(sFileFilter) ? "*.*" : sFileFilter;
oFsWatcher.NotifyFilter = NotifyFilters.FileName;
oFsWatcher.EnableRaisingEvents = true;
oFsWatcher.IncludeSubdirectories = bIncludeSubdirectories;
oFsWatcher.Created += new FileSystemEventHandler(OFsWatcher_Created);

You cannot use the FileSystemWatcher or any other way, because the FTP protocol does not have any API to notify a client about changes in the remote directory.
All you can do is to periodically iterate the remote tree and find changes.
It's actually rather easy to implement, if you use an FTP client library that supports recursive listing of a remote tree. Unfortunately, the built-in .NET FTP client, the FtpWebRequest does not. But for example with WinSCP .NET assembly, you can use the Session.EnumerateRemoteFiles method.
See the article Watching for changes in SFTP/FTP server:
// Setup session options
SessionOptions sessionOptions = new SessionOptions
{
Protocol = Protocol.Ftp,
HostName = "example.com",
UserName = "user",
Password = "password",
};
using (Session session = new Session())
{
// Connect
session.Open(sessionOptions);
List<string> prevFiles = null;
while (true)
{
// Collect file list
List<string> files =
session.EnumerateRemoteFiles(
"/remote/path", "*.*", EnumerationOptions.AllDirectories)
.Select(fileInfo => fileInfo.FullName)
.ToList();
if (prevFiles == null)
{
// In the first round, just print number of files found
Console.WriteLine("Found {0} files", files.Count);
}
else
{
// Then look for differences against the previous list
IEnumerable<string> added = files.Except(prevFiles);
if (added.Any())
{
Console.WriteLine("Added files:");
foreach (string path in added)
{
Console.WriteLine(path);
}
}
IEnumerable<string> removed = prevFiles.Except(files);
if (removed.Any())
{
Console.WriteLine("Removed files:");
foreach (string path in removed)
{
Console.WriteLine(path);
}
}
}
prevFiles = files;
Console.WriteLine("Sleeping 10s...");
Thread.Sleep(10000);
}
}
(I'm the author of WinSCP)
Though, if you actually want to just download the changes, it's a way easier. Just use the Session.SynchronizeDirectories in the loop.
while (true)
{
SynchronizationResult result =
session.SynchronizeDirectories(
SynchronizationMode.Local, "/remote/path", #"C:\local\path", true);
result.Check();
// You can inspect result.Downloads for a list for updated files
Console.WriteLine("Sleeping 10s...");
Thread.Sleep(10000);
}
This will update even modified files, not only new files.
Though using WinSCP .NET assembly from a web application might be problematic. If you do not want to use a 3rd party library, you have to do with limitations of the FtpWebRequest. For an example how to recursively list a remote directory tree with the FtpWebRequest, see my answer to List names of files in FTP directory and its subdirectories.
You have edited your question to say that you have performance problems with the solutions I've suggested. Though you have already asked a new question that covers this:
Get FTP file details based on datetime in C#

Unless you have access to the OS which hosts the service; it will be a bit harder.
FileSystemWatcher places a hook on the filesystem, which will notify your application as soon as something happened.
FTP command specifications does not have such a hook. Besides that it's always initiated by the client.
Therefor, to implement such logic you should periodical perform a NLST to list the FTP-directory contents and track the changes (or hashes, perhaps (MDTM)) yourself.
More info:
FTP return codes
FTP

I have got an alternative solution to do my functionality.
Explanation:
I am downloading the files from FTP (Read permission reqd.) with same folder structure.
So everytime the job/service runs I can check into the physical path same file(Full Path) exists or not If not exists then it can be consider as a new file. And Ii can do some action for the same and download as well.
Its just an alternative solution.
Code Changes:
private static void GetFiles()
{
using (FtpClient conn = new FtpClient())
{
string ftpPath = "ftp://myftp/";
string downloadFileName = #"C:\temp\FTPTest\";
downloadFileName += "\\";
conn.Host = ftpPath;
//conn.Credentials = new NetworkCredential("ftptest", "ftptest");
conn.Connect();
//Get all directories
foreach (FtpListItem item in conn.GetListing(conn.GetWorkingDirectory(),
FtpListOption.Modify | FtpListOption.Recursive))
{
// if this is a file
if (item.Type == FtpFileSystemObjectType.File)
{
string localFilePath = downloadFileName + item.FullName;
//Only newly created files will be downloaded.
if (!File.Exists(localFilePath))
{
conn.DownloadFile(localFilePath, item.FullName);
//Do any action here.
Console.WriteLine(item.FullName);
}
}
}
}
}

DirectoryInfo usage in a UWP project

I am currently working to a Windows 10 UWP project and I keep getting the following exception:
Unable to cast object of type 'System.IO.FileSystemInfo[]' to type 'System.Collections.Generic.IEnumerable`1[System.IO.FileInfo]'.
and this is the code which throws it:
DirectoryInfo dirInfo = new DirectoryInfo(path);
FileInfo[] files = dirInfo.GetFiles(path);
path is a valid one, i verified it several times, I do not know why I am getting this exception. Can the DirectoryInfo class still be used in a UWP application or should I use an equivalent one ?

The DirectoryInfo class is applicable for UWP. However, it has a lot of limitations. Such as whether the path is valid. For more detail you could refer to Skip the path: stick to the StorageFile.
It throw Second path fragment must not be a drive or UNC name exception when I passed path parameter. I found the following description.
The search string to match against the names of files. This parameter can contain
a combination of valid literal path and wildcard (* and ?) characters (see Remarks),
but doesn't support regular expressions. The default pattern is "*", which returns
all files.
So I modify the searchPattern like the following, it works well.
string root = Windows.ApplicationModel.Package.Current.InstalledLocation.Path;
string path = root + #"\Assets\Media";
DirectoryInfo dirinfo = new DirectoryInfo(path);
FileInfo[] files = dirinfo.GetFiles("head.*");
I do not know why I am getting this exception. Can the DirectoryInfo class still be used in a UWP application or should I use an equivalent one ?
The best practice to query files in UWP is to use folder picker to select a folder and enumerate all the files with GetFilesAsync method. For example:
var picker = new Windows.Storage.Pickers.FolderPicker();
picker.SuggestedStartLocation = Windows.Storage.Pickers.PickerLocationId.PicturesLibrary;
picker.FileTypeFilter.Add("*");
picker.ViewMode = Windows.Storage.Pickers.PickerViewMode.Thumbnail;
var folder = await picker.PickSingleFolderAsync();
if(folder != null)
{
StringBuilder outputText = new StringBuilder();
var query = folder.CreateFileQuery();
var files = await query.GetFilesAsync();
foreach (StorageFile file in files)
{
outputText.Append(file.Name + "\n");
}
}

List names of files in FTP directory and its subdirectories

I have been searched in the net and I didn't found any result. Actually I want to get the name of all the files that I have in the root and Directory and Sub Directory. I tried the code as bellow but its give me only the files in the root of my FTP.
The folder that I have in the FTP is like as bellow:
/ds/product/Jan/
/ds/subproduct/Jan/
/ds/category/Jan/
The code that I tried:
FtpWebRequest ftpRequest = (FtpWebRequest)WebRequest.Create("ftp://" + FtpIP);
ftpRequest.Credentials = new NetworkCredential(FtpUser, FtpPass);
ftpRequest.Method = WebRequestMethods.Ftp.ListDirectory;
FtpWebResponse response = (FtpWebResponse)ftpRequest.GetResponse();
StreamReader streamReader = new StreamReader(response.GetResponseStream());
List<string> directories = new List<string>();
string line = streamReader.ReadLine();
while (!string.IsNullOrEmpty(line))
{
// directories.Add(line);
line = streamReader.ReadLine().ToString();
MessageBox.Show(line);
}
streamReader.Close();

It is s not easy to implement this without any external library. Unfortunately, neither the .NET Framework nor PowerShell have any explicit support for recursively listing files in an FTP directory.
You have to implement that yourself:
List the remote directory
Iterate the entries, recursing into subdirectories - listing them again, etc.
Tricky part is to identify files from subdirectories. There's no way to do that in a portable way with the .NET Framework (FtpWebRequest). The .NET Framework unfortunately does not support the MLSD command, which is the only portable way to retrieve directory listing with file attributes in FTP protocol. See also Checking if object on FTP server is file or directory.
Your options are:
Do an operation on a file name that is certain to fail for file and succeeds for directories (or vice versa). I.e. you can try to download the "name".
You may be lucky and in your specific case, you can tell a file from a directory by a file name (i.e. all your files have an extension, while subdirectories do not)
You use a long directory listing (LIST command = ListDirectoryDetails method) and try to parse a server-specific listing. Many FTP servers use *nix-style listing, where you identify a directory by the d at the very beginning of the entry. But many servers use a different format. The following example uses this approach (assuming the *nix format)
static void ListFtpDirectory(string url, NetworkCredential credentials)
{
WebRequest listRequest = WebRequest.Create(url);
listRequest.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
listRequest.Credentials = credentials;
List<string> lines = new List<string>();
using (WebResponse listResponse = listRequest.GetResponse())
using (Stream listStream = listResponse.GetResponseStream())
using (StreamReader listReader = new StreamReader(listStream))
{
while (!listReader.EndOfStream)
{
string line = listReader.ReadLine();
lines.Add(line);
}
}
foreach (string line in lines)
{
string[] tokens =
line.Split(new[] { ' ' }, 9, StringSplitOptions.RemoveEmptyEntries);
string name = tokens[8];
string permissions = tokens[0];
if (permissions[0] == 'd')
{
Console.WriteLine($"Directory {name}");
string fileUrl = url + name;
ListFtpDirectory(fileUrl + "/", credentials);
}
else
{
Console.WriteLine($"File {name}");
}
}
}
Use the function like:
NetworkCredential credentials = new NetworkCredential("user", "mypassword");
string url = "ftp://ftp.example.com/directory/to/list/";
ListFtpDirectory(url, credentials);
If you want to avoid troubles with parsing the server-specific directory listing formats, use a 3rd party library that supports the MLSD command and/or parsing various LIST listing formats.
For example with WinSCP .NET assembly you can list whole directory recursively with a single call to Session.EnumerateRemoteFiles:
// Setup session options
var sessionOptions = new SessionOptions
{
Protocol = Protocol.Ftp,
HostName = "ftp.example.com",
UserName = "user",
Password = "mypassword",
};
using (var session = new Session())
{
// Connect
session.Open(sessionOptions);
// Enumerate files
var options =
EnumerationOptions.EnumerateDirectories |
EnumerationOptions.AllDirectories;
IEnumerable<RemoteFileInfo> fileInfos =
session.EnumerateRemoteFiles("/directory/to/list", null, options);
foreach (var fileInfo in fileInfos)
{
Console.WriteLine(fileInfo.FullName);
}
}
Not only the code is simpler, more robust and platform-independent. It also makes all other file attributes (size, modification time, permissions, ownership) readily available via the RemoteFileInfo class.
Internally, WinSCP uses the MLSD command, if supported by the server. If not, it uses the LIST command and supports dozens of different listing formats.
(I'm the author of WinSCP)

Copying/over-writing one source to another in TFS?

Question Background:
Within my TF Server I have two folders, one is a simple 'HelloWorld.sln' in a folder called 'HelloWorldDev' and the other is a 'HelloWorld.sln' in a folder called 'HelloWorldQA'. Each folder contains its respective .cs files etc.
I want to checkout a file from the HelloWorld QA folder, replace - or update it - with a version from the HelloWorldDev folder with the same file name, then check this file back into the HelloWorldQA folder with the relevant changes.
Question:
I am very new to the TFS API so I'm not 100% if what I'm trying to ask is the correct way to proceed, or if its even possible. Can someone give me an example of achieving this?
Code so far:
string fileName = #"C:\Users\Me\Documents\TfsServer\HelloWorldQA\IHelloWorld.cs";
string fileNameQA = #"C:\Users\Me\Documents\TfsServer\HelloWorld\IHelloWorld.cs";
string uri = #"https://tfsServer.visualstudio.com/";
var workspaceInfo = Workstation.Current.GetLocalWorkspaceInfo(fileName);
var server = new TfsTeamProjectCollection(workspaceInfo.ServerUri);
var workspace = workspaceInfo.GetWorkspace(server);
workspace.PendEdit(fileName);
FileInfo fi = new FileInfo(fileName);
var workspaceInfoQA = Workstation.Current.GetLocalWorkspaceInfo(fileNameQA);
var serverQA = new TfsTeamProjectCollection(workspaceInfo.ServerUri);
var workspaceQA = workspaceInfo.GetWorkspace(serverQA);
workspace.PendEdit(fileNameQA);
FileInfo fiQA = new FileInfo(fileNameQA);

First, instead of using 2 workspaces, you can simply map both folders in the same workspace.
Then you're looking for a merge operation:
var sourcePath = workspace.GetServerItemForLocalItem(fileName);
var targetPath = workspace.GetServerItemForLocalItem(fileNameQA);
var getStatus = workspace.Merge(sourcePath, targetPath, null, null);
if (getStatus.NumUpdated > 0)
{
//OK
}

Moving files based on name to the corresponding folder

Hello everyone and well met! I have tried a lot of different methods/programs to try and solve my problem. I'm a novice programmer and have taken a Visual Basic Class and Visual C# class.
I'm working with this in C#
I started off by making a very basic move file program and it worked fine for one file but as I mentioned I will be needing to move a ton of files based on name
What I am trying to do is move .pst (for example dave.pst) files from my exchange server based on username onto a backup server in the users folder (folder = dave) that has the same name as the .pst file
The ideal program would be:
Get files from the folder with the .pst extension
Move files to appropriate folder that has the same name in front of the .pst file extension
Update:
// String pstFileFolder = #"C:\test\";
// var searchPattern = "*.pst";
// var extension = ".pst";
//var serverFolder = #"C:\test3\";
// String filename = System.IO.Path.GetFileNameWithoutExtension(pstFileFolder);
// Searches the directory for *.pst
DirectoryInfo sourceDirectory = new DirectoryInfo(#"C:\test\");
String strTargetDirectory = (#"C:\test3\");
Console.WriteLine(sourceDirectory);
Console.ReadKey(true);>foreach (FileInfo file in sourceDirectory.GetFiles()) {
Console.WriteLine(file);
Console.ReadKey(true);
// Try to create the directory.
System.IO.Directory.CreateDirectory(strTargetDirectory);
file.MoveTo(strTargetDirectory + "\\" + file.Name);
}
This is just a simple copy procedure. I'm completely aware. The
Console.WriteLine(file);
Console.ReadKey(true);
Are for verification purpose right now to make sure I'm getting the proper files and I am. Now I just need to find the folder based on the name of the .pst file(the folder for the users are already created), make a folder(say 0304 for the year), then copy that .pst based on the name.
Thanks a ton for your help guys. #yuck, thanks for the code.

Have a look at the File and Directory classes in the System.IO namespace. You could use the Directory.GetFiles() method to get the names of the files you need to transfer.

Here's a console application to get you started. Note that there isn't any error checking and it makes some assumptions about how the files are named (e.g. that they end with .pst and don't contain that elsewhere in the name):
private static void Main() {
var pstFileFolder = #"C:\TEMP\PST_Files\";
var searchPattern = "*.pst";
var extension = ".pst";
var serverFolder = #"\\SERVER\PST_Backup\";
// Searches the directory for *.pst
foreach (var file in Directory.GetFiles(pstFileFolder, searchPattern)) {
// Exposes file information like Name
var theFileInfo = new FileInfo(file);
// Gets the user name based on file name
// e.g. DaveSmith.pst would become DaveSmith
var userName = theFileInfo.Name.Replace(extension, "");
// Sets up the destination location
// e.g. \\SERVER\PST_Backup\DaveSmith\DaveSmith.pst
var destination = serverFolder + userName + #"\" + theFileInfo.Name;
File.Move(file, destination);
}
}

System.IO is your friend in this case ;)
First, Determine file name by:
String filename = System.IO.Path.GetFileNameWithoutExtension(SOME_PATH)
To make path to new folder, use Path.Combine:
String targetDir = Path.Combine(SOME_ROOT_DIR,filename);
Next, create folder with name based on given fileName
System.IO.Directory.CreateDirectory(targetDir);
Ah! You need to have name of file, but with extension this time. Path.GetFileName:
String fileNameWithExtension = System.IO.Path.GetFileName(SOME_PATH);
And you can move file (by File.Move) to it:
System.IO.File.Move(SOME_PATH,Path.Combine(targetDir,fileNameWithExtension)
Laster already show you how to get file list in folder.

I personally prefer DirectoryInfo because it is more object-oriented.
DirectoryInfo sourceDirectory = new DirectoryInfo("C:\MySourceDirectoryPath");
String strTargetDirectory = "C:\MyTargetDirectoryPath";
foreach (FileInfo file in sourceDirectory.GetFiles())
{
file.MoveTo(strTargetDirectory + "\\" + file.Name);
}