We Keep Coding

WinSCP.Net, folder is not copied from remote to local

I have been using WinSCP to download files, periodically, from a Unix server to Windows server and it has been working with no issues. I also check if remote file is older or already exists (don't copy).
Now, I have to do the same but this time I have to download files and folders. Files are copied fine but folders aren't. When playing with the settings, I got it to copy the contents of the folder but they get copied to my root local folder; I thought WinSCP would copy everything.
In below code, LocalFolder is Z:\My_Data and LogRootFolder is /xyz/gtc/a00/
Folder structure on remote is /xyz/gtc/a00/ABCD/outcomes/ with subfolder "backup" that has many subfolders named as dates (e.g. /xyz/gtc/a00/ABCD/outcomes/backup/2021-06-23/)
Either none of "backup/2021-xx-xx/" files and folder are copied, or they are all copied to Z:\My_Data\ABCD
After setting up the session, called SFTP_Session:
string sRemotePath = LogRootFolder + "ABCD/outcomes/";
string sLocalFolder = Path.Combine(LocalFolder, #"ABCD\");
if (SFTP_Session.Opened)
{
using (SFTP_Session)
{
SFTP_Session.QueryReceived += (sender, e) =>
{
...
e.Continue();
};
//var opts = EnumerationOptions.EnumerateDirectories | EnumerationOptions.AllDirectories;
//IEnumerable<RemoteFileInfo> fileInfos = SFTP_Session.EnumerateRemoteFiles(sRemotePath, "*.dat", opts); <-- This copies files in folder(s) to m local root folder
Regex mask = new Regex(#"\.(dat|err)$", RegexOptions.IgnoreCase);
IEnumerable<RemoteFileInfo> fileInfos =
SFTP_Session.EnumerateRemoteFiles(sRemotePath, null, EnumerationOptions.AllDirectories)
.Where(fileInfo => mask.Match(fileInfo.Name).Success)
.ToList();
foreach (RemoteFileInfo fileInfo in fileInfos)
{
string localFilePath = Path.Combine(sLocalFolder, fileInfo.Name);
if (fileInfo.IsDirectory)
{
// Create local subdirectory, if it does not exist yet
if (!Directory.Exists(localFilePath))
{
Directory.CreateDirectory(localFilePath);
}
}
else
{
string remoteFilePath = RemotePath.EscapeFileMask(fileInfo.FullName);
// If file does not exist in local folder, download
if (!File.Exists(localFilePath))
{
bDownload = true;
}
else // If file exists in local folder but is older, download; else skip
{
DateTime remoteWriteTime = SFTP_Session.GetFileInfo(remoteFilePath).LastWriteTime;
DateTime localWriteTime = File.GetLastWriteTime(localFilePath);
if (remoteWriteTime > localWriteTime)
{
bDownload = true;
}
else
{
bDownload = false;
}
}
if (bDownload)
{
// Download file
TransferOptions oTrRes = new TransferOptions();
oTrRes.TransferMode = TransferMode.Automatic; //The Transfer Mode - Automatic, Binary, or Ascii
oTrRes.FilePermissions = null; //Permissions applied to remote files; null for default permissions. Can set user, Group, or other Read/Write/Execute permissions.
oTrRes.PreserveTimestamp = false; //Set last write time of destination file to that of source file - basically change the timestamp to match destination and source files.
oTrRes.ResumeSupport.State = TransferResumeSupportState.Off;
TransferOperationResult transferResult = SFTP_Session.GetFiles(remoteFilePath, localFilePath, false, oTrRes);//.Replace("\\","")); // I thought this would get files AND folders
// Throw on any error
transferResult.Check();
foreach (TransferEventArgs transfer in transferResult.Transfers)
{
// Store local file info in a data table for processing later
...
}
SessionRemoteExceptionCollection srec = transferResult.Failures;
foreach (SessionRemoteException sre in srec)
{
// Log errors
}
// Did the download succeeded?
if (!transferResult.IsSuccess)
{
// Log error (but continue with other files)
}
}
}
}
At the end, in local folder I see the files downloaded and copied and subfolders that I created (using above code) but no files in those folders. Can't see what I am missing here.

Your code basically synchronizes a remote directory to a local one.
Instead of fixing your code, you can simply replace most of it with a simple call to Session.SynchronizeDirectories:
https://winscp.net/eng/docs/library_session_synchronizedirectories
Try the synchronization first in WinSCP GUI to see if it does what you need.
If you need to do some processing with the synchronized files, use the SynchronizationResult returned by Session.SynchronizeDirectories. It contains a list of all synchronized files.
If you need to exclude some files from the synchronization, use TransferOptions.FileMask.

Access to the path 'c:\$Recycle.Bin\S-1-5-18' is denied

I have this code to copy all files from source-directory, F:\, to destination-directory.
public void Copy(string sourceDir, string targetDir)
{
//Exception occurs at this line.
string[] files = System.IO.Directory.GetFiles(sourceDir, "*.jpg",
SearchOption.AllDirectories);
foreach (string srcPath in files)
{
File.Copy(srcPath, srcPath.Replace(sourceDir, targetDir), true);
}
}
and getting an exception.
If I omit SearchOption.AllDirectories and it works but only copies files from F:\

Use following function instead of System.IO.Directory.GetFiles:
IEnumerable<String> GetAllFiles(string path, string searchPattern)
{
return System.IO.Directory.EnumerateFiles(path, searchPattern).Union(
System.IO.Directory.EnumerateDirectories(path).SelectMany(d =>
{
try
{
return GetAllFiles(d,searchPattern);
}
catch (UnauthorizedAccessException e)
{
return Enumerable.Empty<String>();
}
}));
}

File system objects are subject to security. Some file system objects are secured in such a way that they can only be accessed by certain users. You are encountering a file to which the user executing the code does not have sufficient rights to access.
The reason that you don't have access rights for this particular folder is to protect the security of the different users on the system. The folder in question is the recycle bin on that drive. And each different user has their own private recycle bin, that only they have permission to access. If anybody could access any other user's recycle bin, then users would be able to read each other's files, a clear violation of the system's security policy.
Perhaps the simplest way around this is to skip hidden folders at the root level of the drive. That simple change would be enough to solve your problem because you surely don't want to copy recycle bins.

That folder is a secure system folder (your bin, each drive has its own bin). Just place your file.copy into a try catch statement and ignore/log all the failures. That way you will only copy actual files and skip system files/folders.
If you really want to avoid the try catch statement. Use the fileinfo and directory info classes to figure out which folders/files are of the system and will throw an exception.

This should do the trick:
private IEnumerable<string> RecursiveFileSearch(string path, string pattern, ICollection<string> filePathCollector = null)
{
try
{
filePathCollector = filePathCollector ?? new LinkedList<string>();
var matchingFilePaths = Directory.GetFiles(path, pattern);
foreach(var matchingFile in matchingFilePaths)
{
filePathCollector.Add(matchingFile);
}
var subDirectories = Directory.EnumerateDirectories(path);
foreach (var subDirectory in subDirectories)
{
RecursiveFileSearch(subDirectory, pattern, filePathCollector);
}
return filePathCollector;
}
catch (Exception error)
{
bool isIgnorableError = error is PathTooLongException ||
error is UnauthorizedAccessException;
if (isIgnorableError)
{
return Enumerable.Empty<string>();
}
throw error;
}
}

Moving a folder (Directory) from one location to another - misbehavior

I want to move a directory from one location to another using C# .NET. I used Directory.Move or even DirectoryInfo (with MoveTo) this simple way:
// source is: "C:\Songs\Elvis my Man"
// newLocation is: "C:\Songs\Elvis"
try
{
// Previous command was: Directory.Move(source, newLocation);
DirectoryInfo dir = new DirectoryInfo(source);
dir.MoveTo(newLocation);
}
catch (Exception e)
{
Console.WriteLine("Error: "+ e.Message);
}
But action that's being done (for both cases) is renaming the folder name from 'source' to 'newLocation'
What I expected? that folder "Elvis my man" will be now in "Elvis" folder.
What has happened? "Elvis my man" was changed to "Elvis" (Renamed). If the directory "Elvis" is already exists, it can't change it to "Elvis" (cause he can't make a duplicate names), therefore I get an exception saying that.
What am I doing wrong??
Many thanks!!!

I would advise putting validation around the Move command to ensure that the source location does exists and the destination location doesn't exists.
I've always found it easier to avoid the exceptions than handle them once they do occur.
You'll probably want to include exception handling as well, just in case the access permissions are a problem or a file is open and can't be moved...
Here's some sample code for you:
string sourceDir = #"c:\test";
string destinationDir = #"c:\test1";
try
{
// Ensure the source directory exists
if (Directory.Exists(sourceDir) == true )
{
// Ensure the destination directory doesn't already exist
if (Directory.Exists(destinationDir) == false)
{
// Perform the move
Directory.Move(sourceDir, destinationDir);
}
else
{
// Could provide the user the option to delete the existing directory
// before moving the source directory
}
}
else
{
// Do something about the source directory not existing
}
}
catch (Exception)
{
// TODO: Handle the exception that has been thrown
}

Even though this works in the command line to move a file, when programming you need to provide the full new name.
So you'd need to change newLocation to "C:\Songs\Elvis\Elvis my Man" to make this work.

From MSDN,
This method throws an IOException if, for example, you try to move c:\mydir to c:\public, and c:\public already exists. You must specify "c:\public\mydir" as the destDirName parameter, or specify a new directory name such as "c:\newdir".
It looks like you need to set newLocation to C:\Songs\Elvis\Elvis my man

private void moveDirectory(string fuente,string destino)
{
if (!System.IO.Directory.Exists(destino))
{
System.IO.Directory.CreateDirectory(destino);
}
String[] files = Directory.GetFiles(fuente);
String[] directories = Directory.GetDirectories(fuente);
foreach (string s in files)
{
System.IO.File.Copy(s, Path.Combine(destino,Path.GetFileName(s)), true);
}
foreach(string d in directories)
{
moveDirectory(Path.Combine(fuente, Path.GetFileName(d)), Path.Combine(destino, Path.GetFileName(d)));
}
}

How do I create a file AND any folders, if the folders don't exist?

Imagine I wish to create (or overwrite) the following file :- C:\Temp\Bar\Foo\Test.txt
Using the File.Create(..) method, this can do it.
BUT, if I don't have either one of the following folders (from that example path, above)
Temp
Bar
Foo
then I get an DirectoryNotFoundException thrown.
So .. given a path, how can we recursively create all the folders necessary to create the file .. for that path? If Temp or Bar folders exists, but Foo doesn't... then that is created also.
For simplicity, lets assume there's no Security concerns -- all permissions are fine, etc.

To summarize what has been commented in other answers:
//path = #"C:\Temp\Bar\Foo\Test.txt";
Directory.CreateDirectory(Path.GetDirectoryName(path));
Directory.CreateDirectory will create the directories recursively and if the directory already exist it will return without an error.
If there happened to be a file Foo at C:\Temp\Bar\Foo an exception will be thrown.

DirectoryInfo di = Directory.CreateDirectory(path);
Console.WriteLine("The directory was created successfully at {0}.",
Directory.GetCreationTime(path));
See this MSDN page.

Use Directory.CreateDirectory before you create the file. It creates the folder recursively for you.

. given a path, how can we recursively create all the folders necessary to create the file .. for that path
Creates all directories and subdirectories as specified by path.
Directory.CreateDirectory(path);
then you may create a file.

You will need to check both parts of the path (directory and filename) and create each if it does not exist.
Use File.Exists and Directory.Exists to find out whether they exist. Directory.CreateDirectory will create the whole path for you, so you only ever need to call that once if the directory does not exist, then simply create the file.

You should use Directory.CreateDirectory.
http://msdn.microsoft.com/en-us/library/54a0at6s.aspx

Assuming that your assembly/exe has FileIO permission is itself, well is not right. Your application may not run with admin rights. Its important to consider Code Access Security and requesting permissions
Sample code:
FileIOPermission f2 = new FileIOPermission(FileIOPermissionAccess.Read, "C:\\test_r");
f2.AddPathList(FileIOPermissionAccess.Write | FileIOPermissionAccess.Read, "C:\\example\\out.txt");
try
{
f2.Demand();
}
catch (SecurityException s)
{
Console.WriteLine(s.Message);
}
Understanding .NET Code Access Security
Is “Code Access Security” of any real world use?

You want Directory.CreateDirectory()
Here is a class I use (converted to C#) that if you pass it a source directory and a destination it will copy all of the files and sub-folders of that directory to your destination:
using System.IO;
public class copyTemplateFiles
{
public static bool Copy(string Source, string destination)
{
try {
string[] Files = null;
if (destination[destination.Length - 1] != Path.DirectorySeparatorChar) {
destination += Path.DirectorySeparatorChar;
}
if (!Directory.Exists(destination)) {
Directory.CreateDirectory(destination);
}
Files = Directory.GetFileSystemEntries(Source);
foreach (string Element in Files) {
// Sub directories
if (Directory.Exists(Element)) {
copyDirectory(Element, destination + Path.GetFileName(Element));
} else {
// Files in directory
File.Copy(Element, destination + Path.GetFileName(Element), true);
}
}
} catch (Exception ex) {
return false;
}
return true;
}
private static void copyDirectory(string Source, string destination)
{
string[] Files = null;
if (destination[destination.Length - 1] != Path.DirectorySeparatorChar) {
destination += Path.DirectorySeparatorChar;
}
if (!Directory.Exists(destination)) {
Directory.CreateDirectory(destination);
}
Files = Directory.GetFileSystemEntries(Source);
foreach (string Element in Files) {
// Sub directories
if (Directory.Exists(Element)) {
copyDirectory(Element, destination + Path.GetFileName(Element));
} else {
// Files in directory
File.Copy(Element, destination + Path.GetFileName(Element), true);
}
}
}
}

Following code will create directories (if not exists) & then copy files.
// using System.IO;
// for ex. if you want to copy files from D:\A\ to D:\B\
foreach (var f in Directory.GetFiles(#"D:\A\", "*.*", SearchOption.AllDirectories))
{
var fi = new FileInfo(f);
var di = new DirectoryInfo(fi.DirectoryName);
// you can filter files here
if (fi.Name.Contains("FILTER")
{
if (!Directory.Exists(di.FullName.Replace("A", "B"))
{
Directory.CreateDirectory(di.FullName.Replace("A", "B"));
File.Copy(fi.FullName, fi.FullName.Replace("A", "B"));
}
}
}

UnauthorizedAccessException in my own directory

I'm trying to iterate over the items on my start menu, but I keep receiving the UnauthorizedAccessException. I'm the directory's owner and my user is an administrator.
Here's my method (it's in a dll project):
// root = C:\Users\Fernando\AppData\Roaming\Microsoft\Windows\Start Menu
private void walkDirectoryTree(DirectoryInfo root) {
try {
FileInfo[] files = root.GetFiles("*.*");
foreach (FileInfo file in files) {
records.Add(new Record {Path = file.FullName});
}
DirectoryInfo[] subDirectories = root.GetDirectories();
foreach (DirectoryInfo subDirectory in subDirectories) {
walkDirectoryTree(subDirectory);
}
} catch (UnauthorizedAccessException e) {
// do some logging stuff
throw; //for debugging
}
}
The code fail when it starts to iterate over the subdirectories. What else should I do? I've already tried to create the manifest file, but it didn't work.
Another point (if is relevant): I'm just running some unit tests with visual studio (which is executed as administrator).

Based on your description, it appears there is a directory to which your user does not have access when running with UAC enabled. There is nothing inherently wrong with your code and the behavior in that situation is by design. There is nothing you can do in your code to get around the fact that your account doesn't have access to those directories in the context it is currently running.
What you'll need to do is account for the directory you don't have access to. The best way is probably by adding a few extension methods. For example
public static FileInfo[] GetFilesSafe(this DirectoryRoot root, string path) {
try {
return root.GetFiles(path);
} catch ( UnauthorizedAccessException ) {
return new FileInfo[0];
}
}