I have the following hard drive backup code that compares the .LastWriteTime() time of each file before copying and it is running slower than I expected. My assumption is that it should run pretty fast (on the order of a few minutes) if there are no files to update. I'm finding that it is still taking over an hour for 210 GB via USB3.0. I'm wondering if there are any unnecessary, time-consuming parts of my code that I can improve. I was also thinking about putting each directorycopy() call on a different thread (at least for the first level of directories, but was unsure if that was bad practice).
The code is mostly borrowed from:
https://learn.microsoft.com/en-us/dotnet/standard/io/how-to-copy-directories
I made changes to ignore the $Recycle Bin folder, log the files that have changed or had issues such as long filenames and being deliberate in how the Exceptions were handled. But most importantly, I added a check to see which file is newer before copying.
private void DirectoryCopy(string sourceDirName, string destDirName, bool copySubDirs)
{
// Get the subdirectories for the specified directory.
DirectoryInfo dir = new DirectoryInfo(sourceDirName);
if (sourceDirName.Contains("$")) // avoids $Recycle Bin
return;
if (!dir.Exists)
{
textb_Status.AppendText("Issue with " + dir.FullName + " This folder will not be compied.");
return;
//throw new DirectoryNotFoundException(
// "Source directory does not exist or could not be found: "
// + sourceDirName);
}
DirectoryInfo[] dirs = dir.GetDirectories();
// If the destination directory doesn't exist, create it.
if (!Directory.Exists(destDirName))
{
Directory.CreateDirectory(destDirName);
}
// Get the files in the directory and copy them to the new location.
FileInfo[] files = dir.GetFiles();
foreach (FileInfo file in files)
{
string temppath = Path.Combine(destDirName, file.Name);
try
{
file.CopyTo(temppath);
}
catch (PathTooLongException)
{
textb_Status.AppendText("Filename Too long \n " + file.FullName + "\n");
}
catch (IOException ex)
{
FileInfo sourcefile = new FileInfo(file.FullName);
FileInfo destFile = new FileInfo(temppath);
int CompareValue = sourcefile.LastWriteTime.CompareTo(destFile.LastWriteTime); //<0==> Earlier (old) =0 ==> same >0 Later (newer)
//textb_Status.AppendText("CompareValue: " + CompareValue + "\n");
if (CompareValue > 0) // Represents newer file
{
file.CopyTo(temppath, true);
textb_Status.AppendText("Updated: " + file.FullName + "\n");
}
}
catch (Exception ex2)
{
textb_Status.AppendText("Issue with " + file.FullName + "\n");
textb_Status.AppendText("Error Message \n");
textb_Status.AppendText(ex2.Message + "\n");
}
}
// If copying subdirectories, copy them and their contents to new location.
if (copySubDirs)
{
foreach (DirectoryInfo subdir in dirs)
{
string temppath = Path.Combine(destDirName, subdir.Name);
DirectoryCopy(subdir.FullName, temppath, copySubDirs);
}
}
}
I'm expecting the backup process to be on the order of a few minutes if there are only a few files to update.
I don't think, it's the amount of data, which slows down the process, but the number of files. The initial file access (check if it exists, get the stats) is pretty expensive, regardless of file size. Furthermore, many people consider using exceptions for control-flow bad style and throwing and catching exceptions may be quite expensive. And from your use case (ie most of the files are unchanged) there are MANY exceptions thrown.
Also depending on your disks (SSD or HDD), multithreaded reads and writes may be a very bad idea and slow down the whole process.
And depending on the implementation of File.Copy() you may be better off, checking the target first, and only do the Copy if it's really necessary. But this is something you can only know after a benchmark.
Thank you #derpirscher. For each file, I checked if it existed before trying to write. This way, no exception is thrown. The drive was checked in about 5 seconds! I modified a few files deep in the source directory to make sure they were being detected and copied. They were.
I had the sense Exceptions were expensive, I just didn't know it was this bad....great lesson!
My code is below. Note: I was receiving errors when trying to get files from my System Volume Information folder, so I started with a check to make sure sourceDirName did not equal that directory.
private void DirectoryCopy(string sourceDirName, string destDirName, bool copySubDirs)
{
if (sourceDirName.Contains("System Volume Information"))
return;
//textb_Status.AppendText("Current Directory: " + sourceDirName +"\n");
DirectoryInfo[] dirs = null;
// Get the subdirectories for the specified directory.
DirectoryInfo dir = new DirectoryInfo(sourceDirName);
if (sourceDirName.Contains("$")) // avoids $Recycle Bin
return;
if (!dir.Exists)
{
textb_Status.AppendText("Issue with " + dir.FullName + " This folder will not be compied.");
return;
//throw new DirectoryNotFoundException(
// "Source directory does not exist or could not be found: "
// + sourceDirName);
}
{
dirs = dir.GetDirectories();
// If the destination directory doesn't exist, create it.
if (!Directory.Exists(destDirName))
{
Directory.CreateDirectory(destDirName);
}
// Get the files in the directory and copy them to the new location.
FileInfo[] files = dir.GetFiles();
foreach (FileInfo file in files)
{
string temppath = Path.Combine(destDirName, file.Name);
try
{
if (File.Exists(temppath)) // Check for newer
{
FileInfo sourcefile = new FileInfo(file.FullName);
FileInfo destFile = new FileInfo(temppath);
int CompareValue = sourcefile.LastWriteTime.CompareTo(destFile.LastWriteTime); //<0==> Earlier (old) =0 ==> same >0 Later (newer)
//textb_Status.AppendText("CompareValue: " + CompareValue + "\n");
if (CompareValue > 0) // Represents newer file
{
file.CopyTo(temppath, true);
textb_Status.AppendText("********** Updated: " + file.FullName + "********* \n");
}
}
else
{
file.CopyTo(temppath);
}
}
catch (PathTooLongException)
{
textb_Status.AppendText("Filename Too long \r\n\n " + file.FullName + "\r\n\n");
}
catch (IOException ex)
{
FileInfo sourcefile = new FileInfo(file.FullName);
FileInfo destFile = new FileInfo(temppath);
int CompareValue = sourcefile.LastWriteTime.CompareTo(destFile.LastWriteTime); //<0==> Earlier (old) =0 ==> same >0 Later (newer)
//textb_Status.AppendText("CompareValue: " + CompareValue + "\n");
if (CompareValue > 0) // Represents newer file
{
file.CopyTo(temppath, true);
textb_Status.AppendText("Updated: " + file.FullName + "\n");
}
}
catch (Exception ex2)
{
textb_Status.AppendText("Issue with " + file.FullName + "\n");
textb_Status.AppendText("Error Message \n");
textb_Status.AppendText(ex2.Message + "\n\n");
}
}
// If copying subdirectories, copy them and their contents to new location.
if (copySubDirs)
{
foreach (DirectoryInfo subdir in dirs)
{
string temppath = Path.Combine(destDirName, subdir.Name);
DirectoryCopy(subdir.FullName, temppath, copySubDirs);
}
}
}
}
Related
I'm having issues moving files from a listbox to a newly created folder. I get: Cannot create a file when that file already exists.
public void CreateFolders()
{
//create folders
string folder1 = pattern.Substring(0, 2);
string folder2 = pattern.Substring(3, 2);
string folder3 = pattern.Substring(6, 2);
Directory.CreateDirectory("c:\\destinationfolder" + "\\" + folder1);
Directory.CreateDirectory("c:\\destinationfolder" + "\\" + folder1 + "\\" + folder2);
Directory.CreateDirectory("c:\\destinationfolder" + "\\" + folder1 + "\\" + folder2 + "\\" + folder3);
var destinationDirectoryFinal = Directory.CreateDirectory("c:\\destinationfolder" + "\\" + folder1 + "\\" + folder2 + "\\" + folder3);
destinationDirectory = destinationDirectoryFinal.FullName.ToString();
}
public void MoveFiles()
{
try
{
//Move files from listbox to newly created folders
foreach (string files in listBox1.Items)
{
File.Move(files, destinationDirectory);
}
}
catch (Exception ex)
{
MessageBox.Show("Error: " + ex);
}
}
The problem you're running into here is that you're trying to move a file to a directory that already contains a file with that name.
There are a couple of options you can choose from:
Option One
Check if the file exists before attempting the Move
foreach (var file in listBox1.Items)
{
// Only move the file if it doesn't already exist
if (!File.Exists(Path.Combine(destinationDirectory, Path.GetFileName(file))))
{
File.Move(file, destinationDirectory);
}
}
Option Two
Always overwrite the file if it exists. We can do this in a two step process - first by calling File.Copy with the "overwrite" parameter set to true, and then by calling File.Delete to delete the file in the original location:
foreach (var file in listBox1.Items)
{
// If the destination file already exists, overwrite it. Then delete the original
File.Copy(file, Path.Combine(destinationDirectory, Path.GetFileName(file)), true);
File.Delete(file);
}
Note:
Another thing you can do to prevent errors (in either case) is to ensure that the source file exists, and that the source directory is not the same as the destination directory before you try anything:
foreach (var file in listBox1.Items)
{
// Ensure that the file exists and that the source
// and destination directories are not the same
if (!(File.Exists(file)) ||
Path.GetDirectoryName(file).Equals(
destinationDirectory, StringComparison.OrdinalIgnoreCase))
{
continue; // Continue to the next loop iteration without doing anything
}
// Rest of loop code here...
}
I am trying to delete some files in a directory from a separate thread, but sometimes the delete doesn't work.
DirectoryInfo dirInfo = new DirectoryInfo(Directory.GetCurrentDirectory());
FileInfo[] fileNames = dirInfo.GetFiles("*.*");
foreach (FileInfo fileName in fileNames)
{
string destinationFilename = cncDestinationDirectory + #"\" + dirInfo.Name + #"\" + fileName.Name;
if (File.Exists(destinationFilename))
File.Delete(destinationFilename);
File.Move(fileName.FullName, destinationFilename);
}
My goal is to move some files in a directory but, as I know the File.Move doesn't work if the destination file already exists. So, I check if the file exists and if it is true, I delete this file, then move to the original.
The File.Delete also cause a prematurely exit from the function.
The current directory is not the same folder as the executable is running because I set previously it into another folder.
How can I avoid this error? And still move the files in the destination directory?
The problem there is that the access to the file is denied because of the read only attribute of the file.
So, I set all my files attributes as normal as the follow:
DirectoryInfo dirInfo = new DirectoryInfo(Directory.GetCurrentDirectory());
FileInfo[] fileNames = dirInfo.GetFiles("*.*");
foreach (FileInfo fileName in fileNames)
{
if (fileName.Extension == ".iso")
return;
string destinationFilename = cncDestinationDirectory + #"\" + dirInfo.Name + #"\" + fileName.Name;
fileName.Attributes = FileAttributes.Normal;
if (File.Exists(destinationFilename))
{
File.SetAttributes(destinationFilename, FileAttributes.Normal);
File.Delete(destinationFilename);
}
File.Move(fileName.FullName, destinationFilename);
}
You need to decide how to handle error cases like you suggest in your question. It's entirely possible that between checking the file exists and then deleting it, that the file has been opened by another process. You can catch an exception around the File.Delete and then not move the origin file if it throws, but you will end up with files that haven't moved. There's nothing you can do about it.
DirectoryInfo dirInfo = new DirectoryInfo(Directory.GetCurrentDirectory());
FileInfo[] fileNames = dirInfo.GetFiles("*.*");
foreach (FileInfo fileName in fileNames)
{
string destinationFilename = cncDestinationDirectory + #"\" + dirInfo.Name + #"\" + fileName.Name;
try
{
if (File.Exists(destinationFilename))
File.Delete(destinationFilename);
File.Move(fileName.FullName, destinationFilename);
}
catch(IOException exception)
{
Console.WriteLine($"Can't move file { filename.FullName}");
}
}
im doing something similar to the cmd tree. And what im planning to check first C:\ than check the first folder in it if the first folder contains something in it go check it and so on until i get DirectoryNotFoundException. If i get such i want to skip the first folder and check the second one how to do that ?
static string path = #"C:\";
static void Main()
{
try
{
DirectoryInfo di = new DirectoryInfo(path);
DirectoryInfo[] fileNames = di.GetDirectories();
}
catch (DirectoryNotFoundException)
{
DirectoryInfo di = new DirectoryInfo(path);
DirectoryInfo[] fileNames = di.GetDirectories();
//i need to edit the picking of filenames.Something like PickSecond/Next
}
Console.ReadKey();
}
If you want to print the directory tree you can use recursion without getting any DirectoryNotFoundException. The following code will print to the console all the sub-directories starting from a specific path. The recursion works this way: every time a folder is found, if it's not empty then go to explore its sub-directories, otherwise go and check the next folder.
static void DirectoryTree(string directory, string indent = "")
{
try
{
DirectoryInfo currentDirectory = new DirectoryInfo(directory);
DirectoryInfo[] subDirectories = currentDirectory.GetDirectories();
// Print all the sub-directories in a recursive way.
for (int n = 0; n < subDirectories.Length; ++n)
{
// The last sub-directory is drawn differently.
if (n == subDirectories.Length - 1)
{
Console.WriteLine(indent + "└───" + subDirectories[n].Name);
DirectoryTree(subDirectories[n].FullName, indent + " ");
}
else
{
Console.WriteLine(indent + "├───" + subDirectories[n].Name);
DirectoryTree(subDirectories[n].FullName, indent + "│ ");
}
}
}
catch (Exception e)
{
// Here you could probably get an "Access Denied" exception,
// that's very likely if you are exploring the C:\ folder.
Console.WriteLine(e.Message);
}
}
static void Main()
{
// Consider exploring a more specific folder. If you just type "C:\"
// you are requesting to view all the folders in your computer.
var path = #"C:\SomeFolder" ;
Console.WriteLine(path);
if(Directory.Exists(path))
DirectoryTree(path);
else
Console.WriteLine("Invalid Directory");
Console.ReadKey();
}
I've asked a very similar question before here. But that was about getting directories, and this is about files. And these codes are a bit different from each other. Ever since I've been trying to convert this to make it look like the answer on my old question, I haven't been able to make it work.
string[] files = Directory.GetFiles(ScanPath, "*.*", System.IO.SearchOption.AllDirectories);
DateTime From = DateTime.Now.AddHours(-24);
DateTime To = DateTime.Now;
foreach (string name in files)
{
FileInfo file = new FileInfo(name);
string fullname = file.FullName;
if (file.LastWriteTime >= From & file.LastWriteTime <= To && file.Length >= ScanSize)
Console.WriteLine(file.FullName + " ; " + "last changed at " + " ; " + file.LastWriteTime.ToString());
}
I've been getting the same errors as I explained in the other question. Because I don't know where to put the code of the foreach in a recursion. Since it's not an enumeration but a Directory.GetFiles().
The error occurs with:
Directory.GetFiles(ScanPath, "*", SearchOption.AllDirectories);
because this gets all the files of the directories at once. But if I remove it, it only gets the files in the given path, without any of the files in the subdirectories. So I was told to apply recursion.
I am the administrator of the system and I plan to run this on the entire data drive. D:\
I'm hoping anyone here knows a good example.
Your app could not have access rights to some folders, for others you can use the following code:
void DiscoverDirs(string where, List<string> files, Func<FileInfo, bool> filter)
{
try
{
var di = new DirectoryInfo(where);
files.AddRange(di.EnumerateFiles().Where(filter).Select(x => x.FullName));
foreach (var dir in Directory.GetDirectories(where))
{
DiscoverDirs(dir, files, filter);
}
}
catch
{
// no access fo this dir, ignore
}
}
Usage:
DateTime From = DateTime.Now.AddHours(-24);
DateTime To = DateTime.Now;
var ScanSize = 5*1024*1024;
var list = new List<string>();
DiscoverDirs(#"C:\", list,
file => file.LastWriteTime >= From & file.LastWriteTime <= To && file.Length >= ScanSize);
foreach (string name in list)
{
FileInfo file = new FileInfo(name);
string fullname = file.FullName;
Console.WriteLine(file.FullName + " ; " + "last changed at " + " ; " + file.LastWriteTime.ToString());
}
You might be getting "UnauthorizedAccessException" while accessing the some of the system directories.List of the directory causing the problems are directories which are actually just redirection to other directory.
May be you can tried out the following code if it helps-
try
{
foreach (String file in Directory.GetFiles(directoryName, pattern, SearchOption.TopDirectoryOnly))
{
// do stuff
}
catch (UnauthorizedAccessException uae)
{
//handle
}
catch (Exception e)
{
//handle
}
Alternative:
string[] directories = Directory.GetDirectories(ScanPath);
foreach (string directory in directories)
{
string[] filesinCurrentDirectory = Directory.GetFiles(directory, "*.*", System.IO.SearchOption.AllDirectories);
foreach (string file in filesinCurrentDirectory)
{
MessageBox.Show(file);
}
}
I'm having trouble, I got this code:
DirectoryInfo di = new DirectoryInfo(dir);
FileInfo[] rgFiles = di.GetFiles();
DirectoryInfo[] d = di.GetDirectories();
if(rgFiles != null && d != null) {
foreach (FileInfo fi in rgFiles)
{
foreach (DirectoryInfo dii in d)
{
using (ZipFile zip = ZipFile.Read(locateZipFile()))
{
zip.AddFile(fi.FullName, "");
zip.AddDirectory(dii.FullName,dii.Name);
toolStripStatusLabel1.Text = "Inserting " + fi.Name;
toolStripStatusLabel1.Text = "Inserting " + dii.Name + " and all of it's contents";
MessageBox.Show("Inserted the file " + fi.Name);
MessageBox.Show("Inserted the folder " + dii.Name + " and all contents in it.");
zip.Save();
}
}
}
Everything works great, but when I'm trying to add a file that is named the same in the zip, it does not overwrite it, which i want it to.. Any ideas on how i can do that? thanks.
You can use the UpdateFile method.
zip.UpdateFile(fi.FullName, "");
This method adds a file to a zip archive, or, if the file already exists in the zip archive, this method Updates the content of that given filename in the zip archive. The UpdateFile method might more accurately be called "AddOrUpdateFile".
Upon success, there is no way for the application to learn whether the file was added versus updated.
Before the line
zip.AddFile(fi.FullName, "");
you must test if the name already exists in entries. If yes, remove it and then insert it again.