I am trying to build a recursive search function for a web service that returns a list of files and folders. I created the two methods so they act as recursive search, it first goes and gets the top level contents, then it adds any files to the fileList, and any sub folders to the subFoldersList. We pass in the access level (in our case root) and then the path which you want the information for. If any folders were found it then removes the top folder because it has begun the search for that folder. Then it calls the processDirectories method, which passes back to getFiles the new path location starting the process all over again. Right now for testing my folder structure is below. When it goes to add the second file (profilepic.png) to the list. I get an error "Collection was modified; enumeration operation may not execute." What is causing this error?
Photos
picture1.png
TestFolder
profilepic.png
my code:
public static List<string> fileList = new List<string>();
public static List<string> subFolderList = new List<string>();
static void processDirectories(string access, string Folder)
{
getFiles(access, Folder);
}
static void getFiles(string access, string Folder)
{
var accessToken = new OAuthToken(token, secret);
var api = new DssAPI(ConsumerKey, ConsumerSecret, accessToken);
var folder = api.GetContents(access, Folder);//Get list from WebService
foreach (var item in folder.Contents)//Contents is an IEnumerable
{
if (item.IsDirectory == true)
subFolderList.Add(item.Path);
else
fileList.Add(item.Path);
}
foreach (var subFolder in subFolderList)
{
subFolderList.RemoveAt(0);
processDirectories(root, subFolder);
}
}
Assuming you're not writing this as an academic exercise, you can use Directory.EnumerateFiles and avoid implementing this yourself.
foreach(var png in Directory.EnumerateFiles(sourceDirectory, "*.png", SearchOption.AllDirectories))
{
// do something with the png file
}
Change that:
foreach (var subFolder in subFolderList)
{
subFolderList.RemoveAt(0);
processDirectories(root, subFolder);
}
To:
while (subFolderList.Count > 0)
{
var subFolder = subFolderList[0];
subFolderList.RemoveAt(0);
processDirectories(root, subFolder);
}
A collection cannot be modified while iterating through it, so when you're foreach-ing it and removing items from it inside the iteration, it causes trouble. The workaround is usually using a for loop and manipulating the loop-variable appropriately, but in your case a while loop is simpler.
the problem is here
foreach (var subFolder in subFolderList)
{
subFolderList.RemoveAt(0);
processDirectories(root, subFolder);
}
You're iterating over subFilderList, and you're removing items from it at the same time. The machine doesn't know how to handle that.
What I would suggest, in this case, is probably doing a regular for-loop
Try this,
Public static void GetFilesLocal( string path)
{
foreach (string f in Directory.GetFiles( path))
{
// Add to subFolderList.
}
foreach (string d in Directory.GetDirectories( path))
{
GetFilesLocal( d );
}
}
You cannot go over the collection and modify it, as the error message says. For example the lower foreach is iterating the subFolderList and then you remove the first item. After that the iterators are not valid.
You should be using for loops, if you want to modify the collections, but then you have to remember to decrease the indexer variable if you delete the first item etc.
Related
I am new to C# . I have a text box where i enter the file to search and a 'search' button. on clock of search i want it to populate the files in the folder but i get the above error. Below is my code:
string[] directories = Directory.GetDirectories(#"d:\",
"*",
SearchOption.AllDirectories);
string file = textBox1.Text;
DataGrid dg = new DataGrid();
{
var files = new List<string>();
foreach (DriveInfo d in DriveInfo.GetDrives().Where(x => x.IsReady))
{
try
{
files.AddRange(Directory.GetFiles(d.RootDirectory.FullName, file , SearchOption.AllDirectories));
}
catch(Exception ex)
{
MessageBox.Show("the exception is " + ex.ToString());
//Logger.Log(e.Message); // Log it and move on
}
}
Please help me resolve it . Thanks
The most important rule when searching on a folder which potentially contains inaccessible subfolder is:
Do NOT use SearchOption.AllDirectories!
Use SearchOption.TopDirectoryOnly instead, combined with recursive search for all the accessible directories.
Using SearchOption.AllDirectories, one access violation will break your entire loop even before any file/directory is processed. But if you use SearchOption.TopDirectoryOnly, you only skip what is inaccessible.
There is more difficult way to use Directory.GetAccessControl() per child directory check to see if you have an access to a Directory before hand (this option is rather hard though - I don't really recommend this unless you know exactly how the access system works).
For recursive search, I have this code implemented for my own use:
public static List<string> GetAllAccessibleDirectories(string path, string searchPattern) {
List<string> dirPathList = new List<string>();
try {
List<string> childDirPathList = Directory.GetDirectories(path, searchPattern, SearchOption.TopDirectoryOnly).ToList(); //use TopDirectoryOnly
if (childDirPathList == null || childDirPathList.Count <= 0) //this directory has no child
return null;
foreach (string childDirPath in childDirPathList) { //foreach child directory, do recursive search
dirPathList.Add(childDirPath); //add the path
List<string> grandChildDirPath = GetAllAccessibleDirectories(childDirPath, searchPattern);
if (grandChildDirPath != null && grandChildDirPath.Count > 0) //this child directory has children and nothing has gone wrong
dirPathList.AddRange(grandChildDirPath.ToArray()); //add the grandchildren to the list
}
return dirPathList; //return the whole list found at this level
} catch {
return null; //something has gone wrong, return null
}
}
This is how you call it
List<string> accessibleDirs = GetAllAccessibleDirectories(myrootpath, "*");
Then, you only need to search/add the files among all accessible directories.
Note: this question is quite classical though. I believe there are some other better solutions out there too.
And in case there are some directories which you particularly want to avoid after you get all your accessible directories, you could also filter the List result by LINQ using part of the directory's name as keyword (i.e. Recycle.Bins).
As Ian has specified in his post, do not use recursive file listing (Directory.GetFiles(path, searchPattern, SearchOption.AllDirectories)) in case like yours, since the first exception will stop further processing.
Also, to somewhat alleviate such issues and for better results in general, you should run this program as an Administrator. This can be done by right-clicking your application in windows explorer, and then checking Run this program as an administrator option on Compatibility tab.
Also, you should use code like below to do your search, so the intermediate exceptions do not stop further searching.
static void Main(string[] args) {
string fileToFind = "*.jpg";
var files = new List<string>();
foreach (DriveInfo d in DriveInfo.GetDrives().Where(x => x.IsReady))
files.AddRange(FindDirectory(fileToFind, d.RootDirectory.FullName));
}
/// <summary>
/// This function returns the full file path of the matches it finds.
/// 1. It does not do any parameter validation
/// 2. It searches recursively
/// 3. It eats up any error that occurs when requesting files and directories within the specified path
/// 4. Supports specifying wildcards in the fileToFind parameter.
/// </summary>
/// <param name="fileToFind">Name of the file to search, without the path</param>
/// <param name="path">The path under which the file needs to be searched</param>
/// <returns>Enumeration of all valid full file paths matching the file</returns>
public static IEnumerable<string> FindDirectory(string fileToFind, string path) {
// Check if "path" directly contains "fileToFind"
string[] files = null;
try {
files = Directory.GetFiles(path, fileToFind);
} catch { }
if (files != null) {
foreach (var file in files)
yield return file;
}
// Check all sub-directories of "path" to see if they contain "fileToFInd"
string[] subDirs = null;
try {
subDirs = Directory.GetDirectories(path);
} catch { }
if (subDirs == null)
yield break;
foreach (var subDir in subDirs)
foreach (var foundFile in FindDirectory(fileToFind, subDir))
yield return foundFile;
}
I want to continue my code when error comes up , but i dont know how ...
here's my code :
foreach(string path in Directory.GetDirectories(#"C:\", "*.*", SearchOption.AllDirectories)
{
Console.WriteLine(path);
}
And the error comes on foreach(string path in Directory.GetDirectories(#"C:\", "*.*", SearchOption.AllDirectories) and i don't know how to continue this loop
and the error :
Unauthorized access
And even i run my code as Administrator this error comes up again
Thanks,
The best is to use recursive search and not using SearchOption.AllDirectories, but rather SearchOption.TopDirectoryOnly
If you use SearchOption.AllDirectories, one access violation will break your entire loop even before any file/directory is processed. But if you use SearchOption.TopDirectoryOnly, you only skip what is inaccessible.
Thus, to do it, you can create a method which receives a directory path as input. And in that method, if the input directory have child directory(ies) (see Directory.GetDirectories(string path) method, you call the method again for each child directory (recursive call) before you process all the files in the directory. Else, get the files (see Directory.GetFiles) in the directory and process them immediately.
Then for the method above, one way is to prevent the code crash when you cannot access certain file/directory is by using try-catch block for each child directory reading and file reading. This way, if one file/folder cannot be accessed, your code will still be running, finding the processing the next file/directory.
Alternatively, you can use Directory.GetAccessControl() per child directory check to see if you have an access to a Directory before hand (this option is rather hard though).
Edit (code added):
Something like this will do:
public static List<string> GetAllAccessibleDirectories(string path, string searchPattern) {
List<string> dirPathList = new List<string>();
try {
List<string> childDirPathList = Directory.GetDirectories(path, searchPattern, SearchOption.TopDirectoryOnly).ToList(); //use TopDirectoryOnly
if (childDirPathList == null || childDirPathList.Count <= 0) //this directory has no child
return null;
foreach (string childDirPath in childDirPathList) { //foreach child directory, do recursive search
dirPathList.Add(childDirPath); //add the path
List<string> grandChildDirPath = GetAllAccessibleDirectories(childDirPath, searchPattern);
if (grandChildDirPath != null && grandChildDirPath.Count > 0) //this child directory has children and nothing has gone wrong
dirPathList.AddRange(grandChildDirPath.ToArray()); //add the grandchildren to the list
}
return dirPathList; //return the whole list found at this level
} catch {
return null; //something has gone wrong, return null
}
}
And to call it, you can do something like this
string rootpath = #"C:\DummyRootFolder";
List<string> dirList = GetAllAccessibleDirectories(rootpath, "*.*"); //you get all accessible directories here
In the dirList you will get all the directories that you search for, and if there is access violation along the way, it will only affects sub-directories search due to the try-catch block.
Note that the rootpath is excluded in the method. But if you want to add it to the list too, you could simply do
dirList.Insert(0, path); //do this after you get dirList
There are also more complicated ways of doing this by using Directory.GetAccessControl and PermissionSet
Hope it may clarify.
According to the documentation, you should look at EnumerateDirectories for performance reasons:
https://msdn.microsoft.com/en-us/library/c1sez4sc(v=vs.110).aspx
Also, it appears that this question has already been answered before:
Directory.EnumerateFiles => UnauthorizedAccessException
Hope this helps!
How about this:
foreach (string path in Directory.GetDirectories(#"C:\", "*.*", SearchOption.AllDirectories)) {
try {
Console.WriteLine(path);
} catch (Exception ex) {
Console.WriteLine("Unable to access directories in path: " + path);
}
}
hi
i am creating a application and i want to know the each and every file which is present under that one folder .i.e. how can i iterate through a root directory and get the each files visit at list once.
If you just need to list them all at once, you can just use the overload for GetFiles that includes the option.
string[] filePaths = Directory.GetFiles(#"c:\MyDir\", "*.*", SearchOption.AllDirectories);
Obviously, in a web app you wouldn't likely have access to "c:\MyDir", so you can replace that with a variable holding the results of a MapPath call like so:
var rootDir = Server.MapPath("~/App_Data");
Use the Directory.EnumerateFiles(String, String, SearchOption) function with SearchOption.AllDirectories:
foreach (var file in Directory.EnumerateFiles(#"c:\", "*.txt", SearchOption.AllDirectories))
{
// Do stuff here
}
EnumerateFiles method is way faster than GetFiles method since it actually just returns the enumerator and does not actually access the files until they are red.
You can use the DirectoryInfo and FileInfo classes as well as a recursive function.
void Main()
{
DirectoryInfo info = new DirectoryInfo(#"C:\Personal");
ListContents(info);
}
public void ListContents(DirectoryInfo info)
{
foreach(var dir in info.GetDirectories())
{
ListContents(dir);
}
foreach(var file in info.GetFiles())
{
Console.WriteLine(file.FullName);
}
}
I want to index all my music files and store them in a database.
I have this function that i call recusively, starting from the root of my music drive.
i.e.
start > ReadFiles(C:\music\);
ReadFiles(path){
foreach(file)
save to index;
foreach(directory)
ReadFiles(directory);
}
This works fine, but while running the program the amount of memory that is used grows and grows and.. finally my system runs out of memory.
Does anyone have a better approach that doesnt need 4GB of RAM to complete this task?
Best Regards, Tys
Alxandr's queue based solution should work fine.
If you're using .NET 4.0, you could also take advantage of the new Directory.EnumerateFiles method, which enumerates files lazily, without loading them all in memory:
void ReadFiles(string path)
{
IEnumerable<string> files =
Directory.EnumerateFiles(
path,
"*",
SearchOption.AllDirectories); // search recursively
foreach(string file in files)
SaveToIndex(file);
}
Did you check for the . and .. entries that show up in every directory except the root?
If you don't skip those, you'll have an infinite loop.
You can implement this as a queue. I think (but I'm not sure) that this will save memory. At least it will free up your stack. Whenever you find a folder you add it to the queue, and whenever you find a file you just read it. This prevents recursion.
Something like this:
Queue<string> dirs = new Queue<string>();
dirs.Enqueue("basedir");
while(dirs.Count > 0) {
foreach(directory)
dirs.Enqueue(directory);
ReadFiles();
}
Beware, though, that EnumerateFiles() will stop running if you don't have access to a file or if a path is too long or if some other exception occurs. This is what I use for the moment to solve those problems:
public static List<string> getFiles(string path, List<string> files)
{
IEnumerable<string> fileInfo = null;
IEnumerable<string> folderInfo = null;
try
{
fileInfo = Directory.EnumerateFiles(str);
}
catch
{
}
if (fileInfo != null)
{
files.AddRange(fileInfo);
//recurse through the subfolders
fileInfo = Directory.EnumerateDirectories(str);
foreach (string s in folderInfo)
{
try
{
getFiles(s, files);
}
catch
{
}
}
}
return files;
}
Example use:
List<string> files = new List<string>();
files = folder.getFiles(path, files);
My solution is based on the code at this page: http://msdn.microsoft.com/en-us/library/vstudio/bb513869.aspx.
Update: A MUCH faster method to get files recursively can be found at http://social.msdn.microsoft.com/Forums/vstudio/en-US/ae61e5a6-97f9-4eaa-9f1a-856541c6dcce/directorygetfiles-gives-me-access-denied?forum=csharpgeneral. Using Stack is new to me (I didn't even know it existed), but the method seems to work. At least it listed all files on my C and D partition with no errors.
It could be junction folders wich leads to infinite loop when doing recursion but i am not sure , check this out and see by yourself . Link: https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/mklink
I am just learning C# (have been fiddling with it for about 2 days now) and I've decided that, for leaning purposes, I will rebuild an old app I made in VB6 for syncing files (generally across a network).
When I wrote the code in VB 6, it worked approximately like this:
Create a Scripting.FileSystemObject
Create directory objects for the source and destination
Create file listing objects for the source and destination
Iterate through the source object, and check to see if it exists in the destination
if not, create it
if so, check to see if the source version is newer/larger, and if so, overwrite the other
So far, this is what I have:
private bool syncFiles(string sourcePath, string destPath) {
DirectoryInfo source = new DirectoryInfo(sourcePath);
DirectoryInfo dest = new DirectoryInfo(destPath);
if (!source.Exists) {
LogLine("Source Folder Not Found!");
return false;
}
if (!dest.Exists) {
LogLine("Destination Folder Not Found!");
return false;
}
FileInfo[] sourceFiles = source.GetFiles();
FileInfo[] destFiles = dest.GetFiles();
foreach (FileInfo file in sourceFiles) {
// check exists on file
}
if (optRecursive.Checked) {
foreach (DirectoryInfo subDir in source.GetDirectories()) {
// create-if-not-exists destination subdirectory
syncFiles(sourcePath + subDir.Name, destPath + subDir.Name);
}
}
return true;
}
I have read examples that seem to advocate using the FileInfo or DirectoryInfo objects to do checks with the "Exists" property, but I am specifically looking for a way to search an existing collection/list of files, and not live checks to the file system for each file, since I will be doing so across the network and constantly going back to a multi-thousand-file directory is slow slow slow.
Thanks in Advance.
The GetFiles() method will only get you files that does exist. It doesn't make up random files that doesn't exist. So all you have to do is to check if it exists in the other list.
Something in the lines of this could work:
var sourceFiles = source.GetFiles();
var destFiles = dest.GetFiles();
foreach (var file in sourceFiles)
{
if(!destFiles.Any(x => x.Name == file.Name))
{
// Do whatever
}
}
Note: You have of course no guarantee that something hasn't changed after you have done the calls to GetFiles(). For example, a file could have been deleted or renamed if you try to copy it later.
Could perhaps be done nicer somehow by using the Except method or something similar. For example something like this:
var sourceFiles = source.GetFiles();
var destFiles = dest.GetFiles();
var sourceFilesMissingInDestination = sourceFiles.Except(destFiles, new FileNameComparer());
foreach (var file in sourceFilesMissingInDestination)
{
// Do whatever
}
Where the FileNameComparer is implemented like so:
public class FileNameComparer : IEqualityComparer<FileInfo>
{
public bool Equals(FileInfo x, FileInfo y)
{
return Equals(x.Name, y.Name);
}
public int GetHashCode(FileInfo obj)
{
return obj.Name.GetHashCode();
}
}
Untested though :p
One little detail, instead of
sourcePath + subDir.Name
I would use
System.IO.Path.Combine(sourcePath, subDir.Name)
Path does reliable, OS independent operations on file- and foldernames.
Also I notice optRecursive.Checked popping out of nowhere. As a matter of good design, make that a parameter:
bool syncFiles(string sourcePath, string destPath, bool checkRecursive)
And since you mention it may be used for large numbers of files, keep an eye out for .NET 4, it has an IEnumerable replacement for GetFiles() that will let you process this in a streaming fashion.