I want to get all files (recursively) in my root directory / of linux OS.
I am using the following code
var files = System.IO.Directory.GetFiles("/", "*", SearchOption.AllDirectories);
But its taking so much time which is making my code inefficient. Is there any way to find files recursively in much more efficient way ?
I thought of this answer but again its taking even more time than System.IO.Directory.GetFiles()
I think Matthew Watson is right, whatever you do is going to work slowly because of the disc. The only thing I would try to do is do it parallely. If you have other things to do while waiting on result, maybe async is a good choiche.
using System;
using System.Threading.Tasks;
using System.IO;
using System.Collections.Concurrent;
namespace Test
{
class Program
{
static void Main(string[] args)
{
GetFilesOnRoot("*");
Console.ReadLine();
}
private static ConcurrentBag<string> FilesBag;
private static void GetFilesOnRoot(string filter)
{
FilesBag = new ConcurrentBag<string>();
DirectoryInfo dirRoot = new DirectoryInfo("/");
GetDirTree(dirRoot, "*");
}
private static void GetDirTree(DirectoryInfo dr, string filter)
{
FileInfo[] files = null;
DirectoryInfo[] subDirs = null;
try
{
files = dr.GetFiles(filter + ".*");
}
catch(Exception) { }
if (files != null)
{
Parallel.ForEach(files, (FileInfo item) => { FilesBag.Add(item.Name); });
subDirs = dr.GetDirectories();
Parallel.ForEach(subDirs, (DirectoryInfo item) => { GetDirTree(item,filter); });
}
}
public static async Task GetFilesOnRootAsync(string filter)
{
await Task.Run(() => {
GetFilesOnRoot(filter);
});
}
}
}
Related
I am trying to figure out how to continue from a UnauthorizedAccessException error. I am trying to list all files in my drives and have used the try/ctch statements and continue but nothing seems to work.
Here is the code:
using System;
using System.IO;
using System.Linq;
public class Program
{
public static void Main(string[] args)
{
foreach(DriveInfo d in DriveInfo.GetDrives().Where(x = > x.IsReady))
{
try
{
string[] files = Directory.GetFiles(d.RootDirectory.FullName, #"*.*", SearchOption.AllDirectories).ToArray();
foreach(string file in files)
{
Console.Write(file);
}
}
catch (UnauthorizedAccessException e)
{
Console.WriteLine(e.Message);
continue;
}
}
}
}
The exeception catches that I cannot access 'C:\Documents and Settings' but then terminates the code instead of listing the rest of the files that I can access. I have read up and know this is a problem/bug with net but cannot find out how to continue even after catching the exception.
You need to use recursion to query one directory at a time. This way you can catch any unauthorized exceptions and continue your search.
static void Main()
{
foreach (DriveInfo info in DriveInfo.GetDrives().Where(c => c.IsReady))
{
foreach (string file in DirSearch(info.RootDirectory.FullName))
{
Console.WriteLine(file);
}
}
Console.ReadKey(true);
}
private static IList<string> DirSearch(string path)
{
List<string> files = new List<string>();
try
{
foreach (string dir in Directory.GetDirectories(path))
{
foreach (string file in Directory.GetFiles(dir))
{
files.Add(file);
}
files.AddRange(DirSearch(dir));
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
return files;
}
I bumped into the problem with copying files/dirs. Been struggling almost for whole day.
I have to copy from root dir files and its dires with files and subdirs.
Actually, i've made something. However, every time i run in the stackoverflow error.
abstract class SystemOperations {
public virtual void SearchFiles() { }
public virtual void SearchDirectories() { }
public abstract void CreateDirectory(string DIR);
public abstract void CloneContent(string DIR);
public abstract void CreateJSON(string DIR);
public void ExecuteCopying(string DIR) {
CreateDirectory(DIR);
CloneContent(DIR);
CreateJSON(DIR);
}
}
class FileOperations : SystemOperations {
DirectoryInfo _MainPath;
public DirectoryInfo MainPath {
get { return _MainPath; }
set { _MainPath = value; }
}
public FileOperations(DirectoryInfo MainPath) {
this.MainPath = MainPath;
}
#region Unnecessary for current task
public override void SearchFiles() {
string path = "";
FileInfo[] files = MainPath.GetFiles();
foreach (FileInfo file in files) {
path = file.Name;
}
}
public override void SearchDirectories() {
string path = "";
DirectoryInfo[] directories = MainPath.GetDirectories();
foreach (DirectoryInfo directory in directories) {
path = directory.Name;
}
}
#endregion
public override void CreateDirectory(string DIR) {
string newFolder = Path.Combine(MainPath + "", DIR);
Directory.CreateDirectory(newFolder);
}
public override void CloneContent(string DIR) {
foreach (var directory in Directory.GetDirectories(MainPath + "")) {
string dir = Path.GetFileName(directory);
CloneContent(Path.Combine(MainPath + "", dir));
}
foreach (var file in Directory.GetFiles(MainPath + "")) {
File.Copy(file, Path.Combine(MainPath + "", Path.GetFileName(file)), true);
}
}
public override void CreateJSON(string DIR) {
if (!Directory.Exists(DIR)) {
var asd = new DirectoryInfo(DIR);
}
}
}
class Program {
static void Main() {
SystemOperations task = new FileOperations(new DirectoryInfo(#"D:\\LAK"));
task.ExecuteCopying("COPY");
}
}
So, the function CloneContent has to copy in each dir/subdirs files. But its recursive func and as i written above, i run to the error. And dont know how to fix this one. Thank u!
There is some kind of problem with the way you are trying to determine which directory you need to search next, the use of MainPath looks wrong to me.
Personally i also always prefer to have a secondary stop condition to avoid a StackOverflowException, like the maxrunCount i use below.
If you want a recursive directory lookup you should rewrite your code to something like
void Main()
{
string MainPath = "D:\\LAK";
// unless your directory is actually named \LAK:) you should use either #"D:\LAK" or "d:\\LAK"
CloneContent(MainPath,1000);
}
public void CloneContent(string directoryToSearch, int maxrunCount)
{
if(maxrunCount==0)
return;
System.Diagnostics.Debug.Print(directoryToSearch);
string[] directories = null;
try
{
directories = Directory.GetDirectories(directoryToSearch);
}
catch(UnauthorizedAccessException ex) {
System.Diagnostics.Debug.Print($"No access to dir {directoryToSearch}");
directories = new string[0];
}
// ensure you have access to the current directoryToSearch
foreach (var directory in directories)
{
CloneContent(directory,--maxrunCount);
}
System.Diagnostics.Debug.Print($"cloning {directoryToSearch}");
// .... do the actual cloning here,
// you will end up here when there are no more subdirectories on the current branch
}
For a recursive method to work, it must have at least one "exit" condition - the point at which it's done its job and can unwind the stack. In our case, it would be when there are no more direcories or files to copy from the source to the destination.
One way of writing this method would take in a source directory and a destination directory, and then it can recursively call itself for each sub-directory:
public static void CloneContent(string sourceDir, string destDir)
{
// If the source directory doesn't exist, return
if (!Directory.Exists(sourceDir)) return;
// Create destination if needed
Directory.CreateDirectory(destDir);
// Copy files from this directory to the new path
foreach (string file in Directory.GetFiles(sourceDir))
{
File.Copy(file, Path.Combine(destDir, Path.GetFileName(file)));
}
// Recursively call this method for each sub directory
foreach (string subDir in Directory.GetDirectories(sourceDir))
{
string dirName = Path.GetFileName(subDir);
string newSource = Path.Combine(sourceDir, dirName);
string newDest = Path.Combine(destDir, dirName);
CloneContent(newSource, newDest);
}
}
I'm trying to write a command line tool that modifies some code using Roslyn. Everything seems to go well: the solution is opened, the solution is changed, the Workspace.TryApplyChanges method returns true. However no actual files are changed on disk. What's up? Below is the top level code I'm using.
static void Main(string[] args)
{
var solutionPath = args[0];
UpdateAnnotations(solutionPath).Wait();
}
static async Task<bool> UpdateAnnotations(string solutionPath)
{
using (var workspace = MSBuildWorkspace.Create())
{
var solution = await workspace.OpenSolutionAsync(solutionPath);
var newSolution = await SolutionAttributeUpdater.UpdateAttributes(solution);
var result = workspace.TryApplyChanges(newSolution);
Console.WriteLine(result);
return result;
}
}
I constructed a short program using your code and received the results I expected - the problem appears to reside within the SolutionAttributeUpdater.UpdateAttributes method. I received these results using the following implementation with your base main and UpdateAnnotations-methods:
public class SolutionAttributeUpdater
{
public static async Task<Solution> UpdateAttributes(Solution solution)
{
foreach (var project in solution.Projects)
{
foreach (var document in project.Documents)
{
var syntaxTree = await document.GetSyntaxTreeAsync();
var root = syntaxTree.GetRoot();
var descentants = root.DescendantNodes().Where(curr => curr is AttributeListSyntax).ToList();
if (descentants.Any())
{
var attributeList = SyntaxFactory.AttributeList(
SyntaxFactory.SingletonSeparatedList(
SyntaxFactory.Attribute(SyntaxFactory.IdentifierName("Cookies"), SyntaxFactory.AttributeArgumentList(SyntaxFactory.SeparatedList(new[] { SyntaxFactory.AttributeArgument(
SyntaxFactory.LiteralExpression(
SyntaxKind.StringLiteralExpression, SyntaxFactory.Literal(#"Sample"))
)})))));
root = root.ReplaceNodes(descentants, (node, n2) => attributeList);
solution = solution.WithDocumentSyntaxRoot(document.Id, root);
}
}
}
return solution;
}
}
It was tested using the following class in the sample solution:
public class SampleClass<T>
{
[DataMember("Id")]
public int Property { get; set; }
[DataMember("Id")]
public void DoStuff()
{
DoStuff();
}
}
And it resulted in the following Output:
public class SampleClass<T>
{
[Cookies("Sample")] public int Property { get; set; }
[Cookies("Sample")] public void DoStuff()
{
DoStuff();
}
}
If you take a look at the UpdateAttributes method I had to replace the nodes with ReplaceNodes and updated the solution by calling WithDocumentSyntaxRoot.
I would assume that either one of those two calls is missing or that nothing was changed at all - if you call workspace.TryApplyChanges(solution) you would still receive true as an Output.
Note that using multiple calls of root.ReplaceNode() instead of root.ReplaceNodes() can also result in an error since only the first update is actually used for the modified document - which might lead you to believe that nothing has changed at all, depending on the implementation.
static IEnumerable<string> GetSubdirectoriesContainingOnlyFiles(string path)
{
try
{
return from subdirectory in Directory.GetDirectories(path, "*", SearchOption.AllDirectories)
where Directory.GetDirectories(subdirectory).Length == 0
select subdirectory;
}
catch
{
}
}
In this case i'm searching in c:\
So some directories are access denied. I added try and catch but now this method dosent have a return.
And how or should i handle at all it when it's getting to the catch ?
I want in the end to get a List of all sub directories so i can get all sub directories names and the Length(the number of sub directories).
UPDATE
I tried this in the class constructor:
if (m_pars.SearchDir != null)
{
ApplyAllFiles(m_pars.SearchDir,ProcessFile);
}
m_pars.SearchDir in this contain C:\
Then in ApplyAllFiles:
static List<string> allsubdirs = new List<string>();
static void ProcessFile(string path) {/* ... */}
public static void ApplyAllFiles(string folder, Action<string> fileAction)
{
foreach (string file in Directory.GetFiles(folder))
{
fileAction(file);
}
foreach (string subDir in Directory.GetDirectories(folder))
{
try
{
ApplyAllFiles(subDir, fileAction);
allsubdirs.Add(subDir);
}
catch
{
// swallow, log, whatever
}
}
}
But the List allsubdirs is empty.
Your problem might be that you don't visit (add to the list) the current directory before recursively visiting its subdirectories. So if you get an exception there, nothing will be added to the list.
The following works for me. (I've also made it a bit more generic by using callbacks and made the exception handling stricter.)
class DirectoryHelper
{
public static void Test()
{
DirectoryHelper.EnumerateSubDirectories(#"c:\windows\system32");
}
public static List<string> EnumerateSubDirectories(string path)
{
// Depending on your use case, it might be
// unecessary to save these in memory
List<string> allSubdirs = new List<string>();
EnumerateSubDirectories(path,
filePath => Console.WriteLine("Visited file: " + filePath),
dirPath => allSubdirs.Add(dirPath),
noAccessPath => Console.WriteLine("No access: " + noAccessPath)
);
return allSubdirs;
}
private static void EnumerateSubDirectories(string root, Action<string> fileAction, Action<string> subdirAction, Action<string> noAccessAction)
{
foreach (string file in Directory.GetFiles(root))
{
fileAction(file);
}
foreach (string dir in Directory.GetDirectories(root))
{
try
{
subdirAction(dir);
EnumerateSubDirectories(dir, fileAction, subdirAction, noAccessAction);
}
catch (UnauthorizedAccessException)
{
noAccessAction(dir);
}
}
}
}
I've created a function used to populate a treeview with a targeted directory. However when I try to implement and error check to skip over folders which may have folder permission restrictions I get an error. Why do I get this error and how do i fix it?
Thank you in advance.
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Forms;
namespace directoryBrowser
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
ListDirectory(treeView1, #"C:\Windows");
}
public void ListDirectory(TreeView treeView, string path)
{
treeView.Nodes.Clear();
var rootDirectoryInfo = new DirectoryInfo(path);
treeView.Nodes.Add(CreateDirectoryNode(rootDirectoryInfo));
}
public static TreeNode CreateDirectoryNode(DirectoryInfo directoryInfo)
{
//try
//{
var directoryNode = new TreeNode(directoryInfo.Name);
foreach (var directory in directoryInfo.GetDirectories())
{
directoryNode.Nodes.Add(CreateDirectoryNode(directory));
}
foreach (var file in directoryInfo.GetFiles())
{
directoryNode.Nodes.Add(new TreeNode(file.Name));
}
return directoryNode;
//}
//catch (UnauthorizedAccessException) { }
}
}
}
You are answering your own question. You get an error because the user account which this function is running under doesn't have permission to access some of the folder.
You should then apply a try/catch statement WITHIN the for loop so if you get this exception, you function will keep on running for following folders.
public static TreeNode CreateDirectoryNode(DirectoryInfo directoryInfo)
{
var directoryNode = new TreeNode(directoryInfo.Name);
foreach (var directory in directoryInfo.GetDirectories())
{
//try
{
directoryNode.Nodes.Add(CreateDirectoryNode(directory));
} catch {
// cannot access directory
}
}
foreach (var file in directoryInfo.GetFiles())
{
directoryNode.Nodes.Add(new TreeNode(file.Name));
}
return directoryNode;
}