We Keep Coding

Files disappear after they fail to be moved

We have a process where people scan documents with photocopiers and drop them in a certain directory on our file server. We then have a hourly service within an .NET Core app, that scans the directory, grabs the file and moves them according to their file name to a certain directory. Here comes the problems.
The code looks something like that:
private string MoveFile(string file, string commNumber)
{
var fileName = Path.GetFileName(file);
var baseFileName = Path.GetFileNameWithoutExtension(fileName).Split("-v")[0];
// 1. Check if the file already exists at destination
var existingFileList = luxWebSamContext.Documents.Where(x => EF.Functions.Like(x.DocumentName, "%" + Path.GetFileNameWithoutExtension(baseFileName) + "%")).ToList();
// If the file exists, check for the current version of file
if (existingFileList.Count > 0)
{
var nextVersion = existingFileList.Max(x => x.UploadVersion) + 1;
var extension = Path.GetExtension(fileName);
fileName = baseFileName + "-v" + nextVersion.ToString() + extension;
}
var from = #file;
var to = Path.Combine(#destinationPath, commNumber,fileName);
try
{
log.Info($"------ Moving File! ------ {fileName}");
Directory.CreateDirectory(Path.Combine(#destinationPath, commNumber));
File.Move(from, to, true);
return to;
}
catch (Exception ex)
{
log.Error($"----- Couldn't MOVE FILE: {file} ----- commission number: {commNumber}", ex);
The interesting part is in the try-block, where the file move takes place. Sometmes we have the problem that the program throws the following exception
2021-11-23 17:15:37,960 [60] ERROR App ----- Couldn't MOVE FILE:
\PATH\PATH\PATH\Filename_423489120.pdf ----- commission number:
05847894
System.IO.IOException: The process cannot access the file because it is being used by another process.
at System.IO.FileSystem.MoveFile(String sourceFullPath, String destFullPath, Boolean overwrite)
at System.IO.File.Move(String sourceFileName, String destFileName, Boolean overwrite)
So far so good. I would expect that after the file cannot be moved, it remains in the directory from it was supposed to be moved. But that's not the case. We had this issue yesterday afternoon and after I looked for the file, it was gone from the directory.
Is this the normal behaviour of the File.Move() method?

First to your question:
Is this the normal behaviour of the File.Move() method?
No, thats not the expected behaviour. The documentation says:
Moving the file across disk volumes is equivalent to copying the file
and deleting it from the source if the copying was successful.
If you try to move a file across disk volumes and that file is in use,
the file is copied to the destination, but it is not deleted from the
source.
Your Exception says, that another process is using the file in the same moment. So you should check, whether other parts of your application may performs a Delete, or someone (if this scenario is valid) is deleting files manually from the file system.

Typically, File.Move() only removes the source file, once the destination file is successfully transferred in place. So the answer to your question is no, it cannot be purely the File.Move(). The interesting part is, why is this file locked? Probaby because some file stream is still open and blocking access to the file. Also, do you have multiple instances of the copy process services running? This may cause several services trying to access the file simultaneously, causing the exception you posted.
There must be a different cause making the files disappear because the File.Move() will certainly not remove the file when the copy process did not succeed.
For debugging purposes, you may try and open the file with a lock on it. This will fail when a different process locks the file providing you a little bit more information.

File is being used by another process in c#

I am trying to delete a file in C#, however I am receiving a message that the file is used from another process. What I want to do is to check if the files exists and close it. I am using the following function in order to check if the file is open:
public static bool IsFileInUse(string path)
{
if (string.IsNullOrEmpty(path))
throw new ArgumentException("'path' cannot be null or empty.", "path");
try
{
using (var stream = new FileStream(path, FileMode.Open, FileAccess.Read)) { }
}
catch (IOException)
{
return true;
}
return false;
}
and I am trying when the file is in use to close it:
bool checking = IsFileInUse(file );
File.Create(file ).Close();
if (File.Exists(file))
{
File.Delete(file );
}
I got issues in File.Create line, I am receiving the message:
File is being used by another process.
EDIT: I am trying to use lock approach in order to delete the file. Am I suppose to delete the file inside a lock statement? How Can I use properly the lock statement?

Why do you suppose that a reading operation will fail if file is in use while a writing operation will not? File.Create() will fail exactly as new FileStream() failed before...
See also IOException: The process cannot access the file 'file path' because it is being used by another process.
Note that your check will fail if the other process didn't open that file exclusively (check FileShare enumeration): file may be open for shared reading, writing and sometimes even for deleting (for example you may be able to read concurrently but not writing however the other process may let you delete that file...).
To close an open file can be really disruptive for the other process, it may crash, nicely handle the problem or...anything else (silently ignore that error and produce random output, open file again and so on...) Is it possible to do it in C#? Yes with some P/Invoke...
1) Let's find the handle for the file you want to unlock. Use NtQuerySystemInformation() and enumerate all handles until you find the one that refers to that file.
2) Duplicate that handle to be valid in your own process using DuplicateHandle().
3) Close just create handle specifying DUPLICATE_CLOSE_SOURCE, it will close both your handle and the original one (of course if your process has enough permissions).
4) Check if file is really closed calling NtQuerySystemInformation() again, if not then you may need to directly close its parent process.

In your code, you don't do anything with the IsFileInUse result.
This File.Create(file ).Close(); will also not close a file that is opened by another process. You need to close the process that has the file open, and if it is your own app, close the file handle before trying to delete the file.
bool checking = IsFileInUse(file );
File.Create(file ).Close();
if (!checking)
{
if (File.Exists(file))
{
File.Delete(file );
}
}

You have no need to check if the file exists, just try do delete it:
https://msdn.microsoft.com/en-us/library/system.io.file.delete(v=vs.110).aspx
If the file to be deleted does not exist, no exception is thrown.
Try and check the exception
try {
File.Delete(file);
}
catch (IOException) {
// File in use and can't be deleted; no permission etc.
}

Concurrent File.Move of the same file

It was clearly stated that File.Move is atomic operation here: Atomicity of File.Move.
But the following code snippet results in visibility of moving the same file multiple times.
Does anyone know what is wrong with this code?
using System;
using System.Collections.Generic;
using System.IO;
using System.Threading.Tasks;
namespace FileMoveTest
{
class Program
{
static void Main(string[] args)
{
string path = "test/" + Guid.NewGuid().ToString();
CreateFile(path, new string('a', 10 * 1024 * 1024));
var tasks = new List<Task>();
for (int i = 0; i < 10; i++)
{
var task = Task.Factory.StartNew(() =>
{
try
{
string newPath = path + "." + Guid.NewGuid();
File.Move(path, newPath);
// this line does NOT solve the issue
if (File.Exists(newPath))
Console.WriteLine(string.Format("Moved {0} -> {1}", path, newPath));
}
catch (Exception e)
{
Console.WriteLine(string.Format(" {0}: {1}", e.GetType(), e.Message));
}
});
tasks.Add(task);
}
Task.WaitAll(tasks.ToArray());
}
static void CreateFile(string path, string content)
{
string dir = Path.GetDirectoryName(path);
if (!Directory.Exists(dir))
{
Directory.CreateDirectory(dir);
}
using (FileStream f = new FileStream(path, FileMode.OpenOrCreate))
{
using (StreamWriter w = new StreamWriter(f))
{
w.Write(content);
}
}
}
}
}
The paradoxical output is below. Seems that file was moved multiple times onto different locations. On the disk only one of them is present. Any thoughts?
Moved test/eb85560d-8c13-41c1-926a-6871be030742 -> test/eb85560d-8c13-41c1-926a-6871be030742.0018d317-ed7c-4732-92ac-3bb974d29017
Moved test/eb85560d-8c13-41c1-926a-6871be030742 -> test/eb85560d-8c13-41c1-926a-6871be030742.3965dc15-7ef9-4f36-bdb7-94a5939b17db
Moved test/eb85560d-8c13-41c1-926a-6871be030742 -> test/eb85560d-8c13-41c1-926a-6871be030742.fb66306a-5a13-4f26-ade2-acff3fb896be
Moved test/eb85560d-8c13-41c1-926a-6871be030742 -> test/eb85560d-8c13-41c1-926a-6871be030742.c6de8827-aa46-48c1-b036-ad4bf79eb8a9
System.IO.FileNotFoundException: Could not find file 'C:\file-move-test\test\eb85560d-8c13-41c1-926a-6871be030742'.
System.IO.FileNotFoundException: Could not find file 'C:\file-move-test\test\eb85560d-8c13-41c1-926a-6871be030742'.
System.IO.FileNotFoundException: Could not find file 'C:\file-move-test\test\eb85560d-8c13-41c1-926a-6871be030742'.
System.IO.FileNotFoundException: Could not find file 'C:\file-move-test\test\eb85560d-8c13-41c1-926a-6871be030742'.
System.IO.FileNotFoundException: Could not find file 'C:\file-move-test\test\eb85560d-8c13-41c1-926a-6871be030742'.
System.IO.FileNotFoundException: Could not find file 'C:\file-move-test\test\eb85560d-8c13-41c1-926a-6871be030742'.
The resulting file is here: eb85560d-8c13-41c1-926a-6871be030742.fb66306a-5a13-4f26-ade2-acff3fb896be
UPDATE. I can confirm that checking File.Exists also does NOT solve the issue - it can report that single file was really moved into several different locations.
SOLUTION. The solution I end up with is following: Prior to operations with source file create special "lock" file, if it succeeded then we can be sure that only this thread got exclusive access to the file and we are safe to do anything we want. The below is right set of parameters to create suck "lock" file.
File.Open(lockPath, FileMode.CreateNew, FileAccess.Write);

Does anyone know what is wrong with this code?
I guess that depends on what you mean by "wrong".
The behavior you're seeing is not IMHO unexpected, at least if you're using NTFS (other file systems may or may not behave similarly).
The documentation for the underlying OS API (MoveFile() and MoveFileEx() functions) is not specific, but in general the APIs are thread-safe, in that they guarantee the file system will not be corrupted by concurrent operations (of course, your own data could be corrupted, but it will be done in a file-system-coherent way).
Most likely what is occurring is that as the move-file operation proceeds, it does so by first getting the actual file handle from the given directory link to it (in NTFS, all "file names" that you see are actually hard links to an underlying file object). Having obtained that file handle, the API then creates a new file name for the underlying file object (i.e. as a hard link), and then deletes the previous hard link.
Of course, as this progresses, there is a window during the time between a thread having obtained the underlying file handle but before the original hard link has been deleted. This allows some but not all of the other concurrent move operations to appear to succeed. I.e. eventually the original hard link doesn't exist and further attempts to move it won't succeed.
No doubt the above is an oversimplification. File system behaviors can be complex. In particular, your stated observation is that you only wind up with a single instance of the file when all is said and done. This suggests that the API does also somehow coordinate the various operations, such that only one of the newly-created hard links survives, probably by virtue of the API actually just renaming the associated hard link after retrieving the file object handle, as opposed to creating a new one and deleting the old one (implementation detail).
At the end of the day, what's "wrong" with the code is that it is intentionally attempting to perform concurrent operations on a single file. While the file system itself will ensure that it remains coherent, it's up to your own code to ensure that such operations are coordinated so that the results are predictable and reliable.

How to copy files from one disk to another location with the same folder structure?

I want to make an exact copy of some files, directories and subdirectories that are on my USB drive I:/ and want them to be in C:/backup (for example)
My USB drive has the following structure:
(just to know, this is an example, my drive has more files, directories and subdirectories)
courses/data_structures/db.sql
games/pc/pc-game.exe
exams/exam01.doc
Well, I am not sure how to start with this but my first idea is to get all the files doing this:
string[] files = Directory.GetFiles("I:");
The next step could be to make a loop and use File.Copy specifying the destination path:
string destinationPath = #"C:/backup";
foreach (string file in files)
{
File.Copy(file, destinationPath + "\\" + Path.GetFileName(file), true);
}
At this point everything works good but not as I wanted cause this doesn't replicate the folder structure. Also some errors happen like the following...
The first one happens because my PC configuration shows hidden files for every folder and my USB has an AUTORUN.INF hidden file that is not hidden anymore and the loop tries to copy it and in the process generates this exception:
Access to the path 'AUTORUN.INF' is denied.
The second one happens when some paths are too long and this generates the following exception:
The specified path, file name, or both are too long. The fully
qualified file name must be less than 260 characters, and the
directory name must be less than 248 characters.
So, I am not sure how to achieve this and validate each posible case of error. I would like to know if there is another way to do this and how (maybe some library) or something more simple like an implemented method with the following structure:
File.CopyDrive(driveLetter, destinationFolder)
(VB.NET answers will be accepted too).
Thanks in advance.

public static void Copy(string src, string dest)
{
// copy all files
foreach (string file in Directory.GetFiles(src))
{
try
{
File.Copy(file, Path.Combine(dest, Path.GetFileName(file)));
}
catch (PathTooLongException)
{
}
// catch any other exception that you want.
// List of possible exceptions here: http://msdn.microsoft.com/en-us/library/c6cfw35a.aspx
}
// go recursive on directories
foreach (string dir in Directory.GetDirectories(src))
{
// First create directory...
// Instead of new DirectoryInfo(dir).Name, you can use any other way to get the dir name,
// but not Path.GetDirectoryName, since it returns full dir name.
string destSubDir = Path.Combine(dest, new DirectoryInfo(dir).Name);
Directory.CreateDirectory(destSubDir);
// and then go recursive
Copy(dir, destSubDir);
}
}
And then you can call it:
Copy(#"I:\", #"C:\Backup");
Didn't have time to test it, but i hope you get the idea...
edit: in the code above, there are no checks like Directory.Exists and such, you might add those if the directory structure of some kind exists at destination path. And if you're trying to create some kind of simple sync app, then it gets a bit harder, as you need to delete or take other action on files/folders that don't exist anymore.

This generally starts with a recursive descent parser. Here is a good example: http://msdn.microsoft.com/en-us/library/bb762914.aspx

You might want to look into the overloaded CopyDirectory Class
CopyDirectory(String, String, UIOption, UICancelOption)
It will recurse through all of the subdirectories.
If you want a standalone application, I have written an application that copies from one selected directory to another, overwriting newer files and adding subdirectories as needed.
Just email me.

querying a directory for files that are complete (not copy-in-progress)

I tried to use FileInfo.CreationTime, but it doesn't represent the copy finish time.
I am trying to get a list of files in a directory. The problem is that the call also returns files which are not yet finished copying.
If I try to use the file, it returns an error stating that the file is in use.
How can you query for files that are fully copied?
As below code. Directory.GetFiles() returns which are not yet finished copying.
my test file size is over 200Mb.
if(String.IsNullOrEmpty(strDirectoryPath)){
txtResultPrint.AppendText("ERROR : Wrong Directory Name! ");
}else{
string[] newFiles = Directory.GetFiles(strDirectoryPath,"*.epk");
_epkList.PushNewFileList(newFiles);
if(_epkList.IsNewFileAdded()){
foreach (var fileName in _epkList.GetNewlyAddedFile()){
txtResultPrint.AppendText(DateTime.Now.Hour + ":" + DateTime.Now.Minute + ":" + DateTime.Now.Second + " => ");
txtResultPrint.AppendText(fileName + Environment.NewLine);
this.Visible = true;
notifyIconMain.Visible = true;
}
}else{
}
}

If performance and best-practices aren't huge concerns then you could simply wrap the failing file operation in an inner-scoped try/catch.
using System.IO;
string[] files = Directory.GetFiles("pathToFiles");
foreach (string file in files) {
FileStream fs = null;
try {
//try to open file for exclusive access
fs = new FileStream(
file,
FileMode.Open,
FileAccess.Read, //we might not have Read/Write privileges
FileShare.None //request exclusive (non-shared) access
);
}
catch (IOException ioe) {
//File is in use by another process, or doesn't exist
}
finally {
if (fs != null)
fs.Close();
}
}
This isn't really the best design advice as you shouldn't be relying on exception handling for this sort of thing, but if you're in a pinch and it's not code for a client or for your boss then this should work alright until a better solution is suggested or found.

Do you have the ability to change thy copying itself?
If yes (and if you can guarantee that your program will always execute on NTFS on Windows Vista or newer), you can use Transactional NTFS to wrap the copy in a single transaction. File(s) being copied will only become visible to the rest of the world after you commit the transaction, so you'll never even see the partially copied files.
Unfortunately Transactional NTFS is not accessible directly from .NET Framework - you'll need to P/Invoke into Win32 APi functions such as: CreateTransaction, CommitTransaction, RollbackTransaction, CopyFileTransacted (and other *Transacted functions).