We Keep Coding

How can I watch for newly created files and wait for them to be unlocked?

First the watch method; I need to watch for any newly created jpg files, since I don't know yet the file names. My program creates each time a new jpg in the directory specified by a TextBox. So my first problem is how to know/get the file name when it's being created?
Second problem, how can I use all these methods, the two methods and the event changed (code below)? I have a button click event when I click it, it will create the new jpg file. Then in the button click event I want to start watching it and give a message on a label something like: "Creating file wait", then when the file is created and ready for use "File created".
private void watch()
{
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = SavePathTextBox.Text;
watcher.NotifyFilter = NotifyFilters.LastWrite;
watcher.Filter = "*.jpg";
watcher.Changed += watcher_Changed;
watcher.EnableRaisingEvents = true;
}
Then the event watcher_Changed:
void watcher_Changed(object sender, FileSystemEventArgs e)
{
}
And the method that checks if the file is locked or not
public static bool IsFileReady(String sFilename)
{
// If the file can be opened for exclusive access it means that the file
// is no longer locked by another process.
try
{
using (FileStream inputStream = File.Open(sFilename, FileMode.Open, FileAccess.Read, FileShare.None))
{
if (inputStream.Length > 0)
{
return true;
}
else
{
return false;
}
}
}
catch (Exception)
{
return false;
}
}
This is what i tried:
In the button click event:
private void TakePhotoButton_Click(object sender, EventArgs e)
{
try
{
if ((string)TvCoBox.SelectedItem == "Bulb") CameraHandler.TakePhoto((uint)BulbUpDo.Value);
else CameraHandler.TakePhoto();
watch();
}
catch (Exception ex) { ReportError(ex.Message, false); }
}
In the watch method:
private void watch()
{
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = SavePathTextBox.Text;
watcher.NotifyFilter = NotifyFilters.LastWrite;
watcher.Filter = "*.JPG";
watcher.Changed += watcher_Changed;
watcher.EnableRaisingEvents = true;
}
The event watcher_Changed
void watcher_Changed(object sender, FileSystemEventArgs e)
{
if (IsFileReady(e.FullPath) == false)
{
this.Invoke((Action)delegate { label6.Text = "Busy"; });
}
else
{
this.Invoke((Action)delegate { label6.Text = "File Ready"; });
}
}
And the method to find if file is locked or not:
public static bool IsFileReady(String sFilename)
{
// If the file can be opened for exclusive access it means that the file
// is no longer locked by another process.
try
{
using (FileStream inputStream = File.Open(sFilename, FileMode.Open, FileAccess.Read, FileShare.None))
{
if (inputStream.Length > 0)
{
return true;
}
else
{
return false;
}
}
}
catch (Exception)
{
return false;
}
}
The problem is that sometimes in most of the cases it's getting to the line inside the watcher_Changed event:
this.Invoke((Action)delegate { label6.Text = "File Ready"; });
And making this line twice or somtimes even 3 times in a row.
I can say that each click my camera take one photo and it's creating two files one for example with the name: IMG_0001.CR2 and the Jpg one: IMG_0001.JPG
But i'm not sure if that's why it's getting to the event and doing the line/s there more then once.
I also checked the file in the e.FullPath is always .jpg and never cr2.
The question is why it's getting there more then once and how can i make sure that the file is really ready ? ("File Ready")
Maybe i need somehow to track the file size from 0kb until the size not change any more and then to decide in the event that it's ready or not ?

I see some problems with the way You use watcher.
Watcher should run before You call CameraHandler.TakePhoto or there is a (small) chance You would miss it.
Either do not create new instance of watcher on every TakePhotoButton_Click or stop the old one. Otherwise You would end up with new running watcher with every click and get as many calls to watcher_Changed as many watchers You have.
I have read there is a chance watcher can be garbage collected. I am not sure if it is true, but rather be safe and save it to some local field.
What You are now waiting for is end of writing to some .jpg file. This may do what You need and in that case it is good. But if You want to wait for file create, You need different setting. This worked for me:
string watchDir = Application.StartupPath;
watcher = new FileSystemWatcher(watchDir, "*.jpg");
watcher.NotifyFilter |= NotifyFilters.Size;
watcher.Created += new FileSystemEventHandler(watcher_Created);
watcher.EnableRaisingEvents = true;
protected void watcher_Created(object sender, FileSystemEventArgs e)
{
string changeType = e.ChangeType.ToString();
if (changeType != "Created")
{
return;
}
// file is created, wait for IsFileReady or whatever You need
}

FileSystemWatcher losing files in its queue

I have written a FileSystemWatcher to call a pgm once for every file. But some of my files are lost. I tested the code with only 10-11 files. Deletion of a file is logged correctly, but not the creation. Some of the files are not logged. Is there maybe any problem in my TASK implementation?
or is there any problem with Window Service?
public static FileSystemWatcher m_Watcher;
static BlockingCollection<string> blockingCollection = new BlockingCollection<string>();
protected override void OnStart(string[] args)
{
current_directory = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
//XmlDocument xml = new XmlDocument();
try
{
strDir = ConfigurationManager.AppSettings["Directory"];
fileMask = ConfigurationManager.AppSettings["FileMask"];
strBatfile = ConfigurationManager.AppSettings["Batch"];
strlog = ConfigurationManager.AppSettings["Log"];
m_Watcher = new FileSystemWatcher();
m_Watcher.Filter = fileMask;
m_Watcher.Path = strDir + "\\";
m_Watcher.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite
| NotifyFilters.FileName | NotifyFilters.DirectoryName;
m_Watcher.Created += new FileSystemEventHandler(OnCreated);
m_Watcher.Deleted += new FileSystemEventHandler(OnDeleated);
m_Watcher.Renamed += new RenamedEventHandler(OnRenamed);
m_Watcher.EnableRaisingEvents = true;
}
catch (Exception exception)
{
CustomException.Write(CustomException.CreateExceptionString(exception.ToString()));
}
}
public static void OnDeleated(object source, FileSystemEventArgs e)
{
try
{
Log.getLogger("File deleated- Filename :" + e.Name + " at timestamp : " + DateTime.Now.ToString(), strlog);
}
catch (Exception exception)
{
CustomException.Write(CustomException.CreateExceptionString(exception, e.Name));
}
}
private static void OnCreated(object source, FileSystemEventArgs e)
{
var exceptions = new ConcurrentQueue<Exception>();
try
{
Task.Factory.StartNew(() =>
{
try
{
blockingCollection.Add(e.Name.ToString());
}
catch (Exception)
{
throw;
}
});
Task.Factory.StartNew(() =>
{
try
{
foreach (string value in blockingCollection.GetConsumingEnumerable())
{
System.Diagnostics.Process.Start(Service1.strBatfile);
Log.getLogger("File Processed after executing batch: Filename ->:" + value + " " + "Batch File Executed- > " + Service1.strBatfile + " at timestamp : " + DateTime.Now.ToString(), Service1.strlog);
}
}
catch (Exception)
{
throw;
}
});
}
catch (AggregateException ae)
{
foreach (var ex in ae.InnerExceptions)
{
CustomException.Write(CustomException.CreateExceptionString(ex, e.Name));
}
}
finally
{
m_Watcher.EnableRaisingEvents = true;
}
}

You are using way to many threads/Tasks to get a clear understanding how the code works. As you stated that you want to only process one file at a time you need just one Thread/Task that lives for the lifetime of the class (and I assume the application).
If I strip down your code to accomplish processing one file at a time whenever a file is dropped in a certain folder this could be one implementation.
Notice how I have one ConcurrentQueue and ONE method that reads that queue. I also use the method WaitForExit on the process instance to prevent running more than one process.
static ConcurrentQueue<string> filenames = new ConcurrentQueue<string>();
static void QueueHandler()
{
bool run = true;
AppDomain.CurrentDomain.DomainUnload += (s, e) =>
{
run = false;
filenames.Enqueue("stop");
};
while(run)
{
string filename;
if (filenames.TryDequeue(out filename) && run)
{
var proc = new Process();
proc.StartInfo.FileName = filename;
proc.Start();
proc.WaitForExit(); // this blocks until the process ends....
}
}
}
Now we need a single Task/Thread that will run QueueHandler and our FileSystemWatcher:
protected override void OnStart(string[] args)
{
// have our queue reader method started
Task.Factory.StartNew(QueueHandler);
var fsw = new FileSystemWatcher();
fsw.Created += (o, e) =>
{
// add a file to the queue
filenames.Enqueue(e.FullPath);
// optionally add polling for missed files
// http://stackoverflow.com/questions/239988/filesystemwatcher-vs-polling-to-watch-for-file-changes
};
fsw.Path = ConfigurationManager.AppSettings["Directory"];
fsw.NotifyFilter = NotifyFilters.FileName;
fsw.Filter = ConfigurationManager.AppSettings["FileMask"];
fsw.EnableRaisingEvents = true;
}
This implementation will use at worst three threads: one main thread, one for the Created events of the FileSystemWatcher and one for the QueueHandler instead if your example code where new Tasks were started every time a new file was created in the folder the FileSystemWatcher was watching

You are starting two tasks in your OnCreated method, where the second task seems to depend on the output from the first task. However, there is no guarantee that the first task will have finished (or even started) when the second task executes.
You could group the two operations into a single task, which would then execute sequentially, or you could await the result of the first task.
There is also a lot of information missing from your code. It clearly isn't the 'real' code because OnDeleated [sic] is misspelled and wouldn't compile. We also can't see what your external process is or how you are attempting to pass the file list to it. There could be lots of problems there. Would it be possible to post the actual code?

C# thread hangs on close file

I have a thread that calls a static method to update file properties using WindowsAPICodePack ShellPropertyWriter and BackgroundWorker. The thread calls the method below for each file in a folder of 1000+ files and hangs on the ShellPropertyWriter.close() after the 700th update or so.
Nothing to do with the file itself, tried using different files that successfully updated before.
public static bool ShellPropertyUpdate(VideoEntry mediaEntry)
{
try
{
ShellFile mediafile = ShellFile.FromFilePath(mediaEntry.FilePath);
ShellPropertyWriter pw = mediafile.Properties.GetPropertyWriter();
pw.WriteProperty(SystemProperties.System.Music.Artist, mediaEntry.Actor);
pw.WriteProperty(SystemProperties.System.Music.Genre, mediaEntry.Genre);
pw.WriteProperty(SystemProperties.System.Rating, mediaEntry.Rating);
pw.Close();
}
catch (Exception ex)
{
return false;
}
return true;
}
private void mnuWriteMetadataToFiles_Click(object sender, EventArgs ev)
{
this.WorkerThread = new BackgroundWorker();
this.WorkerThread.DoWork += new DoWorkEventHandler(WorkerThread_WriteMetadataToFiles);
this.WorkerThread.ProgressChanged += new ProgressChangedEventHandler(WorkerThread_ProgressChanged);
this.WorkerThread.RunWorkerCompleted += (s, e) => WorkerThread_Completed("Writing metadata to files", s, e);
this.WorkerThread.WorkerReportsProgress = true;
this.WorkerThread.WorkerSupportsCancellation = true;
this.WorkerThread.RunWorkerAsync(WMPlayer);
}
private void WorkerThread_WriteMetadataToFiles(object sender, DoWorkEventArgs e)
{
int counter = 0;
BackgroundWorker worker = (BackgroundWorker)sender;
MediaPlayer wmp = (MediaPlayer)e.Argument;
// ... Loop with the foreach video in the library and write it to file.
foreach (VideoEntry entry in wmp.Videos)
{
if (worker.CancellationPending)
{
e.Cancel = true;
}
else
{
worker.ReportProgress(counter, "Updating '" + entry.Filename + "'" + Environment.NewLine + "Processing file");
if (VideoToFile.ShellPropertyUpdate(entry))
{
result &= true;
}
counter++;
}
}
e.Result = result;
}

Never heard of this assembly before, but it smells like handle exhaustion to me. Try this instead:
using (ShellFile mediafile = ShellFile.FromFilePath(mediaEntry.FilePath))
{
ShellPropertyWriter pw = mediafile.Properties.GetPropertyWriter();
pw.WriteProperty(SystemProperties.System.Music.Artist, mediaEntry.Actor);
pw.WriteProperty(SystemProperties.System.Music.Genre, mediaEntry.Genre);
pw.WriteProperty(SystemProperties.System.Rating, mediaEntry.Rating);
pw.Close();
}
Here every file handle is closed immediately, instead of at garbage collector's discretion. ShellFile must implement IDisposable for this to work, otherwise this code will not compile. I'm fairly certain that ShellFile implements it.

Apparently it does have something to do with the files themselves. I took out few problem files and the Thread continued processing until the next problem file. I have no clue what's wrong with the file, however I'm willing to pass on updating problem files. Is there a way to stop/kill the thread? I can't use DoWorkEventArgs.cancel() since the thread is hanging and not coming back.

How do I write a filesystem watcher in C#?

I have written a small application using C# which involves Filesystemwather to watch a particular folder. As soon as a file is updated it open up a serial port and writes the files contents to the serial port. But at times the file isn't updated for more this 4-5hours. And it seems like filesystemwatcher goes to sleep and doesn't respond after the file gets updated.
Here is my code:
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = #"Z:\";
watcher.EnableRaisingEvents = true;
watcher.NotifyFilter = NotifyFilters.LastAccess;
watcher.Filter = "*.wsx";
watcher.Changed += new FileSystemEventHandler(OnChanged);
Console.WriteLine("Press \'q\' to quit the sample.");
while (Console.Read() != 'q') ;
public static string CrL = "\r\n";
private static void OnChanged(object source, FileSystemEventArgs e)
{
string FileName;
FileName = e.FullPath;
Console.WriteLine("File: " + e.FullPath + " " + e.ChangeType);
Console.WriteLine("FILE is changed 1");
SerialPort port = new SerialPort("COM1");
port.Encoding = Encoding.ASCII;
port.Open();
using (System.IO.TextReader reader = System.IO.File.OpenText(e.FullPath))
{
string line;
while ((line = reader.ReadLine()) != null)
{
port.Write(line);
port.Write(CrL);
}
}
port.Close();
Console.WriteLine("FILE is sent to com port");
}
Any pointers on this one.

If there is an issue in dotnet that causes the FileSystemWatcher to "timeout", per say: a quick work-around would be to use a timer control and re-initialize the FileSystemWatcher every so often so it doesn't "timeout". You have access to the dotnet debug symbol servers so you could debug the FileSystemWatcher yourself to see what it does as well.

Maybe the FSW object is being garbage-collected because it's going out of scope.
In your code above you haven't shown where you're defining the "watcher" object.
If it's defined in the same method where you use it, maybe that's the problem. Try defining it at the top of your class (outside of any methods).

c# Unable to open file for reading

I'm writing a program that uses FileSystemWatcher to monitor changes to a given directory, and when it receives OnCreated or OnChanged event, it copies those created/changed files to a specified directories. At first I had problems with the fact that OnChanged/OnCreated events can be sent twice (not acceptable in case it needed to process 500MB file) but I made a way around this and with what I'm REALLY BLOCKED with is getting the following IOException:
The process cannot access the file 'C:\Where are Photos\bookmarks (11).html' because it is being used by another process.
Thus, preventing the program from copying all the files it should.
So as I mentioned, when user uses this program he/she specifes monitored directory, when user copies/creates/changes file in that directory, program should get OnCreated/OnChanged event and then copy that file to few other directories.
Above error happens in all cases, if user copies few files that needs to overwrite other ones in folder being monitored or when copying bulk of several files or even sometimes when copying one file in a monitored directory.
Whole program is quite big so I'm sending the most important parts.
OnCreated:
private void OnCreated(object source, FileSystemEventArgs e) {
AddLogEntry(e.FullPath, "created", "");
// Update last access data if it's file so the same file doesn't
// get processed twice because of sending another event.
if (fileType(e.FullPath) == 2) {
lastPath = e.FullPath;
lastTime = DateTime.Now;
}
// serves no purpose now, it will be remove soon
string fileName = GetFileName(e.FullPath);
// copies file from source to few other directories
Copy(e.FullPath, fileName);
Console.WriteLine("OnCreated: " + e.FullPath);
}
OnChanged:
private void OnChanged(object source, FileSystemEventArgs e) {
// is it directory
if (fileType(e.FullPath) == 1)
return; // don't mind directory changes itself
// Only if enough time has passed or if it's some other file
// because two events can be generated
int timeDiff = ((TimeSpan)(DateTime.Now - lastTime)).Seconds;
if ((timeDiff < minSecsDiff) && (e.FullPath.Equals(lastPath))) {
Console.WriteLine("-- skipped -- {0}, timediff: {1}", e.FullPath, timeDiff);
return;
}
// Update last access data for above to work
lastPath = e.FullPath;
lastTime = DateTime.Now;
// Only if size is changed, the rest will handle other handlers
if (e.ChangeType == WatcherChangeTypes.Changed) {
AddLogEntry(e.FullPath, "changed", "");
string fileName = GetFileName(e.FullPath);
Copy(e.FullPath, fileName);
Console.WriteLine("OnChanged: " + e.FullPath);
}
}
fileType:
private int fileType(string path) {
if (Directory.Exists(path))
return 1; // directory
else if (File.Exists(path))
return 2; // file
else
return 0;
}
Copy:
private void Copy(string srcPath, string fileName) {
foreach (string dstDirectoy in paths) {
string eventType = "copied";
string error = "noerror";
string path = "";
string dirPortion = "";
// in case directory needs to be made
if (srcPath.Length > fsw.Path.Length) {
path = srcPath.Substring(fsw.Path.Length,
srcPath.Length - fsw.Path.Length);
int pos = path.LastIndexOf('\\');
if (pos != -1)
dirPortion = path.Substring(0, pos);
}
if (fileType(srcPath) == 1) {
try {
Directory.CreateDirectory(dstDirectoy + path);
//Directory.CreateDirectory(dstDirectoy + fileName);
eventType = "created";
} catch (IOException e) {
eventType = "error";
error = e.Message;
}
} else {
try {
if (!overwriteFile && File.Exists(dstDirectoy + path))
continue;
// create new dir anyway even if it exists just to be sure
Directory.CreateDirectory(dstDirectoy + dirPortion);
// copy file from where event occured to all specified directories
using (FileStream fsin = new FileStream(srcPath, FileMode.Open, FileAccess.Read, FileShare.Read)) {
using (FileStream fsout = new FileStream(dstDirectoy + path, FileMode.Create, FileAccess.Write)) {
byte[] buffer = new byte[32768];
int bytesRead = -1;
while ((bytesRead = fsin.Read(buffer, 0, buffer.Length)) > 0)
fsout.Write(buffer, 0, bytesRead);
}
}
} catch (Exception e) {
if ((e is IOException) && (overwriteFile == false)) {
eventType = "skipped";
} else {
eventType = "error";
error = e.Message;
// attempt to find and kill the process locking the file.
// failed, miserably
System.Diagnostics.Process tool = new System.Diagnostics.Process();
tool.StartInfo.FileName = "handle.exe";
tool.StartInfo.Arguments = "\"" + srcPath + "\"";
tool.StartInfo.UseShellExecute = false;
tool.StartInfo.RedirectStandardOutput = true;
tool.Start();
tool.WaitForExit();
string outputTool = tool.StandardOutput.ReadToEnd();
string matchPattern = #"(?<=\s+pid:\s+)\b(\d+)\b(?=\s+)";
foreach (Match match in Regex.Matches(outputTool, matchPattern)) {
System.Diagnostics.Process.GetProcessById(int.Parse(match.Value)).Kill();
}
Console.WriteLine("ERROR: {0}: [ {1} ]", e.Message, srcPath);
}
}
}
AddLogEntry(dstDirectoy + path, eventType, error);
}
}
I checked everywhere in my program and whenever I use some file I use it in using block so even writing event to log (class for what I omitted since there is probably too much code already in post) wont lock the file, that is it shouldn't since all operations are using using statement block.
I simply have no clue who's locking the file if not my program "copy" process from user through Windows or something else.
Right now I have two possible "solutions" (I can't say they are clean solutions since they are hacks and as such not desirable). Since probably the problem is with fileType method (what else could lock the file?) I tried changing it to this, to simulate "blocking-until-ready-to-open" operation:
fileType:
private int fileType(string path) {
FileStream fs = null;
int ret = 0;
bool run = true;
if (Directory.Exists(path))
ret = 1;
else {
while (run) {
try {
fs = new FileStream(path, FileMode.Open);
ret = 2;
run = false;
} catch (IOException) {
} finally {
if (fs != null) {
fs.Close();
fs.Dispose();
}
}
}
}
return ret;
}
This is working as much as I could tell (test), but... it's hack, not to mention other deficients.
The other "solution" I could try (I didn't test it yet) is using GC.Collect() somewhere at the end of fileType() method. Maybe even worse "solution" than previous one.
Can someone pleas tell me, what on earth is locking the file, preventing it from opening and how can I fix that? What am I missing to see?
Thanks in advance.

The problem is most likely that the file is still being copied while you already try to access it. This can happen especially on large files.
You can try to check whether the file can be opened with write permissions before you actually start your processing. For details how to do that check here.
If you can influence the process creating the file there might be a better solution. First copy the file with a temporary extension, and then, after the copying is completed, rename it so that the FileSystemWatcher event will be triggered.

You can try with Volume Shadow Copies.
See www.codeproject.com/KB/dotnet/makeshadowcopy.aspx for more details.

FileSystemWatcher events trigger when the file begins the copy, not at the end, so it's common to run into this kind of errors.
Your first approach will work, however, I would recommend spinning the I/O intensive code on another thread, and using an incremental Sleep() instead of the busy waiting you do.
However, if you have access to the software that actually creates the files, the extension changing is a slightly less complicated solution. Just beware, that a xls filter on the FileSystemwatcher will match a file called myfile1.xls.temp, as I found that out the hard way :)