We Keep Coding

System.Timer seems to stop Elapsed event after 2 days or so

I have a very simple console app with the entirety of the high level code shown below:
class Program
{
static Timer timer = new Timer(TimeSpan.FromMinutes(5).TotalMilliseconds);
static void Main(string[] args)
{
timer.Elapsed += Timer_Elapsed;
timer.Start();
Timer_Elapsed(null, null);
Console.ReadKey();
}
private static void Timer_Elapsed(object sender, ElapsedEventArgs e)
{
timer.Stop();
var syncSftp = new Tasks.WindowsServiceTasks.SFTPSync();
Common.Logging.Logger = new Common.Logging.TextFileAndConsoleLogger();
syncSftp.Execute();
timer.Start();
}
}
There seems to be a strange occurrence where after about 2 days the timer seems to stop triggering the Elapsed event.
The quick rundown on this app: It monitors sever folders and sends files out via SFTP. It also goes out and grabs files via SFTP and stores them on local folders. I have logging on every server connection, file move, copy, etc.
It seems that after 2 days the console app stops showing any logging at all. This is what leads me to think the Elapsed event stops firing. Not only that but we stop getting files from inbound SFTP and stop sending outbounds. There are no errors whatsoever. The only thing I can think of here is that somehow the console app may be trying to move or copy a file that is open somewhere else.
What would cause a Timer.Elapsed event to stop firing in simple IO processes?
I'm thinking maybe there is a catch ya here somewhere such as the program running out of memory or something; however, I do not store any incremented variables such as lists or anything of the sort.
In order to avoid polluting the post with extra code that may not be needed I have added the SFTPSync class to a paste bin here: http://pastebin.com/VfWbPpWt

how to wait for filesystemwatcher event to finish

Please excuse me if this is redundant, however all the questions related to this seem to point in different directions, also I am new to multithreaded programming.
I have a FileSystemWatcher class in my code, which watches the created event. It looks like a created event of file system watcher starts it's own thread. So sometimes the calling thread continues it's work before the work initiated in called thread of FileSystemWatcher created event finishes. I don't want this. My workflow needs to be single-threaded, so what I want to achieve is wait for for created event to finish it's work before the calling thread gets an opportunity to resume.
pesudo code:
main() {
FileSystemWatcher fsw = new FileSystemWatcher()
fsw.Path = ini.location;
fsw.Created += new FileSystemEventHandler(OnFileCreation);
fsw.EnableRaisingEvents = true;
main_engine.processDataToFile();
main_engine.processCreatedFile();
}
void OnFileCreation(object sender, FileSystemEventArgs e) {
// do some file processing
// takes time based on size of file and whether file is locked or not etc.
}
void processDataToFile() {
// do some data processing on received data and output to a file.
}
void processCreatedFile() {
// do not want this method to be called, unless OnFileCreation() finish it's work.
}
The reason choose to use FileSystemWatcher was because sometimes files are directly placed for processing instead of main_engine getting the data first and it works on multiple locations, so did not want to roll out a homegrown solution when FileSystemWatcher was available.

If the event fires in the separate thread you cant make it single-threaded. because this is not your code. end of story.
however it is quite simple to wait on:
...
me.WaitOne();
main_engine.processCreatedFile();
}
...
void OnFileCreation(object sender, FileSystemEventArgs e) {
// do some file processing
// takes time based on size of file and whether file is locked or not etc.
...
me.Set();
}
ManualResetEventSlim me = new ManualResetEventSlim(false);

C# - from time to time check if a file exists and read from it

I creating an windows service which will from time to time check if a certain exists and if it does, then reads from it, sends data to a server and move a file to another folder. A file's size is about 1-3 Mb.
I think I will use System.Threading.Timer here to check if a file exists. What do you think of it?
And another question. If a file is being copied then my app must not read from it. It should wait until copying is done. Only after that it must read from it and does other activities.
So the questions:
1) Is that a right decision to use System.Threading.Timer?
2) How do I check a file is being copied and wait until it's done?
3) Must I use multi-threading?

I think I will use System.Threading.Timer here to check if a file exists. What do you think of it?
I think you might take a look at the FileSystemWatcher class which will notify you when the file is created and raise an event instead of you using a Timer that will continuously poll for the existence of the file.

Timer is very much costly . You can use FileSystemWatcher Which Listens to the file system change notifications and raises events when a directory, or file in a directory, changes.
// Create a new FileSystemWatcher and set its properties.
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = /*path*/
/* Watch for changes in LastAccess and LastWrite times, and
the renaming of files or directories. */
watcher.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite
| NotifyFilters.FileName | NotifyFilters.DirectoryName;
// Only watch text files.
watcher.Filter = "*.txt";
// Add event handlers.
watcher.Changed += new FileSystemEventHandler(OnChanged);
watcher.Created += new FileSystemEventHandler(OnChanged);
watcher.Deleted += new FileSystemEventHandler(OnChanged);
watcher.Renamed += new RenamedEventHandler(OnRenamed);
// Begin watching.
watcher.EnableRaisingEvents = true;
Then this would be the OnChanged method:
//This method is called when a file is created, changed, or deleted.
private static void OnChanged(object source, FileSystemEventArgs e)
{
//Show that a file has been created, changed, or deleted.
WatcherChangeTypes wct = e.ChangeType;
Console.WriteLine("File {0} {1}", e.FullPath, wct.ToString());
}
Reference: http://devproconnections.com/net-framework/how-build-folder-watcher-service-c

The usual approach I've used is to monitor the folder/s with FileSystemWatcher, (or the relevant API's if not .NET managed), and try to ensure that the only operations performed on the source/target folders are move/rename between folders on the same physical drive, and delete. If you want to add a file, open/write/flush/close it ito a temp folder on the target filesystem drive and only then move/rename it to the folder being watched. It is vital that the temp folder is on the same physical drive as the target folder so that it can be move/renamed without a data copy.
This works well on non-managed systems, not tried it on C#, but don't see any reason for to not to work OK.
Other solutions involving continual polling and/or checking file sizes are just inconvenient, inflexible, wasteful, messy and latency-ridden at best.
Multithreading - probably yes on any remote filesystem. Network file calls tend to have very long timeouts on unrechability etc. and so block the caller for what seems like forever before issuing an error/exception. If you want to get anything else done, you should probably thread off the lot unless your users can tolerate 'hourglass apps', with windows becoming unresponsive, disappearing to back, getting greyed-out and the OS offering to close them.
Oh, and another thing - best to go on a purge when starting up. Stuff can go wrong at any time, so clean any lingering rubbish from temp folders etc. when running up.

I would not use the FileSystemWatcher it's too flaky
FileSystemWatcher does not work on files created from windows service
I would use the timer set to a reasonable tick period and you should be ok.
Some sample code
Timer_TicK()
{
// remove tick event handler
// do your processing
// add back tick event handler
}
This will keep multiple tick events from happening if you have a rather large amount of processing to do. I wouldn't multi-thread until you find you need to use it due to performance issues.
In C# if you try and read a file while it's being copied by the file system you will get an exception. You will need to check for that exception or check the properties of the file (size, etc) to know when it's done.

Full timer example, where I start new threads each time the interval happens (you don't have to, but it is a good practice), which goes and checks if the file is there and reads it and deletes it if it is:
using System;
using System.ServiceProcess;
using System.Threading;
using System.Timers;
using System.IO;
namespace MyNamespace
{
public partial class Service1: ServiceBase
{
Thread syncThread = null;
System.Timers.Timer timer1;
string filePath = #"C:\myfile.txt";
int interval = 60000; // 1 min -- adjust as necessary
public Service1()
{
InitializeComponent();
}
protected override void OnStart(string[] args)
{
timer1 = new System.Timers.Timer();
timer1.Interval = interval;
timer1.Enabled = true;
timer1.Elapsed += new ElapsedEventHandler(timer1_Elapsed);
timer1.Start();
}
protected override void OnStop()
{
syncThread.Abort();
timer1.Stop();
}
protected void timer1_Elapsed(object sender, ElapsedEventArgs e)
{
syncThread = new Thread(new ThreadStart(doThread));
syncThread.Start();
}
protected void doThread()
{
// whatever you put here, it will
// run for each timer interval that elapses
// in a separate thread, and each thread will
// end when the processing in this function ends
string fileContent = String.Empty;
if (File.Exists(filePath)
{
FileStream fs = new FileStream(filePath, FileMode.Open);
StreamReader sr = new StreamReader(fs);
fileContent = sr.ReadToEnd();
sr.Close();
fs.Close();
}
if (fileContent != String.Empty)
{
// File was present... process the content...
// Then I do this...
File.Delete();
}
}
}
}
I've ran it with no problems. I prefer to start new threads with each time interval so it doesn't cause issues if the previous run hasn't finished yet. Going this route, you have that control, can decide what your interval is, and your process is not always going on - just the service. The other route, with FileSystemWatcher, it is always watching and running whenever it can and you cannot adjust your time interval, like you can with a timer, to keep down the number of events/processes going on, such as when/if the file is created, then quickly modified & saved.
Only downside I see is having to check the file attributes yourself, but those are easy enough checks to make. If you do File.Delete() after processing it like I do, you only have File.Exists(filePath) to do, in order to find out if the file has been re-created. If you have to check for modification, you merely check DateTime lastModifed = File.GetLastWriteTime(filePath) (see http://www.csharp-examples.net/file-creation-modification-time/ ) against the current time + your interval (which would be DateTime lastRun = DateTime.Now.AddMilliseconds(interval)). If lastRun > lastModified, you would have to process it.

Windows Service w/ FileSystemWatcher in C#

I have to create a program that monitors changes in file size. I already made a simple windows service and filesystemwatcher so I am now familiar w/ the concept. I also made a code that checks for the filesize (made it in a form button)but haven't yet implemented in my filesystemwatcher. How do I create a windows service that has a filewatcher that monitors the file size? Do I have to put a filesystemwatcher inside the windows service and call the watcher via the OnStart method?

If you're making a Window's service, then you'll want to do it programmatically. I usually keep forms out of my services and make a separate interface for them to communicate. Now the FileSystemWatcher doesn't have an event to watch solely for size, so you'll want to make a method that ties to FileSystemWatcher.Changed to check for modifications to existing files. Declare and initialize the control in your OnStart method and tie together the events as well. Do any cleanup code in your OnStop method. It should look something like this:
protected override void OnStart(string[] args)
{
FileSystemWatcher Watcher = new FileSystemWatcher("PATH HERE");
Watcher.EnableRaisingEvents = true;
Watcher.Changed += new FileSystemEventHandler(Watcher_Changed);
}
// This event is raised when a file is changed
private void Watcher_Changed(object sender, FileSystemEventArgs e)
{
// your code here
}
Also note, the FileSystemWatcher will fire off multiple events for a single file, so when you're debugging watch for patterns to work around it.

You can simply enable your filesystemwatcher object in the OnStart method by setting
EnableRaisingEvents = true;
Then handle the event. That should do it.

you can create a delegate to handle what has changed like
myWatcher.Changed += new FileSystemHandler(FSWatcherTest_Changed);
private void FSWatcherTest_Changed(object sender,
System.IO.FileSystemEventArgs e)
{
//code here for newly changed file or directory
}
And so on
I would recommend you to read this article http://www.codeproject.com/Articles/18521/How-to-implement-a-simple-filewatcher-Windows-serv
Also have this delegate on_start in windows service

How to copy a file as it is being written in C#

I am currently working on Windows Service to copy data from our security cameras as it is being written to the Google Drive directory on the computer for instant upload. The files are accessible immediately after creation by the provided playback software so we would like if possible to immediately copy the data stream, that way we have some video even if the recording is interrupted (the files are 10 minute time blocks).
I currently have a service created which can watch the directory, however I am having some difficulty determining the best way to watch these files. Since they are modified continuously for 10 minutes, I will receive a large number of changed events. I was hoping there might be a way that I can capture the initial creation and start streaming the data to a second file. My concern here is that I need to ensure that I don't overrun the recording stream.
If this isn't possible or relatively simple, then I will just have to detect when the file is no longer being written to by using some logic with the last write time, but I am looking for suggestions on what the best way to do this might be. I am aware of the solutions proposed Here, but I am unsure if they apply to the situation I am dealing with. There are a large number of files within sub-directories so trying to keep track of which files I have are no longer triggering events could get very messy. Does anyone have any suggestions for how to do either of these methods?

Hmmm... You could try using a timer... This way, you can limit when it fires
private Boolean TimeToCheck=false;
public static void Run()
{ Timer timer=new Timer(2000); //2 seconds
FileSystemWatcher fileWatch=new FileSystemWatcher();
fileWatch.Path="DirToWatch";
fileWatch.Filter="fileToWatch";
fileWatch.Changed += new FileSystemEventHandler(OnChanged);
fileWatch.Created += new FileSystemEventHandler(OnChanged);
fileWatch.Deleted += new FileSystemEventHandler(OnChanged);
//If you want rename, you could use the rename event as well fileWatch.Renamed += new RenamedEventHandler(OnRenamed);
timer.Elapsed += new ElapsedEventHandler(timer_done);
watcher.EnableRaisingEvents = true;
timer.Enabled = true; // Enable it
}
private static void OnChanged(object source, FileSystemEventArgs e)
{
if(TimeToCheck)
{
TimeToCheck=false;
timer.Enabled = false; // Enable it
//move the files
timer.Enabled = true; // Enable it
}
}
private static void OnRenamed(object source, RenamedEventArgs e)
{
if(TimeToCheck)
{
TimeToCheck=false;
timer.Enabled = false; // Enable it
//move the files
timer.Enabled = true; // Enable it
}
}
private static void timer_done(object sender, ElapsedEventArgs e)
{
TimeToCheck=true;
}

You could try to do this but to be honest this seems like a hack and I'm skeptical that Windows has any supported method for doing what you're trying to do. Essentially you're trying to listen in on a write stream.
It sounds like whatever solution you're working with right now is a black box so accessing the stream directly probably isn't an option. However, there is another approach. I would look into how you can create a virtual drive with your app in windows. That way you can have the recording application writing to your virtual drive path which will allow you to handle the streams however you like. Which can include writing them to two separate locations at the same time. Both Google drive and some local storage of some kind for example.
Here's a StackOverflow question on how to create virtual drives that should get you started: C#: Create a virtual drive in Computer

Have you looked at the FileSystemWatcher object? If i'm understanding the question correctly, it may be something you may want to use.... If you were to put this security file within a certain directory, you could then use file.copy to move the updated security log into the google drive folder...