I'm making a program that controls a game server. One of the functions I'm making, is a live server logfile monitor.
There is a logfile (a simple textfile) that gets updated by the server as it runs.
How do I continuously check the logfile and output it's content in a RichTextBox?
I did this simple function just try and get the content of the log. It will of course just get the text row by row and output it to my textbox. Also it will lock the program for as long as the loop runs, so I know it's useless.
public void ReadLog()
{
using (StreamReader reader = new StreamReader("server.log"))
{
String line;
// Read and display lines from the file until the end of the file is reached.
while ((line = reader.ReadLine()) != null)
{
monitorTextBox.AppendText(line + "\n");
CursorDown();
}
}
}
But how would you go about solving the live monitoring as simple as possible?
*** EDIT ***
I'm using Prescots solution. great stuff.
At the moment I'm using a sstreamreader to put the text from the file to my textbox. I ran into the problem is that, whenever I tried to access any of the gui controls in my event handler the program just stopped with no error or warnings.
I found out that it has to do with threading. I solved that like this:
private void OnChanged(object source, FileSystemEventArgs e)
{
if (monitorTextField.InvokeRequired)
{
monitorTextField.Invoke((MethodInvoker)delegate { OnChanged(source, e); });
}
else
{
StreamReader reader = new StreamReader("file.txt");
monitorTextField.Text = "";
monitorTextField.Text = reader.ReadToEnd();
reader.Close();
CursorDown();
}
}
Now my only problem is that the file.txt is used by the server so I can't access it, since it's "being used by another process". I can't control that process, so maybe I'm out of luck.
But the file can be opened in notepad while the server is running, so somehow it must be possible. Perhaps I can do a temp copy of the file when it updates and read the copy. I don't know.
Check out the System.IO.FileSystemWatcher class:
public static Watch()
{
var watch = new FileSystemWatcher();
watch.Path = #"D:\tmp";
watch.Filter = "file.txt";
watch.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite; //more options
watch.Changed += new FileSystemEventHandler(OnChanged);
watch.EnableRaisingEvents = true;
}
/// Functions:
private static void OnChanged(object source, FileSystemEventArgs e)
{
if(e.FullPath == #"D:\tmp\file.txt")
{
// do stuff
}
}
Edit: if you know some details about the file, you could handle the most efficent way to get the last line. For example, maybe when you read the file, you can wipe out what you've read, so next time it's updated, you just grab whatever is there and output. Perhaps you know one line is added at a time, then your code can immediately jump to the last line of the file. Etc.
Although the FileSystemWatcher is the most simple solution I have found it to be unreliable in reality.. often a file can be updated with new contents but the FileSystemWatcher does not fire an event until seconds later and often never.
The only reliable way I have found to approach this is to check for changes to the file on a regular basis using a System.Timers.Timer object and checking the file size.
I have written a small class that demonstrates this available here:
https://gist.github.com/ant-fx/989dd86a1ace38a9ac58
Example Usage
var monitor = new LogFileMonitor("c:\temp\app.log", "\r\n");
monitor.OnLine += (s, e) =>
{
// WARNING.. this will be a different thread...
Console.WriteLine(e.Line);
};
monitor.Start();
The only real disadvantage here (apart from a slight performance delay caused by file size checking) is that because it uses a System.Timers.Timer the callback comes from a different thread.
If you are using a Windows Forms or WPF app you could easily modify the class to accept a SynchronizingObject which would ensure the event handler events are called from the same thread.
As #Prescott suggested, use a FileSystemWatcher. And make sure, you open the file with the appropriate FileShare mode (FileShare.ReadWrite seems to be appropriate), since the file might still be opened by the server. If you try to open the file exclusively while it is still used by another process, the open operation will fail.
Also in order to gain a bit of performance, you could remember the last position up to which you already have read the file and only read the new parts.
Use this answer on another post c# continuously read file.
This one is quite efficient, and it checks once per second if the file size has changed.
You can either run it on another thread (or convert to async code), but in any case you would need to marshall the text back to the main thread to append to the textbox.
Try adding a Timer and have the Timer.Tick set to an Interval of 1 second. On Timer.Tick you run the function.
private void myTimer_Tick(object sender, EventArgs e)
{
ReadLog();
}
Related
Please excuse me if this is redundant, however all the questions related to this seem to point in different directions, also I am new to multithreaded programming.
I have a FileSystemWatcher class in my code, which watches the created event. It looks like a created event of file system watcher starts it's own thread. So sometimes the calling thread continues it's work before the work initiated in called thread of FileSystemWatcher created event finishes. I don't want this. My workflow needs to be single-threaded, so what I want to achieve is wait for for created event to finish it's work before the calling thread gets an opportunity to resume.
pesudo code:
main() {
FileSystemWatcher fsw = new FileSystemWatcher()
fsw.Path = ini.location;
fsw.Created += new FileSystemEventHandler(OnFileCreation);
fsw.EnableRaisingEvents = true;
main_engine.processDataToFile();
main_engine.processCreatedFile();
}
void OnFileCreation(object sender, FileSystemEventArgs e) {
// do some file processing
// takes time based on size of file and whether file is locked or not etc.
}
void processDataToFile() {
// do some data processing on received data and output to a file.
}
void processCreatedFile() {
// do not want this method to be called, unless OnFileCreation() finish it's work.
}
The reason choose to use FileSystemWatcher was because sometimes files are directly placed for processing instead of main_engine getting the data first and it works on multiple locations, so did not want to roll out a homegrown solution when FileSystemWatcher was available.
If the event fires in the separate thread you cant make it single-threaded. because this is not your code. end of story.
however it is quite simple to wait on:
...
me.WaitOne();
main_engine.processCreatedFile();
}
...
void OnFileCreation(object sender, FileSystemEventArgs e) {
// do some file processing
// takes time based on size of file and whether file is locked or not etc.
...
me.Set();
}
ManualResetEventSlim me = new ManualResetEventSlim(false);
I creating an windows service which will from time to time check if a certain exists and if it does, then reads from it, sends data to a server and move a file to another folder. A file's size is about 1-3 Mb.
I think I will use System.Threading.Timer here to check if a file exists. What do you think of it?
And another question. If a file is being copied then my app must not read from it. It should wait until copying is done. Only after that it must read from it and does other activities.
So the questions:
1) Is that a right decision to use System.Threading.Timer?
2) How do I check a file is being copied and wait until it's done?
3) Must I use multi-threading?
I think I will use System.Threading.Timer here to check if a file exists. What do you think of it?
I think you might take a look at the FileSystemWatcher class which will notify you when the file is created and raise an event instead of you using a Timer that will continuously poll for the existence of the file.
Timer is very much costly . You can use FileSystemWatcher Which Listens to the file system change notifications and raises events when a directory, or file in a directory, changes.
// Create a new FileSystemWatcher and set its properties.
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = /*path*/
/* Watch for changes in LastAccess and LastWrite times, and
the renaming of files or directories. */
watcher.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite
| NotifyFilters.FileName | NotifyFilters.DirectoryName;
// Only watch text files.
watcher.Filter = "*.txt";
// Add event handlers.
watcher.Changed += new FileSystemEventHandler(OnChanged);
watcher.Created += new FileSystemEventHandler(OnChanged);
watcher.Deleted += new FileSystemEventHandler(OnChanged);
watcher.Renamed += new RenamedEventHandler(OnRenamed);
// Begin watching.
watcher.EnableRaisingEvents = true;
Then this would be the OnChanged method:
//This method is called when a file is created, changed, or deleted.
private static void OnChanged(object source, FileSystemEventArgs e)
{
//Show that a file has been created, changed, or deleted.
WatcherChangeTypes wct = e.ChangeType;
Console.WriteLine("File {0} {1}", e.FullPath, wct.ToString());
}
Reference: http://devproconnections.com/net-framework/how-build-folder-watcher-service-c
The usual approach I've used is to monitor the folder/s with FileSystemWatcher, (or the relevant API's if not .NET managed), and try to ensure that the only operations performed on the source/target folders are move/rename between folders on the same physical drive, and delete. If you want to add a file, open/write/flush/close it ito a temp folder on the target filesystem drive and only then move/rename it to the folder being watched. It is vital that the temp folder is on the same physical drive as the target folder so that it can be move/renamed without a data copy.
This works well on non-managed systems, not tried it on C#, but don't see any reason for to not to work OK.
Other solutions involving continual polling and/or checking file sizes are just inconvenient, inflexible, wasteful, messy and latency-ridden at best.
Multithreading - probably yes on any remote filesystem. Network file calls tend to have very long timeouts on unrechability etc. and so block the caller for what seems like forever before issuing an error/exception. If you want to get anything else done, you should probably thread off the lot unless your users can tolerate 'hourglass apps', with windows becoming unresponsive, disappearing to back, getting greyed-out and the OS offering to close them.
Oh, and another thing - best to go on a purge when starting up. Stuff can go wrong at any time, so clean any lingering rubbish from temp folders etc. when running up.
I would not use the FileSystemWatcher it's too flaky
FileSystemWatcher does not work on files created from windows service
I would use the timer set to a reasonable tick period and you should be ok.
Some sample code
Timer_TicK()
{
// remove tick event handler
// do your processing
// add back tick event handler
}
This will keep multiple tick events from happening if you have a rather large amount of processing to do. I wouldn't multi-thread until you find you need to use it due to performance issues.
In C# if you try and read a file while it's being copied by the file system you will get an exception. You will need to check for that exception or check the properties of the file (size, etc) to know when it's done.
Full timer example, where I start new threads each time the interval happens (you don't have to, but it is a good practice), which goes and checks if the file is there and reads it and deletes it if it is:
using System;
using System.ServiceProcess;
using System.Threading;
using System.Timers;
using System.IO;
namespace MyNamespace
{
public partial class Service1: ServiceBase
{
Thread syncThread = null;
System.Timers.Timer timer1;
string filePath = #"C:\myfile.txt";
int interval = 60000; // 1 min -- adjust as necessary
public Service1()
{
InitializeComponent();
}
protected override void OnStart(string[] args)
{
timer1 = new System.Timers.Timer();
timer1.Interval = interval;
timer1.Enabled = true;
timer1.Elapsed += new ElapsedEventHandler(timer1_Elapsed);
timer1.Start();
}
protected override void OnStop()
{
syncThread.Abort();
timer1.Stop();
}
protected void timer1_Elapsed(object sender, ElapsedEventArgs e)
{
syncThread = new Thread(new ThreadStart(doThread));
syncThread.Start();
}
protected void doThread()
{
// whatever you put here, it will
// run for each timer interval that elapses
// in a separate thread, and each thread will
// end when the processing in this function ends
string fileContent = String.Empty;
if (File.Exists(filePath)
{
FileStream fs = new FileStream(filePath, FileMode.Open);
StreamReader sr = new StreamReader(fs);
fileContent = sr.ReadToEnd();
sr.Close();
fs.Close();
}
if (fileContent != String.Empty)
{
// File was present... process the content...
// Then I do this...
File.Delete();
}
}
}
}
I've ran it with no problems. I prefer to start new threads with each time interval so it doesn't cause issues if the previous run hasn't finished yet. Going this route, you have that control, can decide what your interval is, and your process is not always going on - just the service. The other route, with FileSystemWatcher, it is always watching and running whenever it can and you cannot adjust your time interval, like you can with a timer, to keep down the number of events/processes going on, such as when/if the file is created, then quickly modified & saved.
Only downside I see is having to check the file attributes yourself, but those are easy enough checks to make. If you do File.Delete() after processing it like I do, you only have File.Exists(filePath) to do, in order to find out if the file has been re-created. If you have to check for modification, you merely check DateTime lastModifed = File.GetLastWriteTime(filePath) (see http://www.csharp-examples.net/file-creation-modification-time/ ) against the current time + your interval (which would be DateTime lastRun = DateTime.Now.AddMilliseconds(interval)). If lastRun > lastModified, you would have to process it.
I am currently working on Windows Service to copy data from our security cameras as it is being written to the Google Drive directory on the computer for instant upload. The files are accessible immediately after creation by the provided playback software so we would like if possible to immediately copy the data stream, that way we have some video even if the recording is interrupted (the files are 10 minute time blocks).
I currently have a service created which can watch the directory, however I am having some difficulty determining the best way to watch these files. Since they are modified continuously for 10 minutes, I will receive a large number of changed events. I was hoping there might be a way that I can capture the initial creation and start streaming the data to a second file. My concern here is that I need to ensure that I don't overrun the recording stream.
If this isn't possible or relatively simple, then I will just have to detect when the file is no longer being written to by using some logic with the last write time, but I am looking for suggestions on what the best way to do this might be. I am aware of the solutions proposed Here, but I am unsure if they apply to the situation I am dealing with. There are a large number of files within sub-directories so trying to keep track of which files I have are no longer triggering events could get very messy. Does anyone have any suggestions for how to do either of these methods?
Hmmm... You could try using a timer... This way, you can limit when it fires
private Boolean TimeToCheck=false;
public static void Run()
{ Timer timer=new Timer(2000); //2 seconds
FileSystemWatcher fileWatch=new FileSystemWatcher();
fileWatch.Path="DirToWatch";
fileWatch.Filter="fileToWatch";
fileWatch.Changed += new FileSystemEventHandler(OnChanged);
fileWatch.Created += new FileSystemEventHandler(OnChanged);
fileWatch.Deleted += new FileSystemEventHandler(OnChanged);
//If you want rename, you could use the rename event as well fileWatch.Renamed += new RenamedEventHandler(OnRenamed);
timer.Elapsed += new ElapsedEventHandler(timer_done);
watcher.EnableRaisingEvents = true;
timer.Enabled = true; // Enable it
}
private static void OnChanged(object source, FileSystemEventArgs e)
{
if(TimeToCheck)
{
TimeToCheck=false;
timer.Enabled = false; // Enable it
//move the files
timer.Enabled = true; // Enable it
}
}
private static void OnRenamed(object source, RenamedEventArgs e)
{
if(TimeToCheck)
{
TimeToCheck=false;
timer.Enabled = false; // Enable it
//move the files
timer.Enabled = true; // Enable it
}
}
private static void timer_done(object sender, ElapsedEventArgs e)
{
TimeToCheck=true;
}
You could try to do this but to be honest this seems like a hack and I'm skeptical that Windows has any supported method for doing what you're trying to do. Essentially you're trying to listen in on a write stream.
It sounds like whatever solution you're working with right now is a black box so accessing the stream directly probably isn't an option. However, there is another approach. I would look into how you can create a virtual drive with your app in windows. That way you can have the recording application writing to your virtual drive path which will allow you to handle the streams however you like. Which can include writing them to two separate locations at the same time. Both Google drive and some local storage of some kind for example.
Here's a StackOverflow question on how to create virtual drives that should get you started: C#: Create a virtual drive in Computer
Have you looked at the FileSystemWatcher object? If i'm understanding the question correctly, it may be something you may want to use.... If you were to put this security file within a certain directory, you could then use file.copy to move the updated security log into the google drive folder...
I am trying to get a notification when a file is updated on disk. I am interested in getting this notifications as soon as a flush occurs, yet, it seems that the FileSystemWatcher would only send an event when the stream opens or closes.
In the code below I am writing to a file and repeatedly flush the buffer to the disk. Yet, the FileSystemWatcher only notified me once in the beginning of the writes and once when they end.
Is there another way to get these notifications? Or should I use polling for that?
The code:
class Program
{
static void Main(string[] args)
{
FileSystemWatcher watcher = new FileSystemWatcher(Environment.CurrentDirectory, "Test.txt");
watcher.Changed += watcher_Changed;
watcher.EnableRaisingEvents = true;
using(TextWriter writer = new StreamWriter("Test.txt"))
{
WriteData(writer);
WriteData(writer);
WriteData(writer);
WriteData(writer);
}
Thread.Sleep(10000);
}
private static void WriteData(TextWriter writer)
{
writer.WriteLine("Hello!");
writer.Flush();
Console.WriteLine(DateTime.Now.ToString("T") + "] Wrote data");
Thread.Sleep(3000);
}
static void watcher_Changed(object sender, FileSystemEventArgs e)
{
Console.WriteLine(DateTime.Now.ToString("T") + "] Watcher changed!");
}
}
UPDATE 1
I've corrected the ToString function of the DateTime to show seconds. Here is the output with the code above:
11:37:47 AM] Watcher changed!
11:37:47 AM] Wrote data
11:37:50 AM] Wrote data
11:37:53 AM] Wrote data
11:37:56 AM] Wrote data
11:37:59 AM] Watcher changed!
Thanks!
It has nothing to do with the FileSystemWatcher. The watcher reacts to updates on the LastWrite attribute for the filesystem.
E.g. NTFS does not update the LastWrite on every write. The value is cached and only written when the stream is closed or at some other unspecified time. This document says
Timestamps are updated at various times and for various reasons. The only guarantee about a file timestamp is that the file time is correctly reflected when the handle that makes the change is closed. [...] The NTFS file system delays updates to the last access time for a file by up to 1 hour after the last access
I assume a similar caching applies for write
I think one of your issues here is that all your writes get executed before the event gets a slice of that cpu-time.
MSDN states
The Changed event is raised when changes are made to the size, system attributes, last write time, last access time, or security permissions of a file or directory in the directory being monitored.
I did a test and inserted Sleeps after each call to WriteData(...), what i got was
09:32] Watcher changed!
09:32] Watcher changed!
09:32] -----> Wrote data
09:32] -----> Wrote data
09:32] -----> Wrote data
09:32] -----> Wrote data
09:32] Watcher changed!
09:32] Watcher changed!
I guess this kind of proves that the even is fired right after you call Flush(), it's just a question of when the event-handler gets executed (and I assume it groups events as well).
I don't know the specific needs of your project, but I wouldn't poll. Seems like a waste since FileSystemWatcher does what you want it to do in my opinion.
Edit: Ok, I guess my brain wasn't quite ready yet for thinking when I posted this.
Your conclusion that it fires when you open and close the stream seems more logical and right.
I guess I was looking for "prove" that it fires when you call flush and therefore found it - somehow.
Update
I just had a poke at the USN-Journal and it seems that wont get you what you want either as it writes the record only when the file is closed. -> http://msdn.microsoft.com/en-us/library/aa363803(VS.85).aspx
I also found a USN-Viewer in C# and http://social.msdn.microsoft.com/Forums/en/csharpgeneral/thread/c1550294-d121-4511-ac32-31551497f64e might be interesting to read as well.
I also ran DiskMon to see if it gets the changes in realtime. It doesn't, but I don't know if that's intentional or not. However, the problem with both is that they require admin-rights to run.
So I guess you are stuck with FileSystemWatcher. (What is it you need the updates for anyway, it's not like you can read the file while it's open/locked by another program.)
ps: I just noticed you are a dev of BugAid, which I was made aware of it only recently - it looks awesome :)
I know this is an old thread but the problem still exists 10 years later. I solved this by using a timer:
private Timer _checkTimer;
private DateTime _lastSaved;
Before initializing the timer, I get the last write time of the file:
_lastSaved = File.GetLastWriteTime(_watchedFile);
_checkTimer = new Timer() { Interval = 1000, AutoReset = true };
_checkTimer.Elapsed += CheckTimer_Elapsed;
_checkTimer.Start();
Then on each elapsed event I check if the last write time is newer than the one that I knew to be the last write time:
private void CheckTimer_Elapsed(object sender, ElapsedEventArgs e)
{
DateTime lastWriteTime = File.GetLastWriteTime(_watchedFile);
if (lastWriteTime > _lastSaved)
{
// Stop the timer to avoid concurrency problems
_syncTimer.Stop();
// Do here your processing
// Now, this is the new last known write time
_lastSaved = File.GetLastWriteTime(_watchedFile);
// Start again the timer
_syncTimer.Start();
}
}
This is how I personally solved this problem that I also had with FileSystemWatcher. This method is not as expensive as reading the file content or computing the file hash every second to check if there is a difference.
I'm using FileSystemWatcher to check when a file is modified or deleted, but I'm wondering if there is any way to check when a file is read by another application.
Example:
I have the file C:\test.txt on my harddrive and am watching it using FileSystemWatcher. Another program (not under my control) goes to read that file; I would like to catch that event and, if possible, check what program is reading the file then modify the contents of the file accordingly.
It sounds like you want to write to your log file when your log file is read externally, or something to that effect. If that is the case, there is a NotifyFilters value, LastAccess. Make sure this is set as one of the flags in your FileSystemWatcher.NotifyFilter property. A change in the last access time will then fire the Changed event on FileSystemWatcher.
Currently, FileSystemWatcher does not allow you to directly differentiate between a read and a change; they both fire the Changed event based on the "change" to LastAccess. So, it would be infeasible to watch for reads to a large number of files. However, you seem to know which file you're watching, so if you had a FileInfo object for that file, and FileSystemWatcher fired its Changed event, you could get a new one and compare LastAccessTime values. If the access time changed, and LastWriteTime didn't, your file is only being read.
Now, in simplest terms, changes you make to the file while it is being read are not going to immediately show up in the other app, nor are you going to be able to "get there first", lock the file and write to it before they see it. So, you cannot use FileSystemWatcher to "intercept" a read request and show the content you want that app to see. The only way the user of another application can see what you just wrote is if the application is also watching the file and re-loads the file. That will fire another Changed event, causing an infinite loop as long as the other application continues to reload the file.
You will also get a Changed event for a read and a write. Opening a file in a text editor (virtually any will do), making some changes, then saving will fire two Changed events if you're looking for changes to Last Access Time. The first one will go off when the file is opened by the editor; at that time, you may not be able to tell that a write will happen, so if you are looking for pure read-only accesses to the file then you're SOL.
The easiest way I can think of to do this would be with a timer (System.Threading.Timer) whose callback checks and stores the last
System.IO.File.GetLastAccessTime(path)
Something like (maybe with a bit more locking...)
public class FileAccessWatcher
{
public Dictionary<string, DateTime> _trackedFiles = new Dictionary<string, DateTime>();
private Timer _timer;
public event EventHandler<EventArgs<string>> FileAccessed = delegate { };
public FileAccessWatcher()
{
_timer = new Timer(OnTimerTick, null, 500, Timeout.Infinite);
}
public void Watch(string path)
{
_trackedFiles[path] = File.GetLastAccessTime(path);
}
public void OnTimerTick(object state)
{
foreach (var pair in _trackedFiles.ToList())
{
var accessed = File.GetLastAccessTime(pair.Key);
if (pair.Value != accessed)
{
_trackedFiles[pair.Key] = accessed;
FileAccessed(this, new EventArgs<string>(pair.Key));
}
}
_timer.Change(500, Timeout.Infinite);
}
}
There is SysInternals' program FileMon... It can trace every file access in the system. If you can find its source, and understand what win32 hooks it uses, you might marshal those functions in C# and get what you want.
You could use FileInfo.LastAccessTime and FileInfo.Refresh() in a polling loop.
http://msdn.microsoft.com/en-us/library/system.io.fileinfo_members.aspx
Yes, using file system filter driver you can catch all read requests, analyze them and even substitute the data being read. Development of such driver yourself is possible, but very time-consuming and complicated. We offer a product called CallbackFilter, which includes a ready to use driver and lets you implement your filtering business logic in user-mode.
A little snippet that I found useful for detecting when another process has a lock:
static bool IsFileUsedbyAnotherProcess(string filename)
{
try
{
using(var file = File.Open(filename, FileMode.Open, FileAccess.Read, FileShare.None))
{
}
}
catch (System.IO.IOException exp)
{
return true;
}
return false;
}