So, I'd like to write a logger in c# for an app I'm working on. However, since I love efficiency, I don't want to be opening and closing a log file over and over again during execution.
I think I'd like to write all events to RAM and then write to the log file once when the app exits. Would this be a good practice? If so, how should I implement it?
If this is not a good practice, what would be?
(And I'm not using Windows' event log at this time.)
However, since I love efficiency, I don't want to be opening and closing a log file over and over again during execution
No. It's since you love premature optimizations.
I think I'd like to write all events to RAM and then write to the log file once when the app exits. Would this be a good practice? If so, how should I implement it?
If you love efficiency, why do you want to waste a lot of memory for log entries?
If this is not a good practice, what would be?
It is if you want to lose all logs when your application crashes (since it cannot write the log to disk then). Why did you create the log in the first place?
You'll have to think of some issues you might encounter:
System being shut down while your application runs -> no log files
An application crash might not invoke your write method
If the log grows large (how long does your application run?), you might get memory problems
If the log grows large, and there is not enough space on the drive, not a single log line will be written
You could simply keep the file open while your application runs (with at least FileShare.Read so you can monitor it), or consider writing batches of log lines, invoking the write method after a group of methods, or even using a timer.
Well if your app crashes, you lose all your logs. Better if you flush the logs to disk at an appropriate moment.. Lazy write:
Queue off the log entries to a seperate logger thread, (ie. store them in some class and queue the class instance to a producer-consumer queue). In the logger thread, wait on the input queue with a timeout. If a log entry comes in, store it in a local cache queue.
If (timeout fires) or (some high water mark of logs stored is reached) then write all cached log entries to the file and flush file buffers.
Rgds,
Martin
Related
Is there any way I can use Microsoft.Diagnostic.Tracing.EventSource package or any other .NET built in types to implement a back up logic for logging?
I have used an EventSession that directs all logs to an .ETL file but there are two problems with this:
The most important one is that, from what I understand, the logs are actually being written to the file when the processing stops. But what happens if the machine shuts down or the processor process gets killed? From my tests the logs are lost.
The second problem is less important and it's just that it would be more convenient for my solution to be able to read from the file as logs are coming in.
Basically I would like to have a file buffer for my logs and I was wondering if there's some built in implementation especially for this.
Yes, EventSource's WriteEvent methods are not blocking, so they do
not guarantee that logs actually have been saved.
And yes, you cannot
read ETL files while they are in use.
ETW is not the option for you in this case, you could use common TraceSource (with two listeners, one is configured to do auto-flush and another one is ETW adaptor, which you could listen to through a real-time ETW session) or log4net (you could choose appender that works for your case).
I've used FileSystemWatcher in the past. However, I am hoping someone can explain how it actually is working behind the scenes.
I plan to utilize it in an application I am making and it would monitor about 5 drives and maybe 300,000 files.
Does the FileSystemWatcher actually do "Checking" on the drive - as in, will it be causing wear/tear on the drive? Also does it impact hard drive ability to "sleep"
This is where I do not understand how it works - if it is like scanning the drives on a timer etc... or if its waiting for some type of notification from the OS before it does anything.
I just do not want to implement something that is going to cause extra reads on a drive and keep the drive from sleeping.
Nothing like that. The file system driver simply monitors the normal file operations requested by other programs that run on the machine against the filters you've selected. If there's a match then it adds an entry to an internal buffer that records the operation and the filename. Which completes the driver request and gets an event to run in your program. You'll get the details of the operation passed to you from that buffer.
So nothing actually happens the operations themselves, there is no extra disk activity at all. It is all just software that runs. The overhead is minimal, nothing slows down noticeably.
The short answer is no. The FileSystemWatcher calls the ReadDirectoryChangesW API passing it an asynchronous flag. Basically, Windows will store data in an allocated buffer when changes to a directory occur. This function returns the data in that buffer and the FileSystemWatcher converts it into nice notifications for you.
I have a text file and multiple threads/processes will write to it (it's a log file).
The file gets corrupted sometimes because of concurrent writings.
I want to use a file writing mode from all of threads which is sequential at file-system level itself.
I know it's possible to use locks (mutex for multiple processes) and synchronize writing to this file but I prefer to open the file in the correct mode and leave the task to System.IO.
Is it possible ? what's the best practice for this scenario ?
Your best bet is just to use locks/mutexex. It's a simple approach, it works and you can easily understand it and reason about it.
When it comes to synchronization it often pays to start with the simplest solution that could work and only try to refine if you hit problems.
To my knowledge, Windows doesn't have what you're looking for. There is no file handle object that does automatic synchronization by blocking all other users while one is writing to the file.
If your logging involves the three steps, open file, write, close file, then you can have your threads try to open the file in exclusive mode (FileShare.None), catch the exception if unable to open, and then try again until success. I've found that tedious at best.
In my programs that log from multiple threads, I created a TextWriter descendant that is essentially a queue. Threads call the Write or WriteLine methods on that object, which formats the output and places it into a queue (using a BlockingCollection). A separate logging thread services that queue--pulling things from it and writing them to the log file. This has a few benefits:
Threads don't have to wait on each other in order to log
Only one thread is writing to the file
It's trivial to rotate logs (i.e. start a new log file every hour, etc.)
There's zero chance of an error because I forgot to do the locking on some thread
Doing this across processes would be a lot more difficult. I've never even considered trying to share a log file across processes. Were I to need that, I would create a separate application (a logging service). That application would do the actual writes, with the other applications passing the strings to be written. Again, that ensures that I can't screw things up, and my code remains simple (i.e. no explicit locking code in the clients).
you might be able to use File.Open() with a FileShare value set to None, and make each thread wait if it can't get access to the file.
I am setting out to create an app that will watch a directory for any files created. pretty straightforward time to use a filesystemwatcher. My question relates to how to utilize it. Is it common practice to use a windows service to ensure the application is always running?
i have been trying to get away from building windows services if i don't have to, but i really don't see an alternative to doing it that way in this instance. normally, i would convert my service to a console app and schedule it using windows scheduler, but that doesn't really apply in this situation.
can anyone recommend a better way of implementing the filesystemwatcher than a windows service?
thanks for any thoughts.
EDIT
in response to the comments below, more specifically, i just have to watch a directory on a server, and when a new file is created, i have to move a copy of that file into a different directory on the same server, perhaps renaming it in the process.
The frequency and amount of files will be quite small. perhaps 5-10 at most in a day.
I'm not sure yet how the file watcher works, but this is what I'm thinking: The file system fires events; I mean like NTFS must be doing that. Your file watcher hooks into those events. The file watcher probably suspends the thread it's running in until an event occurs and the event somehow wakes up the thread. A suspended thread uses pretty much very few cpu cycles (actually none) while it is suspended, so waiting for a file event costs nothing. So a polled approach wastes mucho beaucoup (that's French, it means 'a shit load') of cpu cycles but the file watcher does not. You could probably look at PerfMon to see if this is likely true.
You should describe more about what you want to do, but typically if you have something that needs to run in the background and does not require direct user interaction, then a service often makes sense.
You can use Remoting to connect a front-end to your service as needed if you'd like.
Yes, use a service for this kind of operation, but don't use filesystem watcher. If you poll for files in your service, dont use the Timer class either.
Do check to make sure the file is completed writing and is no longer locked before trying to move it.
Its trivial to poll for file changes (syntax may be off), and eliminates much of the extra programming associated with file system watcher events.
While True 'or your exit condition'
Dim _files() as FileInfo = Directory.GetFiles(yourTargetDirectory)
For Each _file as FileInfo In _files
If _file.LastModifiedDate < DateTime.Now.AddMinutes(1) Then
'move your files'
End If
Next
End While
Using a Windows service to wrap FileSystemWatcher is perfectly fine for what you need.
FSW is the right tool for the job (native code for filesystem watching is a bear to get right), and a service is the right mechanism to deploy it given you need 'always on' operation.
The service credentials will be independent of logged-in user too, which may be useful to you.
I have several processes running concurrently that I want to log to the same file.
We have been using Enterprise Library 4.1 Logging Application Block (with a RollingFlatFileTraceListener), and it works fine, apart from the fact that it prepends a GUID to the log file name when two processes try to write to the log file at the same time (a quirk of System.Diagnostics.TextWriterTraceListener I believe).
I've tried various things, including calling Logger.Writer.Dispose() after writing to the log file, but it's not ideal to do a blocking call each time a log entry is being written.
The EntLib forums suggest using MSMQ with a Distributor Service, but that is not an option as MSMQ is not allowed at my company.
Is there another way I can quickly and easily log from multiple threads/processes to the same file?
Sorry to say but the answer is no. The File TraceListeners lock the output file so only one TraceListener can log to a file.
You can try other Trace Listeners that are not file based (e.g. Database, Event Log).
Another option I can think of would be to write your own logging service (out of process) that would log to the file and accepts LogEntries. Then create a custom trace listener that sends a message to your service.
It might not be a good idea since you would have a bit of custom development plus it could impact performance since it is an out of process call. Basically you are setting up your own simplified-pseudo-distributor-service.
EntLib locks the log file when it writes to it. Therefore, 2 processes cannot write to the same log file.
When we have had this problem, that we needed to log from many difference places, to the same place, we have used database logging.
If you are 100% stuck logging to a text file, then you could log to individual log files, and then write a program to merge these files.
I know this is old, but if you are still curious. log4net supports this:
http://logging.apache.org/log4net/release/faq.html#How do I get multiple process to log to the same file?
The problem occurs when the App Pool Recycles and allows for Overlapping Threads. The closing thread has it still open, and the new thread gets the error. Try disabling the overlapping recycling behavior in IIS, or create your own version of the text writer.