we are using log4net with a AdoNetAppender to write critical logs into an database. Since the AdoNetAppender is a subclass of the BufferedAppender there is a possibility to enable queuing of log events.
What I'd like to do is to save the backup & restore the log buffer to a local file, so that no log entry can get lost it the database is down or the application crashes.
Does somebody know how to this?
Don't think you can save the buffer without writing some code yourself. What I rather would suggest is sending the logs to both a AdoNetAppender and a RollingFileAppender. The first will ensure your regular logging to database while the second will ensure that the latest logs are also written to disk.
Update: in light of your later comments I can see how logging to two different sources (one database and one local store, either a file or local database) gets tough to consolidate.
Imo you should absolutely use log4net for what it is best at: a tried and true framework for collection log data from the application and routing that data to receiving systems. Building a failover system on top of log4net though is not what it is designed for. For instance, there is no process model that can pick up the pieces after an application crash.
Instead, handle failover in the receiving system. Failover at the database level and the network level gets you a long way, still you are not guaranteed 100% uptime. By logging to a local store and then have a process picking up the logs and shipping it to the database would minimize the risk for log data being lost, and at the same time you avoid having to consolidate logs from two different stores. Even better, logging is still simple and fast and thus have a low impact on the application.
An alternative would be logging to a local database and having a database job pull the data into the master database. You could also use queuing. There is a sample MsmqAppender out there to get you started. If you're using MS SQL Server you could even use the Service Broker for its queuing abilities.
Related
I am using NLog as my logging framework. I envisage I will have logs coming in from multiple sources (20, 30+)
I want to be able to live monitor at will.
What Viewers (commercial or free) are the best to use?
I am currently rolling over my days and using C:\Logging as my "base" logging directory.
NLog FileName for trace is as follows:
C:\Logging\${appdomain:format={1\}}\${shortdate}\MyType.xml
I have Trace/Debug/Info,Warn/Error/Fatal all going into their own separate files (Debug.xml/Info.xml/Error.xml etc), all in the above file name format.
I also have a UDP target setup, and that is currently going to Sentinel. This works fine, and would be a great solution for me if sentinel could setup multiple apps/tabs/receiveds. But on the surface, I can only have one it would seem. The other problem is that I have millions of logs pumping through. Last time i left it running for a while, it killed all the memory in my system.
Ideally, What i would like, is an application that i simply add the "C:\Logging" folder to, like a "watch folder" and it keeps pumping out my logs, including detecting when a new file is created (example Fatal.xml), which would also handle date rollovers. Also the addition of multiple receiver types eg UDP
Not sure if Amazon is an option for you, but I ran across an AWS NLog Target that I was looking implement. I am not capturing as many logs as you, but do have logs coming from multiple servers. This would send the items written to the logs to Amazon CloudWatch Logs target and searchable in the console.
I am not sure on the bandwidth required to duplicate log items to AWS but it would put them in one place. CloudWatch Retention has been increased but if you do find issues, you could always go back to the text log files for more details past the retention period.
You could also setup CloudWatch Alarms to let you know if there are issues.
Is there any way I can use Microsoft.Diagnostic.Tracing.EventSource package or any other .NET built in types to implement a back up logic for logging?
I have used an EventSession that directs all logs to an .ETL file but there are two problems with this:
The most important one is that, from what I understand, the logs are actually being written to the file when the processing stops. But what happens if the machine shuts down or the processor process gets killed? From my tests the logs are lost.
The second problem is less important and it's just that it would be more convenient for my solution to be able to read from the file as logs are coming in.
Basically I would like to have a file buffer for my logs and I was wondering if there's some built in implementation especially for this.
Yes, EventSource's WriteEvent methods are not blocking, so they do
not guarantee that logs actually have been saved.
And yes, you cannot
read ETL files while they are in use.
ETW is not the option for you in this case, you could use common TraceSource (with two listeners, one is configured to do auto-flush and another one is ETW adaptor, which you could listen to through a real-time ETW session) or log4net (you could choose appender that works for your case).
So, I have been stuck on the below mentioned issue for more than a day now and am dead confused on how can I go about it.
My boss wants the logs generated by Log4Net, into the database (currently they are being generated in flat files). But hey, he does not want Log4Net to log it directly to the database (which is easy to do :)), as it would be an overhead leading to increased latency.
His requirement is, once the log files are generated, these files should be bulk inserted/copied/imported into the database.
Does anyone have any tips or suggestions I could use?
Note: The lines in the log file are not consistent, most of the times it starts with Date. But at times if there was an exception the Line starts with System.IO.Exception.
Answering this based on the information available from your question
You could just have workers/background processes running (periodically or whenever file updates depending on how much log loss you can tolerate) which would consume this file(s), parse them (you will have to do this regardless, since the files are custom to you and store them in db as you want. Since they are read only operations you will not face any concurrency issues while reading them. Although you will face this issue when writing to your db.
I'm currently coding in C# WinForms on Visual Studio 2008, and we might be implementing SQL database connectivity. However, it has to be very robust for industrial work, meaning the potential for power outages during the writing process.
I was thinking of making a file in between the app and SQL. The program first flips a SAFE flag on the file set to FALSE, then writes to it. Once it has been safely written to that file, it flips the SAFE flag to TRUE, THEN writes it to the SQL database. Is such a thing possible/the best course of action for outage prone environments?
You would want to implement transactions.
If the power goes out , you can set your transaction to hold over to attempt to re-write the information when the system comes back online.
i think that a sql write inside a transaction (begin...commit) is way more robust than anything you can implement as a single developer.
if you get a positive ack from the server that the data has been written then you can be sure it has.
if not then you know that you have to retry.
When you write to your database you can use various technologies including a Message Queue to ensure the message is delivered to its destination, transaction processing to ensure that the whole save is completed or aborted. If you use SQL server it will also manage transactions that have not been saved to the database in the Transaction log thereby solving the problem of recovery after an outage. You therefore need to look at a couple of technologies:
Message Queuing : Microsoft Message Queue
Transaction Processing : Transaction Fundamentals
Transaction Logs : SQL Server Transaction Logs
I have a C# console app which I'm deploying around 20 times (with different config settings) and running. As you might imagine it's hard to keep an eye on what's happening with 20 apps running (I'm eventually going to deploy these as windows services), so is there anything that can show the output of these in one place easily?
I've thought about log files but these could get big quite fast, and it is a lot of files to open and look at - I just want to have some output to check things are still running as expected.
Edit:
I'm going to be writing errors and stop/start information to the database. What I'm talking about here is the general processing information, which isn't all that relevant to revisit, but interesting to look at while its running in the console app.
I have successfully used log4net and its configurable UdpAppender. Then you can point all the UdpAppenders to a single machine where you can receive the Udp messages with Log4View for example.
Since it's configurable, you can use it when you install and debug in production and then increase the logging level to only output ERROR messages instead of DEBUG or INFO messages.
http://logging.apache.org/log4net/
http://www.log4view.com
http://logging.apache.org/log4net/release/config-examples.html
Maybe because I come from a heavy DB background, but how about using SQL Server with a Log table to track activity across different apps?
DBs are geared up towards concurrency and will easily handle multiple applications inserting data into the same Log table, also you get the options of slicing and dicing through the data as much as you would like, taking advantage of the already existing aggregation functions in a DB environment.
If you go down that route, you will probably need to consider maintaining that table (Log retention period, etc.).
You could also potentially start using tools such as Splunk to collate all the log data, and start corresponding app failures to system or environment failures (if these are being tracked).
I'd second Mikael Östberg and recommend using a logger library (log4net, or nlog). There are many options where you can send messages to either a database or queues, etc... Since you can turn the logging on or off easily, you can even keep it in your services as a monitor hook in case something weird happens