I have a C# console app which I'm deploying around 20 times (with different config settings) and running. As you might imagine it's hard to keep an eye on what's happening with 20 apps running (I'm eventually going to deploy these as windows services), so is there anything that can show the output of these in one place easily?
I've thought about log files but these could get big quite fast, and it is a lot of files to open and look at - I just want to have some output to check things are still running as expected.
Edit:
I'm going to be writing errors and stop/start information to the database. What I'm talking about here is the general processing information, which isn't all that relevant to revisit, but interesting to look at while its running in the console app.
I have successfully used log4net and its configurable UdpAppender. Then you can point all the UdpAppenders to a single machine where you can receive the Udp messages with Log4View for example.
Since it's configurable, you can use it when you install and debug in production and then increase the logging level to only output ERROR messages instead of DEBUG or INFO messages.
http://logging.apache.org/log4net/
http://www.log4view.com
http://logging.apache.org/log4net/release/config-examples.html
Maybe because I come from a heavy DB background, but how about using SQL Server with a Log table to track activity across different apps?
DBs are geared up towards concurrency and will easily handle multiple applications inserting data into the same Log table, also you get the options of slicing and dicing through the data as much as you would like, taking advantage of the already existing aggregation functions in a DB environment.
If you go down that route, you will probably need to consider maintaining that table (Log retention period, etc.).
You could also potentially start using tools such as Splunk to collate all the log data, and start corresponding app failures to system or environment failures (if these are being tracked).
I'd second Mikael Östberg and recommend using a logger library (log4net, or nlog). There are many options where you can send messages to either a database or queues, etc... Since you can turn the logging on or off easily, you can even keep it in your services as a monitor hook in case something weird happens
Related
I am using NLog as my logging framework. I envisage I will have logs coming in from multiple sources (20, 30+)
I want to be able to live monitor at will.
What Viewers (commercial or free) are the best to use?
I am currently rolling over my days and using C:\Logging as my "base" logging directory.
NLog FileName for trace is as follows:
C:\Logging\${appdomain:format={1\}}\${shortdate}\MyType.xml
I have Trace/Debug/Info,Warn/Error/Fatal all going into their own separate files (Debug.xml/Info.xml/Error.xml etc), all in the above file name format.
I also have a UDP target setup, and that is currently going to Sentinel. This works fine, and would be a great solution for me if sentinel could setup multiple apps/tabs/receiveds. But on the surface, I can only have one it would seem. The other problem is that I have millions of logs pumping through. Last time i left it running for a while, it killed all the memory in my system.
Ideally, What i would like, is an application that i simply add the "C:\Logging" folder to, like a "watch folder" and it keeps pumping out my logs, including detecting when a new file is created (example Fatal.xml), which would also handle date rollovers. Also the addition of multiple receiver types eg UDP
Not sure if Amazon is an option for you, but I ran across an AWS NLog Target that I was looking implement. I am not capturing as many logs as you, but do have logs coming from multiple servers. This would send the items written to the logs to Amazon CloudWatch Logs target and searchable in the console.
I am not sure on the bandwidth required to duplicate log items to AWS but it would put them in one place. CloudWatch Retention has been increased but if you do find issues, you could always go back to the text log files for more details past the retention period.
You could also setup CloudWatch Alarms to let you know if there are issues.
Problems have been reported to me regarding the performance of a live site. I can't seem to replicate any of these issues on any dev or staging environments, and the profilers I have ran against dev has revealed nothing unusual.
This has led me to turn to a diagnostics trace for a simple timing trace so I can at least try and isolate the cause and try and narrow it down.
I'm quite happy to add
System.Diagnostics.Trace.WriteLine("....");
wherever necessary and add a listener (via web.config entry) to write out to a log file, but could this massively impact the performance of the live environment itself?
Is there anything else I need to consider when, potentially, leaving this to run over the weekend? i.e. is it best that I specify how large the log file is to get before closing and opening a new one?
It depends how much data you are going to log so turn on the logger and check if your application behaves normally. Also if logging to a log file slows down your application consider a faster TraceListener such as EventLogTraceListener (you may create a dedicated event log for this purpose with maximum size and log rolling). In case logging to a file is not a problem get EssentialDiagnostics RollingFileTraceListener. It has many options including setting maximum file size and the number of rolled files.
Use a logging framework like log4NET and make logging like:
LogManager.GetCurrentClassLogger().Debug("...");
When you disable logging afterwards in the configuration, these functions are not executed by the framework.
If you need to do string formatting for your messages: Use "DebugFormat()" which will not do the formatting if it is not needed by the level of logging desired.
we are using log4net with a AdoNetAppender to write critical logs into an database. Since the AdoNetAppender is a subclass of the BufferedAppender there is a possibility to enable queuing of log events.
What I'd like to do is to save the backup & restore the log buffer to a local file, so that no log entry can get lost it the database is down or the application crashes.
Does somebody know how to this?
Don't think you can save the buffer without writing some code yourself. What I rather would suggest is sending the logs to both a AdoNetAppender and a RollingFileAppender. The first will ensure your regular logging to database while the second will ensure that the latest logs are also written to disk.
Update: in light of your later comments I can see how logging to two different sources (one database and one local store, either a file or local database) gets tough to consolidate.
Imo you should absolutely use log4net for what it is best at: a tried and true framework for collection log data from the application and routing that data to receiving systems. Building a failover system on top of log4net though is not what it is designed for. For instance, there is no process model that can pick up the pieces after an application crash.
Instead, handle failover in the receiving system. Failover at the database level and the network level gets you a long way, still you are not guaranteed 100% uptime. By logging to a local store and then have a process picking up the logs and shipping it to the database would minimize the risk for log data being lost, and at the same time you avoid having to consolidate logs from two different stores. Even better, logging is still simple and fast and thus have a low impact on the application.
An alternative would be logging to a local database and having a database job pull the data into the master database. You could also use queuing. There is a sample MsmqAppender out there to get you started. If you're using MS SQL Server you could even use the Service Broker for its queuing abilities.
I'm working with a system that consists of several applications and services, almost all using a SQL database.
Windows services do different things at different times, and I would like to track them. Meaning that on some deployed systems we see the machine running high on CPU, we see that the sql process is running high, but we can't be sure about which service is responsible for it.
I wonder if Performance Counters are good for this job.
Basically I would like to be able to see at a certain moment which service woke up and is processing something.
It seems to me that I can end up having a perfcounter that only has the value 0 or 1 for each service to show if it is doing something, but this doesn't seem like a normal usage for perfcounters.
Are performance counters suitable?
Do you think I should track this in a different way?
If your monitoring framework/approach already centers around monitoring performance counters, this is a viable approach.
Personally I find more detailed instrumentation necessary to really understand what's happening in my services (though maybe that has to do with the nature of my services).
I use .NET Logging Framework because it's simple and can write to multiple targets including log files, the event log, and a TCP socket (I have a simple monitor that listens on the logging socket for each app server and shows me in real-time what's happening).
Performance Counters are attractive because they are really lightweight, but as you say, they only allow you to capture numeric values. Sure, there's a slew of different types of values you can record, such as averages, deltas and totals, but they have to be numbers.
If you need more information than that, you must resort to some other type of instrumentation. In your case, it sounds like your need goes more in that direction.
If your services don't wake up and suspend themselves too often, it sounds like informational message to a custom event log might be a good idea. Create a custom event log for the application if you expect a fair amount of these so as to not flood the regular Application event log.
The .NET Trace API will be a better option if you expect the instrumentation to generate too much data for the normal event log. You can configure your application(s) to trace or not based on app/web.config, although a change will require a restart of the app. This is a good option if you only wish to use the instrumentation for troubleshooting, but it otherwise generates too much data or if tracing itself degrades performance too much. Another good thing about the Tracing API is that you can Trace on multiple levels, so even if you have written code to Trace very verbosely, you will only see that verbose trace data if you enable verbose tracing. That gives you better control of just what is being traced.
Eric J has a good point. I think if you really want to capture "timing" performance you'll have to use some other sort of logging and use start and stop time logs. I personally like log4net though it can be a pain to configure the first time around
I am building a client-server based solution; client being a desktop application and the server being a web application.
Basically, I need to monitor the performance and resource utilization of the client, which is a .NET 2.0 based Windows Desktop application.
The most important thing I need to monitor is the network resources the client uses, i.e. what is the size of the data that flows out from the client to the server and what is the size of the data that the client downloads from the server.
Apart from this, general performance monitoring would help too.
Please guide.
Edit: A few people have suggested using perfmon, but aren't the values shown in perfmon system-wide? I need these network based stats for a single application only...bytes being sent and received by a single desktop application.
The standard tool for network monitoring is Wireshark.
It allows you to filter the network traffic very flexiblely.
This could be quite an overkill for your application though.
If you are using pure .NET, I would suggest that you add performance logging into your networking classes on the server side- if you are using .Net library classes, then inheritate from them your own classes which add statistics when sending and receiving data.
You need to split your monitoring in two parts:
How the system interacts with the server (number of calls performed)
Amount of network traffic (size of exchanged data for any call)
The first part is (in my experience) often negleted while it has a lot of importance, because acquiring a new connection is often much more expensive that data traffic in itself.
You do not tell us anything about the king of connection you're using (low level tcpip calls, web services, WCF or what else) but my suggestion is:
Find a way to determine how many time your application calls the server
Find how much any single call is costing in term of data exchanged
How to monitor these values depends a lot from the technology involved, for some is very simple (if, for example, you're using a web service, setting up Fiddler to monitor the calls and examining an monitoring results is very simple), for other you need to work using a low level traffic analyzer like Wireshark or MS Network Monitor and learn how to filter traffic according to IP address of the server, ports used and other parameters.
If you clarify your application architecture I can try to be more specific.
Regards
Massimo
You can also use Task Manager to do this. Go to the processes tab, then View->"select columns". Check "I/O read bytes" and "I/O write bytes". Then find your program in the processes list and you can observe the cumulative values.
Take a look at this article: http://www.codeproject.com/KB/IP/apptraffwatcher.aspx
You may be able to tear apart the source code, and grab what you need to meassure download/upload for your application's process ID.
It looks like he uses this library to get information about the amount of traffic: http://www.codeproject.com/KB/IP/trafficwatcher.aspx
I tried the perfmon and I was unable to watch our network traffic either. But I was able to in the Performance Explorer in Visual Studio 2005 Team suite.
If you have Team edition Visual Studio you can set up either Sampling/Instrumentation on your desktop application. Then go into options of the session. select Events -> Windows Kernel Trace -> Network. Run your application and let the Visual studio log the data. Then save the report. (I love Microsoft for this "feature") go to the command prompt, navigate to C:\Program Files\Microsoft Visual Studio 8\Team Tools\Performance Tools and run "vsperfreport /CALLTRACE (filename).vsp" This will produce a csv file containing all network packets sent/recieved/size/port etc by the desktop application.
I know this was a long winded solution but I just tried it on my .Net 2.0 application and it captured all of our communication with Oracle Identity Manager and Oracle Database.
It is not clear by your post if you are using HTTP requests. You indicated that the server is a web application, which implies (perhaps incorrectly) to me that you might be using the HTTP protocol to send/receive data from server to client.
If so, one tool that might be of use is Fiddler. This tool will monitor all HTTP traffic in and out of your workstation and it can (I believe) watch specific sessions and applications. The nice part is that you can see individual requests and see the statistics for these requests, including bytes in/out, round trip times, and other useful bits of information.
If you are not HTTP based, then this tool won't help.
I'm surprised nobody has suggested SysInternals (now Microsoft) Process Explorer (technet.microsoft.com/en-us/sysinternals/bb896653.aspx). If you right click on the executable in question and left click properties it will bring up a dialog box. Then you switch to the performance tab and you can monitor I/O of the executable. The Performance Graph tab will show CPU usage and I/O bytes history graphed over time. It's a cool and free tool.
You want to look at perfmon (otherwise called Performance Monitor in admin tools off the start menu).
Open it in its default graph view, add a counter, select network interface, then bytes per second (or a similar counter), click ok and you're done.
You can experiment with the other networking counters as there are many, one of them will do exactly what you want. You can also save the perfmon logs to a file and view them afterwards - you'll see the graph in its entirety and you can "zoom in" on sections. Alternatively, you can save log-style files with just raw numbers.
Here's a quick guide through perfmon as an admin tool, once you understand that, the rest comes easily.
In Vista you can't add individual counters any more, you add the entire set of counters grouped under an object - so for my example, you'd add the Network Interface object, then you'd see all the individual counters on the graph after you click ok.
If you want this built into your client codebase, and not using an external tool, you can use Performance Counters to get access to this and most other things reported by the Performance Monitor, Task Manager, etc.
You should check out ACE Analyst for this use case - think of it as a superintelligent layer on top of Wireshark packet captures. You need to look at the packets to understand the true nature of the application behavior as runs across the network.