I'm sure this has already been done, but Google isn't helping me - I'm getting swamped with answers for similar but different problems:
My boss has asked me to find or build a system that will log uses of our kiosk installations. We build kiosks using java, native c++, c#, python and using things like Unity. We saw another company we worked with using a simple system where a post call with data was logged on a remote site to be checked later. The system allowed the application programmer to decide the contents of the message, and was able to allocate it to either debug or release according to the programmer's wishes.
An example of the log output might be:
[Debug] 28-11-2011 10:10:20 Kiosk1: Pulse
[Debug] 28-11-2011 10:10:25 Kiosk1: Button pressed
[Debug] 28-11-2011 10:10:45 Kiosk1: Widget used
[Debug] 28-11-2011 10:11:20 Kiosk1: Pulse
I looked at log4net/log4j, but that doesn't seem to be compatible with native c++ or python. I'm probably mistaken there :).
Does anyone know of a system that works like this, or that will otherwise be suitable for logging from such diverse languages? If not, I can write my own easily enough. I just don't want to have to support it :)
Regards,
Steve
I'm not sure, but I think what you're looking for is SPLUNK. This can parse almost every log and display it in a unified manner. It can listen to ports, read log files via polling and parses and indexes anything you throw at any point of time.
You can use this to set up you're own multi-language logging server/system. We've been using this and it seamlessly works in our distributed environment.
While writing a specialized logging backend to handle logging both locally and to the network is quite possible, I would advise against it. The reason being that network latency can be to long so it either stops your application, or logging messages can be queued up if using another process/thread to do the actual network pushing.
A much simpler solution is to use little script that is scheduled to run once or a couple of times per day, and that copies the log file(s) to the remote location.
For C++ I highly recommend Poco logging. It allows you to specify the formatting and log level/output using e.g. a properties file.
the python logging library that is included with python is quite similar to log4net, so if you are used to those, the other will be quite easy to understand, but they do not share code (as far as I know)
Use log4j/log4net with a socket appender or log remotely via rsyslog.
You might be interested in something like web beacons. I know it's not exactly what you're asking for, but you ought to think about it for the same reason that web developers do: it's good to know what users are doing.
Related
I am trying to write an application that will make statics over the usage of certain functions and DLLs of Windows API over all the running processes (as well as ones that are created after my application has started).
After searching the internet I have found several tools that may help - such as WinAPIOverride, EasyHook and ProcMon, which use different kind of hooks. Unfortunately, it seems for me that they are not able to make exactly what I need: WinAPIOverride, EasyHook can hook only certain processes that one should choose, and ProcMon doesn't have an interface that I can use to trace calls of API that I need.
I also wonder if this kind of hooking could interface with a Python code, or at least C# environment.
I would like to hear some suggestions for how this could be done.
This article may not be the holy grail for what you're trying to do but certainly will get you further in your quest:
http://www.codeproject.com/Articles/2082/API-hooking-revealed
I'm not 100% sure that is the article I was thinking of for tapping into ProcMon.. After further research I'm pretty sure it was EasyHook I was thinking about: http://www.codeproject.com/Articles/27637/EasyHook-The-reinvention-of-Windows-API-hooking
Also ETW might be another avenue to investigate: http://www.codeproject.com/Articles/570690/Application-Analysis-with-Event-Tracing-for-Window
In some projects I have used MiniDump generation functions to generate dumps.
Creating a MiniDump of a running process
How to create minidump for my process when it crashes?
It is possible to generate these dumps and copy them from the machine to some development/test machines and analyze.
When a program fails at Windows a form is shown stating that would you want to submit the crash information. It is also possible to add some functions to our code to enable this feature.
However the dumps will be sent to Watson servers of Microsoft?
I wonder if it can be possible to write these WER enabling functions in our code and update CorporateWERServer property to make the bugs be forwarded to our servers instead of Microsoft.
WER Reference
So we might be automatically informed when our applications crash, using an already implemented feature of Microsoft. Furthermore, is there a similar utility available for Linux side for programmers to use for such situations.
I've used both mechanisms. I started out "rolling my own" crash reporting system by reading articles on how to write my own unhandled exception filter. I even had the process collect additional files that were helpful in diagnosing the problem. The results were zipped up and http'd to a company server. WER has some advantages over rolling your own solution (eg. collection of crash dumps for you, statistical analysis, automated responses, etc.), so, my company decided to give it a try. We've been using WER for a few years and have had some success and failures. The WER support is pretty terrible. And we can no longer get the additional files as part of our crash report. So, we've decided to look for an alternative solution. There aren't many, but, we did find BugSplat and were pretty impressed. The moral of the story is make sure you know what you expect from such a system before deciding one way or the other.
Recently one of my machines was infected with malware using IRC to communicate back to it's command and control center.
This just sparked interest within me to see if there was a program I could create in C# that is capable of monitoring traffic over TCP that contains strings such as PONG / NICK / USER (strings the irc rfc needs to communicate), and then can tell me the process that that traffic is coming from.
It would be an interesting learning experience for me, and it's something that I want to attempt.
I've done some research and I found something that did make use of WinPcap but i think I'd like to avoid a solution using pcap if possible, can anyone send any suggestions my way?
I know I may be able to view connections between my computer and other hosts using System.Net.NetworkInformation.TcpConnectionInformation and possibly IPGlobalProperties but I'm not sure if there is anyway I can view the information in realtime, or easily trace it back to a process on my PC.
Thank you.
This is going to be hard. The API you need is native, Windows Filtering Platform (WFP). According to Microsoft, this is not accessible from C#, you would have to wrap it in C++/CLI first.
See here and here for discussion. The second thread has some 'could be useful' stuff for you.
I have a C++ application that needs to communicate to a C# application (a windows service) running on the same machine. I want the C++ application to be able to write as many messages as it wants, without knowing or caring when/if the C# app is reading them, or even if it's running. The C# app be able to should just wake up every now and then and request the latest messages, even if the C++ app has been shut down.
What is the simplest way to achieve this? I think this kind of thing is what MSMQ is for, but I haven't found a good way to do it in C++. I'm using Named Pipes right now, but that's not really working out as the way I'm doing it requires a connection between the two apps, and the C++ call to WriteLine blocks until the read takes place.
Currently the best solution I can think of is just writing the messages to a file with a timestamp on each message that the C# application checks periodically against its last update timestamp. That seems a little crude, though.
What is the simplest way to achieve this sort of messaging?
I would use a named pipe.
Well, the simplest way actually is using a file to store the messages. I would suggest using an embedded database like SQLite, though: the advantage will be better performance and a nice way to query for changes (i.e. SELECT * FROM messages WHERE timestamp > last_app_start).
MSMQ definitely sounds like what you want, or the more basic reading and writing files written to a common area but then you need to watch contention on the files.
VC++ help on MSMQ.
The requirement of both apps not always running at the same time but still being able to message each other definitely means you need a third component to store/queue messages. Whether you use a shared database/file or you write a third app that acts as a message store is up to you. Either way you will find sharing always causes contention.
Personally I would look at 0MQ before MSMQ but neither will solve your problem as is. An sqlite database would be my first choice.
I need to build in click and conversion tracking (more specific and focused than IIS log files) to an existing web site. I am expecting pretty high load. I have investigated using log4net, specifically the FileAppender Class, but the docs explicitly state: "This type is not safe for multithreaded operations."
Can someone suggest a robust approach for a solution for this type of heavy logging? I really like the flexibility log4net would give me. Can I get around the lack of safe multi-threading using lock? Would this introduce performance/contention concerns?
While FileAppender itself may not be safe for logging, I'd certainly expect the normal access routes to it via log4net to be thread-safe.
From the FAQ:
log4net is thread-safe.
In other words, either the main log4net framework does enough locking, or it has a dedicated logging thread servicing a producer/consumer queue of log messages.
Any logging framework which wasn't thread-safe wouldn't survive for long.
You could check out the Logging Application Block available in the Microsoft Enterprise Library. It offers a whole host of different types of loggers, as well as a handy GUI configurator that you can point to your app.config\web.config in order to modify it. So there's not need to sift through the XML yourself.
Here's a link to a nice tutorial on how to get started with it:
http://elegantcode.com/2009/01/20/enterprise-library-logging-101/
I'm also interested in the answer, but I'll tell you what I was told when I tried to find a solution.
An easy way around it would be to use something like an SQL database. If the data you want isn't well suited for that, you could have each page access write it's own log file and then periodically merge the log files.
However, I'm sure there's a better solution.
When using syslog, you won't be having any threading issues. Syslog, sends the loglines using UDP to a logdaemon (could potentially be on the same machine).
Works especially great if you have more running processes/services, since all log lines are aggregated in 1 viewing tool.
if you expect really heavy loads, look at how the guys from facebook do it: http://developers.facebook.com/scribe/ You can use their opensource logtool. I don't think you'll hit their kind of load just yet, so you should be safe for some time to come!
R