I've designed a logging service for multiple applications in C#.
Since the thinking of performance saving, all logs should be store in buffer first, and then write to log file when buffer is full.
However, there are some of extension cards (PCI / PCI-e) causing BSoD, which are not in my control. The logs in the buffer will lose when the BSoD occurs, but I want to find a way to keep them.
I've found some articles are discussing about how to dumping data when software crashed. However, the minidump one needs to dump everything by myself, and I think it will cause some performance issues; the other articles (A)(B) are only suitable in single application crash.
Do anyone have any suggestion to save my logs even if BSoD occurs?
EDIT: if there are any suggestion to reduce the loss of data to minimize is also welcome.
Since the buffer you have in your C# application is not written to disk for performance reasons, the only other place left for it is memory (RAM). Since you don't know how Windows manages your memory at the moment of the crash, we have to consider two cases: a) the log is really in RAM and b) that RAM has been swapped to disk (page file). To get access to all RAM of a BSoD, you have to configure Windows to create a full memory dump instead of a kernel minidump.
At the time of a blue screen, the operating system stops relying on almost anything, even most kernel drivers. The only attempt it makes is to write the contents of the physical RAM onto disk. Furthermore, since it cannot even rely on valid NTFS data structures, it writes to the only place of contiguous disk space it knows: the page file. That's also the reason why your page file needs to be at least as large as physical RAM plus some metadata, otherwise it won't be able to hold the information.
At this point we can already give an answer to case b): if your log was actually swapped to the page file, it will likely be overwritten by the dump.
If the buffer was really part of the working set (RAM), that part will be included in the kernel dump. Debugging .NET applications from a kernel dump is almost impossible, because the SOS commands to analyze the .NET heaps only work for a user mode full memory dump. If you can identify your log entries by some other means (e.g. a certain substring), you may of course do a simple string search on the kernel dump.
All in all, what you're trying to achieve sounds like a XY-problem. If you want to test your service, why don't you remove or replace the unrelated problematic PCI cards or test on a different PC?
If blue screen logging is an explicit feature of your logging service, you should have considered this as a risk and evaluated before writing the service. That's a project management issue and off-topic for StackOverflow.
Unfortunately I have to confirm what #MobyDisk said: it's (almost) impossible and at least unreliable.
Related
I am having problems figuring out the source of a memory problem in a complex C# based Windows service. Unfortunately the problem does not occur all the time and I still don't exactly know the conditions which cause it to happen. Sometimes when I check the system ressources used by the service, it takes up multiple gigabytes of memory until the point where it throws OutOfMemory-exceptions everywhere because there isn't any memory left.
I have a paid version of .NET Memory Profiler available but so far it has been useless because the whole system becomes slow and unstable when the service uses too much memory so I cannot attach the memory profiler to the application.
The solution of the application consists of more than 30 individual projects and hundreds of thousands lines of code so there is no way for me to find the source of the problem by simply looking through the source code.
So far the only thing I was able to do is creating a memory dump (.dmp file) of the process while it was using a lot of memory. Is there a way to analyze this dump or anything else that would help me narrow down the source of this problem?
If you could identify some central methods in the main classes of your legacy projects and you have some kind of logging already in place, you could log the total memory (managed and unmanaged, if your application opens such resources) by calling
Process.GetCurrentProcess().PrivateMemorySize64
At least that would give you a feeling if the memory problem is "diffuse", e. g. by not release objects for garbage collection etc., if if the memory problem occurs just in certain use cases (jump in memory consumption, if a certain action happens). Then you could nail it down by more logging and investigating the corresponding code sections. Its tedious, but when you can not use code instrumentation as you have said, I find it effective. If you want to analyze a specific situation with a memory dump, you can use WinDbg for analyzing it, but that takes some effort for the first time to learn, and would be a separate topic (see https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/debugger-download-tools).
I am working to diagnose a series of OutOfMemoryException problems within an application of ours. This is an internal 32-bit (x86) OWIN-hosted WebAPI that runs within a console application and talks to a series of hardware components in parallel. For that reason it's creating around 20 instances of a library, and the sharp increase in "virtual size" memory matches when those instances are created.
From the output of Process Explorer, and dotMemory, it does not appear that we're allocating that much actual memory within this application:
From reading many, many SO answers I think I understand that our problem is either from fragmentation within the G0, G1, G2 & LOH heaps, or we're possibly bumping into the 2GB addressable memory limit for a 32-bit process running on Windows 7. This application works in batches where it collects a bunch of data from hardware devices, creates collections in memory to aggregate that data into a single object, and then saves it to be retrieved by a client app. This activity is the cause of the spikes in the dotMemory visual, but these data structures are not enormous, which I think the dotMemory chart shows.
Looking at the heaps has shown they rarely grow beyond 10-15MB in size, and I don't see much evidence that the LOH is growing too large or being severely fragmented. I'm really struggling with how to proceed to better understand what's happening here.
So my question is two-fold:
Is it conceivable that we could be hitting that 2GB limit for virtual memory, and that's a cause for these memory exceptions?
If that is a possible cause then am I right in thinking a 64-bit build would get around that?
We are exploring moving to a 64-bit build, but that would require updating some low-level libraries we use to also be 64-bit. It's certainly an option we will explore eventually (if not sooner), but we're trying to understand this situation better before investing the time required.
Update after setting the LARGEADDRESSFLAG
Based a recommendation I set that flag on the binary and interestingly saw the virtual size jump immediately to nearly 3GB. I don't know if I should be alarmed by that?!
I will monitor the application with this configuration for the next several hours.
In my case the advice provided by #ThomasWeller was indeed correct and enabling the "large address aware" flag has allowed this application to run for several days without throwing memory exceptions.
If in testing on a computer without a debugger, say a client's computer, I encounter a bug that may have corrupted the state of the program but not actually crashed it, I know I can take a memory dump using the Windows Task Manager (right click on process name, create dump file).
I can use these with WinDbg to peek around in memory, etc., but what would be most useful to me is to be able to restore the dump into memory so that I can continue interacting with the program. Is this possible? If so, how? Is there a tool that can restore it or do I need to write my own.
The typical usermode dumps or minidumps do not contain enough information to do so. While they contain all usermode memory, they do not contain kernel memory, so open handles to kernel resources like files or network sockets will not be included in the dump (and even if they were, the hard disk has most likely changed so just trying to write to the hard disk may corrupt your system even more).
The only way I see to restore a memory dump is restoring the full memory and all other state like hard disk state, which can be done with most virtual machine software (which will, however, disconnect all your network connections on restore; gratefully most programs can handle lost network connectsions better than lost file handles).
I discovered that I could do this with Hyper-V snapshots. If I run my program in a virtual machine, I can optionally dump the memory, create a snapshot, transfer the dump if necessary, come back some time later, restore the snapshot and continue the program.
I understand there are many questions related to this, so I'll be very specific.
I create Console application with two instructions. Create a List with some large capacity and fill it with sample data, and then clear that List or make it equal to null.
What I want to know is if there is a way for me to know/measure/profile while debugging or not, if the actual memory used by the application after the list was cleared and null-ed is about the same as before the list was created and populated. I know for sure that the application has disposed of the information and the GC has finished collecting, but can I know for sure how much memory my application would consume after this?
I understand that during the process of filling the list, a lot of memory is allocated and after it's been cleared that memory may become available to other process if it needs it, but is it possible to measure the real memory consumed by the application at the end?
Thanks
Edit: OK, here is my real scenario and objective. I work on a WPF application that works with large amounts of data read through USB device. At some point, the application allocates about 700+ MB of memory to store all the List data, which it parses, analyzes and then writes to the filesystem. When I write the data to the filesystem, I clear all the Lists and dispose all collections that previously held the large data, so I can do another data processing. I want to know that I won't run into performance issues or eventually use up all memory. I'm fine with my program using a lot of memory, but I'm not fine with it using it all after few USB processings.
How can I go around controlling this? Are memory or process profilers used in case like this? Simply using Task Manager, I see my application taking up 800 MB of memory, but after I clear the collections, the memory stays the same. I understand this won't go down unless windows needs it, so I was wondering if I can know for sure that the memory is cleared and free to be used (by my application or windows)
It is very hard to measure "real memory" usage on Windows if you mean physical memory. Most likley you want something else like:
Amount of memory allocated for the process (see Zooba's answer)
Amount of Managed memory allocated - CLR Profiler, or any other profiler listed in this one - Best .NET memory and performance profiler?
What Task Manager reports for your application
Note that it is not necessary that after garbage collection is finished amount of memory allocated for your process (1) changes - GC may keep allocated memory for future managed allocations (this behavior is not specific to CLR for memory allcation - most memory allocators keep free blocks for later usage unless forced to release it by some means). The http://blogs.msdn.com/b/maoni/ blog is excelent source for details on GC/memory.
Process Explorer will give you all the information you need. Specifically, you will probably be most interested in the "private bytes history" graph for your process.
Alternatively, it is possible to use Window's Performance Monitor to track your specific application. This should give identical information to Process Explorer, though it will let you write the actual numbers out to a separate file.
(A picture because I can...)
I personaly use SciTech Memory Profiler
It has a real time option that you can use to see your memory usage. It has help me find a number of problems with leaking memory.
Try ANTS Profiler. Its not free but you can try the trial version.
http://www.red-gate.com/products/dotnet-development/ants-performance-profiler/
See this SuperUser question. To summarize, VM software lets you save state of arbitrary applications (by saving the whole VM image).
Would it be possible to write some software for Windows that allows you to save and reload arbitrary application state? If so (and presumably so), what would it entail?
I would be looking to implement this, if possible, in a high-level language like C#. I presume if I used something else, I would need to dump memory registers (or maybe dump the entire application memory block) to a file somewhere and load it back somewhere to refresh state.
So how do I build this thing?
Well, it's unlikely that this is going to happen, especially not in C# - addressing the low level requirements in managed code would hardly be possible. Saving the state of a whole virtual machine is actually easier than just saving single processes, all you have to do is dump the whole machine memory and ensure a consistent disk image, which, given the capabilities of Virtualization software, is rather straightforward.
Restoring a single process would imply loading all shared objects that the process refers to, including whatever objects the process refers to in kernel space, i.e. file/memory/mutex/whatever handles, and without the whole machine / operating system being virtual, this would mean poking deep down in the internals of Windows...
All I'm saying is: while it is possible, the effort would be tremendous and probably not worth it.