I am maintaining a video application written in C#.
I need as much control as possible over memory allocation/deallocation
for large memory buffers (hundreds of megabytes).
As it is written, when pixel data needs to be freed, the pixel buffer
is set to null. Is there a better way of freeing up memory?
Is there a large cost to garbage collecting large objects?
Thanks!
Don't throw big buffers like that away, you are lucky to have it. Video gives lots of opportunity for re-use. Don't lose a buffer until you are sure you won't need it anymore. At which point it doesn't matter when it get collected.
The cost of garbage collecting large objects is very high from what I remember. From what I read they automatically become generation 2 on allocation(they are allocated in the large object heap). And since they are large they force frequent generation 2 collections.
So I'd rather implement manual pooling for the bitmap arrays, or even use unmanaged memory. Have some pool class and return the array back to it in the Dispose of your pixels/bitmap class.
With memory blocks that large ("hundreds of megabytes") it should be relativaly easy to know precisely who and where uses them (you can fit just 10-20 of such blocks in memory anyway). As ypu plan to use such amounts of mmeory you need to carefully budget memory usage - i.e. simple copy of whole buffer will take non-trivial time.
When you are done with particular block you can force GC yourself. It sounds like reasonable usage of GC.Collect API - you done with using huge portion of all memory avaialble.
You also may consider switchihng to allocation of smaller (64k) blocks and link them together if it works for your application. This will align better with garbage collection and may provide more flexibility for your application.
Related
If you application is such that it has to do lot of allocation/de-allocation of large size objects (>85000 Bytes), its eventually will cause memory fragmentation and you application will throw an Out of memory exception.
Is there any solution to this problem or is it a limitation of CLR memory management?
Unfortunately, all the info I've ever seen only suggests managing risk factors yourself: reuse large objects, allocate them at the beginning, make sure they're of sizes that are multiples of each other, use alternative data structures (lists, trees) instead of arrays. That just gave me an another idea of creating a non-fragmenting List that instead of one large array, splits into smaller ones. Arrays / Lists seem to be the most frequent culprits IME.
Here's an MSDN magazine article about it:
http://msdn.microsoft.com/en-us/magazine/cc534993.aspx, but there isn't that much useful in it.
The thing about large objects in the CLR's Garbage Collector is that they are managed in a different heap.
The garbage collector uses a mechanism called "Compacting", which is basically fragmentation and re-linkage of objects in the regular heap.
The thing is, since "compacting" large objects (copying and re-linking them) is an expensive procedure, the GC provides a different heap for them, which is never being compacted.
Note also that memory allocation is contiguous. Meaning if you allocate Object #1 and then Object #2, Object #2 will always be placed after Object #1.
This is probably what's causing you to get OutOfMemoryExceptions.
I would suggest having a look at design patterns like Flyweight, Lazy Initialization and Object Pool.
You could also force GC collection, if you're suspecting that some of those large objects are already dead and have not been collected due to flaws in your flow of control, causing them to reach higher generations just before being ready for collection.
A program always bombs on OOM because it is asking for a chunk of memory that's too large, never because it completely exhausted all virtual memory address space. You could argue that's a problem with the LOH getting fragmented, it is just as easy to argue that the program is using too much virtual memory.
Once a program goes beyond allocating half the addressable virtual memory (a gigabyte), it is really time to either consider making its code smarter so it doesn't gobble so much memory. Or making a 64-bit operating system a prerequisite. The latter is always cheaper. It doesn't come out of your pocket either.
Is there any solution to this problem or is it a limitation of CLR memory management?
There is no solution besides reconsidering your design. And it is not a problem of the CLR. Note, the problem is the same for unmanaged applications. It is given by the fact, that too much memory is used by the application at the same time and in segments laying 'disadvantageous' out in memory. If some external culprit has to be pointed at nevertheless, I would rather point at the OS memory manager, which (of course) does not compact its vm address space.
The CLR manages free regions of the LOH in a free list. This in most cases is the best what can be done against fragmentation. But since for really large objects, the number of objects per LOH segment decreases - we eventually end up having only one object per segment. And where those objects are positioned in the vm space is completely up to the memory manager of the OS. This means, the fragmentation mostly happens on the OS level - not on the CLR. This is an often overseen aspect of heap fragmentation and it is not .NET to blame for it. (But it is also true, fragmentation can also occour on the managed side like nicely demonstrated in that article.)
Common solutions have been named already: reuse your large objects. I up to now was not confronted with any situation, where this could not be done by proper design. However, it can be tricky sometimes and therefore may be expensive though.
We were precessing images in multiple threads. With images being large enough, this also caused OutOfMemory exceptions due to memory fragmentation. We tried to solve the problem by using unsafe memory and pre-allocating heap for every thread. Unfortunately, this didn't help completely since we relied on several libraries: we were able to solve the problem in our code, but not 3rd party.
Eventually we replaced threads with processes and let operating system do the hard work. Operating systems have long ago built a solution for memory fragmentation, so it's unwise to ignore it.
I have seen in a different answer that the LOH can shrink in size:
Large Arrays, and LOH Fragmentation. What is the accepted convention?
"
...
Now, having said that, the LOH can shrink in size if the area at its end is completely free of live objects, so the only problem is if you leave objects in there for a long time (e.g. the duration of the application).
...
"
Other then that you can make your program run with extended memory up to 3GB on 32bit system and up to 4 GB on 64bit system.
Just add the flag /LARGEADDRESSAWARE in your linker or this post build event:
call "$(DevEnvDir)..\tools\vsvars32.bat"
editbin /LARGEADDRESSAWARE "$(TargetPath)"
In the end if you are planning to run the program for a long time with lots of large objects you will have to optimize the memory usage and you might even have to reuse allocated objects to avoid garbage collector which is similar in concept, to working with real time systems.
Refers to this Unity documentation and go to section
Large heap with slow but infrequent garbage collection
var tmp = new System.Object[1024];
// make allocations in smaller blocks to avoid them to be treated in a special way, which is designed for large blocks
for (int i = 0; i < 1024; i++)
tmp[i] = new byte[1024];
// release reference
tmp = null;
The trick is to pre-allocate some memory chunks at the program start.
Why does this trick work?
Are the chunks being somekind of "registered" (or "bound") to the application when they are being pre-allocated, so that even though the tmp is being freed when Start() is finished, the OS still treat these chunks as "registered" to the application?
Since the chunks are "registered" to the application, so the heap size of the application is expanded to certain size, and the next time it acquires memory chunk, the OS would just pick it from the heap of this application.
Is my explanation correct? No matter Yes or No could someone please explain in more details, thanks.
It's not really a trick. It's the way that parts of Unity3D handle memory.
In Unity3D you have objects that are handled by Mono and will be garbage collected, and objects that are handled by Unity, that will not be garbage collected. Strings, ints etc are cleaned up by Mono automatically and we do not have to worry about this. Texture(2D)s etc are not, and we have to dispose of these objects manually.
When a request for memory is made the first thing that happens is that the memory manager scans the application's currently allocated memory from the OS for a chunk large enough to store the data you are requesting. If a match is found, that memory is used. If a match is not found, then the application will request additional memory from the OS in order to store your data. When this data is no longer used up it is garbage collected, but the application still retains that memory. In essence, it sets a flag on the memory to say it is 'usable' or re-allocatable. This reduces the requests for memory made to the OS by never returning it.
What this means is two things;
1) Your application's memory will only continue to grow, and will not return memory to the OS. On mobile devices this is dangerous, as if you use too much memory your application will be terminated.
2) Your application may actually be allocated way more memory than it actually needs. This is due to fragmented memory. You may have 10MB of available memory in your application's memory pool, but non of those chunks are large enough to house the data you need to store. Therefore, it is possible that the application will request more memory from the OS because there is not a single piece of contiguous memory available that can be used.
Because you're creating a large object, an therefore requesting memory, when you set that object to null and signal to the garbage collector that memory is no longer needed by the application, it is quicker to reallocate that kept memory to other objects rather than requesting additional memory from the OS. It is this reason why in theory this particular method is fast and will result in less performance spikes as the garbage collector is invoked less often. Especially as this is a large, contiguous memory allocation.
Why does this trick work?
This trick works because the application won't return memory to the OS, unless the OS memory manager is low and explicitly requests them to do so, in which then they will free up as much as possible. There is an assumption that once the memory is allocated, it will be needed again. If it is already allocated, there is no reason to return it back to the OS, unless it really needs to use it.
Now i started work on typical application that is massively using buffers.. i was surprised that i can't find good clear guide on this topic.
I have couple questions.
1) When do i prefer to use buffer in unmanaged heap memory over managed memory?
I know that object allocation is faster on .net then on the unmanaged heap and object destruction is much more expensive on .net because of GC overhead, so i think that it will be little faster to use unmanaged. When should i use fixed{} and when Marshal.AllocHGlobal()?
2) As i understand it is more effective to use week reference for both managed and unmanaged buffers in .net if it buffer possible can be reused after some time (based on user actions), isn't it?
Trying to manually manage your memory allocations using native "buffers" is going to be difficult at best with .NET. You can't allocate managed types into the unmanaged buffers, so they'll only be usable for structured data, in which case, there's little advantage over a simple managed array (which will stay in memory contiguously, etc).
In general, it's typically a better approach to try to manage how you're allocating and letting go of objects, and try to manually reuse them as appropriate (if, and only if, memory pressure is an issue for you).
As for some of your specific points:
I know that object allocation is faster on .net then on the unmanaged heap and object destruction is much more expensive on .net because of GC overhead, so i think that it will be little faster to use unmanaged.
I think your assumptions here are a bit flawed. Object allocation, at the point of allocation, is typically faster in .NET, as the CLR will potentially have preallocated memory it can already use. Object "destruction" is also faster on .NET, though there is a deferred cost due to GC that can be a bit higher (though it's not always). There are a lot of factors here, mainly focused around object lifecycles - if you allow your objects to be promoted to Gen1 or especially Gen2, then things can potentially get difficult to track and measure, as GC compaction costs can be higher.
When should i use fixed{} and when Marshal.AllocHGlobal()?
In general, you would (very) rarely use either in C#. You're typically better off leaving your memory unpinned, and allowing the GC to do its work properly, which in turn tends to lead to better GC heuristics overall.
2) As i understand it is more effective to use week reference for both managed and unmanaged buffers in .net if it buffer possible can be reused after some time (based and user actions), isn't it?
Not necessarily. Reusing objects and keeping them alive longer than necessary has some serious drawbacks, as well. This will probably guarantee that the memory will get promoted into Gen2, which will potentially make life worse, not better.
Typically, my advice would be to trust the system, but measure as you go. If, and only if, you find a real problem, there are almost always ways to address those specific issues (without resorting to unmanaged or manually managing memory buffers). Working with raw memory should be an absolute last resort when dealing with a managed code base.
My understanding is that the CPU does its math operations in conjunction with the CPU caches (L1 etc) and that if a value needed for an operation is not already in the cache a page will need to be got from RAM before the calculation can be performed. It seems reasonable to think, therefore, that managed heap RAM is a better place to be having your Vector data than than any old hole the OS managed to find somewhere in the great expanse of unmanaged stack RAM. I say this because I assume managed memory is held together tighter than unmanaged memory, and therefore there is more likelihood that vectors (x, y, z) for math operations will be stored in same pages loaded into the cache; whereas vectors as structs on the stack might be pages apart. Could anyone explain the pros and cons of class based rather than struct based vector classes in this light?
CPU cache is managed completely by CPU. Memory that is recently accessed is cached by relatively large chunks (i.e. 128 bytes around accessed position).
OSes manage paging to/from physical memory. If you application hitting that process often enough (i.e. size of your data is way bigger that physical RAM) than you have other issues to worry about outside CPU cache line hits and misses.
There is essentially no difference between stack and heap from that point of view. The only meaningful difference is how close the next piece of data to be used to one of recently used once.
In most cases math classes (vector/matrix/points) are stored in sequential blocks of memory for both managed and native implementations. So caching behavior is likely be comparable unless one explicitly does some strange allocations to make individual elements to be far apart in memory.
Summary: make sure to profile your code and keep data compact if performance is of huge concern.
Try and measure different iteration orders across arrays. I.e. if iteration crosses caching lines every time it could be slower - walk by row or by column first in 2d array could show measurable difference for large enough data sets when caches have to be repopulated on most array access...
The stack is faster here is a site that cover it in more detail.
[Update - Sep 30, 2010]
Since I studied a lot on this & related topics, I'll write whatever tips I gathered out of my experiences and suggestions provided in answers over here-
1) Use memory profiler (try CLR Profiler, to start with) and find the routines which consume max mem and fine tune them, like reuse big arrays, try to keep references to objects to minimal.
2) If possible, allocate small objects (less than 85k for .NET 2.0) and use memory pools if you can to avoid high CPU usage by garbage collector.
3) If you increase references to objects, you're responsible to de-reference them the same number of times. You'll have peace of mind and code probably will work better.
4) If nothing works and you are still clueless, use elimination method (comment/skip code) to find out what is consuming most memory.
Using memory performance counters inside your code might also help you.
Hope these help!
[Original question]
Hi!
I'm working in C#, and my issue is out of memory exception.
I read an excellent article on LOH here ->
http://www.simple-talk.com/dotnet/.net-framework/the-dangers-of-the-large-object-heap/
Awesome read!
And,
http://dotnetdebug.net/2005/06/30/perfmon-your-debugging-buddy/
My issue:
I am facing out of memory issue in an enterprise level desktop application. I tried to read and understand stuff about memory profiling and performance counter (tried WinDBG also! - little bit) but am still clueless about basic stuff.
I tried CLR profiler to analyze the memory usage. It was helpful in:
Showing me who allocated huge chunks of memory
What data type used maximum memory
But, both, CLR Profiler and Performance Counters (since they share same data), failed to explain:
The numbers that is collected after each run of the app - how to understand if there is any improvement?!?!
How do I compare the performance data after each run - is lower/higher number of a particular counter good or bad?
What I need:
I am looking for the tips on:
How to free (yes, right) managed data type objects (like arrays, big strings) - but not by making GC.Collect calls, if possible. I have to handle arrays of bytes of length like 500KB (unavoidable size :-( ) every now and then.
If fragmentation occurs, how to compact memory - as it seems that .NET GC is not really effectively doing that and causing OOM.
Also, what exactly is 85KB limit for LOH? Is this the size of the object of the overall size of the array? This is not very clear to me.
What memory counters can tell if code changes are actually reducing the chances of OOM?
Tips I already know
Set managed objects to null - mark them garbage - so that garbage collector can collect them. This is strange - after setting a string[] object to null, the # bytes in all Heaps shot up!
Avoid creating objects/arrays > 85KB - this is not in my control. So, there could be lots of LOH.
3.
Memory Leaks Indicators:
# bytes in all Heaps increasing
Gen 2 Heap Size increasing
# GC handles increasing
# of Pinned Objects increasing
# total committed Bytes increasing
# total reserved Bytes increasing
Large Object Heap increasing
My situation:
I have got 4 GB, 32-bit machine with Wink 2K3 server SP2 on it.
I understand that an application can use <= 2 GB of physical RAM
Increasing the Virtual Memory (pagefile) size has no effect in this scenario.
As its OOM issue, I am only focusing on memory related counters only.
Please advice! I really need some help as I'm stuck because of lack of good documentation!
Nayan, here are the answers to your questions, and a couple of additional advices.
You cannot free them, you can only make them easier to be collected by GC. Seems you already know the way:the key is reducing the number of references to the object.
Fragmentation is one more thing which you cannot control. But there are several factors which can influence this:
LOH external fragmentation is less dangerous than Gen2 external fragmentation, 'cause LOH is not compacted. The free slots of LOH can be reused instead.
If the 500Kb byte arrays are referring to are used as some IO buffers (e.g. passed to some socket-based API or unmanaged code), there are high chances that they will get pinned. A pinned object cannot be compacted by GC, and they are one of the most frequent reasons of heap fragmentation.
85K is a limit for an object size. But remember, System.Array instance is an object too, so all your 500K byte[] are in LOH.
All counters that are in your post can give a hint about changes in memory consumption, but in your case I would select BIAH (Bytes in all heaps) and LOH size as primary indicators. BIAH show the total size of all managed heaps (Gen1 + Gen2 + LOH, to be precise, no Gen0 - but who cares about Gen0, right? :) ), and LOH is the heap where all large byte[] are placed.
Advices:
Something that already has been proposed: pre-allocate and pool your buffers.
A different approach which can be effective if you can use any collection instead of contigous array of bytes (this is not the case if the buffers are used in IO): implement a custom collection which internally will be composed of many smaller-sized arrays. This is something similar to std::deque from C++ STL library. Since each individual array will be smaller than 85K, the whole collection won't get in LOH. The advantage you can get with this approach is the following: LOH is only collected when a full GC happens. If the byte[] in your application are not long-lived, and (if they were smaller in size) would get in Gen0 or Gen1 before being collected, this would make memory management for GC much easier, since Gen2 collection is much more heavyweight.
An advice on the testing & monitoring approach: in my experience, the GC behavior, memory footprint and other memory-related stuff need to be monitored for quite a long time to get some valid and stable data. So each time you change something in the code, have a long enough test with monitoring the memory performance counters to see the impact of the change.
I would also recommend to take a look at % Time in GC counter, as it can be a good indicator of the effectiveness of memory management. The larger this value is, the more time your application spends on GC routines instead of processing the requests from users or doing other 'useful' operations. I cannot give advices for what absolute values of this counter indicate an issue, but I can share my experience for your reference: for the application I am working on, we usually treat % Time in GC higher than 20% as an issue.
Also, it would be useful if you shared some values of memory-related perf counters of your application: Private bytes and Working set of the process, BIAH, Total committed bytes, LOH size, Gen0, Gen1, Gen2 size, # of Gen0, Gen1, Gen2 collections, % Time in GC. This would help better understand your issue.
You could try pooling and managing the large objects yourself. For example, if you often need <500k arrays and the number of arrays alive at once is well understood, you could avoid deallocating them ever--that way if you only need, say, 10 of them at a time, you could suffer a fixed 5mb memory overhead instead of troublesome long-term fragmentation.
As for your three questions:
Is just not possible. Only the garbage collector decides when to finalize managed objects and release their memory. That's part of what makes them managed objects.
This is possible if you manage your own heap in unsafe code and bypass the large object heap entirely. You will end up doing a lot of work and suffering a lot of inconvenience if you go down this road. I doubt that it's worth it for you.
It's the size of the object, not the number of elements in the array.
Remember, fragmentation only happens when objects are freed, not when they're allocated. If fragmentation is indeed your problem, reusing the large objects will help. Focus on creating less garbage (especially large garbage) over the lifetime of the app instead of trying to deal with the nuts and bolts of the gc implementation directly.
Another indicator is watching Private Bytes vs. Bytes in all Heaps. If Private Bytes increases faster than Bytes in all Heaps, you have an unmanaged memory leak. If 'Bytes in all Heaps` increases faster than 'Private Bytes' it is a managed leak.
To correct something that #Alexey Nedilko said:
"LOH external fragmentation is less dangerous than Gen2 external
fragmentation, 'cause LOH is not compacted. The free slots of LOH can
be reused instead."
is absolutely incorrect. Gen2 is compacted which means there is never free space after a collection. The LOH is NOT compacted (as he correctly mentions) and yes, free slots are reused. BUT if the free space is not contiguous to fit the requested allocation, then the segment size is increased - and can continue to grow and grow. So, you can end up with gaps in the LOH that are never filled. This is a common cause of OOMs and I've seen this in many memory dumps I've analyzed.
Though there are now methods in the GC API (as of .NET 4.51) that can be called to programatically compact the LOH, I strongly recommend to avoid this - if app performance is a concern. It is extremely expensive to perform this operation at runtime and and hurt your app performance significantly. The reason that the default implementation of the GC was to be performant which is why they omitted this step in the first place. IMO, if you find that you have to call this because of LOH fragmentation, you are doing something wrong in your app - and it can be improved with pooling techniques, splitting arrays, and other memory allocation tricks instead. If this app is an offline app or some batch process where performance isn't a big deal, maybe it's not so bad but I'd use it sparingly at best.
A good visual example of how this can happen is here - The Dangers of the Large Object Heap and here Large Object Heap Uncovered - by Maoni (GC Team Lead on the CLR)