Background:
I have a service whose purpose in life is to provide objects to requestors - it basically gets complicated data from a database and transforms it once (a bit like a view over data) to produce a simplified record. This then services requests from other services by providing up to 100k records (depending on the nature of the request) on demand.
The idea is that the complicated transformation is done once and is cached by the service - it works out quicker than letting the database work it out each time a view is accessed and for my purposes works just fine. (I believe this is called SSOS by some)
The way data is being cached is in a list of objects which are property bags for standard .Net types. These objects have no references to anything else.
Periodically a record will change, and the cache must be updated which means that the original record must be located, thrown away and replaced.
Now the record in the cache will have been in there for a long time and will have been marked for a Gen 2 collection; pretty much all the collections will happen in the Gen2 phase as these objects are hanging around for ages (on purpose).
So my understanding of Gen2 collections is that they are slow, and if the collections are mainly working on Gen2 then the optimizer is going to do this more often.
I would like to be able to de-reference an object in the list in a way that doesn't end up triggering a full Gen2 collection... I was thinking that maybe there is a way of marking it as Gen0 and then de-referencing it before replacing it - but I don't think that is possible.
I am constrained to using .Net 4 for this and the application is a service which serves data to up to 100 clients who request full lists or changes to the list over a period of time.
Question: Can anyone suggest a way to de-reference long lived objects in a GC friendly way or perhaps another way to approach this problem?
There is no simple answer to this. If you have lots of long-lived objects, then full collections really can hurt, as I discussed here. Since a picture tells a thousand words:
Those vertical spikes are where garbage collection happens and slaughters the response times.
The way we reduced the impact of this was: don't have a gazillion long-lived objects. What we did was to change the classes to structs, which meant that the only object was the array that contained them. We were fortunate here is that the data was simple and didn't involve strings, which would of course themselves be objects. We also did some crazy fixed-size buffer work to reduce things that were previously collections, and changed what were references to indices (into the array). If you do have to use string data, perhaps try to ensure you don't have 20,000 different string instancs with the same value - some kind of manual interner (a Dictionary<string,string> would suffice) can be really useful there.
Note that this needn't impact your public API, since you can always create the old class data from the struct storage - the difference is that this class will only exist briefly as a DTO - so will be collected cheaply in the next gen-0 sweep.
YMMV, but this worked enough well for us.
The problem is: you need to be really careful when working with structs; I strongly advise making them immutable.
Related
I've read the other questions that come up when I ask about memory snapshots, but I might be too thick to really grasp it. I have a windows service that I can produce a memory leak in by doing a pretty straightforward data operation repeatedly. I've taken memory snapshots along the way, and I see that the number of roots is going up (from 2,100 after a successful start to 7,100 after 100 or so data operations). The snapshots were taken at the blue arrow marks:
Before the multiple data operations, the memory snapshot looks like this:
Afterwards, it looks like this:
We're using WCF for data transport and it would appear that Serialization is playing a part in this memory growth, but I don't know where to go from here. If I look at instances of RuntimeType+RuntimeTypeCache, the vast majority of instances look like this:
If anyone can help me figure out the next step to take, I would appreciate it immensely. We have a static instance that has a concurrent dictionary of ServiceHosts that I'm suspicious of, but I don't know how to confirm it.
EDIT:
This also seems significant and is in reference to ServiceHosts. Could we be enabling some unwise proxy generation and instance retention via this static relationship?
Sort your items by Size, and inside that list, watch out for your own class types. Which one is piling up. Have at least a total of a few Megabytes of objects, to be sure to see a real 'Pile' not just some parts of the infrastructure.
The 12.000 existing runtime types, might indicate dynamically created types, maybe the serialization DLL is created for each new call.
You can also try to do GC.Collect() after your critical function call, to enforce Garbage collection.
In my application I use a dictionary (supporting adding, removing, updating and lookup) where both keys and values are or can be made serializable (values can possibly be quite large object graphs). I came to a point when the dictionary became so large that holding it completely in memory started to occasionally trigger OutOfMemoryException (sometimes in the dictionary methods, and sometimes in other parts of code).
After an attempt to completely replace the dictionary with a database, performance dropped down to an unacceptable level.
Analysis of the dictionary usage patterns showed that usually a smaller part of values are "hot" (are accessed quite often), and the rest (a larger part) are "cold" (accessed rarely or never). It is difficult to say when a new value is added if it will be hot or cold, moreover, some values may migrate back and forth between hot and cold parts over time.
I think that I need an implementation of a dictionary that is able to flush its cold values to a disk on a low memory event, and then reload some of them on demand and keep them in memory until the next low memory event occurs when their hot/cold status will be re-assessed. Ideally, the implementation should neatly adjust the sizes of its hot and cold parts and the flush interval depending on the memory usage profile in the application to maximize overall performance. Because several instances of a dictionary exist in the application (with different key/value types), I think, they might need to coordinate their workflows.
Could you please suggest how to implement such a dictionary?
Compile for 64 bit, deploy on 64 bit, add memory. Keep it in memory.
Before you grown your own you may alternatively look at WeakReference http://msdn.microsoft.com/en-us/library/ms404247.aspx. It would of course require you to rebuild those objects that were reclaimed but one should hope that those which are reclaimed are not used much. It comes with the caveat that its own guidleines state to avoid using weak references as an automatic solution to memory management problems. Instead, develop an effective caching policy for handling your application's objects.
Of course you can ignore that guideline and effectively work your code to account for it.
You can implement the caching policy and upon expiry save to database, on fetch get and cache. Use a sliding expiry of course since you are concerned with keeping those most used.
Do remember however that most used vs heaviest is a trade off. Losing an object 10 times a day that takes 5 minutes to restore would annoy users much more than losing an object 10000 times which tool just 5ms to restore.
And someone above mentioned the web cache. It does automatic memory management with callbacks as noted, depends if you want to lug that one around in your apps.
And...last but not least, look at a distributed cache. With sharding you can split that big dictionary across a few machines.
Just an idea - never did that and never used System.Runtime.Caching:
Implement a wrapper around MemoryCache which will:
Add items with an eviction callback specified. The callback will place evicted items to the database.
Fetch item from database and put back into MemoryCache if the item is absent in MemoryCache during retrieval.
If you expect a lot of request for items missing both in database and memory, you'll probably need to implement either bloom filter or cache keys for present/missing items also.
I have a similar problem in the past.
The concept you are looking for is a read through cache with a LRU (Least Recently Used or Most Recently Used) queue.
Is it there any LRU implementation of IDictionary?
As you add things to your dictionary keep track of which ones where used least recently, remove them from memory and persist those to disk.
In C# I had do create my own dynamic memory management. For that reason I have created a static memory manager and a MappableObject. All object that should be dynamic mappable and unmappable from and to the harddisk implement this interface.
This memory management is only done for these large objects that have the ability to unmap/map the data from the harddisk. All other things use of course the regular GC.
Everytime a MappableObject is allocated it asks for memory. If no memory is available that the MemoryManager unmaps some data dynamically to the harddisk to get more memory to make it possible to allocate a new MappableObject.
A problem in my case is that I can have more than 100.000 MappableObject instances (scattered over a few files ~ 10-20 files) and everytime I have to run through a list of all objects if I need to unmap some data. Is there a way to get all allocated objects that are created in my current instance?
In fact I don't know what's easier to keep my own list or to run through the objects (if possible)? How would you solve such things?
Update
The reason is that I have a large amount of data. About 100GB of data that I need to keep during my run. Therefore I need the references on the data, and so the GC is not able to clean the memory. In fact C# manages the memory pretty well, but in such memory exhausting applications the GC gets really bad. Of course I tried to use the MemoryFailPoint, but this slows down my allocations tremendously and does not give correct results for whatever reason. I have also tried MemoryMappedFiles, but since I have to access the data randomly it doesn't help. Also MemoryMappedFiles only allow to have ~5000 file handles (on my system) and this is not enough.
Is there a ROT (Running Object Table) in .Net? The short answer is no.
You would have to maintain this information yourself.
Given your question update, could you not store your data in a database and use some sort of in-memory cache (perhaps with weak references or MFU, etc) to try and keep hot data close to you?
This is an obvious case for a classic cache. Your data is stored in a database or indexed flat file while you maintain a much smaller number of entries in RAM.
To implement a cache for your program I would create a class that implements IDictionary. Reserve a certain amount of slots in your cache, say a number of elements that would cause about 100 MB of RAM to be allocated; make this cache size an adjustable parameter.
When you override this[], if the object requested is in the cache, return it. If the object requested is not in the cache, remove the least recently used cached value, add the requested value as the most recently used value, and return it. Functions like Remove() and Add() not only adjust the memory cache, but also manipulate the underlying database or flat file on disk.
While it's true that your program might hold some references to objects you removed from the cache, if so, your program is still using them. Garbage collection will clean them up as needed.
Caches like this are easier to implement in C# because of its strong OOP features and safety.
I have profiled my App w/ VS2010 profiler, with object lifetime collection enabled.
I was heavilly surprised to see that most instances of a particular struct named "Record" are collected by the GC as Gen 2 instances. I am very upset, as instances of "Record" struct should live less than 500ms each (theoretically).
These structs are simple time series data of 6xInt32 or so, that are read on the flow, Queued/Dequeued in a Queue having a size of 1000, passed to a processor that fires some logic depending on those few millions "Records" sequentially. I do not need to keep more than 50 records at a time.
So my question is : Why could these object live long enough to mainly end up as Second Generation references, and what could I do to ensure they REALLY get dumped off after each computation.
EDIT :
I am asking this because I have noticed a drastic performance dropdown for bigger sample sizes (i.e Records Numbers) : if N take T minutes, 2N takes 2,5T minutes or so, and so on. So there is obviously a leak somewhere.
EDIT 2 :
My Bad : Creating a struct instance cannot cause a garbage collection
I've changed it to classes and did not notice any significant improvement so far.
I'll run the profiler again with classes this time not structs) and see what it gives
EDIT 3 :
Many answers suspect Boxing/Unboxing to take place somewhere.
I DO use typed generic collections and typed Queues. And "Records" are never attached to any class as members. They are individually handled by events.
The ex-Struct (Now Class) implemented an interface and was casted by it when called (this is rather common usage) and I dropped off that interface. No improvement.
EDIT 4 :
I have run again the profiler, replacing struct by class. I have the same results : most instances of CLASS "Record" still end up being collected as Gen2 instances
EDIT 5 :
Producers of the Record classes are many parallel BackGroundWorkers (Byte Readers), and there is one Consumer Thread that dispatches the Records to other methods after performing a few checks. Besides I use Events and Delegates to communicate between the different parts. I do not unregister those events because they are useful all along the process (I may be wrong on that point)
If you stored them just as local variables (which may or may not be possible depending on your scenario) they will never end up on the heap at all.
If that's possible, I recommend trying that. You might get a more in depth response, if you post a code sample.
As a sanity check, are you forgetting to unregister static events, or using some other class that may be doing this (some classes fix this via Dispose)?
Also have you looked into the possibility of using the Flyweight pattern?
Edit- Since you now say you are doing something with events, this is highly likely to be a cause of your issues. Are you forgetting to unregister the events?
If you are "heavy" on memory use, and you are using C# 4.0, you could try the "server" GC. Merge your app.config (or web.config) with:
<configuration>
<runtime>
<gcServer enabled="true"/>
</runtime>
</configuration>
(with merge I mean that if you already have some of these sections, use them, otherwhise create them. "configuration" is the first level element of the app.config). This is better for some apps (apps that don't need heavy interaction with the user)
They end up as Second Generation because the GC just decided not to garbage collect them. I don't think it is really a problem to worry about. If you really wanted to you could just reuse the 1000 objects instead of continuously creating new ones. This would make it so that these objects wouldn't need to get garbage collected. You could force Garbage collection to ensure the objects "REALLY" get dumped but that would degrade performance as GC is an expensive operation or so I am told.
i have a cache that uses WeakReferences to the cached objects to make them automatically removed from the cache in case of memory pressure. My problem is that the cached objects are collected very soon after they have been stored in the cache. The cache runs in a 64-Bit application and in spite of the case that more than 4gig of memory are still available, all the cached objects are collected (they usually are stored in the G2-heap at that moment). There are no garbage collection induced manually as the process explorer shows.
What methods can i apply to make the objects live a litte longer?
Using WeakReferences as the primary means of referencing cached objects is not really a great idea, because as Josh said, your at the mercy of any future behavioral changes to WeakReference and the GC.
However, if your cache needs any kind of resurrection capability, use of WeakReferences for items that are pending purge is useful. When an item meets eviction criteria, rather than immediately evicting it, you change its reference to a weak reference. If anything requests it before it is GC'ed, you restore its strong reference, and the object can live again. I have found this useful for some caches that have hard to predict hit rate patterns with frequent enough "resurrections" to be beneficial.
If you have predictable hit rate patterns, then I would forgoe the WeakReference option and perform explicit evictions.
There is one situation where a WeakReference-based cache may be good: when the usefulness of an item in the class is predicated upon the existence of a reference to it. In such a situation, a weak interning cache may be useful. For example, if one had an application which would deserialize many large immutable objects, many of which were expected to be duplicates, and would have to perform many comparisons between them. If X and Y are references to some immutable class type, testing X.Equals(Y) will be very fast if both variables point to the same instance, but may be very slow if they point to distinct instances that happen to be equal. If a deserialized object happens to match another object to which a reference already exists, fetching a from the dictionary a reference to that latter object (requiring one slow comparison) may expedite future comparisons. On the other hand, if it matched an item in the dictionary but the dictionary was the only reference to that item, there would be little advantage to using the dictionary object instead of simply keeping the object that was read in; probably not enough advantage to justify the cost of the comparison. For an interning cache, having WeakReferences get invalidated as soon as possible once no other references exist to an object would be a good thing.
In .net, a WeakReference is not considered a reference from the GC standpoint at all, so any object that only has weak references will be collected in the next GC run (for the appropriate generation).
That makes weak reference completely inappropriate for caching - as your experience shows.
You need a "real" cache component, and the most important thing about caching is to get one where the eviction policy (that is, the rules about when to drop an object from the cache) are a good match for you application's usage pattern.
No, WeakReference is not good for that because the behavior of the garbage collector can and will change over time and your cache should not be relying on today's behavior. Also many factors outside of your control could affect memory pressure.
There are many implementations of a cache for .NET. You could find probably a dozen on CodePlex. I guess what you need to add to it is something that looks at the application's current working set to use that as a trigger for purging.
One more note about why your objects are being collected so frequently. The GC is very aggressive at cleaning up Gen0 objects. If your objects are very short-lived (up until the only reference to it is a weak reference) then the GC is doing what it's designed to do by cleaning up as quickly as it can.
I believe the problem you are having is that the Garbage Collector removes weakly referenced objects in response not only in response to memory pressure - instead it will do collection quite aggressively sometimes just because the runtime system thinks some objects may likely have become unreachable.
You may be better off using e.g. System.Runtime.Caching.MemoryCache which can be configured with a memory limit, or custom eviction policies for the items.
The answer actually depends on usage characteristics of the cache you are trying to build. I have successfully used WeakReference based caching strategy for improving performance in many of my projects where the cached objects are expected to be used in short bursts of multiple reads. As others pointed out, the weak references are pretty much garbage from GC's point of view and will be collected whenever the next GC cycle is run. It's nothing to do with the memory utilization.
If, however, you need a cache that survives such brutality from GC, you need to use or mimic the functionality provided by System.Runtime.Caching namespace. Keep in mind that you'd need an additional thread that cleans up the cache when the memory usage is crossing your thresholds.
A bit late, but here's a relevant use case:
I need to cache two types of objects: large (deserialised) data files that take 10 minutes to load and cost 15G of ram each, and smaller (dynamically compiled) objects that contain internal references to those data files (the smaller objects are also cached because they take ~10s to generate). These caches are hidden within the factories that supply the objects (the former component having no knowledge of the latter), and have different eviction policies.
When my `data file' cache evicts an object, it replaces it by a weak reference, so if that object is still available when next requested, we can resurrect it (and renew its cache timeout). In this way we avoid losing (or accidentally duplicating) any object before it is truly defunct (i.e. not used anywhere else). Notice that neither cache is required to be aware of the other, and that no other client objects need to be aware that there are any caches at all (eg: we avoid needing 'keepalives', callbacks, registration, retrieve-and-return scopes, etc - things get a lot simpler).
So although using WeakReference by itself (instead of a cache) is a terrible idea (because modern GCs are typically tuned to the size of the L2 CPU cache, and regular code will burn through this many times per minute), it's very useful as a way to hide your caches from the rest of your code.