We Keep Coding

element size influencing C# collection performance?

Given the task to improve the performance of a piece of code, I have came across the following phenomenon. I have a large collection of reference types in a generic Queue and I'm removing and processing the element one by one, then add them to another generic collection.
It seems the larger the elements are the more time it takes to add the element to the collection.
Trying to narrow down the problem to the relevant part of the code, I've written a test (omitting the processing of elements, just doing the insert):
class Small
{
public Small()
{
this.s001 = "001";
this.s002 = "002";
}
string s001;
string s002;
}
class Large
{
public Large()
{
this.s001 = "001";
this.s002 = "002";
...
this.s050 = "050";
}
string s001;
string s002;
...
string s050;
}
static void Main(string[] args)
{
const int N = 1000000;
var storage = new List<object>(N);
for (int i = 0; i < N; ++i)
{
//storage.Add(new Small());
storage.Add(new Large());
}
List<object> outCollection = new List<object>();
Stopwatch sw = new Stopwatch();
sw.Start();
for (int i = N-1; i > 0; --i)
{
outCollection.Add(storage[i];);
}
sw.Stop();
Console.WriteLine(sw.ElapsedMilliseconds);
}
On the test machine, using the Small class, it takes about 25-30 ms to run, while it takes 40-45 ms with Large.
I know that the outCollection has to grow from time to time to be able to store all the items, so there is some dynamic memory allocation. But given an initial collection size even makes the difference more obvious: 11-12 ms with Small and 35-38 ms with Large objects.
I am somewhat surprised, as these are reference types, so I was expecting the collections to work only with references to the Small/Large instances. I have read Eric Lippert's relevant article that and know that references should not be treated as pointers. At the same time, AFAIK currently they are implemented as pointers and their size and the collection's performance should be independent of element size.
I've decided to put up a question here hoping that someone could explain or help me to understand what's happening here. Aside the performance improvement, I'm really curious what is happening behind the scenes.
Update:
Profiling data using the diagnostic tools didn't help me much, although I have to admit I'm not an expert using the profiler. I'll collect more data later today to find where the bottleneck is.
The pressure on the GC is quite high of course, especially with the Large instances. But once the instances are created and stored in the storage collection, and the program enters the loop, there was no collection triggered any more, and memory usage hasn't increased significantly (outCollction already pre-allocated).
Most of the CPU time is of course spent with memory allocation (JIT_New), around 62% and the only other significant entry is Function Name Inclusive Samples Exclusive Samples Inclusive Samples % Exclusive Samples % Module Name
System.Collections.Generic.List`1[System.__Canon].Add with about 7%.
With 1 million items the preallocated outCollection size is 8 million bytes (the same as the size of storage); one can suspect 64 bit addresses being stored in the collections.
Probably I'm not using the tools properly or don't have the experience to interpret the results correctly, but the profiler didn't help me to get closer to the cause.
If the loop is not triggering collections and it only copies pointers between 2 pre-allocated collections, how could the item size cause any difference? Cache hit/miss ratio is supposed to be the more or less the same in both cases, as the loop is iteration over a list of "addresses" in both cases.
Thanks for all the help so far, I will collect more data, and put an update here if anything found.

I suspect that at least one action in the above (maybe some type checks) will require a de-reference. Then the fact that many Smalls are probably sat close together on the heap and thus sharing cache lines could account for some amount of difference (certainly many more of them could share a single cache line than Larges).
Added to which you are also accessing them in the reverse order in which they were allocated which maximises such a benefit.

Ensuring thread safety with ASP.Net cache - a tale of two strategies

I have code that is not currently thread safe:
public byte[] GetImageByteArray(string filepath, string contentType, RImgOptions options)
{
//Our unique cache keys will be composed of both the image's filepath and the requested width
var cacheKey = filepath + options.Width.ToString();
var image = HttpContext.Current.Cache[cacheKey];
//If there is nothing in the cache, we need to generate the image, insert it into the cache, and return it
if (image == null)
{
RImgGenerator generator = new RImgGenerator();
byte[] bytes = generator.GenerateImage(filepath, contentType, options);
CacheItem(cacheKey, bytes);
return bytes;
}
//Image already exists in cache, serve it up!
else
{
return (byte[])image;
}
}
My CacheItem() method checks to see if its max cache size has been reached, and if it has, it will start removing cached items:
//If the cache exceeds its max allotment, we will remove items until it falls below the max
while ((int)cache[CACHE_SIZE] > RImgConfig.Settings.Profile.CacheSize * 1000 * 1000)
{
var entries = (Dictionary<string, DateTime>)cache[CACHE_ENTRIES];
var earliestCacheItem = entries.SingleOrDefault(kvp => kvp.Value == entries.Min(d => d.Value));
int length = ((byte[])cache[earliestCacheItem.Key]).Length;
cache.Remove(earliestCacheItem.Key);
cache[CACHE_SIZE] = (int)cache[CACHE_SIZE] - length;
}
Since one thread could remove an item from the cache as another thread is referencing it, I can think of two options:
Option 1: A lock
lock (myLockObject)
{
if(image == null){ **SNIP** }
}
Option 2: Assign a shallow copy to a local variable
var image = HttpContext.Current.Cache[cacheKey] != null ? HttpContext.Current.Cache[cacheKey].MemberwiseClone() : null;
Both of these options have overhead. The first forces threads to enter that code block one at a time. The second necessitates creating a new object in memory which could be of non-trivial size.
Are there any other strategies I could employ here?

To provide pure consistency of your cache solution you should lock your resource while slowing down the application.
In general, you should try to provide caching strategy, based on your application logic.
Check sliding window caching: while item which is older when some
time span - will reduce locking of different threads - good when you
have large spread of different cached items which not for sure will
be used again.
Consider using least frequently used strategy: the item that is least used
should be removed while reached max cache size - best serves while
you have multiple client hitting frequently same part of cached
content.
Just check which one more suites better your type of BL and use it. It will not remove the locking issue at all, but right choice will significantly remove racing conditions.
In order to reduce shared resource between different threads, use read and write locks on each item and not on entire collection. This will boost your performance as well.
Another point of consideration that should be kept in mind - what if content of image with the same path is changed physically on the disk (different image was saved with the same name) while having this image already cached there is a chance of mistakenly provide not relevant data.
Hope it helped.

how can I determine memory usage of a giant c# data set

I have a giant data set in a c# windows service that uses about 12GB of ram.
Dictionary<DateTime,List<List<Item>>>
There is a constant stream of new data being added, about 1GB per hour. Old data is occasionally removed. This is a high speed buffer for web pages.
I have a parameter in the config file called "MaxSizeMB". I would like to allow the user to enter, say "11000", and my app will delete some old data every time the app exceeds 11GB of ram usage.
This has proved to be frustratingly difficult.
You would think that you can just call GC.GetTotalMemory(false). This would give you the memory usage of .net managed objects (lets pretent it says 10.8GB). Then you just add a constant 200MB as a safety net for all the other stuff allocated in the app.
This doesn't work. In fact, the more data that is loaded, the bigger the difference between GC.GetTotalMemory and task manager. I even tried to work out a constant multiplier value instead of a constant add value, but I cannot get consistent results. The best i have done so far is count the total number of items in the data structure, multiply by 96, and pretend that number is the ram usage. This is also confusing because the Item object is a 32byte struct. This pretend ram usage is also too unstable. Sometimes the app will delete old data at 11GB, but sometimes it will delete data at 8GB ram usage, because my pretend number calculates a false 11GB.
So i can either use this conservative fake ram calculation, and often not use all the ram I am allowed to use (like 2GB lost), or I can use GC.GetTotalMemory and the customer will freak out that the app goes over the ram setting occasionally.
Is there any way I can use as much ram as possible without going over a limit, as it appears in task manager? I don't care if the math is a multiplier, constant add value, power, whatever. I want to stuff data into a data structure and delete data when I hit the max setting.
Note: i already do some memory shrinking techniques such as using a struct as the Item, list.Capacity = list.Count, and GC.Collect(GC.MaxGeneration). Those seem like a separate issue though.

Use System.Diagnostics.PerformanceCounter and monitor your current process memory usage and available memory, based on this, your application should decide to delete something or not..

Several problems
Garbage collection
Getting a good measure of memory
What is the maximum
You assume there is a hard maximum.
But an object needs contiguous memory so that is a soft maximum.
As for an accurate size measure you could record the size of each list and keep a running total.
Then when you purge read the size and reduce from that running total.
Why fight .NET memory limitations and physical memory limitations
I would so go with a database on an SSD
If it is read only and you have known classes then you could use like a RavenDB
Reconsider your design
OK so I am not getting very far with managing .NET memory limitation that you are never going to tame.
Still reconsider your design.
If your PK is a DateTime and assume you only need 24 hours put one per dictionary per hour as that is just one object.
At the end of 23 hours new the prior - let the GC collect the whole thing.

The answer is super simple.
var n0 = System.Diagnostics.Process.GetCurrentProcess().PrivateMemorySize64;
var n1 = System.Diagnostics.Process.GetCurrentProcess().WorkingSet64;
var n2 = System.Diagnostics.Process.GetCurrentProcess().VirtualMemorySize64;
float f0 = ((float)n0)/(1000*1000);
float f1 = ((float)n1)/(1000*1000);
float f2 = ((float)n2)/(1000*1000);
Console.WriteLine("private = " + f0 + " MB");
Console.WriteLine("working = " + f1 + " MB");
Console.WriteLine("virtual = " + f2 + " MB");
results:
private = 931.9096 MB
working = 722.0756 MB
virtual = 1767.146 MB
All this moaning and fussing about task manager and .net object size and the answer is built into .NET in one line of code.
I gave the answer to Sarvesh because he got me started down the right path with PerformanceCounter, but GetCurrentProcess() turned out to be a nice shortcut to simply inspect your own process.

Datastructure choices for highspeed and memory efficient detection of duplicate of strings

I have a interesting problem that could be solved in a number of ways:
I have a function that takes in a string.
If this function has never seen this string before, it needs to perform some processing.
If the function has seen the string before, it needs to skip processing.
After a specified amount of time, the function should accept duplicate strings.
This function may be called thousands of time per second, and the string data may be very large.
This is a highly abstracted explanation of the real application, just trying to get down to the core concept for the purpose of the question.
The function will need to store state in order to detect duplicates. It also will need to store an associated timestamp in order to expire duplicates.
It does NOT need to store the strings, a unique hash of the string would be fine, providing there is no false positives due to collisions (Use a perfect hash?), and the hash function was performant enough.
The naive implementation would be simply (in C#):
Dictionary<String,DateTime>
though in the interest of lowering memory footprint and potentially increasing performance I'm evaluating a custom data structures to handle this instead of a basic hashtable.
So, given these constraints, what would you use?
EDIT, some additional information that might change proposed implementations:
99% of the strings will not be duplicates.
Almost all of the duplicates will arrive back to back, or nearly sequentially.
In the real world, the function will be called from multiple worker threads, so state management will need to be synchronized.

I don't belive it is possible to construct "perfect hash" without knowing complete set of values first (especially in case of C# int with limited number of values). So any kind of hashing requires ability to compare original values too.
I think dictionary is the best you can get with out of box data structures. Since you can store objects with custom comparisons defined you can easily avoid keeping strings in memeory and simply save location where whole string can be obtained. I.e. object with following values:
stringLocation.fileName="file13.txt";
stringLocation.fromOffset=100;
stringLocation.toOffset=345;
expiration= "2012-09-09T1100";
hashCode = 123456;
Where cutomom comparer will return saved hashCode or retrive string from file if needed and perform comparison.

a unique hash of the string would be fine, providing there is no false
positives due to collisions
That's not possible, if you want the hash code to be shorter than the strings.
Using hash codes implies that there are false positives, only that they are rare enough not to be a performance problem.
I would even consider to create the hash code from only part of the string, to make it faster. Even if that means that you get more false positives, it could increase the overall performance.

Provided the memory footprint is tolerable, I would suggest a Hashset<string> for the strings, and a queue to store a Tuple<DateTime, String>. Something like:
Hashset<string> Strings = new HashSet<string>();
Queue<Tuple<DateTime, String>> Expirations = new Queue<Tuple<DateTime, String>>();
Now, when a string comes in:
if (Strings.Add(s))
{
// string is new. process it.
// and add it to the expiration queue
Expirations.Enqueue(new Tuple<DateTime, String>(DateTime.Now + ExpireTime, s));
}
And, somewhere you'll have to check for the expirations. Perhaps every time you get a new string, you do this:
while (Expirations.Count > 0 && Expirations.Peek().Item1 < DateTime.Now)
{
var e = Expirations.Dequeue();
Strings.Remove(e.Item2);
}
It'd be hard to beat the performance of Hashset here. Granted, you're storing the strings, but that's going to be the only way to guarantee no false positives.
You might also consider using a time stamp other than DateTime.Now. What I typically do is start a Stopwatch when the program starts, and then use the ElapsedMilliseconds value. That avoids potential problems that occur during Daylight Saving Time changes, when the system automatically updates the clock (using NTP), or when the user changes the date/time.
Whether the above solution works for you is going to depend on whether you can stand the memory hit of storing the strings.
Added after "Additional information" was posted:
If this will be accessed by multiple threads, I'd suggest using ConcurrentDictionary rather than Hashset, and BlockingCollection rather than Queue. Or, you could use lock to synchronize access to the non-concurrent data structures.
If it's true that 99% of the strings will not be duplicate, then you'll almost certainly need an expiration queue that can remove things from the dictionary.

If memory footprint of storing whole strings is not acceptable, you have only two choices:
1) Store only hashes of strings, which implies possibility of hash collisions (when hash is shorter than strings). Good hash function (MD5, SHA1, etc.) makes this collision nearly impossible to happen, so it only depends whether it is fast enough for your purpose.
2) Use some kind of lossless compression. Strings have usually good compression ratio (about 10%) and some algorithms such as ZIP let you choose between fast (and less efficient) and slow (with high compression ratio) compression. Another way to compress strings is convert them to UTF8, which is fast and easy to do and has nearly 50% compression ratio for non-unicode strings.
Whatever way you choose, it's always tradeoff between memory footprint and hashing/compression speed. You will probably need to make some benchmarking to choose best solution.

Struct vs class memory overhead

I'm writing an app that will create thousands of small objects and store them recursively in array. By "recursively" I mean that each instance of K will have an array of K instances which will have and array of K instances and so on, and this array + one int field are the only properties + some methods. I found that memory usage grows very fast for even small amount of data - about 1MB), and when the data I'm processing is about 10MB I get the "OutOfMemoryException", not to mention when it's bigger (I have 4GB of RAM) :). So what do you suggest me to do? I figured, that if I'd create separate class V to process those objects, so that instances of K would have only array of K's + one integer field and make K as a struct, not a class, it should optimize things a bit - no garbage collection and stuff... But it's a bit of a challenge, so I'd rather ask you whether it's a good idea, before I start a total rewrite :).
EDIT:
Ok, some abstract code
public void Add(string word) {
int i;
string shorter;
if (word.Length > 0) {
i = //something, it's really irrelevant
if (t[i] == null) {
t[i] = new MyClass();
}
shorterWord = word.Substring(1);
//end of word
if(shorterWord.Length == 0) {
t[i].WordEnd = END;
}
//saving the word letter by letter
t[i].Add(shorterWord);
}
}
}

For me already when researching deeper into this I had the following assumptions (they may be inexact; i'm getting old for a programmer). A class has extra memory consumption because a reference is required to address it. Store the reference and an Int32 sized pointer is needed on a 32bit compile. Allocated always on the heap (can't remember if C++ has other possibilities, i would venture yes?)
The short answer, found in this article, Object has a 12bytes basic footprint + 4 possibly unused bytes depending on your class (has no doubt something to do with padding).
http://www.codeproject.com/Articles/231120/Reducing-memory-footprint-and-object-instance-size
Other issues you'll run into is Arrays also have an overhead. A possibility would be to manage your own offset into a larger array or arrays. Which in turn is getting closer to something a more efficient language would be better suited for.
I'm not sure if there are libraries that may provide Storage for small objects in an efficient manner. Probably are.
My take on it, use Structs, manage your own offset in a large array, and use proper packing instructions if it serves you (although i suspect this comes at a cost at runtime of a few extra instructions each time you address unevenly packed data)
[StructLayout(LayoutKind.Sequential, Pack = 1)]

Your stack is blowing up.
Do it iteratively instead of recursively.
You're not blowing the system stack up, your blowing the code stack up, 10K function calls will blow it out of the water.
You need proper tail recursion, which is just an iterative hack.

Make sure you have enough memory in your system. Over 100mb+ etc. It really depends on your system. Linked list, recursive objects is what you are looking at. If you keep recursing, it is going to hit the memory limit and nomemoryexception will be thrown. Make sure you keep track of the memory usage on any program. Nothing is unlimited, especially memory. If memory is limited, save it to a disk.
Looks like there is infinite recursion in your code and out of memory is thrown. Check the code. There should be start and end in recursive code. Otherwise it will go over 10 terrabyte memory at some point.

You can use a better data structure
i.e. each letter can be a byte (a-0, b-1 ... ). each word fragment can be in indexed also especially substrings - you should get away with significantly less memory (though a performance penalty)

Just list your recursive algorithm and sanitize variable names. If you are doing BFS type of traversal and keep all objects in memory, you will run out of mem. For example, in this case, replace it with DFS.
Edit 1:
You can speed up the algo by estimating how many items you will generate then allocate that much memory at once. As the algo progresses, fill up the allocated memory. This reduces fragmentation and reallocation & copy-on-full-array operations.
Nonetheless, after you are done operating on these generated words you should delete them from your datastructure so they can be GC-ed so you don't run out of mem.