We Keep Coding

Balancing Memory Mapped Files for data larger than the RAM

We are having very large data files, for example assume a 3D volume with 2048x2048 matrix size with a slice depth of 20000.
Originally, I had my own central memory management where each data is backed with a "page-file" on the disk. During processing I track the process memory and if the memory is low I check which slices haven't been touched for a while and page them in my own hand-made paging-file. This results of course in some form a zig-zag when you look at the process memory, but this methods works even if the total size of my files is much larger than the available RAM. Of course putting more RAM inside improves the situation but the system is able to adapt to these situation. So if you pay more you get more speed ;-)
But, the method is of course hand-made, I need a separate thread that watches over the allocated files and data, I need special methods that handle the paging etc.
So I decided to have a look at memory mapped files...
When I now create a memory mapped file (Read&Write) and run through the data slice by slice and get access to the data with a ReadOnlySpan<> in a for loop the whole memory is consumed and you see the process memory lineary growing until the whole memory is consumed. Afterwards the system goes into swapping without coming out ever again. Its even so that the system freezes and only a restart helps to recover.
So obviously the MM-Files are not balancing themselves and when it comes to the point where we a reaching the RAM maximum, the system is exhausted and is obviously helpless.
The question is can I control when the system does a swapping and advise which regions can be swapped? Or do you have suggestions for better approaches?
I have created a example that does following
It creates a Volume as input
It creates one or many processor that simply add 10 to each slice in each processor.
Since everything is created only on access, a Parallel.For iterates over the slices and requests the result slice. Depending on the amount N of processors in the chain each slice of the result should have a value of (N+1)*10.
See here: Example Code
Update [26.09.2021]: My latest investigations have shown that there are two independent process running. One process (my process) is writing or reading data into the memory mapped file(s). Anothter system process is trying to manage the memory. While the memory is growing the system is obviously very lazy flushing data really to the backing memory mapped file or the paging file.
This leads to a bad condition. When the maximum RAM is reached, the memory manager starts to empty the working set and flush the data to disk. While the system flushes, I'm still producing data which fill the memory again. So in the end I'm coming to a point where I'm always using the maximum RAM and never come out.
I tried a few things:
VirtualUnlock enables me mark regions as pageable for the memory manager.
You can call a Flush on the ViewAccessor.
Since the workingset is still showing high memory usage, I call EmptyWorkingSet to empty the working set, which immediately returns.
What I need in the end is something to see if the memory manager is currently unmapping so that I can slow down the process.
I have to search more...

OutOfMemoryException when declaring an array of 100mln floats [duplicate]

This is my code:
int size = 100000000;
double sizeInMegabytes = (size * 8.0) / 1024.0 / 1024.0; //762 mb
double[] randomNumbers = new double[size];
Exception:
Exception of type 'System.OutOfMemoryException' was thrown.
I have 4GB memory on this machine 2.5GB is free when I start this running, there is clearly enough space on the PC to handle the 762mb of 100000000 random numbers. I need to store as many random numbers as possible given available memory. When I go to production there will be 12GB on the box and I want to make use of it.
Does the CLR constrain me to a default max memory to start with? and how do I request more?
Update
I thought breaking this into smaller chunks and incrementally adding to my memory requirements would help if the issue is due to memory fragmentation, but it doesn't I can't get past a total ArrayList size of 256mb regardless of what I do tweaking blockSize.
private static IRandomGenerator rnd = new MersenneTwister();
private static IDistribution dist = new DiscreteNormalDistribution(1048576);
private static List<double> ndRandomNumbers = new List<double>();
private static void AddNDRandomNumbers(int numberOfRandomNumbers) {
for (int i = 0; i < numberOfRandomNumbers; i++) {
ndRandomNumbers.Add(dist.ICDF(rnd.nextUniform()));
}
}
From my main method:
int blockSize = 1000000;
while (true) {
try
{
AddNDRandomNumbers(blockSize);
}
catch (System.OutOfMemoryException ex)
{
break;
}
}
double arrayTotalSizeInMegabytes = (ndRandomNumbers.Count * 8.0) / 1024.0 / 1024.0;

You may want to read this: "“Out Of Memory” Does Not Refer to Physical Memory" by Eric Lippert.
In short, and very simplified, "Out of memory" does not really mean that the amount of available memory is too small. The most common reason is that within the current address space, there is no contiguous portion of memory that is large enough to serve the wanted allocation. If you have 100 blocks, each 4 MB large, that is not going to help you when you need one 5 MB block.
Key Points:
the data storage that we call “process memory” is in my opinion best visualized as a massive file on disk.
RAM can be seen as merely a performance optimization
Total amount of virtual memory your program consumes is really not hugely relevant to its performance
"running out of RAM" seldom results in an “out of memory” error. Instead of an error, it results in bad performance because the full cost of the fact that storage is actually on disk suddenly becomes relevant.

Check that you are building a 64-bit process, and not a 32-bit one, which is the default compilation mode of Visual Studio. To do this, right click on your project, Properties -> Build -> platform target : x64. As any 32-bit process, Visual Studio applications compiled in 32-bit have a virtual memory limit of 2GB.
64-bit processes do not have this limitation, as they use 64-bit pointers, so their theoretical maximum address space (the size of their virtual memory) is 16 exabytes (2^64). In reality, Windows x64 limits the virtual memory of processes to 8TB. The solution to the memory limit problem is then to compile in 64-bit.
However, object’s size in .NET is still limited to 2GB, by default. You will be able to create several arrays whose combined size will be greater than 2GB, but you cannot by default create arrays bigger than 2GB. Hopefully, if you still want to create arrays bigger than 2GB, you can do it by adding the following code to you app.config file:
<configuration>
<runtime>
<gcAllowVeryLargeObjects enabled="true" />
</runtime>
</configuration>

You don't have a continuous block of memory in order to allocate 762MB, your memory is fragmented and the allocator cannot find a big enough hole to allocate the needed memory.
You can try to work with /3GB (as others had suggested)
Or switch to 64 bit OS.
Or modify the algorithm so it will not need a big chunk of memory. maybe allocate a few smaller (relatively) chunks of memory.

As you probably figured out, the issue is that you are trying to allocate one large contiguous block of memory, which does not work due to memory fragmentation. If I needed to do what you are doing I would do the following:
int sizeA = 10000,
sizeB = 10000;
double sizeInMegabytes = (sizeA * sizeB * 8.0) / 1024.0 / 1024.0; //762 mb
double[][] randomNumbers = new double[sizeA][];
for (int i = 0; i < randomNumbers.Length; i++)
{
randomNumbers[i] = new double[sizeB];
}
Then, to get a particular index you would use randomNumbers[i / sizeB][i % sizeB].
Another option if you always access the values in order might be to use the overloaded constructor to specify the seed. This way you would get a semi random number (like the DateTime.Now.Ticks) store it in a variable, then when ever you start going through the list you would create a new Random instance using the original seed:
private static int randSeed = (int)DateTime.Now.Ticks; //Must stay the same unless you want to get different random numbers.
private static Random GetNewRandomIterator()
{
return new Random(randSeed);
}
It is important to note that while the blog linked in Fredrik Mörk's answer indicates that the issue is usually due to a lack of address space it does not list a number of other issues, like the 2GB CLR object size limitation (mentioned in a comment from ShuggyCoUk on the same blog), glosses over memory fragmentation, and fails to mention the impact of page file size (and how it can be addressed with the use of the CreateFileMapping function).
The 2GB limitation means that randomNumbers must be less than 2GB. Since arrays are classes and have some overhead them selves this means an array of double will need to be smaller then 2^31. I am not sure how much smaller then 2^31 the Length would have to be, but Overhead of a .NET array? indicates 12 - 16 bytes.
Memory fragmentation is very similar to HDD fragmentation. You might have 2GB of address space, but as you create and destroy objects there will be gaps between the values. If these gaps are too small for your large object, and additional space can not be requested, then you will get the System.OutOfMemoryException. For example, if you create 2 million, 1024 byte objects, then you are using 1.9GB. If you delete every object where the address is not a multiple of 3 then you will be using .6GB of memory, but it will be spread out across the address space with 2024 byte open blocks in between. If you need to create an object which was .2GB you would not be able to do it because there is not a block large enough to fit it in and additional space cannot be obtained (assuming a 32 bit environment). Possible solutions to this issue are things like using smaller objects, reducing the amount of data you store in memory, or using a memory management algorithm to limit/prevent memory fragmentation. It should be noted that unless you are developing a large program which uses a large amount of memory this will not be an issue. Also, this issue can arise on 64 bit systems as windows is limited mostly by the page file size and the amount of RAM on the system.
Since most programs request working memory from the OS and do not request a file mapping, they will be limited by the system's RAM and page file size. As noted in the comment by Néstor Sánchez (Néstor Sánchez) on the blog, with managed code like C# you are stuck to the RAM/page file limitation and the address space of the operating system.
That was way longer then expected. Hopefully it helps someone. I posted it because I ran into the System.OutOfMemoryException running a x64 program on a system with 24GB of RAM even though my array was only holding 2GB of stuff.

I'd advise against the /3GB windows boot option. Apart from everything else (it's overkill to do this for one badly behaved application, and it probably won't solve your problem anyway), it can cause a lot of instability.
Many Windows drivers are not tested with this option, so quite a few of them assume that user-mode pointers always point to the lower 2GB of the address space. Which means they may break horribly with /3GB.
However, Windows does normally limit a 32-bit process to a 2GB address space.
But that doesn't mean you should expect to be able to allocate 2GB!
The address space is already littered with all sorts of allocated data. There's the stack, and all the assemblies that are loaded, static variables and so on. There's no guarantee that there will be 800MB of contiguous unallocated memory anywhere.
Allocating 2 400MB chunks would probably fare better. Or 4 200MB chunks. Smaller allocations are much easier to find room for in a fragmented memory space.
Anyway, if you're going to deploy this to a 12GB machine anyway, you'll want to run this as a 64-bit application, which should solve all the problems.

Changing from 32 to 64 bit worked for me - worth a try if you are on a 64 bit pc and it doesn't need to port.

If you need such large structures, perhaps you could utilize Memory Mapped Files.
This article could prove helpful:
http://www.codeproject.com/KB/recipes/MemoryMappedGenericArray.aspx
LP,
Dejan

Rather than allocating a massive array, could you try utilizing an iterator? These are delay-executed, meaning values are generated only as they're requested in an foreach statement; you shouldn't run out of memory this way:
private static IEnumerable<double> MakeRandomNumbers(int numberOfRandomNumbers)
{
for (int i = 0; i < numberOfRandomNumbers; i++)
{
yield return randomGenerator.GetAnotherRandomNumber();
}
}
...
// Hooray, we won't run out of memory!
foreach(var number in MakeRandomNumbers(int.MaxValue))
{
Console.WriteLine(number);
}
The above will generate as many random numbers as you wish, but only generate them as they're asked for via a foreach statement. You won't run out of memory that way.
Alternately, If you must have them all in one place, store them in a file rather than in memory.

32bit windows has a 2GB process memory limit. The /3GB boot option others have mentioned will make this 3GB with just 1gb remaining for OS kernel use. Realistically if you want to use more than 2GB without hassle then a 64bit OS is required. This also overcomes the problem whereby although you may have 4GB of physical RAM, the address space requried for the video card can make a sizeable chuck of that memory unusable - usually around 500MB.

Well, I got a similar problem with large data set and trying to force the application to use so much data is not really the right option. The best tip I can give you is to process your data in small chunk if it is possible. Because dealing with so much data, the problem will come back sooner or later. Plus, you cannot know the configuration of each machine that will run your application so there's always a risk that the exception will happens on another pc.

I had a similar problem, it was due to a StringBuilder.ToString();

Convert your solution to x64. If you still face an issue, grant max length to everything that throws an exception like below :
var jsSerializer = new JavaScriptSerializer();
jsSerializer.MaxJsonLength = Int32.MaxValue;

If you do not need the Visual Studio Hosting Process:
Uncheck the option: Project->Properties->Debug->Enable the Visual Studio Hosting Process
And then build.
If you still face the problem:
Go to Project->Properties->Build Events->Post-Build Event Command line and paste the following:
call "$(DevEnvDir)..\..\vc\vcvarsall.bat" x86
"$(DevEnvDir)..\..\vc\bin\EditBin.exe" "$(TargetPath)" /LARGEADDRESSAWARE
Now, build the project.

Increase the Windows process limit to 3gb. (via boot.ini or Vista boot manager)

Why does .NET reserve so much memory for my application?

When I run my application, in a profiler I see that is uses about 80MB of memory (total committed bytes, performance counter). But when I look at the size of the allocated memory, it is over 400MB!
So my question is, why is .NET reserving so much memory for my application? Is this normal?

you should read Memory Mystery. I had similar questions a while ago and stopped asking myself after reading this.
I read other sources, but I cant find now, use keywords "unreasonable allocation of memory windows OS". In a nutshell, OS gives more than your app require depending upon physically available memory resources
for e.g. if you are running your app on two machines with different RAM, it can be guaranteed that both these machines will have different memory allocations

As you no doubt know, there is a massive difference between actual memory used and allocated. An application's allocated memory doesn't mean that it's actually being used anywhere; all it really means is that the OS has 'marked' a zone of virtual memory (which is exactly that - virtual) ready for use by the application.
The memory isn't necessarily being used or starving other processes - it just could if the app starts to fill it.
This allocated number, also, will likely scale based on the overall memory ecosystem of the machine. If there's plenty of room when an app starts up, then it'll likely grab a larger allocation than if there's less.
That principle is the same as the one which says it's good practise to create a List<T>, say, with a reasonable initial capacity that'll mean a decent number of items can be added before resizing needs to take place. The OS takes the same approach with memory usage.

"Reserving" memory is by no means the same as "allocated" ram. Read the posts Steve and Krishna linked to.
The part your client needs to look at is Private Bytes. But even that isn't exactly a hard number as parts of your app may be swapped to the virtual disk.
In short, unless your Private Bytes section is pretty well out of control OR you have leaks (ie: undisposed unmanaged resources) you (and your client) should ignore this and let the OS manage what is allocated, what's in physical ram and what's swapped out to disk.

It's fairly common for software to issue one large memory request to the underlying operating system, then internally manage its own use of the allocated memory block. So common, in fact, that Windows' (and other operating systems') memory manager explicitly supports the concept, called "uncommitted memory" -- memory that the process has requested but hasn't made use of yet. That memory really doesn't exist as far as bits taking up space on your DRAM chips until the process actually makes use of it. The preallocation of memory effectively costs nothing.
Applications do this for many reasons -- though it's primarily done for performance reasons. An application with knowledge of its own memory usage patterns can optimize its allocator for that pattern; similarly, for address locality reasons, as successive memory requests from the OS won't always be 'next' to each other in memory, which can affect the performance of the CPU cache and could even preclude you from using some optimizations.
.NET in particular allocates space for the managed heap ahead of time, for both of the reasons listed above. In most cases, allocating memory on the managed heap merely involves incrementing a top-of-heap pointer, which is incredibly fast --- and also not possible with the standard memory allocator (which has a more general design to perform acceptably in a fragmented heap, whereas the CLR's GC uses memory compaction to sharply limit the fragmentation of the managed heap), and also not possible if the managed heap itself is fragmented across the process address space due to multiple allocations at different points in time.

When and how is the .NET managed heap getting swapped?

My small stress test, which allocates random length arrays (100..200MB each) in a loop, shows different behaviour on a 64 bit Win7 machine and on a 32 bit XP (in a VM). Both systems first normally allocate as much arrays as will fit into the LOH. Then the LOH gets bigger and bigger until the virtual address space available is filled up. Expected behaviour so far. But than - on further requests - both behave differently:
While on Win7 an OutOfMemoryException (OOM) is thrown, on XP it seems, the heap gets increased and even swapped to disk - at least no OOM is thrown. (Dont know, if this may have to do with XP running in a virtual box.)
Question:
How does the runtime (or the OS?) decide, whether for managed memory allocation requests, if it is too large to get allocated, a OOM is generated or the large object heap is getting increased - eventually even swapped to disk?
If it is swapped, when does an OOM occour than?
IMO this question is important to all production environments, potentially dealing with larger datasets. Somehow it feels more "safe" to know, the system would rather slow down dramatically in such situations (by swapping) than simply throwing an OOM. At least, it should somehow be deterministically, right?
#Edit: the app is a 32 bit application, therefore running in 32 bit mode on Win 7.

The normal rules apply, a managed process is not treated differently by the Windows memory manager. The ultimate source for chunks of memory is the Windows memory manager. If it cannot find a hole in the virtual memory address space to fit the requested memory allocation then it fails the VirtualAlloc() call and the CLR generates OOM.
Same for swapping behavior, if pages in RAM are needed to map pages of other processes or even pages of the same process then they'll get swapped out. This is not otherwise associated with OOM.
You cannot assume it will work exactly the same on XP as it does on Win7 x64. Getting OOM on x64 when you build your program targeting AnyCPU is quite unusual, a 64-bit operating system has a very large virtual memory address space. The upper limit is set by the maximum size of the paging file. A 32-bit program will run in the WOW emulation layer, it can have a 4 GB address space if you set the LARGEADDRESSAWARE option bit with Editbin.exe.
You can use SysInteral's VMMap utility to see how the address space of your process is carved up.

.NET OutOfMemoryException

Why does this:
class OutOfMemoryTest02
{
static void Main()
{
string value = new string('a', int.MaxValue);
}
}
Throw the exception; but this wont:
class OutOfMemoryTest
{
private static void Main()
{
Int64 i = 0;
ArrayList l = new ArrayList();
while (true)
{
l.Add(new String('c', 1024));
i++;
}
}
}
Whats the difference?

Have you looked up int.MaxValue in the docs? it's the equivalent of 2GB, which is probably more RAM than you have available for a contiguous block of 'a' characters - that is what you are asking for here.
http://msdn.microsoft.com/en-us/library/system.int32.maxvalue.aspx
Your infinite loop will eventually cause the same exception (or a different one indirectly related to overuse of RAM), but it will take a while. Try increasing 1024 to 10 * 1024 * 1024 to reproduce the symptom faster in the loop case.
When I run with this larger string size, I get the exception in under 10 seconds after 68 loops (checking i).

Your
new string('a', int.MaxValue);
throws an OutOfMemoryException simply because .NET's string has a length limitation. The "Remarks" section in the MSDN docs says:
The maximum size of a String object in memory is 2 GB, or about 1 billion characters.
On my system (.NET 4.5 x64) new string('a', int.MaxValue/2 - 31) throws, whereas new string('a', int.MaxValue/2 - 32) works.
In your second example, the infinite loop allocates ~2048 byte blocks until your OS cannot allocate any more block in the virtual address space. When this is reached, you'll get an OutOfMemoryException too.
(~2048 byte = 1024 chars * 2 bytes per UTF-16 code point + string overhead bytes)
Try this great article of Eric.

Because int.MaxValue is 2,147,483,647, or, 2 gigabytes which needs to be allocated contiguously.
In the second example, the OS only needs to find 1024 bytes to allocate each time and can swap to hard-drive. I am sure if you left it running long enough you'd end up in a dark place :)

The String object can use a backing shared string pool to reduce memory usage. In the former case, you're generating one string thats several gigabytes. In the second case, its likely the compiler is auto-interning the string, so you're generating a 1024 byte string, and then referencing that same string many times.
That being said, an ArrayList of that size should run you out of memory, but its likely you haven't let the code run long enough for it to run out of memory.

The 2nd snippet will crash as well. It just takes a wholeheckofalot longer since it is consuming memory much slower. Pay attention to your hard disk access light, it's furiously blinking while Windows chucks pages out of RAM to make room. The first string constructor immediately fails since the heap manager won't allow you to allocate 4 gigabytes.

Both versions will cause an OOM exception, it's just that (on a 32bit machine) you will get it immediately with the first version when you try to allocate a "single" very large object.
The second version will take much longer however as there will be a lot of thrashing to get to the OOM condition for a couple of factors:
You will be allocating millions of small objects which are all reachable by the GC. Once you start putting the system under pressure, the GC will spend an inordinate amount of time scanning generations with millions and millions of objects in. This will take a considerable amount of time and start to play havoc with paging as cold and hot memory will be constantly paged in and out as generations are scanned.
There will be page thrashing as GC scans millions of objects in generations to try and free memory. Scanning will cause huge amounts of memory to be paged in and out constantly.
The thrashing will cause the system to grind to a halt processing overhead and so the OOM condition will take a long time to be reached. Most time will be spent thrashing on the GC and paging for the second version.

In your first sample you are trying to create a 2g string at one time
In the second example you keep adding 1k to an array. You will need to loop more than 2 million times to reach the same amount of consumption.
And it's also not all stored at once, in one variable. Thus, some of your memory usage can be persisted to disk to make room for the new data, I think.

Because a single object cannot have more than 2 GB:
First some background; in the 2.0 version of the .Net runtime (CLR) we made a conscious design decision to keep the maximum object size allowed in the GC Heap at 2GB, even on the 64-bit version of the runtime
In your first example, you try to allocate one object that 2 GB, with the object overhead (8 Bytes?) it's simply too big.
I don't know how the ArrayList works internally, but you allocate multiple objects of 2 GB each and the ArrayList - to my knowledge - only holds pointers which are 4 (8 on x64?) Bytes, regardless how big the object they point to is.
To quote another article:
Also, objects that have references to other objects store only the reference. So if you have an object that holds references to three other objects, the memory footprint is only 12 extra bytes: one 32-bit pointer to each of the referenced objects. It doesn't matter how large the referenced object is.

One reason your system could be coming to a halt is because .NET's code runs closer to the metal and you're in a tight loop which should consume 100% CPU provided the process priority allows it to. If you would like to prevent the application from consuming too much CPU while it performs the tight loop you should add something like System.Threading.Thread.Sleep(10) to the end of the loop, which will forcibly yield processing time to other threads.
One major difference between the JVM and .NET's CLR (Common Language Runtime) is that the CLR does not limit the size of your memory on an x64 system/application (in 32bit applications, without the Large Address Aware flag the OS limits any application to 2GB due to addressing limitations). The JIT compiler creates native windows code for your processing architecture and then runs it in the same scope that any other windows application would run. The JVM is a more isolated sandbox which constrains the application to a specified size depending on configuration/command line switches.
As for differences between the two algorithms:
The single string creation is not guaranteed to fail when running in an x64 environment with enough contiguous memory to allocate the 4GB necessary to contain int.MaxValue characters (.NET strings are Unicode by default, which requires 2 bytes per character). A 32 bit application will always fail, even with the Large Address Aware flag set because the maximum memory is still something like 3.5GB).
The while loop version of your code will likely consume more overall memory, provided you have plenty available, before throwing the exception because your strings can be allocated in smaller fragments, but it is guaranteed to hit the error eventually (although if you have plenty of resources, it could happen as a result of the ArrayList exceeding the maximum number of elements in an array rather than the inability to allocate new space for a small string). Kent Murra is also correct about string interning; you will either need to randomize the length of the string or the character contents to avoid interning, otherwise you're simply creating pointers to the same string. Steve Townsend's recommendation to increase string length would also make finding large enough contiguous blocks of memory harder to come by, which will allow the exception to happen more quickly.
EDIT:
Thought I'd give some links people may find handy for understanding .NET memory:
These two articles are a little older, but very good in depth reading:
Garbage Collection: Automatic Memory Management in the Microsoft .NET Framework
Garbage Collection Part 2: Automatic Memory Management in the Microsoft .NET Framework
These are blogs from a .NET Garbage Collection developer for information about newer version of .NET memory management:
So, what’s new in the CLR 4.0 GC?
CLR 4.5: Maoni Stephens - Server Background GC
This SO Question may help you observe the inner workings of .NET memory:
.NET Memory Profiling Tools