I have the following code:
const int bufferSize = 1024 * 1024;
var buffer = new byte[bufferSize];
for (int i = 0; i < 10; i++)
{
const int writesCount = 400;
using (var stream = new MemoryStream(writesCount * bufferSize))
{
for (int j = 0; j < writesCount; j++)
{
stream.Write(buffer, 0, buffer.Length);
}
stream.Close();
}
}
which I run on a 32-bit machine.
The first iteration finishes just fine and then on the next iteration I get a System.OutOfMemoryException exception on the line that news the MemoryStream.
Why isn't the previous MemoryStream memory reclaimed despite using statement? How do I force release of memory used by the MemoryStream?
I don't think the problem is the garbage collector not doing its job. If the GC is under memory pressure it should run and reclaim the 400 MBs you've just allocated.
This is more likely down to the GC not finding a contigious 400 MB block.
Rather, an “out of memory” error happens because the process is unable
to find a large enough section of contiguous unused pages in its
virtual address space to do the requested mapping.
You should read Eric Lippert's blog entry "Out Of Memory" Does Not Refer to Physical Memory
You're far better off doing both of the below.
Reusing the memory block you've allocated (why are you creating another with the exact same size)
Allocating much smaller chunks (less than 85KBs)
Prior to Dotnet 4.5, Dotnet constructed two heaps, Small Object Heap (SOH) and Large Object Heap (LOH). See Large Object Hearp Improvements in .NET 4.5 by Brandon Bray. Your MemoryStream is being allocated in LOH, and not compacted (defragmented) for the duration of the process, making it much more likely that multiple calls to allocate this large amount of memory will throw an OutOfMemoryException
The CLR manages two different heaps for allocation, the small object
heap (SOH) and the large object heap (LOH). Any allocation greater
than or equal to 85,000 bytes goes on the LOH. Copying large objects
has a performance penalty, so the LOH is not compacted unlike the SOH.
Another defining characteristic is that the LOH is only collected
during a generation 2 collection. Together, these have the built-in
assumption that large object allocations are infrequent.
Looks like you're allocating too much than your system can handle. Your code runs fine on my machine, but if I change it like this :
const int bufferSize = 1024 * 1024 * 2;
I get the same error as you.
But if I change the target processor to x64, then the code runs, which seems logical as you can address lot more memory.
Detailed explanation on this article : http://www.guylangston.net/blog/Article/MaxMemory
And some information on this question : Maximum Memory a .NET process can allocate
First of all, Dispose() does not guarantee that memory will be released (it does not mark objects for GC collection, in case of MemoryStream - it releases nothing, as MemoryStream has no unmanaged resources). The only reliable way to free memory used by MemoryStream is to lose all references to it and wait for garbage collection to occur (and if you have OutOfMemoryException - garbage collector already tried but failed to free enough memory). Also, allocating such large objects (anything > 85000 bytes) have some consequences - these objects are going to large object heap (LOH), which can get fragmented (and cannot be compacted). As .NET object must occupy a contiguous sequence of bytes, it can lead to a situation where you have enough memory, but there is no room for large object. Garbage collector won't help in this case.
It seems like the main problem here is that reference to a stream object is kept on stack, preventing garbage collection of stream object (even forcing garbage collection won't help, as GC considers that object is still alive, you can check this creating a WeakRefrence to it). Refactoring this sample can fix it:
static void Main(string[] args)
{
const int bufferSize = 1024 * 1024 * 2;
var buffer = new byte[bufferSize];
for(int i = 0; i < 10; i++)
{
const int writesCount = 400;
Write(buffer, writesCount, bufferSize);
}
}
static void Write(byte[] buffer, int writesCount, int bufferSize)
{
using(var stream = new MemoryStream(writesCount * bufferSize))
{
for(int j = 0; j < writesCount; j++)
{
stream.Write(buffer, 0, buffer.Length);
}
}
}
Here is a sample which proves that object can't be garbage collected:
static void Main(string[] args)
{
const int bufferSize = 1024 * 1024 * 2;
var buffer = new byte[bufferSize];
WeakReference wref = null;
for(int i = 0; i < 10; i++)
{
if(wref != null)
{
// force garbage collection
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();
// check if object is still alive
Console.WriteLine(wref.IsAlive); // true
}
const int writesCount = 400;
using(var stream = new MemoryStream(writesCount * bufferSize))
{
for(int j = 0; j < writesCount; j++)
{
stream.Write(buffer, 0, buffer.Length);
}
// weak reference won't prevent garbage collection
wref = new WeakReference(stream);
}
}
}
Try to force garbage collection when you are sure that it is necessary to clean unreferenced objects.
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();
Another alternative is to use the Stream with external storage: FileStream, for example.
But, in general case, it would be better to use one small enough buffer (array, allocated one time) and use it for read/write calls. Avoid having many large objects in .NET (see CLR Inside Out: Large Object Heap Uncovered).
Update
Assuming that the writesCount is the constant, the why not allocate one buffer and reuse it?
const int bufferSize = 1024 * 1024;
const int writesCount = 400;
byte[] streamBuffer = new byte[writesCount * bufferSize];
byte[] buffer = new byte[bufferSize];
for (int i = 0; i < 10; i++)
{
using (var stream = new MemoryStream(streamBuffer))
{
for (int j = 0; j < writesCount; j++)
{
stream.Write(buffer, 0, buffer.Length);
}
}
}
Related
I have a byte array list. And, I am using it to generate bitmap images via memory stream.
While saving images, memory usage goes very high. And at some point, it causes out of memory exception.
I tried to comment out saving files to see if that causing this problem. Or, called GC manually. Nothing changed, still using high memory. My latest code is like this:
List<byte[]> byteArrayList = helper.GetArrayList(); // Gets approximately 10k items.
for (int i = 0; i < byteArrayList.Count; i++)
{
using (MemoryStream ms = new MemoryStream(byteArrayList[i]))
{
using (Bitmap bm = new Bitmap(ms))
{
bm.Save(fileLocation);
bm.Dispose();
}
ms.Dispose();
}
byteArrayList[i] = null;
byteArrayList.Remove(byteArrayList[i]);
}
byteArrayList.Dispose();
How can i solve this issue?
I have tested your code and saw that the system cannot collect your garbage in a LOOP. so if you create so many bitmaps in a loop, the memory increases to the peak levels (such 2-3-4 gbs) until garbage collector runs. But when loop ends, the memory level decreases to the normal which is too late. So When I test your code in a BACKGROUNDWORKER instead of main thread, GC doesnt stuck to the loop and runs as it is supposed to and it converts the byte arrays to the bitmaps and save them without any extreme memory consumption.
If you change the helper method to return a Queue<T> instead...
Queue<byte[]> byteArrayQueue = helper.GetQueue(); // Gets approximately 10k items.
while (byteArrayQueue.Any())
{
using (var ms = new MemoryStream(byteArrayQueue.Dequeue()))
{
using (var bm = new Bitmap(ms))
{
bm.Save(fileLocation);
}
}
}
Let's simplify the model.
class Container
{
//other members
public byte[] PNG;
}
class Producer
{
public byte[] Produce(byte[] ImageOutside)
{
using (MemoryStream bmpStream = new MemoryStream(ImageOutside),
pngStream = new MemoryStream())
{
System.Drawing.Bitmap bitmap = new System.Drawing.Bitmap(bmpStream);
bitmap.Save(pngStream, System.Drawing.Imaging.ImageFormat.Png);
pngStream.Seek(0, System.IO.SeekOrigin.Begin);
byte[] PNG = new byte[pngStream.Length];
pngStream.Read(PNG, 0, (int)pngStream.Length);
bitmap.Dispose();
GC.Collect();
GC.WaitForPendingFinalizers();
return PNG;
}
}
}
The main function keep making Container container = new Container(); produce PNG for container.PNG, and Queue.Enqueue(container)
use using() clause doesn't work at all.
While this repeat for about 40+ times(it varies), it throws an exception. Sometime it's OutOfMemoryException and sometime It's something like "GDI+ normal error"(I am not sure how it is exactly in English, I just translated it).
But If I try and catch the exception and simply ignore it, it can still continue producing more but not unlimitedly though, just more forward.
The occupied memory shown in task manager is only about 600 - 700 MB when the first exception is thrown and it finally stops at about 1.2GB. I have tried this:
while (true)
{
Bitmap b = new Bitmap(4500, 5000);
list.Add(b);
Invoke((MethodInvoker)delegate { textBox1.Text = list.Count.ToString(); });
}
It never throws any exception though 99% memory(about 11GB) has been allocated for the program, and all happen is the number in textBox1 no longer raise.
The way to avoid this may be not to produce so many things, but I still want to know the internal principle and reason and thank you for your help.
with byte[] PNG = new byte[pngStream.Length]; you allocate a large portion of memory to store the image.
The follow call it's useless, you have already disposed the stream.
GC.Collect();
GC.WaitForPendingFinalizers();
The memory used by PNG array cannot be released, because there is active reference in function return.
I suggest to return a stream instead of an array of bytes.
Otherwise after you call the method Produce remember to remove the reference to PNG before call again.
sample:
while (true)
{
Byte[] b = new Byte[1000];
b = this.Produce(b);
//Use your array as you need, but you can't assign it to external property, otherwise memory cannot be released
b = null; //remove the reference, (in reality, in this example assign null is not necessary, because b will be overwritten at next loop.
GC.Collect(); //Force garbage collector, probably not necessarry, but can be useful
GC.WaitForPendingFinalizers();
}
The platform compilation can affect the maximum available memory:
In a 32 bit application, you have a maximum of 2 GiB of available memory
In a 64 bit application you have 2 Tib of available memory, but
single object (class) cannot exceed 2 Gib.
In a UWP application there are other limitation in dependence of the
device
Any CPU is complied just in time, when you launch the application,
and can be run both 32-bit and 64, it depends from machine architecture
and system configuration.
so I just found a really weird issue in my app and it turns out it was caused by the .NET Native compiler for some reason.
I have a method that compares the content of two files, and it works fine. With two 400KBs files, it takes like 0.4 seconds to run on my Lumia 930 in Debug mode. But, when in Release mode, it takes up to 17 seconds for no apparent reason. Here's the code:
// Compares the content of the two streams
private static async Task<bool> ContentEquals(ulong size, [NotNull] Stream fileStream, [NotNull] Stream testStream)
{
// Initialization
const int bytes = 8;
int iterations = (int)Math.Ceiling((double)size / bytes);
byte[] one = new byte[bytes];
byte[] two = new byte[bytes];
// Read all the bytes and compare them 8 at a time
for (int i = 0; i < iterations; i++)
{
await fileStream.ReadAsync(one, 0, bytes);
await testStream.ReadAsync(two, 0, bytes);
if (BitConverter.ToUInt64(one, 0) != BitConverter.ToUInt64(two, 0)) return false;
}
return true;
}
/// <summary>
/// Checks if the content of two files is the same
/// </summary>
/// <param name="file">The source file</param>
/// <param name="test">The file to test</param>
public static async Task<bool> ContentEquals([NotNull] this StorageFile file, [NotNull] StorageFile test)
{
// If the two files have a different size, just stop here
ulong size = await file.GetFileSizeAsync();
if (size != await test.GetFileSizeAsync()) return false;
// Open the two files to read them
try
{
// Direct streams
using (Stream fileStream = await file.OpenStreamForReadAsync())
using (Stream testStream = await test.OpenStreamForReadAsync())
{
return await ContentEquals(size, fileStream, testStream);
}
}
catch (UnauthorizedAccessException)
{
// Copy streams
StorageFile fileCopy = await file.CreateCopyAsync(ApplicationData.Current.TemporaryFolder);
StorageFile testCopy = await file.CreateCopyAsync(ApplicationData.Current.TemporaryFolder);
using (Stream fileStream = await fileCopy.OpenStreamForReadAsync())
using (Stream testStream = await testCopy.OpenStreamForReadAsync())
{
// Compare the files
bool result = await ContentEquals(size, fileStream, testStream);
// Delete the temp files at the end of the operation
Task.Run(() =>
{
fileCopy.DeleteAsync(StorageDeleteOption.PermanentDelete).Forget();
testCopy.DeleteAsync(StorageDeleteOption.PermanentDelete).Forget();
}).Forget();
return result;
}
}
}
Now, I have absolutely no idea why this same exact method goes from 0.4 seconds all the way up to more than 15 seconds when compile with the .NET Native toolchain.
I fixed this issue using a single ReadAsync call to read the entire files, then I generated two MD5 hashes from the results and compared the two. This approach worked in around 0.4 seconds on my Lumia 930 even in Release mode.
Still, I'm curious about this issue and I'd like to know why it was happening.
Thank you in advance for your help!
EDIT: so I've tweaked my method in order to reduce the number of actual IO operations, this is the result and it looks like it's working fine so far.
private static async Task<bool> ContentEquals(ulong size, [NotNull] Stream fileStream, [NotNull] Stream testStream)
{
// Initialization
const int bytes = 102400;
int iterations = (int)Math.Ceiling((double)size / bytes);
byte[] first = new byte[bytes], second = new byte[bytes];
// Read all the bytes and compare them 8 at a time
for (int i = 0; i < iterations; i++)
{
// Read the next data chunk
int[] counts = await Task.WhenAll(fileStream.ReadAsync(first, 0, bytes), testStream.ReadAsync(second, 0, bytes));
if (counts[0] != counts[1]) return false;
int target = counts[0];
// Compare the first bytes 8 at a time
int j;
for (j = 0; j < target; j += 8)
{
if (BitConverter.ToUInt64(first, j) != BitConverter.ToUInt64(second, j)) return false;
}
// Compare the bytes in the last chunk if necessary
while (j < target)
{
if (first[j] != second[j]) return false;
j++;
}
}
return true;
}
Reading eight bytes at a time from an I/O device is a performance disaster. That's why we are using buffered reading (and writing) in the first place. It takes time for an I/O request to be submitted, processed, executed and finally returned.
OpenStreamForReadAsync appears to not be using a buffered stream. So your 8-byte requests are actually requesting 8 bytes at a time. Even with the solid-state drive, this is very slow.
You don't need to read the whole file at once, though. The usual approach is to find a reasonable buffer size to pre-read; something like reading 1 kiB at a time should fix your whole issue without requiring you to load the whole file in memory at once. You can use BufferedStream between the file and your reading to handle this for you. And if you're feeling adventurous, you could issue the next read request before the CPU processing is done - though it's very likely that this isn't going to help your performance much, given how much of the work is just I/O.
It also seems that .NET native has a lot bigger overhead than managed .NET for asynchronous I/O in the first place, which would make those tiny asynchronous calls all the more of a problem. Fewer requests of larger data will help.
I have an small application receives data from thousands for agents and upload those data to another server, the data from agents are pretty small, usually 10KB but agents write speed is very fast, so my application has an internal 4M buffer, once buffer is full, it creates a new 4M buffer, and pass the old buffer to a Task to perform HTTP upload. code like this:
lock (Locker)
{
if (input.Length > this.buffer.Length - this.bufferDataLen)
{
// Save the current buffer and create a new one.
// We should release locker as fast as we can.
byte[] tempBuffer = this.buffer;
int oldBufferDataLen = this.bufferDataLen;
this.buffer = new byte[tempBuffer.Length]; // save size buffer, 4MB
this.bufferDataLen = 0;
Task.Factory.StartNew(
() =>
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(this._uploadUrl);
request.Method = "POST";
request.ContentType = this._contentType;
request.ContentLength = oldBufferDataLen;
request.KeepAlive = false;
request.Proxy = null;
UploaderState state = new UploaderState(tempBuffer, oldBufferDataLen, request);
IAsyncResult result = request.BeginGetRequestStream(this.OnGetRequestStreamComplete, state);
ThreadPool.RegisterWaitForSingleObject(result.AsyncWaitHandle, this.TimeoutCallback, state, this._timeoutMs, true);
});
}
// Copy incoming data to either old buffer or new buffer
Buffer.BlockCopy(input.Buffer, 0, this.buffer, this.bufferDataLen, input.Length);
this.bufferDataLen += input.Length;
}
I expect that tempBuffer will be disposed correctly by GC. However, when I ran my application, I noticed that the memory usage of application increased very fast, from the memory dump, there are 305 byte[] object on managed heap, total size is 469,339,928 B, inclusive size is 469,339,928 B, and that dump is captured when the application ran only few minutes.
My question is why GC didn't free those byte[]? Should I explicitly call GC.Collect? In my case should I manage a buffer pool myself?
I have never used pointers before and now I need to and even though I've read up on them I still have no clue on how to use them!
I am using a third party product (Mitov Signal Lab)and I am using a control that has an event that will allow me to intercept the data. When I enter the event I have access to "args" and one of the statements will return a pointer. Now this points to a buffer of 4096 double values (one a real number and the next an imaginiary number etc). So this is the code:
private void genericComplex1_ProcessData(object Sender, Mitov.SignalLab.ProcessComplexNotifyArgs Args)
{
unsafe
{
IntPtr pointer = Args.InBuffer.Read();
// now what???? I want to get each double value, do some calculations on them
// and return them to the buffer.
}
args.sendOutPutData = true;
}
I'm assuming that i need to assign pointer to Args.InBuffer.Read()
But I've tried some examples I've seen and I just get errors. I'm using visual studio 2010 and i'm compiling with the /unsafe.
Here's an update. There are additional methods such as Args.Inbuffer.Write(), and Args.Inbuffer.Modify() but I still can't figure out how to access the individual components of the buffer.
Ok here are more details. I used the Marshall.copy. Here is the code:
private void genericComplex1_ProcessData(object Sender, Mitov.SignalLab.ProcessComplexNotifyArgs Args)
{
//used to generate a phase corrected Q signal
//equations:
// real = real
// imaginary = (1 + amplitudeimbalance)(real*cos(phaseImbalance) + imaginary*sin(phaseImbalance)
//but amplitude imbalance is already corrected therefore
//imaginary = real * cos(phaseCorrection) + imaginary*sin(phaseCorrection)
double real;
double imaginary;
double newImaginary;
var bufferSize = Convert.ToInt32(Args.InBuffer.GetSize());
Mitov.SignalLab.ComplexBuffer ComplexDataBuffer = new Mitov.SignalLab.ComplexBuffer(bufferSize);
ComplexDataBuffer.Zero();
IntPtr pointer = Args.InBuffer.Read();
IntPtr pointer2 = ComplexDataBuffer.Write();
var data = new double[8192];
Marshal.Copy(pointer, data, 0, 8192);
for (int i = 0; i < 8192; i+=2)
{
data[i] = data[i];
data[i + 1] = data[i] * Math.Cos(phaseCorrection) + data[i+1] * Math.Sin(phaseCorrection);
}
Marshal.Copy(data, 0, pointer2, 8192);
Args.SendOutputData = false;
genericComplex1.SendData(ComplexDataBuffer);
}
Now, this does not work. The reason is it takes too long. I am sampling audio from a sound card at 192,000 samples a second. The event above fires each time there is a 4096 buffer available. Thus every 20 milliseconds a new buffer arrives. This buffer is then sent to a fast fourier transform routine that produces a graph of signal strength vs frequency. The imaginary data is actually the same as the real data but it is phase shifted by 90 degrees. This is how it is in a perfect world. However the phase is never 90 degrees due to imbalances in the signal, different electronics paths, different code and so on. Thus one needs a correction factor and that is what the equations do. Now if I replace the
data[i] = data[i];
data[i + 1] = data[i] * Math.Cos(phaseCorrection) + data[i+1] * Math.Sin(phaseCorrection);
with
data[i] = data[i];
data[i+1] = data[i+1];
it works ok.
So I'm going to need pointers. But will this be substantially faster or will the sin and cosine functions just take too long? Cn I send the data directly to the processor (now thats getting real unsafe!).
You can't access unmanaged data directly in safe context, so you need to do a copy of data to managed container:
using System.Runtime.InteropServices;
// ...
var data = new double[4096];
Marshal.Copy(pointer, data, 0, 4096);
// process data here
And as far as I see it, unsafe context is not required for this part of code.
As for the writing data back it's actually quite the same, but note the parameters order:
// finished processing, copying data back
Marshal.Copy(data, 0, pointer, 4096);
As for your last notes on perfprmance. As I already mentioned, move var data = new double[8192]; to a data class member and initialize it in class constructor, so it's not allocated each time (and allocation is pretty slow). You can do Math.Cos(phaseCorrection) outside of cycle since phaseCorrection is unchanged, same is for Math.Sin(phaseCorrection) of course. Something like this:
Marshal.Copy(pointer, this.data, 0, 8192);
var cosCache = Math.Cos(phaseCorrection);
var sinCache = Math.Sin(phaseCorrection);
for (int i = 0; i < 8192; i+=2)
{
data[i] = data[i];
data[i + 1] = data[i] * cosCache + data[i+1] * sinCache;
}
Marshal.Copy(this.data, 0, pointer2, 8192);
All this should speed up things tremendously.
If you have an IntPtr, you can get a pointer from it. Using your example:
unsafe
{
IntPtr pointer = Args.InBuffer.Read();
// create a pointer to a double
double* pd = (double*)pointer.ToPointer();
// Now you can access *pd
double firstDouble = *pd;
pd++; // moves to the next double in the memory block
double secondDouble = *pd;
// etc.
}
Now, if you want to put data back into the array, it's much the same:
double* pd = (double*)pointer.ToPointer();
*pd = 3.14;
++pd;
*pd = 42.424242;
You need to read and understand Unsafe Code and Pointers. Be very careful with this; you're programming without a safety net here.