We Keep Coding

Why does my program's memory usage rise quickly and then go flat? [duplicate]

This question already has answers here:
What strategies and tools are useful for finding memory leaks in .NET?
(15 answers)
Understanding garbage collection in .NET
(2 answers)
Memory Leak in C#
(21 answers)
Memory Leak in C# WPF
(4 answers)
Closed 3 years ago.
Hi everyone I'm making a C# WinForms application that searches for duplicate images in a directory. It starts by calling a constructor with every image in the directory.
There is a lot of files in the directory and memory was quickly rising to 2gb and then the program would throw an out of memory exception.
Now I've added a check in my for loop to check if the memory has exceeded 800 Megabits and I force a garbage collection. But I've noticed after the first forced collection the memory no longer rises.
(The forced garbage collection occurs at loop ~180 out of ~800 then it never occurs again)
(It looks a little like a shark fin swimming through the water leaving waves in its wake.)
I'm stumped as to why this is happening and have come here in search of help.
private void GeneratePrints()
{
for (int i = 0; i < files.Count; i++)
{
if (imageFileExtensions.Contains(Path.GetExtension(files[i])))
prints.Add(new FilePrint(directory + "/" + files[i]));
//800 Megabits
const long MAX_GARBAGE = 800 * 125000;
if (GC.GetTotalMemory(false) > MAX_GARBAGE)
{
GC.Collect();
GC.WaitForPendingFinalizers();
}
}
Console.WriteLine("File Prints Complete.");
}
GeneratePrints() is called 1 time, once a directory is selected.
I will also show you the constructor for the FilePrint class.
I'm pretty sure this all has something to do with the MemoryStream object.
public FilePrint(string filename)
{
Bitmap img;
using (var fs = new FileStream(filename, FileMode.Open))
{
using (var ms = new MemoryStream())
{
fs.CopyTo(ms);
ms.Position = 0;
img = (Bitmap)Bitmap.FromStream(ms);
ms.Close();
}
fs.Close();
}
this.size = img.Size;
img = ResizeImage(img, 8, 8);
img = MakeGrayscale(img);
//I do some basic for-loop arithmetic here
//calculating the average color of the image, not worth posting.
img.Dispose();
}
So basically I'm wondering how can I make it so that the 'shark-fin-like' memory usage spike at the start never happens so that I do not have to force a garbage collection.
Here is a memory snapshot I have taken when the forced garbage collection occurs (Am I not disposing of the MemoryStreams properly?):
Thank you for your replies, ideas and answers in advance!

You don't show the methods ResizeImage(img, 8, 8) and MakeGrayscale(img), but most likely they simply create and return a new image based on the old. If that's true, your code constructs two Bitmap objects that it never explicitly disposes, so try disposing them e.g. as follows:
using (var old = img)
img = ResizeImage(old, 8, 8);
using (var old = img)
img = MakeGrayscale(old);
You might also want to guarantee that the final img is disposed using a try/finally:
Bitmap img = null;
try
{
img = new Bitmap(filename); // Here I simplified the code, but this will leave the file locked until `img` is disposed after resizing.
this.size = img.Size;
using (var old = img)
img = ResizeImage(old, 8, 8);
using (var old = img)
img = MakeGrayscale(old);
//I do some basic for-loop arithmetic here
//calculating the average color of the image, not worth posting.
}
finally
{
if (img != null)
img.Dispose();
}
The possible reason you get the long buildup in memory use then a precipitous drop is that eventually the unmanaged resources of the undisposed images will get finalized, however because the GC is unaware of unmanaged memory owned by the undisposed Bitmap objects, it doesn't necessarily kick in, identify the bitmaps as unreferenced and pass them on to the finalizer thread for quite a while. It's not always easy to predict when or even if the finalizer thread will spin up and start working; see Are .net finalizers always executed? to which the answer is not necessarily. But by calling GC.WaitForPendingFinalizers(); you may be kickstarting that process.
Incidentally, a MemoryStream doesn't actually need to be disposed in the current implementation as it lacks unmanaged resources. See this answer by Jon Skeet to Is a memory leak created if a MemoryStream in .NET is not closed? for confirmation. (It's still good practice to do so, though in the case of Bitmap there's that pesky file/stream lock that makes it impossible.)

Memory not getting released in WPF Image

I am loading and unloading images in Canvas. I used the below code to load the Image.
Before loading my Image the memory consumption is 14.8MB.
Canvas c = new Canvas();
Image im = new Image();
ImageSource src = new BitmapImage(new Uri(#"E:Capture.png"));
im.Source = src;
im.Height = 800;
im.Width = 800;
c.Children.Add(im);
homegrid.Children.Add(c); //homegrid is my grid's name
The Image displayed correctly and the memory consumption now is 20.8MB. Then I unloaded the Image by the below code:
foreach (UIElement element in homegrid.Children)
{
if (element is Canvas)
{
Canvas page = element as Canvas;
if (page.Children.Count > 0)
{
for (int i = page.Children.Count - 1; i >= 0; i--)
{
if (page.Children[i] is Image)
(page.Children[i] as Image).Source = null;
page.Children.RemoveAt(i);
}
}
page.Children.Clear();
page = null;
}
}
homegrid.Children.RemoveAt(2);
InvalidateVisual();
The Image gets removed after this, but the memory is still 20.8 MB.
Can anyone help me out this?

First of all you should test by explicitly invoking GC.Collect() to collect memory and see that memory releases or not because GC collection is indeterministic. You can't be sure that after your method execution GC runs and reclaim the memory.
So , at end put this code to explicitly force GC to run to check if actually memory is released or not:
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();
However, there is some known memory leak issues in BitmapImage creation which you can refer here, here and here.
Actually under the covers WPF keeps a strong reference between the static BitmapImage and the Image and hook some events on Bitmap image. So, you should freeze the bitmapImage before assigning to image. WPF doesn't hook events on freezed bitmapImage. Also set CacheOption to avoid any caching memory leak of bitmapImage.
Image im = new Image();
BitmapImage bi = new BitmapImage();
bi.BeginInit();
bi.CacheOption = BitmapCacheOption.OnLoad;
bi.UriSource = new Uri(#"E:Capture.png");
bi.EndInit();
bi.Freeze();
ImageSource src = bi;
im.Source = src;
im.Height = 800;
im.Width = 800;

In .Net there is something called the garbage collector (GC) that is in charge of managing the memory you're using.
When you create an instance of an object, it requires some more memory.
When you remove your ImageSource from the Children collection, you don't actually free any memory, you just say "I don't want to use this instance anymore".
At this point the GC will help you. It'll automatically detect instances that are not used anymore, and will free the associated memory for you.
Do note it's an automatic process and you shouldn't (and you don't want to) take care of the memory management.
You can call GC.Collect(); to force the garbage collector to do its job right now, you'll see the memory will be released. NOTE: GC.Collect(); should be used in debug to detect memory leaks, but 99% of times you shouldn't call it explicitly in production code. GC.Collect(); is an operation that can use a lot of CPU time.

OutOfMemoryException # WriteableBitmap # background agent

I have a Windows Phone 8 App, which uses Background agent to:
Get data from internet;
Generate image based on a User Control that uses the data from step 1 as data source;
In the User Control I have Grid & StackPanel & some Text and Image controls;
When some of the Images use local resources from the installation folder (/Assets/images/...)
One of them which I used as a background is selected by user from the phone's photo library, so I have to set the source using C# code behind.
However, when it runs under the background, it get the OutOfMemoryException, some troubleshooting so far:
When I run the process in the "front", everything works fine;
If I comment out the update progress, and create the image directly, it also works fine;
If I don't set the background image, it also works fine;
The OutOfMemoryException was thrown out during var bmp = new WriteableBitmap(480, 800);
I already shrink the image size from 1280*768 to 800*480, I think it is the bottom line for a full screen background image, isn't it?
After some research, I found out this problem occurs because it exceeded the 11 MB limitation for a Periodic Task.
I tried use the DeviceStatus.ApplicationCurrentMemoryUsage to track the memory usage：
-- the limitation is 11,534,336 (bit)
-- when background agent started, even without any task in it, the memory usage turns to be 4,648,960
-- When get update from internet, it grew up to 5,079,040
-- when finished, it dropped back to 4,648,960
-- When the invoke started (to generate image from the User Control), it grew up to 8,499,200
Well, I guess that's the problem, there is little memory available for it to render the image via WriteableBitmap.
Any idea how to work out this problem?
Is there a better method to generate an image from a User Control / or anything else?
Actually the original image might only be 100 kb or around, however, when rendering by WriteableBitmap, the file size (as well as the required memory size I guess) might grew up to 1-2MB.
Or can I release the memory from anywhere?
==============================================================
BTW, when this Code Project article says I can use only 11MB memory in a Periodic Task;
However, this MSDN article says that I can use up to 20 MB or 25MB with Windows Phone 8 Update 3;
Which is correct? And why am I in the first situation?
==============================================================
Edit:
Speak of the debugger, it also stated in the MSDN article:
When running under the debugger, memory and timeout restrictions are suspended.
But why would I still hit the limitation?
==============================================================
Edit:
Well, I found something seems to be helpful, I will check on them for now, suggestions are still welcome.
http://writeablebitmapex.codeplex.com/
http://suchan.cz/2012/07/pro-live-tiles-for-windows-phone/
http://notebookheavy.com/2011/12/06/microsoft-style-dynamic-tiles-for-windows-phone-mango/
==============================================================
The code to generate the image:
Deployment.Current.Dispatcher.BeginInvoke(() =>
{
var customBG = new ImageUserControl();
customBG.Measure(new Size(480, 800));
var bmp = new WriteableBitmap(480, 800); //Thrown the **OutOfMemoryException**
bmp.Render(customBG, null);
bmp.Invalidate();
using (var isf = IsolatedStorageFile.GetUserStoreForApplication())
{
filename = "/Shared/NewBackGround.jpg";
using (var stream = isf.OpenFile(filename, System.IO.FileMode.OpenOrCreate))
{
bmp.SaveJpeg(stream, 480, 800, 0, 100);
}
}
}
The XAML code for the ImageUserControl:
<UserControl blabla... d:DesignHeight="800" d:DesignWidth="480">
<Grid x:Name="LayoutRoot">
<Image x:Name="nBackgroundSource" Stretch="UniformToFill"/>
//blabla...
</Grid>
</UserControl>
The C# code behind the ImageUserControl:
public ImageUserControl()
{
InitializeComponent();
LupdateUI();
}
public void LupdateUI()
{
DataInfo _dataInfo = new DataInfo();
LayoutRoot.DataContext = _dataInfo;
try
{
using (var isoStore = IsolatedStorageFile.GetUserStoreForApplication())
{
using (var isoFileStream = isoStore.OpenFile("/Shared/BackgroundImage.jpg", FileMode.Open, FileAccess.Read))
{
BitmapImage bi = new BitmapImage();
bi.SetSource(isoFileStream);
nBackgroundSource.Source = bi;
}
}
}
catch (Exception) { }
}
When DataInfo is another Class within the settings page that hold data get from the internet:
public class DataInfo
{
public string Wind1 { get { return GetValueOrDefault<string>("Wind1", "N/A"); } set { if (AddOrUpdateValue("Wind1", value)) { Save(); } } }
public string Wind2 { get { return GetValueOrDefault<string>("Wind2", "N/A"); } set { if (AddOrUpdateValue("Wind2", value)) { Save(); } } }
//blabla...
}

If I comment out the update progress, and create the image directly, it also works fine I think you should focus on that part. It seems to indicate some memory isn't freed after the update. Make sure all the references used during the update process go out of scope before rendering the picture. Forcing a garbage collection can help too:
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect(); // Frees the memory that was used by the finalizers
Another thing to consider is that the debugger is also using a lot of memory. Do a real test by compiling your project in "Release" mode and deploying on a phone to make sure you're running out of memory.
Still, I have already been in that situation so I know it may not be enough. The point is: some libraries in the .NET Framework are loaded lazily. For instance, if your update process involves downloading some data, then the background agent will load the network libraries. Those libraries can't be unloaded and will waste some of your agent's memory. That's why, even by freeing all the memory you used during the update process, you won't reach back the same amount of free memory you had when starting the background agent. Seeing this, what I did in one of my app was to span the workload of the background agent across two executions. Basically, when the agents executes:
Check in the isolated storage if there's pending data to be processed. If not, just execute the update process and store all needed data in the isolated storage
If there is pending data (that is, in the next execution), generate the picture and clear the data
It means the picture will be generated only once every hour instead of once every 30 minutes, so use this workaround only if everything else fail.

The larger memory limit is for background audio agents, it clearly states that in the documentation. You're stuck with 11 MB, which can really be a big pain when you're trying to do something smart with pictures in the background.
480x800 adds a MB to your memory because it takes 4 bytes for every pixels, so in the end it's around 1.22MB. When compressed in JPEG, then yes - it makes sense that it's only around 100KB. But whenever you use WriteableBitmap, it gets loaded into memory.
One of the things you could try before forcing the GC.Collect as mentioned in another answer is to null things out even before they go out of scope - whether it's a BitmapImage or a WriteableBitmap. Other than that you can try removing the Image object from the Grid programmatically when you're done and setting the Source of it also to be null.
Are there any other WriteableBitmap, BitmapImage or Image objects you're not showing us?
Also, try without the debugger. I've read somewhere that it adds another 1-2MB which is a lot when you have only 11 MB. Although, if it crashes so quickly with the debugger, I wouldn't risk it even if it suddenly seams OK without the debugger. But just for testing purposes you can give it a shot.
Do you need to use the ImageUserControl? Can you try doing step by step creating Image and all the other objects, without the XAML, so you can measure memory in each and every step to see at which point it goes through the roof?

BitmapPalette Insufficient memory exception

I have the following piece of code that runs in a loop.
public void Test(Bitmap bmp)
{
FormatConvertedBitmap fBitmapSource = new FormatConvertedBitmap();
PngBitmapEncoder pngBitmapEncoder = new PngBitmapEncoder();
BitmapImage bi = new BitmapImage();
using (MemoryStream ms = new MemoryStream())
{
bmp.Save(ms, ImageFormat.Png);
bmp.Dispose();
bmp = null;
bi.BeginInit();
bi.StreamSource = ms;
bi.EndInit();
BitmapPalette pallete = new BitmapPalette(bi, 256);
...
Last line
BitmapPalette pallete = new BitmapPalette(bi, 256);
Sometimes throws the following exception
Insufficient memory to continue the execution of the program.at System.Windows.Media.Imaging.BitmapPalette..ctor(BitmapSource bitmapSource, Int32 maxColorCount)
Any ideas ? I clearly have enough memory to continue execution.

There are other sources of OutOfMemoryException in a managed program that don't have anything to do with running out of managed memory. The exception is also raised when it translates error codes returned by legacy native code. Like the E_OUTOFMEMORY error that can be returned by COM method calls. And relevant in your case, by GDI+. Which has only 20 distinct error codes to indicate failure, you'll find them documented in this answer. One of them is OutOfMemory.
Which can mean more than one thing. Running out of unmanaged memory, the kind used by GDI+ to store bitmap pixels is certainly a possibility. It can also mean that your process has run out of available GDI object handles, Windows imposes a handle quota of 10,000 GDI handles. Which is an enormous number btw, exceeding that quota almost always indicates a bug in the code. A handle leak. Which in the case of a managed program is almost always caused by forgetting to use the Image.Dispose() method and not having the garbage collector run often enough to allow the finalizer to release handles.
Sadly it can even be triggered by corrupted bitmap data, not likely in your case since you bomb on allocating the palette. Which indicates a handle leak, which ought to be readily visible in Taskmgr.exe, Processes tab. View + Select columns and tick GDI Objects. Keep an eye on the displayed value for your process while you test it. A steadily increasing number spells trouble, the show is over when it reaches 10,000. Also look at the "Commit size" column, that can show you trouble with consuming too much unmanaged memory.

out of memory exception when use control.BackgroundImage = Image.FromStream(memStream);

I write a code that read a png image from file and show with control.
I want read image from stream and set
control.BackgroundImage = Image.FromStream(memStream);
but when use this code , occur "out of memory" exception. but when use
control.Image = Image.FromStream(memStream);
or
control.BackgroundImage = Image.FromFile(fileSource);
, that is work.
image file size is 5KB.
if (System.IO.File.Exists(imgSource))
{
using (FileStream localFileStream = new FileStream(imgSource, FileMode.Open))
{
using (MemoryStream memStream = new MemoryStream())
{
int bytesRead;
byte[] buffer = new byte[1024];
while ((bytesRead = localFileStream.Read(buffer, 0, buffer.Length)) > 0)
{
memStream.Write(buffer, 0, bytesRead);
}
retIMG = Image.FromStream(memStream);
pictureBox1.Image = retIMG; // is work
label1.Image = retIMG; // is work
button1.Image = retIMG; // is work
button1.BackgroundImage = retIMG; // don't work
groupBox1.BackgroundImage = retIMG; // don't work
panel1.BackgroundImage = retIMG; // don't work
}
}
}
I think a bug in .net framework.
please you help me?

Read the remarks on Image.FromStream on MSDN:
You must keep the stream open for the lifetime of the Image.
So if you remove the using around the creation of your MemoryStream your code works fine.
Of course you should preferrably dispose the MemoryStream once you no longer need the Image you created, although there is likely no harm in this case in not calling Dispose() and leaving it up to the GC to collect it once unused.
The fact that it seems to work with some of your code is likely pure luck and should not be considered a working solution. Always read the documentation to find out about quirks like this.

Giving some background to add to DeCaf's correct answer. GDI+ tries very hard to avoid copying the pixels of a bitmap. That's expensive, bitmaps taking dozens of megabytes is not unusual. When you load a bitmap from a file with the Bitmap constructor or Image.FromFile() then GDI+ creates a memory-mapped file. The pixels are paged-in on demand, only when needed. Very efficient but it puts a lock on the file. Clearly you were trying to avoid that in lock in this code.
You indeed avoid that lock by loading the bytes into memory yourself with a MemoryStream. But the same principle still applies, GDI+ still doesn't copy the pixels and only reads from the stream when it needs to. This goes wrong when you Dispose() the stream. Very hard to diagnose because the exception occurs later, typically when the bitmap needs to be drawn. It bombs in the painting code, you don't have any code to look at but Application.Run(). With a crappy exception message, GDI+ only has a handful of error codes. You are not out of memory, it only looks that way to GDI+, it cannot otherwise figure out why the stream suddenly isn't readable anymore.
At least part of the problem is caused by the very awkward implementation of MemoryStream.Dispose(). Dispose is meant to release unmanaged resources. A memory stream doesn't have any, it only owns memory. That's already taken care of by the garbage collector. Unfortunately they implemented it anyway. Not by actually disposing anything, since there's nothing to dispose, but by marking the MemoryStream unreadable. Which triggers the error in GDI+ when it tries to read while drawing the bitmap.
So simply remove the using statement to avoid disposing the MemoryStream to solve your problem. And don't fret about disposing it later when the bitmap is no longer in use. There's nothing to dispose, the garbage collector automatically frees the memory.

Two things which together resolved this intermittent issue, which has nothing to do with image size.
First, ensure that the image is in RGB mode and definitely not CMYK mode. In our experience, the RGB rendering is actually larger.
Second, erase (and dispose of if possible) any previous image in the image container before loading the new image, such as
Control.Image = Nothing