I am experiencing a strange memory stack-up in my c# windows form program that occurs all the time on slow PCs, and when the windows form loses focus or is otherwise interrupted on faster PCs.
The program I have written uses Aforge to get images from my webcam, which I then display in an Aforge picturebox control (CurrImagePic in code) in the windows form. The images are switched into the picture box and then disposed at the camera's native framerate, so it appears as video to the user, not still images. The picture box is 1080x1920, but the space for it in the form is smaller and so I allow the user to scroll around the picture.
After about ~30 seconds of memory-stable operation on slower PCs, the problem begins. On faster PCs, the problem only occurs when holding down scroll bar arrows or clicking and dragging around either scroll bar, and if I lock the PC or bring up the Ctrl+Alt+Delete menu.
The problem itself is that memory used by the program starts to increase in very large chunks, leading to an out of memory crash. This is unstoppable on slower PCs, but on the faster PCs if you stop scrolling or return from the lock/Ctrl+alt+delete menu, the program stabilizes at the higher memory usage level. The memory that was accrued during scrolling or while in the lock menu is never collected by the garbage collector. I've even tried to put in a button that forces a GC.collect() when pressed, and it does not reduce this memory usage.
I've run perfmon and found that the memory increase is on the unmanaged heap, but I don't know if it's coming from bitmaps which are not being disposed or what it could be from. It's been impossible to track down since it does not occur except in the above conditions. I've tried various solutions (like moving my image processing out of the event handler and even using both global flags and a "lock" statement to try and ensure that only one thread or frame can access the image processing and displaying method at at time, but I have seen no change. In fact, I am now seeing some unexplained small jumps in memory usage that I wasn't seeing before I put in the lock and moved the processing out of the handler.
Has anyone run into situations like this? I am at a loss for what I need to fix, and I am not getting much help from the Aforge forums. I think the problem is based around my Aforge event handler and image processing method if it is in my code at all - but I also have a suspicion that this is something deeper in the windows form code that I am either misusing or that can't keep up with the demands of my code. Code below:
//Applicable Globals to this code snippet
private bool ALLOWFRAME = true;
private Object FRAMEKEY = new Object();
private VideoCaptureDevice COMPVID;
private Bitmap TMPLTCAP;
private System.Drawing.Image OLDIMAGE;
private bool RCRDPIC = false;
private void COMPVID_NewFrame(object sender, NewFrameEventArgs eventArgs)
{
//Only process a frame when another is done processing
if (ALLOWFRAME == true)
{
ALLOWFRAME = false;
Bitmap PassFrame = AForge.Imaging.Image.Clone(eventArgs.Frame);
ProcessFrame(PassFrame);
PassFrame.Dispose();
}
}
private void ProcessFrame(Bitmap frameIn)
{
lock (FRAMEKEY)
{
if (OLDIMAGE != null) { OLDIMAGE.Dispose(); }
//Call comparison method if flag is set.
if (COMPON == true)
{
Difference TmpltFilter = new Difference(TMPLTCAP);
TmpltFilter.ApplyInPlace(frameIn);
OLDIMAGE = CurrImagePic.Image;
CurrImagePic.Image = AForge.Imaging.Image.Clone(frameIn);
OLDIMAGE.Dispose();
}
else
{
OLDIMAGE = CurrImagePic.Image;
CurrImagePic.Image = AForge.Imaging.Image.Clone(frameIn);
OLDIMAGE.Dispose();
//Toggle the flag back to false to show it's safe (i.e., comparisons have stopped)
//for the result-recording method to copy from the picture box if it is attempting to copy
if (RCRDPIC == true)
{
RCRDPIC = false;
}
}
ALLOWFRAME = true;
}
}
One approach that has often improved performance is to queue up he images in memory and use a timer control to dequeue/display them in a picture box. This way, you get control over proper disposal and allow the NewFrame event to return faster rather than being tied up in image processing.
Also, in the Timer_Tick event, try and do the following:
this.Timer.Stop();
Bitmap image = null;
var temp = this.PictureBox.Image;
lock (FRAMEKEY)
{
if (this.ImageQueue.Any())
{
image = this.ImageQueue.Dequeue();
if (temp != null) { temp.Dispose(); }
}
}
this.PictureBox.Image = image;
if (temp != null) { temp.Dispose(); }
this.Timer.Start();
Related
I'm working on a project using EmguCV which requires capturing images from a camera and processing them, looking for certain things in the image. The processing takes up about ~.2 seconds, and is the biggest time-sink in the application so we're looking into making the capturing/processing different threads to speed up the overall process.
We've already tried the VideoCapture.ImageGrabbed event handler, and calling Retrieve, as well as setting up a threaded loop of our own calling QueryFrame and any other capturing method we can find.
Most of the threaded solutions end up causing empty images (not caught by the .IsEmpty property of the Mat, though, for some reason), which end up as images saved with 0 bytes.
After this, we tried simplifying but ran into an issue where the camera is always a few seconds behind, due to the buffer internal to the library. This leads to my question: is there a way to refresh the buffer, or clear it of memory? We cannot dispose of the capture object because of the time overhead for creating the object. Any tips about threading the capturing and processing are welcome as well, though the other suggestions I've found around this site about similar situations have not led to much success.
A brief example of the code we're using
_capture.ImageGrabbed += GrabFrames;
...
public void GrabFrames()
{
Mat image = new Mat();
_capture.Retrieve(image);
ProcessThread = new Thread(new ParameterizedThreadStart(StartProcess));
ProcessThread.Start(image);
}
...
public void StartProcess(object image)
{
Mat img = (Mat)image;
Process(image);
img.Dispose();
}
I have two high speed USB3 cameras (Ximea) and want to code an application for image recording. Framerates are up to 500fps at VGA resolution but I also want to use the 2Mpx resolution at 170fps.
Their .Net SDK tells me that I should simply "get" the images in a loop.
My problem is that I have no idea how to get the images and save them while still showing a live preview. Everytime I add some code to actually update the picturebox the frame rate drops drastically.
At the moment I utilize a recording function that is called with
Task.Run(() => Record());
and inside the Record() I have a loop getting the bitmaps
while(record == true)
{
Camera.GetImage(out myImage, timeout); //From Ximea .Net SDK
Info = Camera.GetLastImageParams();
Timestamp = Info.GetTimeStamp();
ThreadPool.QueueUserWorkItem(state => SaveImage(myImage, filepath, Timestamp));
}
with the SaveImage being
private void SaveImage(Bitmap myImage, string filepath, double Timestamp)
{
try
{
lock(myImage)
{
myImage.Save(filepath + Timestamp.ToString("0.00000") + ".tif");
}
}
catch{}
}
How can I show a live preview while recording and how can I make the entire code more stable (at the moment there are some dropped frames because of "object already in use"-errors or "generic error in GDI+" at the Image.Save() call, that I skip with the try/catch statement)?
I believe you can tell the Ximea API how many image buffers you want in the incoming queue... use XI_PRM_BUFFER_POLICY and XI_PRM_BUFFERS_QUEUE_SIZE appropriately to make that queue length somewhat long.
Then, have a thread that, when activated, copies an image out of a XI_IMG struct into your own buffer. Activate that thread every n frames (based on the size of the Ximea image buffer queue size)... but don't do any memory copies in the loop that actually calls xiGetImage. You should probably block in your thread to avoid tearing (because the Ximea code could get back around to using the same buffer again if you're not fast enough at copying the data out)... but you could then dynamically adjust the number of buffers so you can finish your copy within the time you have. Also, you may consider copying the image data to ANOTHER buffer if you're doing something that takes a long time...
pseudo-code (sorry, it's C-ish):
// sync objects and a global image buffer pointer
CRITICAL_SECTION cs;
void *buf;
HANDLE ev;
int CopyImageThreadProc(...)
{
while (true)
{
if (WaitOnSingleObject(ev) == WAIT_OBJ_0)
{
EnterCriticalSection(cs);
// copy the image data at buf where ever you want
LeaveCriticalSection(cs);
}
}
}
int main(...)
{
// set up ximea api with appropriate buffering
// create event and critsec, start thread
while (!done)
{
XI_IMG img;
xiGetImage(dev, 10, &img);
// every 15 frames, tell your thread to go...
// if you find that the critsec is causing a hiccup, you can adjust this
// but remember to adjust the queue length, too
// if you change this to TRY entercriticalsection, you can determine that
if ((img.acq_nframe % 15) == 0)
{
EnterCriticalSection(cs);
buf = img.bp;
SetEvent(ev);
LeaveCriticalSection(cs);
}
}
// clean up
}
Add each captured frame to a queue, then have a worker thread that takes those images, one at a time, and saves them. Trying to write multiple images to disk at the same time will most likely be slower. Also, always Dispose of any GDI objects or you will run into trouble really fast. I would think not doing that is what is giving you the exceptions.
As for showing the images, make sure you are not trying to display every image. Your monitor most likely runs at 60 Hz, so anything faster than that will be a waste. I also suspect that (with the performance of GDI), you won't necessarily be able to achieve even that. So I suggest you have a second queue with images to display, and if you see that queue getting too large, your program will need to slow down a bit and not push as many frames to the queue.
Edit: And of course, as #Franck mentions, if your disk can't keep up, your queue/buffer will fill up quickly. Compressing the images might help, assuming they have suitable content for compression and that your processor can keep up.
Edit: What you need is a producer-consumer pattern. There are many ways of doing this, but one might be something like:
// blocking collection
private BlockingCollection<Bitmap> m_Queue = ...
// camera thread
while( run )
{
var bitmap = GrabFrame();
m_Queue.Add( bitmap );
}
// worker thread
try
{
while( true )
{
// Take() will block if the queue is empty
var bitmap = m_Queue.Take();
bitmap.Save( ... );
bitmap.Dispose();
}
catch( InvalidOperationException )
{
// you'll end up here if you call `m_Queue.CompleteAdding()`
// (after the queue has been emptied, of course)
}
As for displaying the images, you could probably use something similar, with the addition of having some code that determines if it's time to push a new image or not.
Currently my program reads images and text in a record from an xml file, displays them on the screen, and then the click of the previous/next buttons moves to the next record. However, it seems to need a few seconds loading time between each photo and I'd like it to be instant, like how Windows Photo Gallery would...or Facebook photos (bear in mind this is not a web app).
I searched found a few similar situations to mine but none seemed to fit my situation. I tried making a class, based on my search, to deal with background loading and calling it in my program, but it's fraught with error and probably won't even do what I want it do:
//ImageManager.cs
class ImageManager
{
private Dictionary<string, Image> images = new Dictionary<string, Image>();
public Image get(string s)
{ // blocking call, returns the image
return load(s);
}
private Image load(string s)
{ // internal, thread-safe helper
lock (images)
{
if (!images.ContainsKey(s))
{
Image img = images.Add(s, img); //load the image s - ERROR cannot implicitly convert type void to image. Void??
return img;
}
return images[s];
}
}
public void preload(params string[] imgs)
{ // non-blocking preloading call
foreach (string img in imgs)
{
BackgroundWorker bw = new BackgroundWorker();
bw.DoWork += (s, e) => { load(img); }; // discard the actual image return
bw.RunWorkerAsync();
}
}
}
//MainWindow.cs
ImageManager im = new ImageManager();
im.preload("Data/Images"); // Errors - im is a field but used like a type/token '('
Many thanks in advance
Your ImageManager should work with ImageSources, not Images. Even if you get your current code to work you'll find that your UI still hangs because you have no choice but to perform the work on the UI thread. If you instead deal with ImageSources, you can load them on a background thread and then freeze them in order to use them from the UI thread. This frees you to pre-emptively load images, or to show a loading animation whilst they load.
BitmapFrame.Create is likely the method you want to be using to load the images.
Consider caching scaled down images - 1:1 of what you want to show, or even smaller. This way loading of preview will be much faster and if user looks at the image long enough you can load full image.
With modern photos original size of the image is usually way bigger than can be normally diaplayed. So if you always read original images you spend large amount of disk IO on something that will never be shown.
Usual note: it may not be case in your program. As with any performance issues measure, than optimize.
public partial class Form1 : Form
{
bool AfterDocumentCompleted = false;
int steps = 0;
public Form1()
{
InitializeComponent();
webBrowser1.DocumentCompleted += new WebBrowserDocumentCompletedEventHandler(DocCompletedHandlerCopy);
webBrowser1.ScriptErrorsSuppressed = true;
}
private void DocCompletedHandlerCopy(object sender, WebBrowserDocumentCompletedEventArgs e)
{
if (webBrowser1.ReadyState == WebBrowserReadyState.Complete && e.Url == webBrowser1.Url)
{
AfterDocumentCompleted = true;
}
}
private void NavigateAndWait(string urlString)
{
AfterDocumentCompleted = false;
webBrowser1.Navigate(urlString);
while (AfterDocumentCompleted == false) Application.DoEvents();
steps += 1;
label1.Text = string.Format("{0:00000} / {1}MB", steps, Environment.WorkingSet / (1024 * 1024));
}
private void button1_Click(object sender, EventArgs e)
{
while (true)
{
NavigateAndWait("http://www.crucial.com/");
NavigateAndWait("http://www.google.com/");
NavigateAndWait("http://www.microsoft.com/");
NavigateAndWait("http://www.stackoverflow.com/");
NavigateAndWait("http://www.yahoo.com/");
}
}
}
When I click Button1 (so it calls button1_Click) and wait for about 1 hr, the ram consumed according to Task Manager and label1 is about 1000MB (is about 20MB as soon as I click it and the rate of grow is somewhat linear). Is the WebBrowser some kind of alpha version of a browser that is not supposed to be used at all by anyone, or am I doing something wrong. If you wonder why in the world do I want to navigate for ever to those pages, this is just the isolation of a problem I was having (see my other question here).
Edit: About a week ago I installed and uninstalled IE9 beta. I think that might have originated the problem. Just tested it on a Windows Vista IE8 and it didn't grow any bigger than 80-90MB. I'll just reinstall Windows and I hope I don't need to downgrade to Windows Vista.
Edit2: I finally found the cause of the problem. It was not IE9 beta. It was the fact that I set IE not to show any pictures. I thought not showing pictures would make navigation faster and lighter, but apparently it activated some bug and the memory consumed started growing like crazy. Still, with pictures the memory grows but a lot slower.
and wait for about 10 minutes, the ram consumed according to Task Manager is about 200MB
200 MB is not a lot
TaskManager is not suitable for measuring memory use
The WebBrowser you use through the control is IE. It will consume quite a bit of memory, mostly to cache content. But you don't really have a problem with memory.
Additional:
I ran your App and changed only this method (add Label1) :
int counter = 0;
private void NavigateAndWait(string urlString)
{
AfterDocumentCompleted = false;
webBrowser1.Navigate(urlString);
while (AfterDocumentCompleted == false) Application.DoEvents();
counter += 1;
label1.Text = string.Format("{0:000} {1}", counter,
Environment.WorkingSet/ (1024*1024));
}
When I let it run for a while Memory stabilizes at a 75-85 MB range.
In a comment you mention 1.6 GB, that is a problem. But that was not with this code. So you have a start: look for the differences.
Edit 2
Upgrade your version of IE. I am running this with Vista/32 and IE8 and the WorkingSet won't go over 90MB. (600 pages)
or am I doing something wrong
Well, this is not going to be a good idea:
while (AfterDocumentCompleted == false) Application.DoEvents();
You're introducing reentrancy and a tight loop in the high priority UI thread. I don't know to what extent that will hamper the garbage collector, but I wouldn't be surprised if it did. Basically you're abusing the UI thread, which is bound to cause oddities. (As Henk says, 200MB in 10 minutes isn't exactly a runaway resource leak anyway.)
This just isn't a good idea. "Isolating" a problem by adding code like this isn't going to help.
You already have a DocumentCompleted handler... why not use that to navigate to the next page, instead of this tight loop?
I'm not sure why it would be unclear why you are using lots of memory; you are reloading web pages over and over again endlessly. Right now my browser is using 140megs of memory, and I'm not reloading pages constantly.
I have a C# desktop application in which one thread that I create continously gets an image from a source(it's a digital camera actually) and puts it on a panel(panel.Image = img) in the GUI(which must be another thread as it is the code-behind of a control.
The application works but on some machines I get the following error at random time intervals(unpredictable)
************** Exception Text **************
System.InvalidOperationException: The object is currently in use elsewhere.
Then the panel turns into a red cross, red X - i think this is the invalid picture icon that is editable from the properties. The application keeps working but the panel is never updated.
From what I can tell this error comes from the control's onpaint event where I draw something else on the picture.
I tried using a lock there but no luck :(
The way I call the function that puts the image on the panel is as follows:
if (this.ReceivedFrame != null)
{
Delegate[] clients = this.ReceivedFrame.GetInvocationList();
foreach (Delegate del in clients)
{
try
{
del.DynamicInvoke(new object[] { this,
new StreamEventArgs(frame)} );
}
catch { }
}
}
this is the delegate:
public delegate void ReceivedFrameEventHandler(object sender, StreamEventArgs e);
public event ReceivedFrameEventHandler ReceivedFrame;
and this is how the function inside the control code-behind registers to it:
Camera.ReceivedFrame +=
new Camera.ReceivedFrameEventHandler(camera_ReceivedFrame);
I also tried
del.Method.Invoke(del.Target, new object[] { this, new StreamEventArgs(b) });
instead of
del.DynamicInvoke(new object[] { this, new StreamEventArgs(frame) });
but no luck
Does anyone know how I could fix this error or at least catch the error somehow and make the thread put the images on the panel once again?
This is because Gdi+ Image class is not thread safe. Hovewer you can avoid InvalidOperationException by using lock every time when you need to Image access, for example for painting or getting image size:
Image DummyImage;
// Paint
lock (DummyImage)
e.Graphics.DrawImage(DummyImage, 10, 10);
// Access Image properties
Size ImageSize;
lock (DummyImage)
ImageSize = DummyImage.Size;
BTW, invocation is not needed, if you will use the above pattern.
I had a similar problem with the same error message but try as I might, locking the bitmap didn't fix anything for me. Then I realized I was drawing a shape using a static brush. Sure enough, it was the brush that was causing the thread contention.
var location = new Rectangle(100, 100, 500, 500);
var brush = MyClass.RED_BRUSH;
lock(brush)
e.Graphics.FillRectangle(brush, location);
This worked for my case and lesson learned: Check all the reference types being used at the point where thread contention is occurring.
Seems to me, that the same Camera object is used several times.
E.g. try to use a new buffer for each received frame. It seems to me, that while the picture box is drawing the new frame, your capture library fills that buffer again. Therefore on faster machines this might not be an issue, with slower machines it might be an issue.
I've programmed something similar once, after each received frame, we had to request to receive the next frame and set the NEW frame receive buffer in that request.
If you can not do that, copy the received frame from the camera first to a new buffer and append that buffer to a queue, or just use 2 alternating buffers and check for overruns. Either use myOutPutPanel.BeginInvoke to call the camera_ReceivedFrame method, or better have a thread running, which checks the queue, when it has a new entry it calls mnyOutPutPanel.BeginInvoke to invoke your method to set the new buffer as image on the panel.
Furthermore, once you received the buffer, use the Panel Invoke Method to invoke the setting of the image (guarantee that it runs in the window thread and not the thread from your capture library).
The example below can be called from any thread (capture library or other separate thread):
void camera_ReceivedFrame(object sender, StreamEventArgs e)
{
if(myOutputPanel.InvokeRequired)
{
myOutPutPanel.BeginInvoke(
new Camera.ReceivedFrameEventHandler(camera_ReceivedFrame),
sender,
e);
}
else
{
myOutPutPanel.Image = e.Image;
}
}
I think this is multithreading problem
Use windows golden rule and update the panel in the main thread use panel.Invoke
This should overcome cross threading exception