Before you mark this thread as a duplicate, please consider that I have been attempting to debug the following code for a few days already, and still cannot seem to find a solution.
I'm working on a camera class that has one public method: to return the current frame. In the background, it keeps updating the current frame every time a new frame event occurs (please see code below).
Through commenting code out, I have come to the conclusion that a leak is happening here: this.currentFrame = (Bitmap)eventArgs.Frame.Clone();. Moreover, I have conducted a few memory performance diagnostics in Visual Studio, and it seems the the this.currentFrame object keeps growing in size (seems to behave similarly to a linked list).
I guess my question is why is this.currentFrame growing? I am using the Image.Clone() method so nothing else should be referring to the memory space that it is referring to. In addition, before I update it and I make it refer to null just in case. So why is it not being garbage collected when the this.currentFrame pointer points to a new memory location?
I doubt there is a bug in the AForge code... I know it's probably something small in my code, but I can't see it...
public WebCamClass(VideoCaptureDevice camera)
{
this.cam = camera;
this.currentFrame = new Bitmap(Project.Properties.Resources.defaultImage);
this.cam.NewFrame += new NewFrameEventHandler(cam_NewFrame);
this.cam.Start();
}
private VideoCaptureDevice cam;
private Bitmap currentFrame;
private void cam_NewFrame(object sender, NewFrameEventArgs eventArgs)
{
this.currentFrame = null;
this.currentFrame = (Bitmap)eventArgs.Frame.Clone();
}
public Image saveFrame()
{
return this.currentFrame;
}
I have also tried this with no success:
public Image saveFrame()
{
return (Bitmap)this.currentFrame.Clone();
}
Simple, you just need to use "using", it automatically dispose the bitmap object.
private void cam_NewFrame(object sender, NewFrameEventArgs eventArgs)
{
using(Bitmap bmp = (Bitmap)eventArgs.Frame.Clone())
{
//your code using bmp object
}
}
When I tried the using in a windows forms app the app crashed drawing the screen with a null reference error, in the en I declared Bitmap bmp as a global variable and then on cam_NewFrame put
if (bmp != null) bmp.Dispose();
this fixed the memory leak
Related
I am using AForge to capture video from a camera and display it in a WinForms PictureBox. AForge supplies an event that gets triggered on every new frame--so, for my camera, 30 times per second.
I'm finding that even when I am careful to dispose of the PictureBox's image, memory usage climbs rapidly (and in a weird pattern... see image below). My guess is that the event is being called more often than it can dispose of the bitmap, but I'm not sure.
Here is what I have inside the newframe event. I'm using Monitor.TryEnter per this answer, to avoid having events pile up waiting for the PictureBox to unlock. I had hoped that setting the timeout to 1 millisecond would effectively prevent any events from piling up.
private void Camera_NewFrame(object sender, NewFrameEventArgs e)
{
if (Monitor.TryEnter(_pictureBox, millisecondsTimeout: 1))
{
try
{
_pictureBox.Image?.Dispose();
_pictureBox.Image = AForge.Imaging.Image.Clone(newImage);
}
finally
{
Monitor.Exit(_pictureBox);
}
}
}
This is the memory usage pattern that results when I run the code.
Video capture started
Unknown event causes memory usage to increase
Unknown event causes memory usage to stabilize
Unknown event causes memory usage to increase
And that repeats. Note that if I change the TryEnter timeout to a higher value, the stable memory usage periods get shorter.
If anyone can offer a solution for this memory leak, I would be very grateful. I did see a similar question here, but it wasn't really answered: C# Picturebox memory leak
This is not the perfect way to do this, but I found a workaround that solves the problem of skyrocketing memory usage. I basically just limited the frequency with which the PictureBox image can update.
private const int MAX_UPDATE_TIME_MS = 70; // Effectively limits displayed video frame rate. Without this, events pile up faster than we can dispose of the PictureBox's current image, leading to a HUGE memory leak
private DateTime _lastUpdateTimeStamp = DateTime.Now;
private void Camera_NewFrame(object sender, NewFrameEventArgs e)
{
if (DateTime.Now < _lastUpdateTimeStamp.AddMilliseconds(MAX_UPDATE_TIME_MS))
{
return;
}
else
{
_lastUpdateTimeStamp = DateTime.Now;
try
{
UpdatePictureBoxImage(_pictureBox, e.Frame);
}
catch
{
// Do nothing, else PictureBox could freeze
}
}
}
private void UpdatePictureBoxImage(
PictureBox pictureBox,
Bitmap newImage)
{
if (Monitor.TryEnter(pictureBox, millisecondsTimeout: 1))
{
try
{
pictureBox.Image?.Dispose();
pictureBox.Image = AForge.Imaging.Image.Clone(newImage);
}
finally
{
Monitor.Exit(pictureBox);
}
}
}
Additional note specific to AForge
When stopping the video stream, AForge says you should do something like the StopVideoCapture() method I wrote below. I found that the memory AForge was using to control the camera, which was not insignificant, did not get deallocated in a timely way. So I added GC.Collect() to a new method called DisposeVideoSource() that I run when I'm done with the camera.
public void StopVideoCapture()
{
while (_videoCaptureDevice.IsRunning)
{
_videoCaptureDevice.SignalToStop();
_videoCaptureDevice.WaitForStop();
}
}
public void DisposeVideoSource()
{
StopVideoCapture();
GC.Collect();
}
Before adding GC.Collect():
After adding GC.Collect():
I want to recognize ArUco Marker in Unity3D and attach a GameObject to their position. I know there are packages in the Asset Store, but as other people got it working I was looking for an free existing solution or try it by myself.
ArucoUnity The detection worked like a charm, but Unity crashes when accessing the data of the rvecs, tvecs. The error somewhere occur in the Get(int i) in the Vec3d class, when the c++ method au_cv_Vec3d_get(CppPtr, i, CppPtr); is called
OpenCv plus Unity This implementation seems not to be complete as there exist no function to EstimatePoseSingleMarker or similar to get the rvecs and tvecs. Also the last updated was Jan 2019.
As Unity (C#) provides the possibility to access unmanaged code, I followed this great tutorial and for the begging I was able to forward the cv::VideoCaputre stream to Unity. The only problem that occured was a FPS drop to around 35-40 whereas normally I get around 90-100.
The C# code:
void Update()
{
MatToTexture2D();
image.texture = tex;
}
void MatToTexture2D()
{
OpenCVInterop.GetRawImageBytes(pixelPtr);
//Update the Texture2D with array updated in C++
tex.SetPixels32(pixel32);
tex.Apply();
}
The C++ code:
extern "C" void __declspec(dllexport) __stdcall GetRawImageBytes(unsigned char* data)
{
_capture >> _currentFrame;
cv::Mat resizedMat(camHeight, camWidth, _currentFrame.type());
cv::resize(_currentFrame, resizedMat, resizedMat.size(), cv::INTER_CUBIC);
//Convert from RGB to ARGB
cv::Mat argb_img;
cv::cvtColor(resizedMat, argb_img, cv::COLOR_BGR2BGRA);
std::vector<cv::Mat> bgra;
cv::split(argb_img, bgra);
std::swap(bgra[0], bgra[3]);
std::swap(bgra[1], bgra[2]);
std::memcpy(data, argb_img.data, argb_img.total() * argb_img.elemSize());
}
The cause seems to be the first line _capture >> _currentFrame;, but as the others projects have to do the same (at least I guess so), I wonder if there is another reason.
If I don't manage to fix this issue I have to look for alternative approaches.
Just adding to / building on Mars' answer:
For the threading problem I would actually use a thread-save ConcurrentStack<Color32[]>. A stack is "last-in | first-out" so that the first item returned is always the last image data added by the thread.
The thread uses Push(pixel32) to add a new entry with image data
in Update you only use the latest entry (TryPop) for updating the texture.
The rest you ignore (Clear).
So something like
// the OpenCV thread will add(push) entries
// the Unity main thread will work on the entries
private ConcurrentStack<Color32[]> stack = new ConcurrentStack<Color32[]>();
public RawImage image;
public Texture2D tex;
private Thread thread;
void Start()
{
// Wherever you get your tex from
tex = new Texture2D(...);
// it should be enough to do this only once
// the texture stays the same, you only update its content
image.texture = tex;
}
// do things in OnEnable so everytime the object gets enabled start the thread
void OnEnable()
{
stack.Clear();
if(thread != null)
{
thread.Abort();
}
thread = new Thread(MatToTexture2D);
thread.Start();
}
void Update()
{
// here in the main thread work the stack
if (stack.TryPop(out var pixels32))
{
// Only use SetPixels and Apply when really needed
tex.SetPixels32(pixels32);
tex.Apply();
}
// Erase older data
stack.Clear();
}
// Make sure to terminate the thread everytime this object gets disabled
private void OnDisable()
{
if(thread == null) return;
thread.Abort();
thread = null;
}
// Runs in a thread!
void MatToTexture2D()
{
while(true)
{
try
{
// Do what you already have
OpenCVInterop.GetRawImageBytes(pixelPtr);
// However you convert the pixelPtr into Color32
Color32[] pixel32 = GetColorArrayFromPtr(pixelPtr);
// Now add this data to the stack
stack.Push(pixel32);
}
catch (ThreadAbortException ex)
{
// This exception is thrown when calling Abort on the thread
// -> ignore the exception since it is produced on purpose
}
}
}
If I recall correctly, the C++ call to get an image (_capture >> _currentFrame;) is blocking/synchronous, meaning your code won't continue until it actually retrieves the image. You probably want to run your MatToTexture2D code asynchronously.
※This will mean that your frame rate will be higher than your image retrieval rate.
Have your MatToTexture2D function run continuously as needed, updating tex. Then just continue to set your texture to the latest tex, which may be the same value 2-3 frames in a row.
Edit:
#derHugo's answer is much more solid for the programming side, so I'll hide that part. The basic issue is explained above, and derHugo's work-around is much better than my pseudo-code :)
I am working on one application where I want to use IP camera for displaying video streaming and and some other major operations on image captured by the IP Camera.
Libraries used in Camera capture
For Camera Capture : Emgu.CV Library
Below is the code which I am using in C#.
Variable Declaration
private Capture capture; //takes images from camera as image frames
private Emgu.CV.UI.ImageBox img; // Dynamic Picture Controls
private int nCam; // no of cameras
Code for Processing Image
private void ProcessFrame(object sender, EventArgs arg)
{
try
{
// Live Streaming Display
Image<Bgr, Byte> ImageFrame = capture.QueryFrame();
// If Ip camera try to reinitialize the IP camera
if(ImageFrame == null)
{
capture.Dispose();
capture = new Capture(URL);
ImageFrame = capture.QueryFrame();
}
ImageFrame = ImageFrame.Resize(img.Width, img.Height, Emgu.CV.CvEnum.INTER.CV_INTER_LINEAR);
img.Image = ImageFrame;
// Here I am doing some other operations like
// 1. Save Image captured from the IP Camera
// 2. Detect faces in Image
// 3. Draw Face markers on Image
// 4. Some database based on result of Face Detection
// 4. Delete image File
// continue Looping for other Ip Cameras
}
catch (NullReferenceException e)
{
}
}
Now, The Problem is after some time the QueryFrame() provide null value and camera Stop streaming.
Can any one tell me why this is happening?
How I can resolve this problem?
If any more information is needed Please Let me know.
Thanks in Advance.
Sorry about the delay but I have provide an example that works with several public IP cameras. It will need the EMGU reference replacing with your current version and the target build directory should be set to "EMGU Version\bin" alternatively extract it to the examples folder.
http://sourceforge.net/projects/emguexample/files/Capture/CameraCapture%20Public%20IP.zip/download
Rather than using the older QueryFrame() method it uses the RetrieveBgrFrame() method. It has worked reasonably well and I have had no null exceptions. However if you do replace the ProcessFrame() method with something like this
You should not be attempting to do any operations if the frame returned is Image is a nullable field and should not have a problem if _capture.RetrieveBgrFrame(); returns null if there is a problem then there is a bigger issue.
private void ProcessFrame(object sender, EventArgs arg)
{
//If you want to access the image data the use the following method call
//Image<Bgr, Byte> frame = new Image<Bgr,byte>(_capture.RetrieveBgrFrame().ToBitmap());
if (RetrieveBgrFrame.Checked)
{
Image<Bgr, Byte> frame = _capture.RetrieveBgrFrame();
//because we are using an autosize picturebox we need to do a thread safe update
if(frame!=null)
{
DisplayImage(frame.ToBitmap());
Image<Bgr, Byte> ImageFrame = frame.Resize(img.Width, img.Height, Emgu.CV.CvEnum.INTER.CV_INTER_LINEAR);
// Here I am doing some other operations like
// 1. Save Image captured from the IP Camera
// 2. Detect faces in Image
// 3. Draw Face markers on Image
// 4. Some database based on result of Face Detection
// 4. Delete image File
// continue Looping for other Ip Cameras
}
//else do nothing as we have no image
}
else if (RetrieveGrayFrame.Checked)
{
Image<Gray, Byte> frame = _capture.RetrieveGrayFrame();
//because we are using an autosize picturebox we need to do a thread safe update
if (frame != null) DisplayImage(frame.ToBitmap());
}
}
On a separate note your comment 'continue Looping for other Ip Cameras' may cause several issues. You should have a new Capture constructor for each camera camera you are using. How many camera are you using? and what public ip camera are you using so I can attempt to replicate the issue? The reason for the separate constructor is that ip cameras take a while to negotiate connections with and constantly Disposing of the original construct and replacing it will play havoc with the garbage collector and introduce no end if timing issues.
Cheers
Chris
[EDIT]
If your camera is returning null frames after a timeout period then I would check to see if there is an issue with the setup or maybe your connection is so slow it disconnects you to reduce lag to others there are various causes but this is not a code problem. You can use c# alone to acquire the data to a bitmap and then pass this to an Image type variable. There is a great article here:
http://www.codeproject.com/Articles/15537/Camera-Vision-video-surveillance-on-C
I've adapted this so you can use a HttpWebRequest as a final check to see if the stream is alive although there are still null exceptions that will be produced here:
using System.Net;
using System.IO;
string url;
private void ProcessFrame(object sender, EventArgs arg)
{
//***If you want to access the image data the use the following method call***/
//Image<Bgr, Byte> frame = new Image<Bgr,byte>(_capture.RetrieveBgrFrame().ToBitmap());
if (RetrieveBgrFrame.Checked)
{
Image<Bgr, Byte> frame = _capture.RetrieveBgrFrame();
//because we are using an autosize picturebox we need to do a thread safe update
if (frame != null)
{
DisplayImage(frame.ToBitmap());
}
else
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
// get response
WebResponse resp = req.GetResponse();
//get stream
Stream stream = resp.GetResponseStream();
if (!stream.CanRead)
{
//try reconnecting the camera
captureButtonClick(null, null); //pause
_capture.Dispose();//get rid
captureButtonClick(null, null); //reconnect
}
}
}
else if (RetrieveGrayFrame.Checked)
{
Image<Gray, Byte> frame = _capture.RetrieveGrayFrame();
//because we are using an autosize picturebox we need to do a thread safe update
if (frame != null) DisplayImage(frame.ToBitmap());
}
}
private void captureButtonClick(object sender, EventArgs e)
{
url = Camera_Selection.SelectedItem.ToString(); //add this
... the rest of the code
}
To display multiple webcams you would create a class to handle the Capture construct and processframe event. Ideally you would raise an purpose built event call that would include a camera identifier as the frameready event call does not call this. I have to make things easier created a form with as a MDI parent and opened an object to manage the capture variables and frame ready event. The Alpha version is available here:
http://sourceforge.net/projects/emguexample/files/Capture/CameraCapture%20Public%20IP%20Multipl%20Display%20Alpha.zip/download
I need to know how to properly dispose of Bitmaps so that I don't have a memory leak.
I am grabbing video in a BackgroundWorker and assigning it to a PictureBox as such:
private void bwVideo_ReadCamera(object sender, DoWorkEventArgs e)
{
Bitmap temp = null;
while (true)
{
Image<Bgr, Byte> frame = logitec.QueryFrame();
if (temp != null)
temp.Dispose();
temp = frame.ToBitmap();
pictureBox2.Image = temp;
}
}
The problem is that I still get a "Out of Memory Exception" with this code. I have tried freeing the pictureBox2.Image variable by using a BackgroundWorker ReportProgress and waiting in the above code for the dispose to finish (you need to be synched with the gui to call dispose on the PictureBox image). I have also tried to use the "Bitmap" property of the Image class which shares data between the Image and Bitmap.
So my questions is, in this situation, what is the proper way to dispose of my image?
You should probably have a using statement for your Image<Bgr, Byte> declaration. See the documentation.
I am using the AForge.NET library to acquire image data from my webcam periodically. For debugging reasons, I have to draw out the filtered images to the screen.
Currently I have 6 WPF Images on my main form, and on every second I handle an event which gives me an UnmanagedImage which I convert into System.Drawing.Bitmap and then to BitmapSource - my code looks something like this:
private void OnImageFiltered(object sender, FilterEventArgs e)
{
var bitmapSource = e.UnmanagedImage.ToManagedImage().ToBitmapSource();
pictureBox.Source = bitmapSource;
}
But as I said before, I have 6 Images, and it slows the whole program down.
How can I make it faster?
My ToBitmapSource extension method is here:
public static BitmapSource ToBitmapSource(this System.Drawing.Image source)
{
System.Drawing.Bitmap bitmap = new System.Drawing.Bitmap(source);
var bitSrc = bitmap.ToBitmapSource();
bitmap.Dispose();
bitmap = null;
return bitSrc;
}
Sorry for writing an answer, but I can't comment yet because of my low reputation.
If the problem still persists, use the Freeze method to make the BitmapSource readonly and therefor passable to a different thread.