We Keep Coding

C# PictureBox: Memory leak when updating multiple times per second

I am using AForge to capture video from a camera and display it in a WinForms PictureBox. AForge supplies an event that gets triggered on every new frame--so, for my camera, 30 times per second.
I'm finding that even when I am careful to dispose of the PictureBox's image, memory usage climbs rapidly (and in a weird pattern... see image below). My guess is that the event is being called more often than it can dispose of the bitmap, but I'm not sure.
Here is what I have inside the newframe event. I'm using Monitor.TryEnter per this answer, to avoid having events pile up waiting for the PictureBox to unlock. I had hoped that setting the timeout to 1 millisecond would effectively prevent any events from piling up.
private void Camera_NewFrame(object sender, NewFrameEventArgs e)
{
if (Monitor.TryEnter(_pictureBox, millisecondsTimeout: 1))
{
try
{
_pictureBox.Image?.Dispose();
_pictureBox.Image = AForge.Imaging.Image.Clone(newImage);
}
finally
{
Monitor.Exit(_pictureBox);
}
}
}
This is the memory usage pattern that results when I run the code.
Video capture started
Unknown event causes memory usage to increase
Unknown event causes memory usage to stabilize
Unknown event causes memory usage to increase
And that repeats. Note that if I change the TryEnter timeout to a higher value, the stable memory usage periods get shorter.
If anyone can offer a solution for this memory leak, I would be very grateful. I did see a similar question here, but it wasn't really answered: C# Picturebox memory leak

This is not the perfect way to do this, but I found a workaround that solves the problem of skyrocketing memory usage. I basically just limited the frequency with which the PictureBox image can update.
private const int MAX_UPDATE_TIME_MS = 70; // Effectively limits displayed video frame rate. Without this, events pile up faster than we can dispose of the PictureBox's current image, leading to a HUGE memory leak
private DateTime _lastUpdateTimeStamp = DateTime.Now;
private void Camera_NewFrame(object sender, NewFrameEventArgs e)
{
if (DateTime.Now < _lastUpdateTimeStamp.AddMilliseconds(MAX_UPDATE_TIME_MS))
{
return;
}
else
{
_lastUpdateTimeStamp = DateTime.Now;
try
{
UpdatePictureBoxImage(_pictureBox, e.Frame);
}
catch
{
// Do nothing, else PictureBox could freeze
}
}
}
private void UpdatePictureBoxImage(
PictureBox pictureBox,
Bitmap newImage)
{
if (Monitor.TryEnter(pictureBox, millisecondsTimeout: 1))
{
try
{
pictureBox.Image?.Dispose();
pictureBox.Image = AForge.Imaging.Image.Clone(newImage);
}
finally
{
Monitor.Exit(pictureBox);
}
}
}
Additional note specific to AForge
When stopping the video stream, AForge says you should do something like the StopVideoCapture() method I wrote below. I found that the memory AForge was using to control the camera, which was not insignificant, did not get deallocated in a timely way. So I added GC.Collect() to a new method called DisposeVideoSource() that I run when I'm done with the camera.
public void StopVideoCapture()
{
while (_videoCaptureDevice.IsRunning)
{
_videoCaptureDevice.SignalToStop();
_videoCaptureDevice.WaitForStop();
}
}
public void DisposeVideoSource()
{
StopVideoCapture();
GC.Collect();
}
Before adding GC.Collect():
After adding GC.Collect():

OpenCV dll calls from Unity3D lead to FPS drop

I want to recognize ArUco Marker in Unity3D and attach a GameObject to their position. I know there are packages in the Asset Store, but as other people got it working I was looking for an free existing solution or try it by myself.
ArucoUnity The detection worked like a charm, but Unity crashes when accessing the data of the rvecs, tvecs. The error somewhere occur in the Get(int i) in the Vec3d class, when the c++ method au_cv_Vec3d_get(CppPtr, i, CppPtr); is called
OpenCv plus Unity This implementation seems not to be complete as there exist no function to EstimatePoseSingleMarker or similar to get the rvecs and tvecs. Also the last updated was Jan 2019.
As Unity (C#) provides the possibility to access unmanaged code, I followed this great tutorial and for the begging I was able to forward the cv::VideoCaputre stream to Unity. The only problem that occured was a FPS drop to around 35-40 whereas normally I get around 90-100.
The C# code:
void Update()
{
MatToTexture2D();
image.texture = tex;
}
void MatToTexture2D()
{
OpenCVInterop.GetRawImageBytes(pixelPtr);
//Update the Texture2D with array updated in C++
tex.SetPixels32(pixel32);
tex.Apply();
}
The C++ code:
extern "C" void __declspec(dllexport) __stdcall GetRawImageBytes(unsigned char* data)
{
_capture >> _currentFrame;
cv::Mat resizedMat(camHeight, camWidth, _currentFrame.type());
cv::resize(_currentFrame, resizedMat, resizedMat.size(), cv::INTER_CUBIC);
//Convert from RGB to ARGB
cv::Mat argb_img;
cv::cvtColor(resizedMat, argb_img, cv::COLOR_BGR2BGRA);
std::vector<cv::Mat> bgra;
cv::split(argb_img, bgra);
std::swap(bgra[0], bgra[3]);
std::swap(bgra[1], bgra[2]);
std::memcpy(data, argb_img.data, argb_img.total() * argb_img.elemSize());
}
The cause seems to be the first line _capture >> _currentFrame;, but as the others projects have to do the same (at least I guess so), I wonder if there is another reason.
If I don't manage to fix this issue I have to look for alternative approaches.

Just adding to / building on Mars' answer:
For the threading problem I would actually use a thread-save ConcurrentStack<Color32[]>. A stack is "last-in | first-out" so that the first item returned is always the last image data added by the thread.
The thread uses Push(pixel32) to add a new entry with image data
in Update you only use the latest entry (TryPop) for updating the texture.
The rest you ignore (Clear).
So something like
// the OpenCV thread will add(push) entries
// the Unity main thread will work on the entries
private ConcurrentStack<Color32[]> stack = new ConcurrentStack<Color32[]>();
public RawImage image;
public Texture2D tex;
private Thread thread;
void Start()
{
// Wherever you get your tex from
tex = new Texture2D(...);
// it should be enough to do this only once
// the texture stays the same, you only update its content
image.texture = tex;
}
// do things in OnEnable so everytime the object gets enabled start the thread
void OnEnable()
{
stack.Clear();
if(thread != null)
{
thread.Abort();
}
thread = new Thread(MatToTexture2D);
thread.Start();
}
void Update()
{
// here in the main thread work the stack
if (stack.TryPop(out var pixels32))
{
// Only use SetPixels and Apply when really needed
tex.SetPixels32(pixels32);
tex.Apply();
}
// Erase older data
stack.Clear();
}
// Make sure to terminate the thread everytime this object gets disabled
private void OnDisable()
{
if(thread == null) return;
thread.Abort();
thread = null;
}
// Runs in a thread!
void MatToTexture2D()
{
while(true)
{
try
{
// Do what you already have
OpenCVInterop.GetRawImageBytes(pixelPtr);
// However you convert the pixelPtr into Color32
Color32[] pixel32 = GetColorArrayFromPtr(pixelPtr);
// Now add this data to the stack
stack.Push(pixel32);
}
catch (ThreadAbortException ex)
{
// This exception is thrown when calling Abort on the thread
// -> ignore the exception since it is produced on purpose
}
}
}

If I recall correctly, the C++ call to get an image (_capture >> _currentFrame;) is blocking/synchronous, meaning your code won't continue until it actually retrieves the image. You probably want to run your MatToTexture2D code asynchronously.
※This will mean that your frame rate will be higher than your image retrieval rate.
Have your MatToTexture2D function run continuously as needed, updating tex. Then just continue to set your texture to the latest tex, which may be the same value 2-3 frames in a row.
Edit:
#derHugo's answer is much more solid for the programming side, so I'll hide that part. The basic issue is explained above, and derHugo's work-around is much better than my pseudo-code :)

How to extract timestamps from each frame obtained by USB camera?

This scene is common in real time video proccessing. And I need timestamps to synchronize with other devices.
I have tried cv::VideoCapture, but it can not extract the timestamps frome video stream.
So I have two questions here:
Does video stream provided by USB camera indeed contains the timestamp information ?
If it has. What should I do to extract it ? A C# solution is best, while C++ is OK.
Addition:
Using these two properties doesn't work:
secCounter = (long) cap.get(CAP_PROP_POS_MSEC);
frameNumber = (long) cap.get(CAP_PROP_POS_FRAMES);
It always gives the following result:
VIDEOIO ERROR: V4L2: getting property #1 is not supported
msecCounter = 0
frameNumber = -1

OpenCV's VideoCapture class is a very high level interface to retrieve frames from a camera, so it "hides" a lot of the details that are necessary to connect to the camera, retrieve frames from the camera, and decode those frames in to a useful color space like BGR. This is nice because you don't have to worry about the details of grabbing frames, but the downside is that you don't have direct access to other data you might want, like the frame number or frame timestamp. That doesn't mean it's impossible to get the data you want, though!
Here's a sample frame grabbing loop that will get you what you want, loosely based on the example code from here. This is in C++.
#include "opencv2/opencv.hpp"
using namespace cv;
int main(int, char**)
{
VideoCapture cap(0); // open the default camera
if(!cap.isOpened()) // check if we succeeded
return -1;
// TODO: change the width, height, and capture FPS to your desired
// settings.
cap.set(CAP_PROP_FRAME_WIDTH, 1920);
cap.set(CAP_PROP_FRAME_HEIGHT, 1080);
cap.set(CAP_PROP_FPS, 30);
Mat frame;
long msecCounter = 0;
long frameNumber = 0;
for(;;)
{
// Instead of cap >> frame; we'll do something different.
//
// VideoCapture::grab() tells OpenCV to grab a frame from
// the camera, but to not worry about all the color conversion
// and processing to convert that frame into BGR.
//
// This means there's less processing overhead, so the time
// stamp will be more accurate because we are fetching it
// immediately after.
//
// grab() should also wait for the next frame to be available
// based on the capture FPS that is set, so it's okay to loop
// continuously over it.
if(cap.grab())
{
msecCounter = (long) cap.get(CAP_PROP_POS_MSEC);
frameNumber = (long) cap.get(CAP_PROP_POS_FRAMES);
// VideoCapture::retrieve color converts the image and places
// it in the Mat that you provide.
if(cap.retrieve(&frame))
{
// Pass the frame and parameters to your processing
// method.
ProcessFrame(&frame, msecCounter, frameNumber);
}
}
// TODO: Handle your loop termination condition here
}
// the camera will be deinitialized automatically in VideoCapture destructor
return 0;
}
void ProcessFrame(Mat& frame, long msecCounter, long frameNumber)
{
// TODO: Make a copy of frame if you are going to process it
// asynchronously or put it in a buffer or queue and then return
// control from this function. This is because the reference Mat
// being passed in is "owned" by the processing loop, and on each
// iteration it will be destructed, so any references to it will be
// invalid. Hence, if you do any work async, you need to copy frame.
//
// If all your processing happens synchronously in this function,
// you don't need to make a copy first because the loop is waiting
// for this function to return.
// TODO: Your processing logic goes here.
}
If you're using C# and Emgu CV it will look a bit different. I haven't tested this code, but it should work or be very close to the solution.
using System;
using Emgu.CV;
using Emgu.CV.CvEnum;
static class Program
{
[STAThread]
static void Main()
{
VideoCapture cap = new VideoCapture(0);
if(!cap.IsOpened)
{
return;
}
cap.SetCaptureProperty(CapProp.FrameWidth, 1920);
cap.SetCaptureProperty(CapProp.FrameHeight, 1080);
cap.SetCaptureProperty(CapProp.Fps, 30);
Mat frame = new Mat();
long msecCounter = 0;
long frameNumber = 0;
for(;;)
{
if(cap.Grab())
{
msecCounter = (long) cap.GetCaptureProperty(CapProp.PosMsec);
frameNumber = (long) cap.GetCaptureProperty(CapProp.PosFrames);
if(cap.Retrieve(frame))
{
ProcessFrame(frame, msecCounter, frameNumber);
}
}
// TODO: Determine when to quit the processing loop
}
}
private static void ProcessFrame(Mat frame, long msecCounter, long frameNumber)
{
// Again, copy frame here if you're going to queue the frame or
// do any async processing on it.
// TODO: Your processing code goes here.
}
}
Emgu's VideoCapture implementation also allows for asynchronous Grab operations to be done for you, and notifications when a grabbed frame is ready to be used with Retrieve. That looks like this:
using System;
using Emgu.CV;
using Emgu.CV.CvEnum;
static class Program
{
private static Mat s_frame;
private static VideoCapture s_cap;
private static object s_retrieveLock = new object();
[STAThread]
static void Main()
{
s_cap = new VideoCapture(0);
if(!s_cap.IsOpened)
{
return;
}
s_frame = new Mat();
s_cap.SetCaptureProperty(CapProp.FrameWidth, 1920);
s_cap.SetCaptureProperty(CapProp.FrameHeight, 1080);
s_cap.SetCaptureProperty(CapProp.Fps, 30);
s_cap.ImageGrabbed += FrameIsReady;
s_cap.Start();
// TODO: Wait here until you're done with the capture process,
// the same way you'd determine when to exit the for loop in the
// above example.
s_cap.Stop();
s_cap.ImageGrabbed -= FrameIsReady;
}
private static void FrameIsReady(object sender, EventArgs e)
{
// This function is being called from VideoCapture's thread,
// so if you rework this code to run with a UI, be very careful
// about updating Controls here because that needs to be Invoke'd
// back to the UI thread.
// I used a lock here to be extra careful and protect against
// re-entrancy, but this may not be necessary if Emgu's
// VideoCapture thread blocks for completion of this event
// handler.
lock(s_retrieveLock)
{
msecCounter = (long) s_cap.GetCaptureProperty(CapProp.PosMsec);
frameNumber = (long) s_cap.GetCaptureProperty(CapProp.PosFrames);
if(s_cap.Retrieve(s_frame))
{
ProcessFrame(s_frame, msecCounter, frameNumber);
}
}
}
private static void ProcessFrame(Mat frame, long msecCounter, long frameNumber)
{
// Again, copy frame here if you're going to queue the frame or
// do any async processing on it.
// TODO: Your processing code goes here.
}
}

reduce CPU overhead while proccessing Video Stream

I am developing C# WPF Auto Number Plate Recognition Using an OCR.
The Flow is, i am getting a pictures from a video stream MJPEG and this images should be passed to the OCR to get the plate number and other details.
The problem is : the Video stream is producing about 30 Frame/second and the CPU can't handle this much of processing also it will take around 1 Sec to process 1 frame, Also when i will get many frames on the Queue the CPU will be 70% used (Intel I7 4th G).
Can anyone suggest solution and better implementation.
//This is the queue where it will hold the frames
// produced from the video streaming(video_Newfram1)
private readonly Queue<byte[]> _anpr1Produces = new Queue<byte[]>();
//I am using AForg.Video to read the MJPEG Streaming
//this event will be triggered for every frame
private void video_NewFrame1(object sender, NewFrameEventArgs eventArgs)
{
var frameDataAnpr = new Bitmap(eventArgs.Frame);
AnprCam1.Source = GetBitmapimage(frameDataAnpr);
//add current fram to the queue
_anpr1Produces.Enqueue(imgByteAnpr);
//this worker is the consumer that will
//take the frames from the queue to the OCR processing
if (!_workerAnpr1.IsBusy)
{
_workerAnpr1.RunWorkerAsync(imgByteAnpr);
}
}
//This is the consumer, it will take the frames from the queue to the OCR
private void WorkerAnpr1_DoWork(object sender, DoWorkEventArgs e)
{
while (true)
{
if (_anpr1Produces.Count <= 0) continue;
BgWorker1(_anpr1Produces.Dequeue());
}
}
//This method will process the frames that sent from the consumer
private void BgWorker1(byte[] imageByteAnpr)
{
var anpr = new cmAnpr("default");
var objgxImage = new gxImage("default");
if (imageByteAnpr != null)
{
objgxImage.LoadFromMem(imageByteAnpr, 1);
if (anpr.FindFirst(objgxImage) && anpr.GetConfidence() >= Configs.ConfidanceLevel)
{
var vehicleNumber = anpr.GetText();
var vehicleType = anpr.GetType().ToString();
if (vehicleType == "0") return;
var imagename = string.Format("{0:yyyy_MMM_dd_HHmmssfff}", currentDateTime) + "-1-" +
vehicleNumber + ".png";
//this task will run async to do the rest of the process which is saving the vehicle image, getting vehicle color, storing to the database ... etc
var tsk = ProcessVehicle("1", vehicleType, vehicleNumber, imageByteAnpr, imagename, currentDateTime, anpr, _anpr1Produces);
}
else
{
GC.Collect();
}
}
}

What you should do is this:
First, figure out if a frame is worth processing or not. If you're using a compressed video stream, you can usually quickly read the frame's compressed size. It stores the difference between the current frame and the previous one.
When it's small, not much changed (i.e: no car drove by).
That's a low-tech way to do motion detection, without even having to decode a frame, and it should be extremely fast.
That way, you can probably decide to skip 80% of the frames in a couple of milliseconds.
Once and a while you'll get frames that need processing. Make sure that you can buffer enough frames so that you can keep recording while you're doing your slow processing.
The next thing to do is find a region of interest, and focus on those first. You could do that by simply looking at areas where the color changed, or try to find rectangular shapes.
Finally, one second of processing is SLOW if you need to process 30 fps. You need to make things faster, or you'll have to build up a gigantic buffer, and hope that you'll ever catch up if it's busy on the road.
Make sure to make proper use of multiple cores if they are available, but in the end, knowing which pieces of the image are NOT relevant is the key to faster performance here.

parallel image processing artifacts

I capture images from a webcam, do some heavy processing on them, and then show the result. To keep the framerate high, i want to have the processing of different frames run in parallel.
So, I have a 'Producer', which captures the images and adds these to the 'inQueue'; also it takes an image from the 'outQueue' and displays it:
public class Producer
{
Capture capture;
Queue<Image<Bgr, Byte>> inQueue;
Queue<Image<Bgr, Byte>> outQueue;
Object lockObject;
Emgu.CV.UI.ImageBox screen;
public int frameCounter = 0;
public Producer(Emgu.CV.UI.ImageBox screen, Capture capture, Queue<Image<Bgr, Byte>> inQueue, Queue<Image<Bgr, Byte>> outQueue, Object lockObject)
{
this.screen = screen;
this.capture = capture;
this.inQueue = inQueue;
this.outQueue = outQueue;
this.lockObject = lockObject;
}
public void produce()
{
while (true)
{
lock (lockObject)
{
inQueue.Enqueue(capture.QueryFrame());
if (inQueue.Count == 1)
{
Monitor.PulseAll(lockObject);
}
if (outQueue.Count > 0)
{
screen.Image = outQueue.Dequeue();
}
}
frameCounter++;
}
}
}
There are different 'Consumers' who take an image from the inQueue, do some processing, and add them to the outQueue:
public class Consumer
{
Queue<Image<Bgr, Byte>> inQueue;
Queue<Image<Bgr, Byte>> outQueue;
Object lockObject;
string name;
Image<Bgr, Byte> image;
public Consumer(Queue<Image<Bgr, Byte>> inQueue, Queue<Image<Bgr, Byte>> outQueue, Object lockObject, string name)
{
this.inQueue = inQueue;
this.outQueue = outQueue;
this.lockObject = lockObject;
this.name = name;
}
public void consume()
{
while (true)
{
lock (lockObject)
{
if (inQueue.Count == 0)
{
Monitor.Wait(lockObject);
continue;
}
image = inQueue.Dequeue();
}
// Do some heavy processing with the image
lock (lockObject)
{
outQueue.Enqueue(image);
}
}
}
}
Rest of the important code is this section:
private void Form1_Load(object sender, EventArgs e)
{
Consumer[] c = new Consumer[consumerCount];
Thread[] t = new Thread[consumerCount];
Object lockObj = new object();
Queue<Image<Bgr, Byte>> inQueue = new Queue<Image<Bgr, Byte>>();
Queue<Image<Bgr, Byte>> outQueue = new Queue<Image<Bgr, Byte>>();
p = new Producer(screen1, capture, inQueue, outQueue, lockObj);
for (int i = 0; i < consumerCount; i++)
{
c[i] = new Consumer(inQueue, outQueue, lockObj, "c_" + Convert.ToString(i));
}
for (int i = 0; i < consumerCount; i++)
{
t[i] = new Thread(c[i].consume);
t[i].Start();
}
Thread pt = new Thread(p.produce);
pt.Start();
}
The parallelisation actually works fine, I do get a linear speed increase with each added thread (up to a certain point of course). The problem is that I get artifacts in the output, even if running only one thread. The artifacts look like part of the picture is not in the right place.
Example of the artifact (this is without any processing to keep it clear, but the effect is the same)
Any ideas what causes this?
Thanks

Displaimer: This post isn't supposed to fully describe an answer, but instead give some hints on why the artifact is being shown.
A quick analysis show that the the actifact is, in fact, a partial, vertically mirrored snippet of a frame. I copied it, mirrored, and placed it back over the image, and added an awful marker to show its placement:
Two things immediately come to attention:
The artifact is roughly positioned on the 'correct' place it would be, only that the position is also vertically mirrored;
The image is slightly different, indicating that it may belong to a different frame.
It's been a while since I played around with raw capture and ran into a similar issue, but I remember that depending on how the driver is implemented (or set up - this particular issue happened when setting a specific imaging device for interlaced capture) it may fill its framebuffer alternating between 'top-down' and 'bottom-up' scans - as soon as the frame is full, the 'cursor' reverts direction.
It seems to me that you're running into a race condition/buffer underrun situation, where the transfer from the framebuffer to your application is happening before the full frame is transferred by the device.
In that case, you'd receive a partial image, and the area still not refreshed would show a bit of the previously transferred frame.
If I'd have to bet, I'd say that the artifact may appear on sequential order, not on the same position but 'fluctuating' on a specific direction (up or down), but always as a mirrored bit.

Well, I think the problem is here . The section of code is not guarantee that you will be access by one thread in here between two queue. The image is pop by inQueue is not actually received in order in outQueue
while (true)
{
lock (lockObject)
{
if (inQueue.Count == 0)
{
Monitor.Wait(lockObject);
continue;
}
image = inQueue.Dequeue();
}
// Do some heavy processing with the image
lock (lockObject)
{
outQueue.Enqueue(image);
}
}

Similar to #OnoSendai, I'm not trying to solve the exact problem as stated. I would have to write an app and I just don't have the time. But, the two things that I would change right away would be to use the ConcurrentQueue class so that you have thread-safety. And, I would use the Task library functions in order to create parallel tasks on different processor cores. These are found in the System.Net and System.Net.Task namespaces.
Also, vertically flipping a chunk like that looks like more than an artifact to me. If it also happens when executing in a single thread as you mentioned, then I would definitely re-focus on the "heavy processing" part of the equation.
Good luck! Take care.

You may have two problems:
1) parallism doesn't ensure that images are added to the out queue in the right order. I imagine that displaying image 8 before image 6 and 7 can produce some artifacts. In consumer thread, you have to wait previous consumer have posted its image to the out queue to post next image. Tasks can help greatly for that because of their inherent synchronisation mecanism.
2) You may also have problems in the rendering code.