I am using AForge to capture video from a camera and display it in a WinForms PictureBox. AForge supplies an event that gets triggered on every new frame--so, for my camera, 30 times per second.
I'm finding that even when I am careful to dispose of the PictureBox's image, memory usage climbs rapidly (and in a weird pattern... see image below). My guess is that the event is being called more often than it can dispose of the bitmap, but I'm not sure.
Here is what I have inside the newframe event. I'm using Monitor.TryEnter per this answer, to avoid having events pile up waiting for the PictureBox to unlock. I had hoped that setting the timeout to 1 millisecond would effectively prevent any events from piling up.
private void Camera_NewFrame(object sender, NewFrameEventArgs e)
{
if (Monitor.TryEnter(_pictureBox, millisecondsTimeout: 1))
{
try
{
_pictureBox.Image?.Dispose();
_pictureBox.Image = AForge.Imaging.Image.Clone(newImage);
}
finally
{
Monitor.Exit(_pictureBox);
}
}
}
This is the memory usage pattern that results when I run the code.
Video capture started
Unknown event causes memory usage to increase
Unknown event causes memory usage to stabilize
Unknown event causes memory usage to increase
And that repeats. Note that if I change the TryEnter timeout to a higher value, the stable memory usage periods get shorter.
If anyone can offer a solution for this memory leak, I would be very grateful. I did see a similar question here, but it wasn't really answered: C# Picturebox memory leak
This is not the perfect way to do this, but I found a workaround that solves the problem of skyrocketing memory usage. I basically just limited the frequency with which the PictureBox image can update.
private const int MAX_UPDATE_TIME_MS = 70; // Effectively limits displayed video frame rate. Without this, events pile up faster than we can dispose of the PictureBox's current image, leading to a HUGE memory leak
private DateTime _lastUpdateTimeStamp = DateTime.Now;
private void Camera_NewFrame(object sender, NewFrameEventArgs e)
{
if (DateTime.Now < _lastUpdateTimeStamp.AddMilliseconds(MAX_UPDATE_TIME_MS))
{
return;
}
else
{
_lastUpdateTimeStamp = DateTime.Now;
try
{
UpdatePictureBoxImage(_pictureBox, e.Frame);
}
catch
{
// Do nothing, else PictureBox could freeze
}
}
}
private void UpdatePictureBoxImage(
PictureBox pictureBox,
Bitmap newImage)
{
if (Monitor.TryEnter(pictureBox, millisecondsTimeout: 1))
{
try
{
pictureBox.Image?.Dispose();
pictureBox.Image = AForge.Imaging.Image.Clone(newImage);
}
finally
{
Monitor.Exit(pictureBox);
}
}
}
Additional note specific to AForge
When stopping the video stream, AForge says you should do something like the StopVideoCapture() method I wrote below. I found that the memory AForge was using to control the camera, which was not insignificant, did not get deallocated in a timely way. So I added GC.Collect() to a new method called DisposeVideoSource() that I run when I'm done with the camera.
public void StopVideoCapture()
{
while (_videoCaptureDevice.IsRunning)
{
_videoCaptureDevice.SignalToStop();
_videoCaptureDevice.WaitForStop();
}
}
public void DisposeVideoSource()
{
StopVideoCapture();
GC.Collect();
}
Before adding GC.Collect():
After adding GC.Collect():
Related
I want to recognize ArUco Marker in Unity3D and attach a GameObject to their position. I know there are packages in the Asset Store, but as other people got it working I was looking for an free existing solution or try it by myself.
ArucoUnity The detection worked like a charm, but Unity crashes when accessing the data of the rvecs, tvecs. The error somewhere occur in the Get(int i) in the Vec3d class, when the c++ method au_cv_Vec3d_get(CppPtr, i, CppPtr); is called
OpenCv plus Unity This implementation seems not to be complete as there exist no function to EstimatePoseSingleMarker or similar to get the rvecs and tvecs. Also the last updated was Jan 2019.
As Unity (C#) provides the possibility to access unmanaged code, I followed this great tutorial and for the begging I was able to forward the cv::VideoCaputre stream to Unity. The only problem that occured was a FPS drop to around 35-40 whereas normally I get around 90-100.
The C# code:
void Update()
{
MatToTexture2D();
image.texture = tex;
}
void MatToTexture2D()
{
OpenCVInterop.GetRawImageBytes(pixelPtr);
//Update the Texture2D with array updated in C++
tex.SetPixels32(pixel32);
tex.Apply();
}
The C++ code:
extern "C" void __declspec(dllexport) __stdcall GetRawImageBytes(unsigned char* data)
{
_capture >> _currentFrame;
cv::Mat resizedMat(camHeight, camWidth, _currentFrame.type());
cv::resize(_currentFrame, resizedMat, resizedMat.size(), cv::INTER_CUBIC);
//Convert from RGB to ARGB
cv::Mat argb_img;
cv::cvtColor(resizedMat, argb_img, cv::COLOR_BGR2BGRA);
std::vector<cv::Mat> bgra;
cv::split(argb_img, bgra);
std::swap(bgra[0], bgra[3]);
std::swap(bgra[1], bgra[2]);
std::memcpy(data, argb_img.data, argb_img.total() * argb_img.elemSize());
}
The cause seems to be the first line _capture >> _currentFrame;, but as the others projects have to do the same (at least I guess so), I wonder if there is another reason.
If I don't manage to fix this issue I have to look for alternative approaches.
Just adding to / building on Mars' answer:
For the threading problem I would actually use a thread-save ConcurrentStack<Color32[]>. A stack is "last-in | first-out" so that the first item returned is always the last image data added by the thread.
The thread uses Push(pixel32) to add a new entry with image data
in Update you only use the latest entry (TryPop) for updating the texture.
The rest you ignore (Clear).
So something like
// the OpenCV thread will add(push) entries
// the Unity main thread will work on the entries
private ConcurrentStack<Color32[]> stack = new ConcurrentStack<Color32[]>();
public RawImage image;
public Texture2D tex;
private Thread thread;
void Start()
{
// Wherever you get your tex from
tex = new Texture2D(...);
// it should be enough to do this only once
// the texture stays the same, you only update its content
image.texture = tex;
}
// do things in OnEnable so everytime the object gets enabled start the thread
void OnEnable()
{
stack.Clear();
if(thread != null)
{
thread.Abort();
}
thread = new Thread(MatToTexture2D);
thread.Start();
}
void Update()
{
// here in the main thread work the stack
if (stack.TryPop(out var pixels32))
{
// Only use SetPixels and Apply when really needed
tex.SetPixels32(pixels32);
tex.Apply();
}
// Erase older data
stack.Clear();
}
// Make sure to terminate the thread everytime this object gets disabled
private void OnDisable()
{
if(thread == null) return;
thread.Abort();
thread = null;
}
// Runs in a thread!
void MatToTexture2D()
{
while(true)
{
try
{
// Do what you already have
OpenCVInterop.GetRawImageBytes(pixelPtr);
// However you convert the pixelPtr into Color32
Color32[] pixel32 = GetColorArrayFromPtr(pixelPtr);
// Now add this data to the stack
stack.Push(pixel32);
}
catch (ThreadAbortException ex)
{
// This exception is thrown when calling Abort on the thread
// -> ignore the exception since it is produced on purpose
}
}
}
If I recall correctly, the C++ call to get an image (_capture >> _currentFrame;) is blocking/synchronous, meaning your code won't continue until it actually retrieves the image. You probably want to run your MatToTexture2D code asynchronously.
※This will mean that your frame rate will be higher than your image retrieval rate.
Have your MatToTexture2D function run continuously as needed, updating tex. Then just continue to set your texture to the latest tex, which may be the same value 2-3 frames in a row.
Edit:
#derHugo's answer is much more solid for the programming side, so I'll hide that part. The basic issue is explained above, and derHugo's work-around is much better than my pseudo-code :)
I'm writing a simple optical measurement application using Xamarin for Android and the OpenCV C# bindings library.
In an effort to separate the frame grabber from the processing, I've created some blocking collections to pass around raw, and then processed imagery between different threads. I have an issue where over the period of about 30 seconds, the GUI shows beautifully smooth processed video (15s) down to choppy video (10s), then a crash.
The code below shows the definition of the collections. OnCameraFrame (bottom of the code) shoves each new frame into camframes collection. In OnCreate, I run a task called CamProcessor that takes the frame, does many things, and stuffs it into outframes collection. OnCameraFrame then takes that processed frame and shows it to the GUI. For the purposes of this post and testing, I've completely commented out all my processing, so this issue exists simply by passing raw data through the collections.
One other note is that my collections seem to be running very fast. At no point do I ever have more than 1 frame in there, so it's not an overflow issue (I think).
Can anyone point to why this strategy isn't working well?
BlockingCollection<Mat> camframes = new BlockingCollection<Mat>(10);
BlockingCollection<Mat> outframes = new BlockingCollection<Mat>(10);
public CameraBridgeViewBase mOpenCvCameraView { get; private set; }
protected override void OnCreate(Bundle savedInstanceState)
{
//LayoutStuff
mOpenCvCameraView = FindViewById<CameraBridgeViewBase>(Resource.Id.squish_cam);
Task.Run(() => camProcessor());
}
public void camProcessor()
{
while (!camframes.IsCompleted)
{
Mat frame = new Mat();
try
{
frame = camframes.Take();
}
catch (InvalidOperationException) { }
Mat frameT = frame.T();
Core.Flip(frame.T(), frameT, 1);
Imgproc.Resize(frameT, frameT, frame.Size());
outframes.Add(frameT);
}
}
public Mat OnCameraFrame(CameraBridgeViewBase.ICvCameraViewFrame inputFrame)
{
mRgba = inputFrame.Rgba();
Mat frame = new Mat();
Task.Run(() => camframes.Add(mRgba));
try
{
frame = outframes.Take();
}
catch (InvalidOperationException) { }
return frame;
}
After looking into the Android SDK monitor.bat output, I discovered this was a memory leak. Turns out it's common to the Java openCV wrapper, and is a result of OpenCV's mat heap being much larger than C# expects it to be, so it's not getting garbage collected.
The solution was to append these at every frame grab:
GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
GC.Collect();
GC.WaitForPendingFinalizers();
I am developing C# WPF Auto Number Plate Recognition Using an OCR.
The Flow is, i am getting a pictures from a video stream MJPEG and this images should be passed to the OCR to get the plate number and other details.
The problem is : the Video stream is producing about 30 Frame/second and the CPU can't handle this much of processing also it will take around 1 Sec to process 1 frame, Also when i will get many frames on the Queue the CPU will be 70% used (Intel I7 4th G).
Can anyone suggest solution and better implementation.
//This is the queue where it will hold the frames
// produced from the video streaming(video_Newfram1)
private readonly Queue<byte[]> _anpr1Produces = new Queue<byte[]>();
//I am using AForg.Video to read the MJPEG Streaming
//this event will be triggered for every frame
private void video_NewFrame1(object sender, NewFrameEventArgs eventArgs)
{
var frameDataAnpr = new Bitmap(eventArgs.Frame);
AnprCam1.Source = GetBitmapimage(frameDataAnpr);
//add current fram to the queue
_anpr1Produces.Enqueue(imgByteAnpr);
//this worker is the consumer that will
//take the frames from the queue to the OCR processing
if (!_workerAnpr1.IsBusy)
{
_workerAnpr1.RunWorkerAsync(imgByteAnpr);
}
}
//This is the consumer, it will take the frames from the queue to the OCR
private void WorkerAnpr1_DoWork(object sender, DoWorkEventArgs e)
{
while (true)
{
if (_anpr1Produces.Count <= 0) continue;
BgWorker1(_anpr1Produces.Dequeue());
}
}
//This method will process the frames that sent from the consumer
private void BgWorker1(byte[] imageByteAnpr)
{
var anpr = new cmAnpr("default");
var objgxImage = new gxImage("default");
if (imageByteAnpr != null)
{
objgxImage.LoadFromMem(imageByteAnpr, 1);
if (anpr.FindFirst(objgxImage) && anpr.GetConfidence() >= Configs.ConfidanceLevel)
{
var vehicleNumber = anpr.GetText();
var vehicleType = anpr.GetType().ToString();
if (vehicleType == "0") return;
var imagename = string.Format("{0:yyyy_MMM_dd_HHmmssfff}", currentDateTime) + "-1-" +
vehicleNumber + ".png";
//this task will run async to do the rest of the process which is saving the vehicle image, getting vehicle color, storing to the database ... etc
var tsk = ProcessVehicle("1", vehicleType, vehicleNumber, imageByteAnpr, imagename, currentDateTime, anpr, _anpr1Produces);
}
else
{
GC.Collect();
}
}
}
What you should do is this:
First, figure out if a frame is worth processing or not. If you're using a compressed video stream, you can usually quickly read the frame's compressed size. It stores the difference between the current frame and the previous one.
When it's small, not much changed (i.e: no car drove by).
That's a low-tech way to do motion detection, without even having to decode a frame, and it should be extremely fast.
That way, you can probably decide to skip 80% of the frames in a couple of milliseconds.
Once and a while you'll get frames that need processing. Make sure that you can buffer enough frames so that you can keep recording while you're doing your slow processing.
The next thing to do is find a region of interest, and focus on those first. You could do that by simply looking at areas where the color changed, or try to find rectangular shapes.
Finally, one second of processing is SLOW if you need to process 30 fps. You need to make things faster, or you'll have to build up a gigantic buffer, and hope that you'll ever catch up if it's busy on the road.
Make sure to make proper use of multiple cores if they are available, but in the end, knowing which pieces of the image are NOT relevant is the key to faster performance here.
Before you mark this thread as a duplicate, please consider that I have been attempting to debug the following code for a few days already, and still cannot seem to find a solution.
I'm working on a camera class that has one public method: to return the current frame. In the background, it keeps updating the current frame every time a new frame event occurs (please see code below).
Through commenting code out, I have come to the conclusion that a leak is happening here: this.currentFrame = (Bitmap)eventArgs.Frame.Clone();. Moreover, I have conducted a few memory performance diagnostics in Visual Studio, and it seems the the this.currentFrame object keeps growing in size (seems to behave similarly to a linked list).
I guess my question is why is this.currentFrame growing? I am using the Image.Clone() method so nothing else should be referring to the memory space that it is referring to. In addition, before I update it and I make it refer to null just in case. So why is it not being garbage collected when the this.currentFrame pointer points to a new memory location?
I doubt there is a bug in the AForge code... I know it's probably something small in my code, but I can't see it...
public WebCamClass(VideoCaptureDevice camera)
{
this.cam = camera;
this.currentFrame = new Bitmap(Project.Properties.Resources.defaultImage);
this.cam.NewFrame += new NewFrameEventHandler(cam_NewFrame);
this.cam.Start();
}
private VideoCaptureDevice cam;
private Bitmap currentFrame;
private void cam_NewFrame(object sender, NewFrameEventArgs eventArgs)
{
this.currentFrame = null;
this.currentFrame = (Bitmap)eventArgs.Frame.Clone();
}
public Image saveFrame()
{
return this.currentFrame;
}
I have also tried this with no success:
public Image saveFrame()
{
return (Bitmap)this.currentFrame.Clone();
}
Simple, you just need to use "using", it automatically dispose the bitmap object.
private void cam_NewFrame(object sender, NewFrameEventArgs eventArgs)
{
using(Bitmap bmp = (Bitmap)eventArgs.Frame.Clone())
{
//your code using bmp object
}
}
When I tried the using in a windows forms app the app crashed drawing the screen with a null reference error, in the en I declared Bitmap bmp as a global variable and then on cam_NewFrame put
if (bmp != null) bmp.Dispose();
this fixed the memory leak
I have an application where performance-sensitive drawings occur using a WriteableBitmap. An event is called with CompositionTarget.Rendering to actually update the back buffer of the WriteableBitmap. From the MSDN documentation, that means the event is fired once per frame, right before the control is rendered.
The issue that I am having is that the WriteableBitmap's Lock() function takes an extremely long time, especially at larger bitmap sizes. I have previously read that AddDirtyRegion() has a bug that causes the entire bitmap to invalidate, leading to poor performance. However, that doesn't seem to be the case here. From a good bit of low-level checking, it seems that Lock() opens the bitmap's backbuffer for writing on the render thread, which means every time my event handler is called, it has to thread block until the render thread is ready for it. This leads to a noticeable lag when updating the graphics of the bitmap.
I have already tried adding a timeout to the event handler, using TryLock(), so that it won't block for such a long time and cause the performance degradation. This, however, causes a similar effect in that it appears to lag, because larger numbers of bitmap updates get lumped together.
Here is the relevant code from the event handler to show what exactly I am doing. The UpdatePixels() function was written to avoid using the potentially bugged AddDirtyRect():
void updateBitmap(object sender, EventArgs e)
{
if (!form.ResizingWindow)
{
// Lock and unlock are important... Make sure to keep them outside of the loop for performance reasons.
if (canvasUpdates.Count > 0)
{
//bool locked = scaledDrawingMap.TryLock(bitmapLockDuration);
scaledDrawingMap.Lock();
//if (locked)
//{
unsafe
{
int* pixData = (int*)scaledDrawingMap.BackBuffer;
foreach (Int32Rect i in canvasUpdates)
{
// The graphics object isn't directly shown, so this isn't actually necessary. We do a sort of manual copy from the drawingMap, which acts similarly
// to a back buffer.
Int32Rect temp = GetValidDirtyRegion(i);
UpdatePixels(temp, pixData);
}
scaledDrawingMap.Unlock();
canvasUpdates.Clear();
}
//}
}
}
}
private unsafe void UpdatePixels(Int32Rect temp, int* pixData)
{
//int* pixData = (int*)scaledDrawingMap.BackBuffer;
// Directly copy the backbuffer into a new buffer, to use WritePixels().
var stride = temp.Width * scaledDrawingMap.Format.BitsPerPixel / 8;
int[] relevantPixData = new int[stride * temp.Height];
int srcIdx = 0;
int pWidth = scaledDrawingMap.PixelWidth;
int yLess = temp.Y + temp.Height;
int xLess = temp.X + temp.Width;
for (int y = temp.Y; y < yLess; y++)
{
for (int x = temp.X; x < xLess; x++)
{
relevantPixData[srcIdx++] = pixData[y * pWidth + x];
}
}
scaledDrawingMap.WritePixels(temp, relevantPixData, stride, 0);
}
I can't seem to figure out how to avoid the issue of thread blocking with the WriteableBitmap, and I can't see any obvious faults in the code I have written. Any help or pointers would be much appreciated.
Looks like you are not actually using the BackBuffer to write - only to read.
WritePixels writes to the "front" buffer and does not require a lock.
I don't know if you have some other reason to lock it (other threads doing something), but for the code that's here i don't see why you would need to.
I guess I was wrong about not needing a lock to read from BackBuffer (*pixData) - I thought it was only for writes, but I am positive you do not need to to call Lock for WritePixels.
As far as I can tell, you are doing:
Lock the back buffer
Copy something from it to an array
Call WritePixels using this new array
Unlock the back buffer.
How about switching 3 and 4?
WritePixels may internally cause rendering thread (which has higher priority on the message queue) to get a lock on its behalf which is probably a factor in the delay you are seeing.