We Keep Coding

How to minimize the lag from the live stream using OpenCV/EmguCV WPF

I am using EmguCV to get a live stream from a high-resolution webcam. (full HD)
The issue is that there is a significant lag in the video.
I compared with the Windows camera app and my application is much more delayed.
Here is the code snippet I am using to get the live stream and show on the canvas.
I am wondering if I am missing anything important to minimize the lag.
If anybody is experienced, please help me.
void init_camera()
{
m_capture= new VideoCapture(0);
m_capture.ImageGrabbed += ProcessFrame;
}
void ProcessFrame()
{
if (m_capture_in_progress)
return;
m_capture_in_progress = true;
if (m_capture != null && m_capture.Ptr != IntPtr.Zero)
{
m_capture.Retrieve(m_frame, 0);
if (m_frame != null && m_frame.Bitmap != null)
{
if (IsRecording && m_video_writer != null && m_video_writer.IsOpened)
{
try
{
m_video_writer.Write(m_frame);
}
catch (Exception ex)
{
log(ex.Message + "\n" + ex.StackTrace);
}
}
this.Dispatcher.Invoke(() =>
{
using (var stream = new MemoryStream())
{
m_frame.Bitmap.Save(stream, System.Drawing.Imaging.ImageFormat.Jpeg);
using (var outstream = new MemoryStream(stream.ToArray()))
{
BitmapImage bitmap = new BitmapImage();
bitmap.BeginInit();
bitmap.StreamSource = new MemoryStream(stream.ToArray());
bitmap.CacheOption = BitmapCacheOption.OnLoad;
bitmap.EndInit();
ui_canvas.Background = new ImageBrush(bitmap);
}
};
});
}
}
m_capture_in_progress = false;
}

OP is referring some kind of laggs while drawing a camera capture stream from EmguCv into canvas wpf control. OP also states that lagg disappears when streaming with third party software.
This is caused by low performance in OP's code.
After wataching the posted code sample I would suggest the following improvements:
Increasing FPS rate from EmguCV parameters
Maybe your device is supporting higher fps rates, try to increase this value.
Drawing the bitmap over another control like image
As far as I know, drawing a bitmap over a canvas is slower than drawing it over an imagecontrol. A good solution would be to overlap both controls so you can paint the frames over the imageand draw your shapes over the canvasas they are well overlapped.
Cleaning up the rendering method
Keep simple and try to avoid so much control structures when you are inside a critical program block. Specially in program blocks that are executed when an external devices fires an event.
Fixing possible program lock
As far as I can see, you are locking the ProcessFrame event with a bool variable called m_capture_in_progress which is set to true while you are drawing the frame and it is freed after you finish the drawing. This can cause that next incoming frames are not painted because the method is still blocked by the previous frame. This can cause a low performance issue as many frames get lost and not painted.

Use the Image control but set its Source Property to a WriteableBitmap instead of a BitmapImage. Then lock the bitmap, copy the pixel data from your EmguCV Mat to the backbuffer of the WriteableBitmap, call AddDirtyRect method and finally unlock the Bitmap. The final unlock call in combination with the AddDirtyRect will trigger a redraw of the UI image.
Adavantages:
It does not generate a new BitmapImage object each time you want to draw a new frame.
You copy the pixel data only once
Your copy, encode and decode data to often:
// Copies and encodes pixel data to jpeg stream
m_frame.Bitmap.Save(stream, System.Drawing.Imaging.ImageFormat.Jpeg);
// Copies jpeg stream
using (var outstream = new MemoryStream(stream.ToArray()))
{
BitmapImage bitmap = new BitmapImage();
bitmap.BeginInit();
//Copies jpeg stream again
bitmap.StreamSource = new MemoryStream(stream.ToArray());
bitmap.CacheOption = BitmapCacheOption.OnLoad;
// Triggers jpeg stream decoding
bitmap.EndInit();
ui_canvas.Background = new ImageBrush(bitmap);
}

Memory exception in image processing loop (Need better solution than GC.collect)

I am making a small application that shows a live webcam feed in a windows form, and also stores watermarked images to drive at a specified interval (Creating a timelapse video is the end goal).
I am using the AForge library for image and video processing.
I have problems were there seems to be a memory leak, even though i try to make sure to use "using" statements at every location where image processing occurs.
Below is the code were the image processing takes place (The NewFrame event)
private void Video_NewFrame(object sender, NewFrameEventArgs eventArgs)
{
try
{
if (ImageProcessing) // If the previous frame is not done processing, let this one go
return;
else
ImageProcessing = true;
using (Bitmap frame = (Bitmap)eventArgs.Frame)
{
// Update the GUI picturebox to show live webcam feed
Invoke((Action)(() =>
{
webcam_PictureBox.Image = (Bitmap)frame.Clone();
}));
// During tests, store images to drive at a certain interval
if (ImageStoreTimer.Elapsed.TotalSeconds > ImageStoreTime)
{
DateTime dt = DateTime.Now;
using (Graphics graphics = Graphics.FromImage(frame))
{
PointF firstLocation = new PointF(frame.Width / 2, frame.Height / 72);
PointF secondLocation = new PointF(frame.Width / 2, frame.Height / 15);
StringFormat drawFormat = new StringFormat();
drawFormat.Alignment = StringAlignment.Center;
using (Font arialFont = new Font("Arial", 15))
{
graphics.DrawString(dt.ToString(), arialFont, Brushes.Red, firstLocation, drawFormat);
graphics.DrawString(Pressure.ToString("F0") + " mbar", arialFont, Brushes.Red, secondLocation, drawFormat);
}
}
// Place images in a folder with the same name as the test
string filePath = Application.StartupPath + "\\" + TestName + "\\";
// Name images by number 1....N
string fileName = (Directory.GetFiles(filePath).Length + 1).ToString() + ".jpeg";
frame.Save(filePath + fileName, ImageFormat.Jpeg);
ImageStoreTimer.Restart();
}
}
//GC.Collect(); <----- I dont want this
}
catch
{
if (ProgramClosing == true){}
// Empty catch for exceptions caused by the program being closed incorrectly
else
throw;
}
finally
{
ImageProcessing = false;
}
}
Now, when running the program, i see memory usage going up and down, usually it gets to about 900MB before dropping. But occasionally it will rise to 2GB. Occasionally, i even get an out of memory exception at this line:
Graphics graphics = Graphics.FromImage(frame)
So after spending an hour or so to trying to reshape the code and looking for my memory leak, i at last tried the GC.Collect line that is commented out in the code (Shame). After that, my memory usage stays constant, at less than 60MB. And i can run the program for 24 hours without any problems.
So i read a bit about GC.Collect, and how bad it is, for example that it could take a lot of processing power to do it to often in a program. But when i compare the CPU power used by my program, it does not really change regardless if i comment the line out or leave it. But the memory problem is gone if i collect at the end of the new frame event.
I would like to find a solution to my problem that does not involve the GC.collect function, as i know it is bad programming practice and i should instead find the underlying problem source.
Thank you all in advance!

I'm not good with win forms but I think that this line:
webcam_PictureBox.Image = (Bitmap)frame.Clone();
Will leave previous image undisposed, which leaks memory (unmanaged memory hold by Bitmap). Since Bitmap has finalizer - it will be reclaimed by GC at some future time (or when you call GC.Collect), but as you already understand - it's not a good practice to rely on GC in such case. So try to do it like this instead:
if (webcam_PictureBox.Image != null)
webcam_PictureBox.Image.Dispose();
webcam_PictureBox.Image = (Bitmap)frame.Clone();
Reasonable comment by Larse: it might be better to not dispose image while it's still being assigned to PictureBox.Image, because who knows, maybe PictureBox control does anything with old image when you are assigning a new one. So alternative is then:
var oldImage = webcam_PictureBox.Image;
webcam_PictureBox.Image = (Bitmap)frame.Clone();
if (oldImage != null)
oldImage.Dispose();

This worked well for us, https://stackoverflow.com/a/70914235/10876657
before that, we tried using statements and all sorts but one solution might work on one machine, but not on the other, but by referencing the image currently set in picturebox and disposing of it afterwords has worked well, not entirely sure why picture.image is not disposed of automatically when a new image is set (frustrating!) but hey, at least this simple workaround exists

Application fails visualizing more than 600 Images

I wrote a WPF app that should swap (fast) between a large set of images (600+, 190Kb average size), but I'm finding some difficulties.
private int appendImages(Canvas c, int start, int end)
{
int tot = 0;
for (int i = start; i < end; i++)
{
BitmapImage bi = new BitmapImage();
bi.BeginInit();
//bi.CacheOption = BitmapCacheOption.OnLoad;
bi.UriSource = new Uri(appFolder+#"/"+imgFolder+"/"+filename(i)+".jpg");
bi.EndInit();
Image img = new Image
{
Width = imgWidth,
Height = imgHeight,
Source = bi,
Name = name(i),
Visibility = i == startImg ? Visibility.Visible : Visibility.Hidden
};
c.Children.Add(img);
tot++;
}
}
Apparently the inizialization is fine, but if I try to swap the images like this:
private void changeImageTo(int n)
{
Image img = findImage(n);
Image old = findImage(prevImg);
if (img != null)
{
img.Visibility = Visibility.Visible;
if (old != null && old != img)
old.Visibility = Visibility.Hidden;
prevImg = n;
}
}
..then the app shows the first 200/300 images (depending on the sources I use), and the others are just empty/blank (i can see the canvas underneath).
I suspect it's a memory issue, but I'm not really sure what causes it.
By the way, if I uncomment the commented line (BitmapCacheOption.OnLoad) sometimes I get a vshost error when launching the app.
Any help would be MUCH appreciated, since I couldn't find anything useful browsing around.
Thanks in advance!

It looks like you're loading all the images at once, and putting them into WinForms/WPF controls. That is a very bad idea with that many images, as each one takes resources even if it's not shown.
Rough back of the envelope calculation, assuming 640x480 images, 24bpp being the native GDI+ format, shows a bit over 2gb for loading all the images at once, and that would, of course, increase exponentially with image size.
What I would do instead, is have only one Image. Move the actual image loading code into your changeImageTo function, build the file name based on n, and set the loaded image to the Image there.

Mark correct sign on file or folder icon at bottom left after upload same as Dropbox does

Assuming that you know about drop box(no problem if you don't know).
In my desktop application there is an upload functionality. I want to mark correct sign on file icon at bottom left after upload.(same as Dropbox does).
How can I do that? What's that trick?

DropBox is a shell extension, so it uses the OS icons and overlay them.
In your case, if it's a desktop application you can overlay your icons using something similar to this:
private static object mOverlayLock = new object();
public static Image GetOverlayedImage(Image baseImage, Image overlay)
{
Image im = null;
lock (mOverlayLock)
{
try
{
im = baseImage.Clone() as Image;
Graphics g = Graphics.FromImage(im);
g.DrawImage(overlay, 0, 0, im.Width, im.Height);
g.Dispose();
}
catch
{
// LOG EXCEPTION!!
}
}
return im;
}
This is a basic example. You can also play with the overlay position, (topleft, middleleft ...), that requires a little bit more programming.
Then, from your application you can call this method to get the result image. For example
...
Image folderIcon = GetFolderIcon();
Image upToDateOverlay = GetUpToDateOverlay();
Image folderUptoDate = GetOlverlayedImage(folderIcon, upToDateOverlay);
// Then assign this image to your control item (treelistnode, listViewnode, whatever)

Out of memory with multi images in one picturebox

I have a problem with out of memory when I'm trying load a few images into one picturebox.
public void button2_Click(object sender, EventArgs e)
{
FolderBrowserDialog dialog = new FolderBrowserDialog();
dialog.ShowDialog();
string selected = dialog.SelectedPath;
string[] imageFileList = Directory.GetFiles(selected);
int iCtr = 0,zCtr = 0;
foreach(string imageFile in imageFileList)
{
if (Image.FromFile(imageFile) != null)
{
Image.FromFile(imageFile).Dispose();
}
PictureBox eachPictureBox = new PictureBox();
eachPictureBox.Size = new Size(100,100);
// if (iCtr % 8 == 0)
//{
// zCtr++;
// iCtr = 0;
//}
eachPictureBox.Location = new Point(iCtr * 100 + 1, 1);
eachPictureBox.Image = Image.FromFile(imageFile);
iCtr++;
panel1.Controls.Add(eachPictureBox);
}
}`enter code here`

if (Image.FromFile(imageFile) != null)
{
Image.FromFile(imageFile).Dispose();
}
Bad. You're loading the image from the file, checking to see if the result is null...then loading it again into a new result so that you can dispose it. While the latter portion is silly, it isn't harmful. The first portion is, however, as the resulting Image is never properly disposed of (if/when the GC collects it, the finalizer on the Image type should dispose of the unmanaged resources, but this is not a wise thing to rely on).
Incidentally, Image.FromFile will never return null. If it cannot read the image, then it will throw an OutOfMemoryException.
The code also appears to do nothing, since there's no else block and nothing meaningful is done in the if block.
My guess is that your OutOfMemoryException is coming from the fact that one or more of the files in that directory is stored in a corrupted or unsupported image format, or isn't an image at all.
Try replacing the code in your foreach with this:
try
{
Image image = Image.FromFile(imageFile);
PictureBox eachPictureBox = new PictureBox();
eachPictureBox.Size = new Size(100,100);
eachPictureBox.Location = new Point(iCtr * 100 + 1, 1);
eachPictureBox.Image = Image.FromFile(imageFile);
iCtr++;
panel1.Controls.Add(eachPictureBox);
}
catch(OutOfMemoryException) { } // skip the file

The picture box internally holds a reference to the bitmap that you place in it. Unless you get rid of the picture box, it's holding a reference to every bitmap you load into it.

Something you have to consider that regardless of the type of picture stored on disk, when you open it for display the picture will become a bitmap and require 4 bytes per displayed pixel.
Your code seems to suggest an attempt at a thumbnail operation. You are in fact loading 70 files into memory and regardless of the display size, in memory they will be very large.
For example let's say you have 70 jpegs at 32bit color depth and say 1920x1080 pixels in size. Your memory requirement to load that many images all at once then is:
70 pics x 1920 pixels x 1080 pixels x 4 bytes/pixel = 580,608,000 bytes!
And that's a fairly low estimate.
You could consider loading many fewer pictures or trying for a real thumbnailing solution.