We Keep Coding

How to minimize the lag from the live stream using OpenCV/EmguCV WPF

I am using EmguCV to get a live stream from a high-resolution webcam. (full HD)
The issue is that there is a significant lag in the video.
I compared with the Windows camera app and my application is much more delayed.
Here is the code snippet I am using to get the live stream and show on the canvas.
I am wondering if I am missing anything important to minimize the lag.
If anybody is experienced, please help me.
void init_camera()
{
m_capture= new VideoCapture(0);
m_capture.ImageGrabbed += ProcessFrame;
}
void ProcessFrame()
{
if (m_capture_in_progress)
return;
m_capture_in_progress = true;
if (m_capture != null && m_capture.Ptr != IntPtr.Zero)
{
m_capture.Retrieve(m_frame, 0);
if (m_frame != null && m_frame.Bitmap != null)
{
if (IsRecording && m_video_writer != null && m_video_writer.IsOpened)
{
try
{
m_video_writer.Write(m_frame);
}
catch (Exception ex)
{
log(ex.Message + "\n" + ex.StackTrace);
}
}
this.Dispatcher.Invoke(() =>
{
using (var stream = new MemoryStream())
{
m_frame.Bitmap.Save(stream, System.Drawing.Imaging.ImageFormat.Jpeg);
using (var outstream = new MemoryStream(stream.ToArray()))
{
BitmapImage bitmap = new BitmapImage();
bitmap.BeginInit();
bitmap.StreamSource = new MemoryStream(stream.ToArray());
bitmap.CacheOption = BitmapCacheOption.OnLoad;
bitmap.EndInit();
ui_canvas.Background = new ImageBrush(bitmap);
}
};
});
}
}
m_capture_in_progress = false;
}

OP is referring some kind of laggs while drawing a camera capture stream from EmguCv into canvas wpf control. OP also states that lagg disappears when streaming with third party software.
This is caused by low performance in OP's code.
After wataching the posted code sample I would suggest the following improvements:
Increasing FPS rate from EmguCV parameters
Maybe your device is supporting higher fps rates, try to increase this value.
Drawing the bitmap over another control like image
As far as I know, drawing a bitmap over a canvas is slower than drawing it over an imagecontrol. A good solution would be to overlap both controls so you can paint the frames over the imageand draw your shapes over the canvasas they are well overlapped.
Cleaning up the rendering method
Keep simple and try to avoid so much control structures when you are inside a critical program block. Specially in program blocks that are executed when an external devices fires an event.
Fixing possible program lock
As far as I can see, you are locking the ProcessFrame event with a bool variable called m_capture_in_progress which is set to true while you are drawing the frame and it is freed after you finish the drawing. This can cause that next incoming frames are not painted because the method is still blocked by the previous frame. This can cause a low performance issue as many frames get lost and not painted.

Use the Image control but set its Source Property to a WriteableBitmap instead of a BitmapImage. Then lock the bitmap, copy the pixel data from your EmguCV Mat to the backbuffer of the WriteableBitmap, call AddDirtyRect method and finally unlock the Bitmap. The final unlock call in combination with the AddDirtyRect will trigger a redraw of the UI image.
Adavantages:
It does not generate a new BitmapImage object each time you want to draw a new frame.
You copy the pixel data only once
Your copy, encode and decode data to often:
// Copies and encodes pixel data to jpeg stream
m_frame.Bitmap.Save(stream, System.Drawing.Imaging.ImageFormat.Jpeg);
// Copies jpeg stream
using (var outstream = new MemoryStream(stream.ToArray()))
{
BitmapImage bitmap = new BitmapImage();
bitmap.BeginInit();
//Copies jpeg stream again
bitmap.StreamSource = new MemoryStream(stream.ToArray());
bitmap.CacheOption = BitmapCacheOption.OnLoad;
// Triggers jpeg stream decoding
bitmap.EndInit();
ui_canvas.Background = new ImageBrush(bitmap);
}

ImageMagick gif creation is very slow (~40 seconds)

I'm trying to create a gif out of some Bitmap images and it takes a really long time for it to load. I followed the example on the library github page for this.
Here is my code:
public void SaveAsGif(Stream stream, ICollection<Bitmap> images, float fps, bool loop)
{
ICollection<IMagickImage> magickImages = new System.Collections.ObjectModel.Collection<IMagickImage>();
float exactDelay = 100 / fps;
foreach (Bitmap bitmap in images)
{
MagickImage image = new MagickImage(bitmap);
image.AnimationDelay = (int) exactDelay;
if (!loop)
{
image.AnimationIterations = 1;
}
magickImages.Add(image);
}
using (MagickImageCollection collection = new MagickImageCollection(magickImages))
{
QuantizeSettings settings = new QuantizeSettings();
settings.Colors = 256;
collection.Quantize(settings);
collection.Optimize();
collection.Write(stream, MagickFormat.Gif);
}
}
I tested and the conversion of the images from bitmap to MagickImages doesn't take very long, at most 5 seconds. And my images are about 4000x3000.
All the images have the same size. Writing to stream doesn't take long either. Any way to improve the timing?

If I understand your question correctly, the slowness is in displaying the GIF, not generating it. If that is the case, try resizing the image prior to calling the Add method.
e.g.:
image.Resize(400,300);

Memory exception in image processing loop (Need better solution than GC.collect)

I am making a small application that shows a live webcam feed in a windows form, and also stores watermarked images to drive at a specified interval (Creating a timelapse video is the end goal).
I am using the AForge library for image and video processing.
I have problems were there seems to be a memory leak, even though i try to make sure to use "using" statements at every location where image processing occurs.
Below is the code were the image processing takes place (The NewFrame event)
private void Video_NewFrame(object sender, NewFrameEventArgs eventArgs)
{
try
{
if (ImageProcessing) // If the previous frame is not done processing, let this one go
return;
else
ImageProcessing = true;
using (Bitmap frame = (Bitmap)eventArgs.Frame)
{
// Update the GUI picturebox to show live webcam feed
Invoke((Action)(() =>
{
webcam_PictureBox.Image = (Bitmap)frame.Clone();
}));
// During tests, store images to drive at a certain interval
if (ImageStoreTimer.Elapsed.TotalSeconds > ImageStoreTime)
{
DateTime dt = DateTime.Now;
using (Graphics graphics = Graphics.FromImage(frame))
{
PointF firstLocation = new PointF(frame.Width / 2, frame.Height / 72);
PointF secondLocation = new PointF(frame.Width / 2, frame.Height / 15);
StringFormat drawFormat = new StringFormat();
drawFormat.Alignment = StringAlignment.Center;
using (Font arialFont = new Font("Arial", 15))
{
graphics.DrawString(dt.ToString(), arialFont, Brushes.Red, firstLocation, drawFormat);
graphics.DrawString(Pressure.ToString("F0") + " mbar", arialFont, Brushes.Red, secondLocation, drawFormat);
}
}
// Place images in a folder with the same name as the test
string filePath = Application.StartupPath + "\\" + TestName + "\\";
// Name images by number 1....N
string fileName = (Directory.GetFiles(filePath).Length + 1).ToString() + ".jpeg";
frame.Save(filePath + fileName, ImageFormat.Jpeg);
ImageStoreTimer.Restart();
}
}
//GC.Collect(); <----- I dont want this
}
catch
{
if (ProgramClosing == true){}
// Empty catch for exceptions caused by the program being closed incorrectly
else
throw;
}
finally
{
ImageProcessing = false;
}
}
Now, when running the program, i see memory usage going up and down, usually it gets to about 900MB before dropping. But occasionally it will rise to 2GB. Occasionally, i even get an out of memory exception at this line:
Graphics graphics = Graphics.FromImage(frame)
So after spending an hour or so to trying to reshape the code and looking for my memory leak, i at last tried the GC.Collect line that is commented out in the code (Shame). After that, my memory usage stays constant, at less than 60MB. And i can run the program for 24 hours without any problems.
So i read a bit about GC.Collect, and how bad it is, for example that it could take a lot of processing power to do it to often in a program. But when i compare the CPU power used by my program, it does not really change regardless if i comment the line out or leave it. But the memory problem is gone if i collect at the end of the new frame event.
I would like to find a solution to my problem that does not involve the GC.collect function, as i know it is bad programming practice and i should instead find the underlying problem source.
Thank you all in advance!

I'm not good with win forms but I think that this line:
webcam_PictureBox.Image = (Bitmap)frame.Clone();
Will leave previous image undisposed, which leaks memory (unmanaged memory hold by Bitmap). Since Bitmap has finalizer - it will be reclaimed by GC at some future time (or when you call GC.Collect), but as you already understand - it's not a good practice to rely on GC in such case. So try to do it like this instead:
if (webcam_PictureBox.Image != null)
webcam_PictureBox.Image.Dispose();
webcam_PictureBox.Image = (Bitmap)frame.Clone();
Reasonable comment by Larse: it might be better to not dispose image while it's still being assigned to PictureBox.Image, because who knows, maybe PictureBox control does anything with old image when you are assigning a new one. So alternative is then:
var oldImage = webcam_PictureBox.Image;
webcam_PictureBox.Image = (Bitmap)frame.Clone();
if (oldImage != null)
oldImage.Dispose();

This worked well for us, https://stackoverflow.com/a/70914235/10876657
before that, we tried using statements and all sorts but one solution might work on one machine, but not on the other, but by referencing the image currently set in picturebox and disposing of it afterwords has worked well, not entirely sure why picture.image is not disposed of automatically when a new image is set (frustrating!) but hey, at least this simple workaround exists

Paint method throwing System.OutOfMemoryException

I'm having a hard time trying to figure out how to prevent memory leak when repainting PictureBox
That's how my drawing method looks like:
Bitmap image;
image = new Bitmap((Bitmap)baseImage.Clone());
Graphics g = Graphics.FromImage(image);
//here I'm drawing using created "g"
//reason why am doing it on new bitmap, not on paintBox.Image is that..
//..I don't want this drawings to be permanently added to my base image
g.Dispose();
paintBox.Image = image;
Then I'm using method which is changing baseImage and refreshing paintBox many times (hundreds).
Calling this method gives me 'System.OutOfMemoryException' in System.Drawing.dll
This method is recursive, however I'm pretty sure it's not causing this exception, because when I'm changing this method to only modify base image without refreshing paint box it works fine (however I would like to see changes it's making up to date).
So, what's the best method to prevent memory leak in this case?
I was trying something like this:
paintBoxx.Image.Dispose();
paintBox.Image = image;
but it is giving me 'System.NullReferenceException' (because I'm trying to use disposed image).
Any advice would be appreciated.

Change your code to this:
Bitmap image;
image = new Bitmap((Bitmap)baseImage.Clone());
using (Graphics g = Graphics.FromImage(image) )
{
// I am drawing on the bitmap so I don't permanently change my base image
// do your draw stuff here..
g.FillEllipse(Brushes.Yellow, 3, 3, 9, 9);
// ..
}
// don't leak the image and..
// ..don't Dispose without checking for null
if (paintBox.Image != null) paintBox.Image.Dispose();
paintBox.Image = image;
Note the using clause, which will dispose of the Graphics object, even if the drawing runs into problems.

Have you tried using MemoryStream?
Take a look at my example code:
image = new Bitmap((Bitmap)baseImage.Clone());
using (MemoryStream imageStream = new MemoryStream())
{
// put iimagem in memory stream
image.Save(imageStream, System.Drawing.Imaging.ImageFormat.Gif);
// create an array of bytes with image length
byte[] imageContent = new Byte[imageStream.Length];
// reset memory stream
imageStream.Position = 0;
// load array of bytes with the imagem
imageStream.Read(imageContent, 0, (int)imageStream.Length);
// change header page "content-type" to "image/jpeg" and print the image.
Response.ContentType = "image/gif";
Response.BinaryWrite(imageContent);
}

WPF Image control memory leak

My program has a lot of small images (Image controls are small, not the images themselves) and by saying a lot I mean more than 500. These images are generated asynchronously and then assigned to the Image controls, which were initialized before.
Basically my code does the following:
filename = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, string.Format("{0}.JPG", Guid.NewGuid().GetHashCode().ToString("x2")));
converter.ConvertPdfPageToImage(filename, i);
//Fire the ThumbnailCreated event
onThumbnailCreated(filename, (i - 1));
There is no memory leak in code that creates the images, I have the following code:
string[] files = Directory.GetFiles("C:\\Users\\Daniel\\Pictures", "*.jpg");
for(int i=0; i<files.Length; i++){
onThumbnailCreated(files[i], i);
}
Still the problem persists.
This happens in the event handler method:
void Thumbnails_ThumbnailCreated(ThumbnailCreatedEventArgs e, object sender)
{
//Since we generate the images async, we need to use Invoke
this.parent.Dispatcher.Invoke(new SetImageDelegate(SetImage), e.Filename, e.PageNumber);
}
private void SetImage(string filename, int pageNumber)
{
BitmapImage bitmap = new BitmapImage();
bitmap.BeginInit();
//I am trying to make the Image control use as less memory as possible
//so I prevent caching
bitmap.CacheOption = BitmapCacheOption.None;
bitmap.UriSource = new Uri(filename);
bitmap.EndInit();
//We set the bitmap as the source for the Image control
//and show it to the user
this.images[pageNumber].Source = bitmap;
}
With 468 images the program uses about 1Gb of memory and then runs out of it at all. Is my task even possible to achieve using WPF or is the number of images too high? Maybe there is something wrong with my code?
Thanks in advance

You should freeze these images and set their width (or height) to that will be actually used in the application if possible:
// ...
bitmap.DecodePixelWidth = 64; // "displayed" width, this improves memory usage
bitmap.EndInit();
bitmap.Freeze();
this.images[pageNumber].Source = bitmap;

Try this:
private void SetImage(string filename, int pageNumber)
{
using (BitmapImage bitmap = new BitmapImage())
{
bitmap.BeginInit();
//I am trying to make the Image control use as less memory as possible
//so I prevent caching
bitmap.CacheOption = BitmapCacheOption.None;
bitmap.UriSource = new Uri(filename);
bitmap.EndInit();
this.images[pageNumber].Source = bitmap;
}
}
That will dispose of your bitmaps when you're done with them.

It may be the event handlers that are causing your memory leaks.
See this SO question:
Why and How to avoid Event Handler memory leaks?