I have an application where performance-sensitive drawings occur using a WriteableBitmap. An event is called with CompositionTarget.Rendering to actually update the back buffer of the WriteableBitmap. From the MSDN documentation, that means the event is fired once per frame, right before the control is rendered.
The issue that I am having is that the WriteableBitmap's Lock() function takes an extremely long time, especially at larger bitmap sizes. I have previously read that AddDirtyRegion() has a bug that causes the entire bitmap to invalidate, leading to poor performance. However, that doesn't seem to be the case here. From a good bit of low-level checking, it seems that Lock() opens the bitmap's backbuffer for writing on the render thread, which means every time my event handler is called, it has to thread block until the render thread is ready for it. This leads to a noticeable lag when updating the graphics of the bitmap.
I have already tried adding a timeout to the event handler, using TryLock(), so that it won't block for such a long time and cause the performance degradation. This, however, causes a similar effect in that it appears to lag, because larger numbers of bitmap updates get lumped together.
Here is the relevant code from the event handler to show what exactly I am doing. The UpdatePixels() function was written to avoid using the potentially bugged AddDirtyRect():
void updateBitmap(object sender, EventArgs e)
{
if (!form.ResizingWindow)
{
// Lock and unlock are important... Make sure to keep them outside of the loop for performance reasons.
if (canvasUpdates.Count > 0)
{
//bool locked = scaledDrawingMap.TryLock(bitmapLockDuration);
scaledDrawingMap.Lock();
//if (locked)
//{
unsafe
{
int* pixData = (int*)scaledDrawingMap.BackBuffer;
foreach (Int32Rect i in canvasUpdates)
{
// The graphics object isn't directly shown, so this isn't actually necessary. We do a sort of manual copy from the drawingMap, which acts similarly
// to a back buffer.
Int32Rect temp = GetValidDirtyRegion(i);
UpdatePixels(temp, pixData);
}
scaledDrawingMap.Unlock();
canvasUpdates.Clear();
}
//}
}
}
}
private unsafe void UpdatePixels(Int32Rect temp, int* pixData)
{
//int* pixData = (int*)scaledDrawingMap.BackBuffer;
// Directly copy the backbuffer into a new buffer, to use WritePixels().
var stride = temp.Width * scaledDrawingMap.Format.BitsPerPixel / 8;
int[] relevantPixData = new int[stride * temp.Height];
int srcIdx = 0;
int pWidth = scaledDrawingMap.PixelWidth;
int yLess = temp.Y + temp.Height;
int xLess = temp.X + temp.Width;
for (int y = temp.Y; y < yLess; y++)
{
for (int x = temp.X; x < xLess; x++)
{
relevantPixData[srcIdx++] = pixData[y * pWidth + x];
}
}
scaledDrawingMap.WritePixels(temp, relevantPixData, stride, 0);
}
I can't seem to figure out how to avoid the issue of thread blocking with the WriteableBitmap, and I can't see any obvious faults in the code I have written. Any help or pointers would be much appreciated.
Looks like you are not actually using the BackBuffer to write - only to read.
WritePixels writes to the "front" buffer and does not require a lock.
I don't know if you have some other reason to lock it (other threads doing something), but for the code that's here i don't see why you would need to.
I guess I was wrong about not needing a lock to read from BackBuffer (*pixData) - I thought it was only for writes, but I am positive you do not need to to call Lock for WritePixels.
As far as I can tell, you are doing:
Lock the back buffer
Copy something from it to an array
Call WritePixels using this new array
Unlock the back buffer.
How about switching 3 and 4?
WritePixels may internally cause rendering thread (which has higher priority on the message queue) to get a lock on its behalf which is probably a factor in the delay you are seeing.
Related
I am making a small application that shows a live webcam feed in a windows form, and also stores watermarked images to drive at a specified interval (Creating a timelapse video is the end goal).
I am using the AForge library for image and video processing.
I have problems were there seems to be a memory leak, even though i try to make sure to use "using" statements at every location where image processing occurs.
Below is the code were the image processing takes place (The NewFrame event)
private void Video_NewFrame(object sender, NewFrameEventArgs eventArgs)
{
try
{
if (ImageProcessing) // If the previous frame is not done processing, let this one go
return;
else
ImageProcessing = true;
using (Bitmap frame = (Bitmap)eventArgs.Frame)
{
// Update the GUI picturebox to show live webcam feed
Invoke((Action)(() =>
{
webcam_PictureBox.Image = (Bitmap)frame.Clone();
}));
// During tests, store images to drive at a certain interval
if (ImageStoreTimer.Elapsed.TotalSeconds > ImageStoreTime)
{
DateTime dt = DateTime.Now;
using (Graphics graphics = Graphics.FromImage(frame))
{
PointF firstLocation = new PointF(frame.Width / 2, frame.Height / 72);
PointF secondLocation = new PointF(frame.Width / 2, frame.Height / 15);
StringFormat drawFormat = new StringFormat();
drawFormat.Alignment = StringAlignment.Center;
using (Font arialFont = new Font("Arial", 15))
{
graphics.DrawString(dt.ToString(), arialFont, Brushes.Red, firstLocation, drawFormat);
graphics.DrawString(Pressure.ToString("F0") + " mbar", arialFont, Brushes.Red, secondLocation, drawFormat);
}
}
// Place images in a folder with the same name as the test
string filePath = Application.StartupPath + "\\" + TestName + "\\";
// Name images by number 1....N
string fileName = (Directory.GetFiles(filePath).Length + 1).ToString() + ".jpeg";
frame.Save(filePath + fileName, ImageFormat.Jpeg);
ImageStoreTimer.Restart();
}
}
//GC.Collect(); <----- I dont want this
}
catch
{
if (ProgramClosing == true){}
// Empty catch for exceptions caused by the program being closed incorrectly
else
throw;
}
finally
{
ImageProcessing = false;
}
}
Now, when running the program, i see memory usage going up and down, usually it gets to about 900MB before dropping. But occasionally it will rise to 2GB. Occasionally, i even get an out of memory exception at this line:
Graphics graphics = Graphics.FromImage(frame)
So after spending an hour or so to trying to reshape the code and looking for my memory leak, i at last tried the GC.Collect line that is commented out in the code (Shame). After that, my memory usage stays constant, at less than 60MB. And i can run the program for 24 hours without any problems.
So i read a bit about GC.Collect, and how bad it is, for example that it could take a lot of processing power to do it to often in a program. But when i compare the CPU power used by my program, it does not really change regardless if i comment the line out or leave it. But the memory problem is gone if i collect at the end of the new frame event.
I would like to find a solution to my problem that does not involve the GC.collect function, as i know it is bad programming practice and i should instead find the underlying problem source.
Thank you all in advance!
I'm not good with win forms but I think that this line:
webcam_PictureBox.Image = (Bitmap)frame.Clone();
Will leave previous image undisposed, which leaks memory (unmanaged memory hold by Bitmap). Since Bitmap has finalizer - it will be reclaimed by GC at some future time (or when you call GC.Collect), but as you already understand - it's not a good practice to rely on GC in such case. So try to do it like this instead:
if (webcam_PictureBox.Image != null)
webcam_PictureBox.Image.Dispose();
webcam_PictureBox.Image = (Bitmap)frame.Clone();
Reasonable comment by Larse: it might be better to not dispose image while it's still being assigned to PictureBox.Image, because who knows, maybe PictureBox control does anything with old image when you are assigning a new one. So alternative is then:
var oldImage = webcam_PictureBox.Image;
webcam_PictureBox.Image = (Bitmap)frame.Clone();
if (oldImage != null)
oldImage.Dispose();
This worked well for us, https://stackoverflow.com/a/70914235/10876657
before that, we tried using statements and all sorts but one solution might work on one machine, but not on the other, but by referencing the image currently set in picturebox and disposing of it afterwords has worked well, not entirely sure why picture.image is not disposed of automatically when a new image is set (frustrating!) but hey, at least this simple workaround exists
This question already has answers here:
How to Cancel a Thread?
(4 answers)
Closed 5 years ago.
I'm improving the performance of a 3d engine i created, introducing LockBits and parallel processing. I have a class for the engine with the following method to update a bitmap with the result:
public void draw() {
clearbmp();
// locking bmp
BitmapData bitmapData = bmp.LockBits(new Rectangle(0, 0, W, H), ImageLockMode.ReadWrite, bmp.PixelFormat);
IntPtr FirstPixel = bitmapData.Scan0;
int bytes = Math.Abs(bitmapData.Stride) * H;
bpp = Bitmap.GetPixelFormatSize(bmp.PixelFormat) / 8;
rgbarray = new byte[bytes];
System.Runtime.InteropServices.Marshal.Copy(FirstPixel, rgbarray, 0, bytes);
int count = 0;
for (int k = 0; k < solidos.Count; k++)
{
if (solidos[k] != null && solidos[k].draw)
{
// separating faces of the solid into different threads
Parallel.ForEach(solidos[k].side, f =>
{
// ... do the operations...
});
}
}
// copy the array to the bitmap and unlock
System.Runtime.InteropServices.Marshal.Copy(rgbarray, 0, FirstPixel, bytes);
bmp.UnlockBits(bitmapData);
The code runs as intended to generate a image, but fails when the main program requires it to update several times in a quick succession, the error occurs in bmp.UnlockBits(bitmapData) with the excepion "Generic error in GDI+"
From what i gathered from my research I suppose this happens because the method runs a second time before it finished the first one, thus trying to unlock data that is already unlocked.
If that is correct then how can i abort the running thread when the new one is created? The last call is always the important one
Before starting a new call, wait for the existing call to complete. You could do that by simply using a lock region. This solves the correctness issue. Maybe it's easier to make each computation run into a Task using Task.Run. The resulting Task object is a handle to that computation and can be used to wait for it to complete.
If you want to speed up finishing an old computation run that is no longer needed, add a cancellation mechanism. Here, you could use a volatile bool cancel = false;. Set it to true to cancel. In your parallel loop (and possibly elsewhere), check that boolean variable periodically and terminate if it is found true. You can also use a CancellationTokenSource.
We have such a situation. We have a canvas, on which some ammount of figures are rendered. It may be 1 or many more (for example thousand) and we need to animate their translation to another location (on button click) using storyboard:
internal void someStoryBoard(figure someFigure, double coordMoveToValue)
{
string sbName = "StoryBoard_" + figure.ID;
string regName = "figure_" + figure.ID;
try
{
cnvsGame.Resources.Remove(sbName);
cnvsGame.UnregisterName(regName);
}
catch{ }
someCanvas.RegisterName(regName, someFigure.Geometry);
var moveFigureYAnimation = new PointAnimation();
moveFigureYAnimation.From = new Point(someFigure.Geometry.Center.X, someFigure.Geometry.Center.Y);
moveFigureYAnimation.To = new Point(someFigure.eGeometry.Center.X, coordMoveToValue);
moveFigureYAnimation.Duration = TimeSpan.FromSeconds(0.5);
var sbFigureMove = new Storyboard();
Storyboard.SetTargetName(sbFigureMove, regName);
Storyboard.SetTargetProperty(sbFigureMove, new PropertyPath(Geometry.CenterProperty));
sbFigureMove.Children.Add(moveFigureYAnimation);
cnvsGame.Resources.Add(sbName, sbFigureMove);
sbFigureMove.Begin();
}
Figures are stored in list. We are calling this StoryBoard using for loop:
for(int i = 0; i<listOfFigures.Count; i++)
{
someStoryBoard(listOfFigures[i], someCoord);
}
But here's the problem: if we have a little amount of figures - code completes quickly. But if ammount is big - there is a delay after a button is clicked and before the figures begin to move.
So, here's the question: is it possible to call someStoryBoard method asynchronously? Is next algorithm possible -> When someStoryBoard is called it begins to move figure instantly, not waiting for whole for loop to complete.?
You can add actions into Dispatcher queue by calling Dispatcher.InvokeAsync. You can also specify dispatcher priority, depending on your requirements.
Please note that moving thousands of items can't be reliably fast, so you may need to rethink the drawing logic. If even starting animation is slow, it's highly likely animating won't be fast enough too.
You can try use async/await modifier
async internal Task someStoryBoard(figure someFigure, double coordMoveToValue)
In my code I retrieve frames from a camera with a pointer to an unmanaged object, make some calculations on it and then I make it visualized on a picturebox control.
Before I go further in this application with all the details, I want to be sure that the base code for this process is good.
In particular I would like to:
- keep execution time minimal and avoid unnecessary operations, such as
copying more images than necessary. I want to keep only essential
operations
- understand if a delay in the calculation process on every frame could have detrimental effects on the way images are shown (i.e. if it is not printed what I expect) or some image is skipped
- prevent more serious errors, such as ones due to memory or thread management, or to image display.
For this purpose, I set up a few experimental lines of code (below), but I’m not able to explain the results of what I found. If you have the executables of OpenCv you can make a try by yourself.
using System;
using System.Drawing;
using System.Drawing.Imaging;
using System.Windows.Forms;
using System.Runtime.InteropServices;
using System.Threading;
public partial class FormX : Form
{
private delegate void setImageCallback();
Bitmap _bmp;
Bitmap _bmp_draw;
bool _exit;
double _x;
IntPtr _ImgBuffer;
bool buffercopy;
bool copyBitmap;
bool refresh;
public FormX()
{
InitializeComponent();
_x = 10.1;
// set experimemental parameters
buffercopy = false;
copyBitmap = false;
refresh = true;
}
private void buttonStart_Click(object sender, EventArgs e)
{
Thread camThread = new Thread(new ThreadStart(Cycle));
camThread.Start();
}
private void buttonStop_Click(object sender, EventArgs e)
{
_exit = true;
}
private void Cycle()
{
_ImgBuffer = IntPtr.Zero;
_exit = false;
IntPtr vcap = cvCreateCameraCapture(0);
while (!_exit)
{
IntPtr frame = cvQueryFrame(vcap);
if (buffercopy)
{
UnmanageCopy(frame);
_bmp = SharedBitmap(_ImgBuffer);
}
else
{ _bmp = SharedBitmap(frame); }
// make calculations
int N = 1000000; /*1000000*/
for (int i = 0; i < N; i++)
_x = Math.Sin(0.999999 * _x);
ShowFrame();
}
cvReleaseImage(ref _ImgBuffer);
cvReleaseCapture(ref vcap);
}
private void ShowFrame()
{
if (pbCam.InvokeRequired)
{
this.Invoke(new setImageCallback(ShowFrame));
}
else
{
Pen RectangleDtPen = new Pen(Color.Azure, 3);
if (copyBitmap)
{
if (_bmp_draw != null) _bmp_draw.Dispose();
//_bmp_draw = new Bitmap(_bmp); // deep copy
_bmp_draw = _bmp.Clone(new Rectangle(0, 0, _bmp.Width, _bmp.Height), _bmp.PixelFormat);
}
else
{
_bmp_draw = _bmp; // add reference to the same object
}
Graphics g = Graphics.FromImage(_bmp_draw);
String drawString = _x.ToString();
Font drawFont = new Font("Arial", 56);
SolidBrush drawBrush = new SolidBrush(Color.Red);
PointF drawPoint = new PointF(10.0F, 10.0F);
g.DrawString(drawString, drawFont, drawBrush, drawPoint);
drawPoint = new PointF(10.0F, 300.0F);
g.DrawString(drawString, drawFont, drawBrush, drawPoint);
g.DrawRectangle(RectangleDtPen, 12, 12, 200, 400);
g.Dispose();
pbCam.Image = _bmp_draw;
if (refresh) pbCam.Refresh();
}
}
public void UnmanageCopy(IntPtr f)
{
if (_ImgBuffer == IntPtr.Zero)
_ImgBuffer = cvCloneImage(f);
else
cvCopy(f, _ImgBuffer, IntPtr.Zero);
}
// only works with 3 channel images from camera! (to keep code minimal)
public Bitmap SharedBitmap(IntPtr ipl)
{
// gets unmanaged data from pointer to IplImage:
IntPtr scan0;
int step;
Size size;
OpenCvCall.cvGetRawData(ipl, out scan0, out step, out size);
return new Bitmap(size.Width, size.Height, step, PixelFormat.Format24bppRgb, scan0);
}
// based on older version of OpenCv. Change dll name if different
[DllImport( "opencv_highgui246", CallingConvention = CallingConvention.Cdecl)]
public static extern IntPtr cvCreateCameraCapture(int index);
[DllImport("opencv_highgui246", CallingConvention = CallingConvention.Cdecl)]
public static extern void cvReleaseCapture(ref IntPtr capture);
[DllImport("opencv_highgui246", CallingConvention = CallingConvention.Cdecl)]
public static extern IntPtr cvQueryFrame(IntPtr capture);
[DllImport("opencv_core246", CallingConvention = CallingConvention.Cdecl)]
public static extern void cvGetRawData(IntPtr arr, out IntPtr data, out int step, out Size roiSize);
[DllImport("opencv_core246", CallingConvention = CallingConvention.Cdecl)]
public static extern void cvCopy(IntPtr src, IntPtr dst, IntPtr mask);
[DllImport("opencv_core246", CallingConvention = CallingConvention.Cdecl)]
public static extern IntPtr cvCloneImage(IntPtr src);
[DllImport("opencv_core246", CallingConvention = CallingConvention.Cdecl)]
public static extern void cvReleaseImage(ref IntPtr image);
}
results [dual core 2 Duo T6600 2.2 GHz]:
A. buffercopy = false; copyBitmap = false; refresh = false;
This is the simpler configuration. Each frame is retrieved in turn, operations are made (in the reality they are based on the same frame, here just calculations), then the result of the calculations is printed on top of the image and finally it is displayed on a picturebox.
OpenCv documentation says:
OpenCV 1.x functions cvRetrieveFrame and cv.RetrieveFrame return image
stored inside the video capturing structure. It is not allowed to
modify or release the image! You can copy the frame using
cvCloneImage() and then do whatever you want with the copy.
But this doesn’t prevent us from doing experiments.
If the calculation are not intense (low number of iterations, N), everything is just ok and the fact that we manipulate the image buffer own by the unmanaged frame retriever doesn’t pose a problem here.
The reason is that probably they advise to leave untouched the buffer, in case people would modify its structure (not its values) or do operations asynchronously without realizing it. Now we retrieve frames and modify their content in turn.
If N is increased (N=1000000 or more), when the number of frames per second is not high, for example with artificial light and low exposure, everything seems ok, but after a while the video is lagged and the graphics impressed on it are blinking. With a higher frame rate the blinking appears from the beginning, even when the video is still fluid.
Is this because the mechanism of displaying images on the control (or refreshing or whatever else) is somehow asynchronous and when the picturebox is fetching its buffer of data it is modified in the meanwhile by the camera, deleting the graphics?
Or is there some other reason?
Why is the image lagged in that way, i.e. I would expect that the delay due to calculations only had the effect of skipping the frames received by the camera when the calculation are not done yet, and de facto only reducing the frame rate; or alternatively that all frames are received and the delay due to calculations brings the system to process images gotten minutes before, because the queue of images to process rises over time.
Instead, the observed behavior seems hybrid between the two: there is a delay of a few seconds, but this seems not increased much as the capturing process goes on.
B. buffercopy = true; copyBitmap = false; refresh = false;
Here I make a deep copy of the buffer into a second buffer, following the advice of the OpenCv documentation.
Nothing changes. The second buffer doesn’t change its address in memory during the run.
C. buffercopy = false; copyBitmap = true; refresh = false;
Now the (deep) copy of the bitmap is made allocating every time a new space in memory.
The blinking effect has gone, but the lagging keep arising after a certain time.
D. buffercopy = false; copyBitmap = false; refresh = true;
As before.
Please help me explain these results!
If I may be so frank, it is a bit tedious to understand all the details of your questions, but let me make a few points to help you analyse your results.
In case A, you say you perform calculations directly on the buffer. The documentation says you shouldn't do this, so if you do, you can expect undefined results. OpenCV assumes you won't touch it, so it might do stuff like suddenly delete that part of memory, let some other app process it, etc. It might look like it works, but you can never know for sure, so don't do it *slaps your wrist* In particular, if your processing takes a long time, the camera might overwrite the buffer while you're in the middle of processing it.
The way you should do it is to copy the buffer before doing anything. This will give you a piece of memory that is yours to do with whatever you wish. You can create a Bitmap that refers to this memory, and manually free the memory when you no longer need it.
If your processing rate (frames processed per second) is less than the number of frames captured per second by the camera, you have to expect some frames will be dropped. If you want to show a live view of the processed images, it will lag and there's no simple way around it. If it is vital that your application processes a fluid video (e.g. this might be necessary if you're tracking an object), then consider storing the video to disk so you don't have to process in real-time. You can also consider multithreading to process several frames at once, but the live view would have a latency.
By the way, is there any particular reason why you're not using EmguCV? It has abstractions for the camera and a system that raises an event whenever the camera has captured a new frame. This way, you don't need to continuously call cvQueryFrame on a background thread.
I think that you still have a problem with your UnmanageCopy method in that you only clone the image the first time this is called and you subsequently copy it. I believe that you need to do a cvCloneImage(f) every time as copy performs only a shallow copy, not a deep copy as you seem to think.
I have the following function to convert a byte array of pixels into an image. However, I have a memory leak in the line:
unpackedImage = UIImage.FromImage(context.ToImage());
When I comment out the line above, the leak goes away. The leak is so bad that iOS is killing my app within about 30 seconds of startup. It's because of this line of code.
How do I prevent this memory leak? Is there a better way to do what I am trying to do?
public static void DrawCustomImage2(IntPtr buffer, int width, int height, int bytesPerRow, CGColorSpace colSpace, byte[] rawPixels, ref UIImage unpackedImage)
{
GCHandle pinnedArray = GCHandle.Alloc(rawPixels, GCHandleType.Pinned);
IntPtr pointer = pinnedArray.AddrOfPinnedObject();
// Set a grayscale drawing context using the image buffer
CGBitmapContext context = new CGBitmapContext(pointer, width, height, 8, bytesPerRow, colSpace, CGImageAlphaInfo.None);
// Turning off interpolation and Antialiasing is supposed to speed things up
context.InterpolationQuality = CGInterpolationQuality.None;
context.SetAllowsAntialiasing(false);
try
{
unpackedImage = UIImage.FromImage(context.ToImage()); // Convert the drawing context to an image and set it as the unpacked image
} finally
{
pinnedArray.Free();
if (context != null)
context.Dispose();
}
}
Here is the profiling screenshot (checked items all disappear when the critical line of code is commented out). You can see how the checked items (especially the Malloc) grow over time.
Here is the zoom-in view on the Malloc 1.50KB. You can see in the Extended Detail pane on the right that it is calling CGBitmapContextCreateImage and CGDataProviderCreateWithCopyOfData and then a malloc.
Here is the profiling screenshot with Rolf's suggestion. I ran the image loop twice. You can see that it cleans up the extra memory at the end of the first loop, but the system didn't clean it up fast enough the second time and iOS killed my app (you can see the low memory warning flags in the top right corner).
Do it like this:
using (var pool = new NSAutoreleasePool ()) {
using (var img = context.ToImage ()) {
unpackedImage = UIImage.FromImage (img);
}
}