In my code I retrieve frames from a camera with a pointer to an unmanaged object, make some calculations on it and then I make it visualized on a picturebox control.
Before I go further in this application with all the details, I want to be sure that the base code for this process is good.
In particular I would like to:
- keep execution time minimal and avoid unnecessary operations, such as
copying more images than necessary. I want to keep only essential
operations
- understand if a delay in the calculation process on every frame could have detrimental effects on the way images are shown (i.e. if it is not printed what I expect) or some image is skipped
- prevent more serious errors, such as ones due to memory or thread management, or to image display.
For this purpose, I set up a few experimental lines of code (below), but I’m not able to explain the results of what I found. If you have the executables of OpenCv you can make a try by yourself.
using System;
using System.Drawing;
using System.Drawing.Imaging;
using System.Windows.Forms;
using System.Runtime.InteropServices;
using System.Threading;
public partial class FormX : Form
{
private delegate void setImageCallback();
Bitmap _bmp;
Bitmap _bmp_draw;
bool _exit;
double _x;
IntPtr _ImgBuffer;
bool buffercopy;
bool copyBitmap;
bool refresh;
public FormX()
{
InitializeComponent();
_x = 10.1;
// set experimemental parameters
buffercopy = false;
copyBitmap = false;
refresh = true;
}
private void buttonStart_Click(object sender, EventArgs e)
{
Thread camThread = new Thread(new ThreadStart(Cycle));
camThread.Start();
}
private void buttonStop_Click(object sender, EventArgs e)
{
_exit = true;
}
private void Cycle()
{
_ImgBuffer = IntPtr.Zero;
_exit = false;
IntPtr vcap = cvCreateCameraCapture(0);
while (!_exit)
{
IntPtr frame = cvQueryFrame(vcap);
if (buffercopy)
{
UnmanageCopy(frame);
_bmp = SharedBitmap(_ImgBuffer);
}
else
{ _bmp = SharedBitmap(frame); }
// make calculations
int N = 1000000; /*1000000*/
for (int i = 0; i < N; i++)
_x = Math.Sin(0.999999 * _x);
ShowFrame();
}
cvReleaseImage(ref _ImgBuffer);
cvReleaseCapture(ref vcap);
}
private void ShowFrame()
{
if (pbCam.InvokeRequired)
{
this.Invoke(new setImageCallback(ShowFrame));
}
else
{
Pen RectangleDtPen = new Pen(Color.Azure, 3);
if (copyBitmap)
{
if (_bmp_draw != null) _bmp_draw.Dispose();
//_bmp_draw = new Bitmap(_bmp); // deep copy
_bmp_draw = _bmp.Clone(new Rectangle(0, 0, _bmp.Width, _bmp.Height), _bmp.PixelFormat);
}
else
{
_bmp_draw = _bmp; // add reference to the same object
}
Graphics g = Graphics.FromImage(_bmp_draw);
String drawString = _x.ToString();
Font drawFont = new Font("Arial", 56);
SolidBrush drawBrush = new SolidBrush(Color.Red);
PointF drawPoint = new PointF(10.0F, 10.0F);
g.DrawString(drawString, drawFont, drawBrush, drawPoint);
drawPoint = new PointF(10.0F, 300.0F);
g.DrawString(drawString, drawFont, drawBrush, drawPoint);
g.DrawRectangle(RectangleDtPen, 12, 12, 200, 400);
g.Dispose();
pbCam.Image = _bmp_draw;
if (refresh) pbCam.Refresh();
}
}
public void UnmanageCopy(IntPtr f)
{
if (_ImgBuffer == IntPtr.Zero)
_ImgBuffer = cvCloneImage(f);
else
cvCopy(f, _ImgBuffer, IntPtr.Zero);
}
// only works with 3 channel images from camera! (to keep code minimal)
public Bitmap SharedBitmap(IntPtr ipl)
{
// gets unmanaged data from pointer to IplImage:
IntPtr scan0;
int step;
Size size;
OpenCvCall.cvGetRawData(ipl, out scan0, out step, out size);
return new Bitmap(size.Width, size.Height, step, PixelFormat.Format24bppRgb, scan0);
}
// based on older version of OpenCv. Change dll name if different
[DllImport( "opencv_highgui246", CallingConvention = CallingConvention.Cdecl)]
public static extern IntPtr cvCreateCameraCapture(int index);
[DllImport("opencv_highgui246", CallingConvention = CallingConvention.Cdecl)]
public static extern void cvReleaseCapture(ref IntPtr capture);
[DllImport("opencv_highgui246", CallingConvention = CallingConvention.Cdecl)]
public static extern IntPtr cvQueryFrame(IntPtr capture);
[DllImport("opencv_core246", CallingConvention = CallingConvention.Cdecl)]
public static extern void cvGetRawData(IntPtr arr, out IntPtr data, out int step, out Size roiSize);
[DllImport("opencv_core246", CallingConvention = CallingConvention.Cdecl)]
public static extern void cvCopy(IntPtr src, IntPtr dst, IntPtr mask);
[DllImport("opencv_core246", CallingConvention = CallingConvention.Cdecl)]
public static extern IntPtr cvCloneImage(IntPtr src);
[DllImport("opencv_core246", CallingConvention = CallingConvention.Cdecl)]
public static extern void cvReleaseImage(ref IntPtr image);
}
results [dual core 2 Duo T6600 2.2 GHz]:
A. buffercopy = false; copyBitmap = false; refresh = false;
This is the simpler configuration. Each frame is retrieved in turn, operations are made (in the reality they are based on the same frame, here just calculations), then the result of the calculations is printed on top of the image and finally it is displayed on a picturebox.
OpenCv documentation says:
OpenCV 1.x functions cvRetrieveFrame and cv.RetrieveFrame return image
stored inside the video capturing structure. It is not allowed to
modify or release the image! You can copy the frame using
cvCloneImage() and then do whatever you want with the copy.
But this doesn’t prevent us from doing experiments.
If the calculation are not intense (low number of iterations, N), everything is just ok and the fact that we manipulate the image buffer own by the unmanaged frame retriever doesn’t pose a problem here.
The reason is that probably they advise to leave untouched the buffer, in case people would modify its structure (not its values) or do operations asynchronously without realizing it. Now we retrieve frames and modify their content in turn.
If N is increased (N=1000000 or more), when the number of frames per second is not high, for example with artificial light and low exposure, everything seems ok, but after a while the video is lagged and the graphics impressed on it are blinking. With a higher frame rate the blinking appears from the beginning, even when the video is still fluid.
Is this because the mechanism of displaying images on the control (or refreshing or whatever else) is somehow asynchronous and when the picturebox is fetching its buffer of data it is modified in the meanwhile by the camera, deleting the graphics?
Or is there some other reason?
Why is the image lagged in that way, i.e. I would expect that the delay due to calculations only had the effect of skipping the frames received by the camera when the calculation are not done yet, and de facto only reducing the frame rate; or alternatively that all frames are received and the delay due to calculations brings the system to process images gotten minutes before, because the queue of images to process rises over time.
Instead, the observed behavior seems hybrid between the two: there is a delay of a few seconds, but this seems not increased much as the capturing process goes on.
B. buffercopy = true; copyBitmap = false; refresh = false;
Here I make a deep copy of the buffer into a second buffer, following the advice of the OpenCv documentation.
Nothing changes. The second buffer doesn’t change its address in memory during the run.
C. buffercopy = false; copyBitmap = true; refresh = false;
Now the (deep) copy of the bitmap is made allocating every time a new space in memory.
The blinking effect has gone, but the lagging keep arising after a certain time.
D. buffercopy = false; copyBitmap = false; refresh = true;
As before.
Please help me explain these results!
If I may be so frank, it is a bit tedious to understand all the details of your questions, but let me make a few points to help you analyse your results.
In case A, you say you perform calculations directly on the buffer. The documentation says you shouldn't do this, so if you do, you can expect undefined results. OpenCV assumes you won't touch it, so it might do stuff like suddenly delete that part of memory, let some other app process it, etc. It might look like it works, but you can never know for sure, so don't do it *slaps your wrist* In particular, if your processing takes a long time, the camera might overwrite the buffer while you're in the middle of processing it.
The way you should do it is to copy the buffer before doing anything. This will give you a piece of memory that is yours to do with whatever you wish. You can create a Bitmap that refers to this memory, and manually free the memory when you no longer need it.
If your processing rate (frames processed per second) is less than the number of frames captured per second by the camera, you have to expect some frames will be dropped. If you want to show a live view of the processed images, it will lag and there's no simple way around it. If it is vital that your application processes a fluid video (e.g. this might be necessary if you're tracking an object), then consider storing the video to disk so you don't have to process in real-time. You can also consider multithreading to process several frames at once, but the live view would have a latency.
By the way, is there any particular reason why you're not using EmguCV? It has abstractions for the camera and a system that raises an event whenever the camera has captured a new frame. This way, you don't need to continuously call cvQueryFrame on a background thread.
I think that you still have a problem with your UnmanageCopy method in that you only clone the image the first time this is called and you subsequently copy it. I believe that you need to do a cvCloneImage(f) every time as copy performs only a shallow copy, not a deep copy as you seem to think.
Related
I have a winforms application, which has a gif on it for letting users know about stalling processes.
The problem is it plays much slower than it seems on other applications (chrome, internet explorer).
I have tried the gif on PictureBox and Label but resulting speed is same. Then after a little research I've come accross this question and the answer of legendary #Hans Passant, but unfortunately applying the boilerplate code suggested by him didn't make any difference.
Below is the simple reproducing code:
public partial class Form1 : Form
{
public Form1 ()
{
InitializeComponent();
timeBeginPeriod(timerAccuracy);
}
protected override void OnFormClosed ( FormClosedEventArgs e )
{
timeEndPeriod(timerAccuracy);
base.OnFormClosed(e);
}
// Pinvoke:
private const int timerAccuracy = 10;
[System.Runtime.InteropServices.DllImport("winmm.dll")]
private static extern int timeBeginPeriod ( int msec );
[System.Runtime.InteropServices.DllImport("winmm.dll")]
public static extern int timeEndPeriod ( int msec );
}
And the designer code if needed:
partial class Form1
{
private System.ComponentModel.IContainer components = null;
protected override void Dispose ( bool disposing )
{
if (disposing && (components != null))
{
components.Dispose();
}
base.Dispose(disposing);
}
#region Windows Form Designer generated code
private void InitializeComponent ()
{
System.ComponentModel.ComponentResourceManager resources = new System.ComponentModel.ComponentResourceManager(typeof(Form1));
this.pictureBox1 = new System.Windows.Forms.PictureBox();
this.label1 = new System.Windows.Forms.Label();
((System.ComponentModel.ISupportInitialize)(this.pictureBox1)).BeginInit();
this.SuspendLayout();
//
// pictureBox1
//
this.pictureBox1.BorderStyle = System.Windows.Forms.BorderStyle.FixedSingle;
this.pictureBox1.Image = ((System.Drawing.Image)(resources.GetObject("pictureBox1.Image")));
this.pictureBox1.Location = new System.Drawing.Point(8, 9);
this.pictureBox1.Name = "pictureBox1";
this.pictureBox1.Size = new System.Drawing.Size(166, 119);
this.pictureBox1.SizeMode = System.Windows.Forms.PictureBoxSizeMode.CenterImage;
this.pictureBox1.TabIndex = 0;
this.pictureBox1.TabStop = false;
//
// label1
//
this.label1.BorderStyle = System.Windows.Forms.BorderStyle.FixedSingle;
this.label1.FlatStyle = System.Windows.Forms.FlatStyle.Flat;
this.label1.Image = ((System.Drawing.Image)(resources.GetObject("label1.Image")));
this.label1.Location = new System.Drawing.Point(180, 9);
this.label1.Name = "label1";
this.label1.Size = new System.Drawing.Size(158, 119);
this.label1.TabIndex = 1;
//
// Form1
//
this.AutoScaleDimensions = new System.Drawing.SizeF(6F, 13F);
this.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font;
this.ClientSize = new System.Drawing.Size(346, 134);
this.Controls.Add(this.label1);
this.Controls.Add(this.pictureBox1);
this.Name = "Form1";
this.Text = "Form1";
((System.ComponentModel.ISupportInitialize)(this.pictureBox1)).EndInit();
this.ResumeLayout(false);
}
#endregion
private System.Windows.Forms.PictureBox pictureBox1;
private System.Windows.Forms.Label label1;
}
Both gifs play at same speed, but lower than the actual gif. Is there any other points that I should be aware of while applying this code?
You can only get guesses, I doubt anybody will have much luck getting a repro:
timeBeginPeriod() can technically fail, although that is very unusual when you ask for 10 msec, verify that it returns 0.
If the image is large then it might just not be able to update fast enough. Or your UI thread is occupied too much with other duties. The pixel format of a gif is a poor match with the pixel format of the video adapter on a modern machine. The conversion is done every time the frame is updated. It is fairly expensive, especially so if you also force the image to be rescaled (i.e. PictureBox.SizeMode != Normal). Use Task Manager to verify that your UI thread is not burning 100% core.
You can get a second opinion about the effective timer period by running powercfg /energy from an elevated command prompt. Do so while your app is running. It will trundle for a minute and then generate an HTML file that you can look at with your browser. Reported under the "Platform Timer Resolution:Timer Request Stack" heading, the Requested Period value should be 10000. Beware that other processes or drivers might also have made requests.
Update 2021-04-02
The underlying reason behind PictureBox animating at a low framerate is because it uses the ImageAnimator class behind the scenes which only ever animates at 20 FPS. It has a hard-coded 50ms Thread.Sleep() in its worker thread method here:
https://referencesource.microsoft.com/#System.Drawing/commonui/System/Drawing/ImageAnimator.cs,333
I didn't have access to the Windows Forms source originally when the below was written, but now I'd probably subclass PictureBox and make it use a different ImageAnimator implementation to reduce that Thread.Sleep() to a reasonable value. (ImageAnimator is a sealed class so you'd have to copy the code into a new class and reference that in your PictureBox instead for smoother animations.)
Original Answer:
PictureBox is quite a heavyweight control and I'd recommend using something like Panel to house your animated GIF instead. In addition, I've read that PictureBox's internal animation timer is low resolution, meaning selecting an update interval of <100ms results in it rounding up to a 100ms update.
Instead you can control the painting and animating yourself. This uses PInvoke because it utilises some kernel timer methods. Example code is below:
using System;
using System.Diagnostics;
using System.Drawing;
using System.Drawing.Imaging;
using System.Runtime.InteropServices;
using System.Windows.Forms;
...
public partial class Form1 : Form
{
[DllImport("kernel32.dll")]
static extern bool CreateTimerQueueTimer(out IntPtr phNewTimer,
IntPtr TimerQueue, WaitOrTimerDelegate Callback, IntPtr Parameter,
uint DueTime, uint Period, uint Flags);
[DllImport("kernel32.dll")]
static extern bool ChangeTimerQueueTimer(IntPtr TimerQueue, IntPtr Timer,
uint DueTime, uint Period);
[DllImport("kernel32.dll")]
static extern bool DeleteTimerQueueTimer(IntPtr TimerQueue,
IntPtr Timer, IntPtr CompletionEvent);
public delegate void WaitOrTimerDelegate(IntPtr lpParameter,
bool TimerOrWaitFired);
// Holds a reference to the function to be called when the timer
// fires
public static WaitOrTimerDelegate UpdateFn;
public enum ExecuteFlags
{
/// <summary>
/// The callback function is queued to an I/O worker thread. This flag should be used if the function should be executed in a thread that waits in an alertable state.
/// The callback function is queued as an APC. Be sure to address reentrancy issues if the function performs an alertable wait operation.
/// </summary>
WT_EXECUTEINIOTHREAD = 0x00000001,
};
private Image gif;
private int frameCount = -1;
private UInt32[] frameIntervals;
private int currentFrame = 0;
private static object locker = new object();
private IntPtr timerPtr;
public Form1()
{
InitializeComponent();
// Attempt to reduce flicker - all control painting must be
// done in overridden paint methods
this.SetStyle(ControlStyles.AllPaintingInWmPaint |
ControlStyles.OptimizedDoubleBuffer, true);
// Set the timer callback
UpdateFn = new WaitOrTimerDelegate(UpdateFrame);
}
private void Form1_Load(object sender, EventArgs e)
{
// Replace this with whatever image you're animating
gif = (Image)Properties.Resources.SomeAnimatedGif;
// How many frames of animation are there in total?
frameCount = gif.GetFrameCount(FrameDimension.Time);
// Retrieve the frame time property
PropertyItem propItem = gif.GetPropertyItem(20736);
int propIndex = 0;
frameIntervals = new UInt32[frameCount];
// Each frame can have a different timing - retrieve each of them
for (int i = 0; i < frameCount; i++)
{
// NB: intervals are given in hundredths of a second, so need
// multiplying to match the timer's millisecond interval
frameIntervals[i] = BitConverter.ToUInt32(propItem.Value,
propIndex) * 10;
// Point to the next interval stored in this property
propIndex += 4;
}
// Show the first frame of the animation
ShowFrame();
// Start the animation. We use a TimerQueueTimer which has better
// resolution than Windows Forms' default one. It should be used
// instead of the multimedia timer, which has been deprecated
CreateTimerQueueTimer(out this.timerPtr, IntPtr.Zero, UpdateFn,
IntPtr.Zero, frameIntervals[0], 100000,
(uint)ExecuteFlags.WT_EXECUTEINIOTHREAD);
}
private void UpdateFrame(IntPtr lpParam, bool timerOrWaitFired)
{
// The timer has elapsed
// Update the number of the frame to show next
currentFrame = (currentFrame + 1) % frameCount;
// Paint the frame to the panel
ShowFrame();
// Re-start the timer after updating its interval to that of
// the new frame
ChangeTimerQueueTimer(IntPtr.Zero, this.timerPtr,
frameIntervals[currentFrame], 100000);
}
private void ShowFrame()
{
// We need to use a lock as we cannot update the GIF at the
// same time as it's being drawn
lock (locker)
{
gif.SelectActiveFrame(FrameDimension.Time, currentFrame);
}
this.panel1.Invalidate();
}
private void panel1_Paint(object sender, PaintEventArgs e)
{
base.OnPaint(e);
lock (locker)
{
e.Graphics.DrawImage(gif, panel1.ClientRectangle);
}
}
private void Form1_FormClosing(object sender, FormClosingEventArgs e)
{
DeleteTimerQueueTimer(IntPtr.Zero, timerPtr, IntPtr.Zero);
}
}
Note: we set the Period on the timer calls to 100000 because if you set it to 0 (to indicate a one-off timing), it will only fire once, even if you subsequently call ChangeTimerQueueTimer.
The timers are still not suitable for super-accurate timings, but this should still give you a faster update than would otherwise have been possible with PictureBox.
I have an application where performance-sensitive drawings occur using a WriteableBitmap. An event is called with CompositionTarget.Rendering to actually update the back buffer of the WriteableBitmap. From the MSDN documentation, that means the event is fired once per frame, right before the control is rendered.
The issue that I am having is that the WriteableBitmap's Lock() function takes an extremely long time, especially at larger bitmap sizes. I have previously read that AddDirtyRegion() has a bug that causes the entire bitmap to invalidate, leading to poor performance. However, that doesn't seem to be the case here. From a good bit of low-level checking, it seems that Lock() opens the bitmap's backbuffer for writing on the render thread, which means every time my event handler is called, it has to thread block until the render thread is ready for it. This leads to a noticeable lag when updating the graphics of the bitmap.
I have already tried adding a timeout to the event handler, using TryLock(), so that it won't block for such a long time and cause the performance degradation. This, however, causes a similar effect in that it appears to lag, because larger numbers of bitmap updates get lumped together.
Here is the relevant code from the event handler to show what exactly I am doing. The UpdatePixels() function was written to avoid using the potentially bugged AddDirtyRect():
void updateBitmap(object sender, EventArgs e)
{
if (!form.ResizingWindow)
{
// Lock and unlock are important... Make sure to keep them outside of the loop for performance reasons.
if (canvasUpdates.Count > 0)
{
//bool locked = scaledDrawingMap.TryLock(bitmapLockDuration);
scaledDrawingMap.Lock();
//if (locked)
//{
unsafe
{
int* pixData = (int*)scaledDrawingMap.BackBuffer;
foreach (Int32Rect i in canvasUpdates)
{
// The graphics object isn't directly shown, so this isn't actually necessary. We do a sort of manual copy from the drawingMap, which acts similarly
// to a back buffer.
Int32Rect temp = GetValidDirtyRegion(i);
UpdatePixels(temp, pixData);
}
scaledDrawingMap.Unlock();
canvasUpdates.Clear();
}
//}
}
}
}
private unsafe void UpdatePixels(Int32Rect temp, int* pixData)
{
//int* pixData = (int*)scaledDrawingMap.BackBuffer;
// Directly copy the backbuffer into a new buffer, to use WritePixels().
var stride = temp.Width * scaledDrawingMap.Format.BitsPerPixel / 8;
int[] relevantPixData = new int[stride * temp.Height];
int srcIdx = 0;
int pWidth = scaledDrawingMap.PixelWidth;
int yLess = temp.Y + temp.Height;
int xLess = temp.X + temp.Width;
for (int y = temp.Y; y < yLess; y++)
{
for (int x = temp.X; x < xLess; x++)
{
relevantPixData[srcIdx++] = pixData[y * pWidth + x];
}
}
scaledDrawingMap.WritePixels(temp, relevantPixData, stride, 0);
}
I can't seem to figure out how to avoid the issue of thread blocking with the WriteableBitmap, and I can't see any obvious faults in the code I have written. Any help or pointers would be much appreciated.
Looks like you are not actually using the BackBuffer to write - only to read.
WritePixels writes to the "front" buffer and does not require a lock.
I don't know if you have some other reason to lock it (other threads doing something), but for the code that's here i don't see why you would need to.
I guess I was wrong about not needing a lock to read from BackBuffer (*pixData) - I thought it was only for writes, but I am positive you do not need to to call Lock for WritePixels.
As far as I can tell, you are doing:
Lock the back buffer
Copy something from it to an array
Call WritePixels using this new array
Unlock the back buffer.
How about switching 3 and 4?
WritePixels may internally cause rendering thread (which has higher priority on the message queue) to get a lock on its behalf which is probably a factor in the delay you are seeing.
I capture images from a webcam, do some heavy processing on them, and then show the result. To keep the framerate high, i want to have the processing of different frames run in parallel.
So, I have a 'Producer', which captures the images and adds these to the 'inQueue'; also it takes an image from the 'outQueue' and displays it:
public class Producer
{
Capture capture;
Queue<Image<Bgr, Byte>> inQueue;
Queue<Image<Bgr, Byte>> outQueue;
Object lockObject;
Emgu.CV.UI.ImageBox screen;
public int frameCounter = 0;
public Producer(Emgu.CV.UI.ImageBox screen, Capture capture, Queue<Image<Bgr, Byte>> inQueue, Queue<Image<Bgr, Byte>> outQueue, Object lockObject)
{
this.screen = screen;
this.capture = capture;
this.inQueue = inQueue;
this.outQueue = outQueue;
this.lockObject = lockObject;
}
public void produce()
{
while (true)
{
lock (lockObject)
{
inQueue.Enqueue(capture.QueryFrame());
if (inQueue.Count == 1)
{
Monitor.PulseAll(lockObject);
}
if (outQueue.Count > 0)
{
screen.Image = outQueue.Dequeue();
}
}
frameCounter++;
}
}
}
There are different 'Consumers' who take an image from the inQueue, do some processing, and add them to the outQueue:
public class Consumer
{
Queue<Image<Bgr, Byte>> inQueue;
Queue<Image<Bgr, Byte>> outQueue;
Object lockObject;
string name;
Image<Bgr, Byte> image;
public Consumer(Queue<Image<Bgr, Byte>> inQueue, Queue<Image<Bgr, Byte>> outQueue, Object lockObject, string name)
{
this.inQueue = inQueue;
this.outQueue = outQueue;
this.lockObject = lockObject;
this.name = name;
}
public void consume()
{
while (true)
{
lock (lockObject)
{
if (inQueue.Count == 0)
{
Monitor.Wait(lockObject);
continue;
}
image = inQueue.Dequeue();
}
// Do some heavy processing with the image
lock (lockObject)
{
outQueue.Enqueue(image);
}
}
}
}
Rest of the important code is this section:
private void Form1_Load(object sender, EventArgs e)
{
Consumer[] c = new Consumer[consumerCount];
Thread[] t = new Thread[consumerCount];
Object lockObj = new object();
Queue<Image<Bgr, Byte>> inQueue = new Queue<Image<Bgr, Byte>>();
Queue<Image<Bgr, Byte>> outQueue = new Queue<Image<Bgr, Byte>>();
p = new Producer(screen1, capture, inQueue, outQueue, lockObj);
for (int i = 0; i < consumerCount; i++)
{
c[i] = new Consumer(inQueue, outQueue, lockObj, "c_" + Convert.ToString(i));
}
for (int i = 0; i < consumerCount; i++)
{
t[i] = new Thread(c[i].consume);
t[i].Start();
}
Thread pt = new Thread(p.produce);
pt.Start();
}
The parallelisation actually works fine, I do get a linear speed increase with each added thread (up to a certain point of course). The problem is that I get artifacts in the output, even if running only one thread. The artifacts look like part of the picture is not in the right place.
Example of the artifact (this is without any processing to keep it clear, but the effect is the same)
Any ideas what causes this?
Thanks
Displaimer: This post isn't supposed to fully describe an answer, but instead give some hints on why the artifact is being shown.
A quick analysis show that the the actifact is, in fact, a partial, vertically mirrored snippet of a frame. I copied it, mirrored, and placed it back over the image, and added an awful marker to show its placement:
Two things immediately come to attention:
The artifact is roughly positioned on the 'correct' place it would be, only that the position is also vertically mirrored;
The image is slightly different, indicating that it may belong to a different frame.
It's been a while since I played around with raw capture and ran into a similar issue, but I remember that depending on how the driver is implemented (or set up - this particular issue happened when setting a specific imaging device for interlaced capture) it may fill its framebuffer alternating between 'top-down' and 'bottom-up' scans - as soon as the frame is full, the 'cursor' reverts direction.
It seems to me that you're running into a race condition/buffer underrun situation, where the transfer from the framebuffer to your application is happening before the full frame is transferred by the device.
In that case, you'd receive a partial image, and the area still not refreshed would show a bit of the previously transferred frame.
If I'd have to bet, I'd say that the artifact may appear on sequential order, not on the same position but 'fluctuating' on a specific direction (up or down), but always as a mirrored bit.
Well, I think the problem is here . The section of code is not guarantee that you will be access by one thread in here between two queue. The image is pop by inQueue is not actually received in order in outQueue
while (true)
{
lock (lockObject)
{
if (inQueue.Count == 0)
{
Monitor.Wait(lockObject);
continue;
}
image = inQueue.Dequeue();
}
// Do some heavy processing with the image
lock (lockObject)
{
outQueue.Enqueue(image);
}
}
Similar to #OnoSendai, I'm not trying to solve the exact problem as stated. I would have to write an app and I just don't have the time. But, the two things that I would change right away would be to use the ConcurrentQueue class so that you have thread-safety. And, I would use the Task library functions in order to create parallel tasks on different processor cores. These are found in the System.Net and System.Net.Task namespaces.
Also, vertically flipping a chunk like that looks like more than an artifact to me. If it also happens when executing in a single thread as you mentioned, then I would definitely re-focus on the "heavy processing" part of the equation.
Good luck! Take care.
You may have two problems:
1) parallism doesn't ensure that images are added to the out queue in the right order. I imagine that displaying image 8 before image 6 and 7 can produce some artifacts. In consumer thread, you have to wait previous consumer have posted its image to the out queue to post next image. Tasks can help greatly for that because of their inherent synchronisation mecanism.
2) You may also have problems in the rendering code.
I have the following function to convert a byte array of pixels into an image. However, I have a memory leak in the line:
unpackedImage = UIImage.FromImage(context.ToImage());
When I comment out the line above, the leak goes away. The leak is so bad that iOS is killing my app within about 30 seconds of startup. It's because of this line of code.
How do I prevent this memory leak? Is there a better way to do what I am trying to do?
public static void DrawCustomImage2(IntPtr buffer, int width, int height, int bytesPerRow, CGColorSpace colSpace, byte[] rawPixels, ref UIImage unpackedImage)
{
GCHandle pinnedArray = GCHandle.Alloc(rawPixels, GCHandleType.Pinned);
IntPtr pointer = pinnedArray.AddrOfPinnedObject();
// Set a grayscale drawing context using the image buffer
CGBitmapContext context = new CGBitmapContext(pointer, width, height, 8, bytesPerRow, colSpace, CGImageAlphaInfo.None);
// Turning off interpolation and Antialiasing is supposed to speed things up
context.InterpolationQuality = CGInterpolationQuality.None;
context.SetAllowsAntialiasing(false);
try
{
unpackedImage = UIImage.FromImage(context.ToImage()); // Convert the drawing context to an image and set it as the unpacked image
} finally
{
pinnedArray.Free();
if (context != null)
context.Dispose();
}
}
Here is the profiling screenshot (checked items all disappear when the critical line of code is commented out). You can see how the checked items (especially the Malloc) grow over time.
Here is the zoom-in view on the Malloc 1.50KB. You can see in the Extended Detail pane on the right that it is calling CGBitmapContextCreateImage and CGDataProviderCreateWithCopyOfData and then a malloc.
Here is the profiling screenshot with Rolf's suggestion. I ran the image loop twice. You can see that it cleans up the extra memory at the end of the first loop, but the system didn't clean it up fast enough the second time and iOS killed my app (you can see the low memory warning flags in the top right corner).
Do it like this:
using (var pool = new NSAutoreleasePool ()) {
using (var img = context.ToImage ()) {
unpackedImage = UIImage.FromImage (img);
}
}
I’m having a problem using StretchBlt (or BitBlt) to copy an image from a PictureBox to a Bitmap for use in creating an AVI file.
The AVI file is an unimportant aspect I believe - I can just display the Bitmap on another PictureBox and see the problem.
The problem is that occasionally (not often) a single image is flipped (mirrored across the X axis, never the Y axis).
I’m not sure if this is a known problem with StretchBlt I haven’t yet found mention of or if I am doing something wrong.
Note this is NOT due to the intended functionality with StretchBlt of "If the signs of source and destination height or width are different then it creates a mirror image".
UPDATE: I changed things to force the source/destination to be the same size, and am using BitBlt with the same behavior.
I’ve included some code (c#), hopefully all of the important parts.
Stepping through the code I can see this happen for a single image that has exactly the same information being passed to StretchBlt (other than the hdc to copy to) as the previous image and the next image (and next, next) all of which are fine.
It doesn't happen often, and I dont see any reason when it does. Or a way to detect it happened (so I can flip it back).
I have a work around that doesn't use StretchBlt, but it is much slower and really degrades performance.
Another possibly useful bit: this flipped image is rare in normal usage (less than 1 in 100 frames). But when run in the IDE, stepping through image by image, it happens very regularly (maybe 1 in 10).
Any ideas what could be causing this or what I could be doing wrong? Or other FAST methods to copy the Bitmap including re-sizing.
NOTE: The bitmaps do vary in size (can't use BitBlt), but not by a lot.
Thank you!
Kate
// --- Import statement
[DllImport("GDI32.DLL", CharSet = CharSet.Auto, SetLastError = true, ExactSpelling = true)]
public static extern bool StretchBlt(
IntPtr hdcDest, int nXDest, int nYDest, int nDestWidth, int nDestHeight,
IntPtr hdcSrc, int nXSrc, int nYSrc, int nSrcWidth, int nSrcHeight, Int32 dwRop );
// --- A small class for creating/storing Bitmap and Graphics objects, which get reused
public class CAviImageInfo
{
private Bitmap mAviBitmap = null;
private Graphics mAviGraphics = null;
public CAviImageInfo(int width, int height )
{
mAviBitmap = new Bitmap(width, height);
mAviGraphics = Graphics.FromImage(mAviBitmap);
}
~CAviImageInfo()
{
mAviGraphics.Dispose();
mAviGraphics = null;
}
public Bitmap AviBitmap
{ get { return mAviBitmap; } }
public Graphics AviGraphics
{ get { return mAviGraphics; } }
}
// --- Two PictureBoxs placed on form at design time (one is just to watch for these mirrored images):
PictureBox mMainPictureBox; // --- Displays the images to be copied
PictureBox DebugPictureBox;
// --- The following done one time:
Graphics mMainPictureBoxGraphics = mMainPictureBox.CreateGraphics();
IntPtr mMainPictureBoxHdc = mMainPictureBoxGraphics.GetHdc();
// --- Method that does the copying. Called each time image on panel is updated.
Public void UpdateAviRecording()
{
// --- Gets unused Bitmap and Graphics objects (these are reused)
CAviImageInfo aviImageInfo = GetUnusedAviImageInfo();
IntPtr destinationHdc = aviImageInfo.AviGraphics.GetHdc();
StretchBlt(
destinationHdc,
0, 0, aviImageInfo.AviBitmap.Width, aviImageInfo.AviBitmap.Height,
mMainPictureBoxHdc,
0, 0, mMainPictureBox.Width, mMainPictureBox.Height, SRCCOPY);
// --- Show the copied Bitmap on the debug PictureBox
// --- (normally would pass it to be written to avi file)
DebugPictureBox.Image = aviImageInfo.AviBitmap;
DebugPictureBox.Refresh();
aviImageInfo.AviGraphics.ReleaseHdc(destinationHdc);
}
this is just a suggestion. I don't know if it will work..
Set the width and height using Math.Abs() of the image in the call to StretchBlt.
That might prevent possible memory corruption and inversion of signs (as GSerg mentioned).
..
StretchBlt(
destinationHdc,
0, 0, Math.Abs(aviImageInfo.AviBitmap.Width),
Math.Abs(aviImageInfo.AviBitmap.Height),
mMainPictureBoxHdc,
0, 0, Math.Abs(mMainPictureBox.Width),
Math.Abs(mMainPictureBox.Height), SRCCOPY);