We Keep Coding

Experiment on displaying a Bitmap retrieved from a camera on a Picturebox

In my code I retrieve frames from a camera with a pointer to an unmanaged object, make some calculations on it and then I make it visualized on a picturebox control.
Before I go further in this application with all the details, I want to be sure that the base code for this process is good.
In particular I would like to:
- keep execution time minimal and avoid unnecessary operations, such as
copying more images than necessary. I want to keep only essential
operations
- understand if a delay in the calculation process on every frame could have detrimental effects on the way images are shown (i.e. if it is not printed what I expect) or some image is skipped
- prevent more serious errors, such as ones due to memory or thread management, or to image display.
For this purpose, I set up a few experimental lines of code (below), but I’m not able to explain the results of what I found. If you have the executables of OpenCv you can make a try by yourself.
using System;
using System.Drawing;
using System.Drawing.Imaging;
using System.Windows.Forms;
using System.Runtime.InteropServices;
using System.Threading;
public partial class FormX : Form
{
private delegate void setImageCallback();
Bitmap _bmp;
Bitmap _bmp_draw;
bool _exit;
double _x;
IntPtr _ImgBuffer;
bool buffercopy;
bool copyBitmap;
bool refresh;
public FormX()
{
InitializeComponent();
_x = 10.1;
// set experimemental parameters
buffercopy = false;
copyBitmap = false;
refresh = true;
}
private void buttonStart_Click(object sender, EventArgs e)
{
Thread camThread = new Thread(new ThreadStart(Cycle));
camThread.Start();
}
private void buttonStop_Click(object sender, EventArgs e)
{
_exit = true;
}
private void Cycle()
{
_ImgBuffer = IntPtr.Zero;
_exit = false;
IntPtr vcap = cvCreateCameraCapture(0);
while (!_exit)
{
IntPtr frame = cvQueryFrame(vcap);
if (buffercopy)
{
UnmanageCopy(frame);
_bmp = SharedBitmap(_ImgBuffer);
}
else
{ _bmp = SharedBitmap(frame); }
// make calculations
int N = 1000000; /*1000000*/
for (int i = 0; i < N; i++)
_x = Math.Sin(0.999999 * _x);
ShowFrame();
}
cvReleaseImage(ref _ImgBuffer);
cvReleaseCapture(ref vcap);
}
private void ShowFrame()
{
if (pbCam.InvokeRequired)
{
this.Invoke(new setImageCallback(ShowFrame));
}
else
{
Pen RectangleDtPen = new Pen(Color.Azure, 3);
if (copyBitmap)
{
if (_bmp_draw != null) _bmp_draw.Dispose();
//_bmp_draw = new Bitmap(_bmp); // deep copy
_bmp_draw = _bmp.Clone(new Rectangle(0, 0, _bmp.Width, _bmp.Height), _bmp.PixelFormat);
}
else
{
_bmp_draw = _bmp; // add reference to the same object
}
Graphics g = Graphics.FromImage(_bmp_draw);
String drawString = _x.ToString();
Font drawFont = new Font("Arial", 56);
SolidBrush drawBrush = new SolidBrush(Color.Red);
PointF drawPoint = new PointF(10.0F, 10.0F);
g.DrawString(drawString, drawFont, drawBrush, drawPoint);
drawPoint = new PointF(10.0F, 300.0F);
g.DrawString(drawString, drawFont, drawBrush, drawPoint);
g.DrawRectangle(RectangleDtPen, 12, 12, 200, 400);
g.Dispose();
pbCam.Image = _bmp_draw;
if (refresh) pbCam.Refresh();
}
}
public void UnmanageCopy(IntPtr f)
{
if (_ImgBuffer == IntPtr.Zero)
_ImgBuffer = cvCloneImage(f);
else
cvCopy(f, _ImgBuffer, IntPtr.Zero);
}
// only works with 3 channel images from camera! (to keep code minimal)
public Bitmap SharedBitmap(IntPtr ipl)
{
// gets unmanaged data from pointer to IplImage:
IntPtr scan0;
int step;
Size size;
OpenCvCall.cvGetRawData(ipl, out scan0, out step, out size);
return new Bitmap(size.Width, size.Height, step, PixelFormat.Format24bppRgb, scan0);
}
// based on older version of OpenCv. Change dll name if different
[DllImport( "opencv_highgui246", CallingConvention = CallingConvention.Cdecl)]
public static extern IntPtr cvCreateCameraCapture(int index);
[DllImport("opencv_highgui246", CallingConvention = CallingConvention.Cdecl)]
public static extern void cvReleaseCapture(ref IntPtr capture);
[DllImport("opencv_highgui246", CallingConvention = CallingConvention.Cdecl)]
public static extern IntPtr cvQueryFrame(IntPtr capture);
[DllImport("opencv_core246", CallingConvention = CallingConvention.Cdecl)]
public static extern void cvGetRawData(IntPtr arr, out IntPtr data, out int step, out Size roiSize);
[DllImport("opencv_core246", CallingConvention = CallingConvention.Cdecl)]
public static extern void cvCopy(IntPtr src, IntPtr dst, IntPtr mask);
[DllImport("opencv_core246", CallingConvention = CallingConvention.Cdecl)]
public static extern IntPtr cvCloneImage(IntPtr src);
[DllImport("opencv_core246", CallingConvention = CallingConvention.Cdecl)]
public static extern void cvReleaseImage(ref IntPtr image);
}
results [dual core 2 Duo T6600 2.2 GHz]:
A. buffercopy = false; copyBitmap = false; refresh = false;
This is the simpler configuration. Each frame is retrieved in turn, operations are made (in the reality they are based on the same frame, here just calculations), then the result of the calculations is printed on top of the image and finally it is displayed on a picturebox.
OpenCv documentation says:
OpenCV 1.x functions cvRetrieveFrame and cv.RetrieveFrame return image
stored inside the video capturing structure. It is not allowed to
modify or release the image! You can copy the frame using
cvCloneImage() and then do whatever you want with the copy.
But this doesn’t prevent us from doing experiments.
If the calculation are not intense (low number of iterations, N), everything is just ok and the fact that we manipulate the image buffer own by the unmanaged frame retriever doesn’t pose a problem here.
The reason is that probably they advise to leave untouched the buffer, in case people would modify its structure (not its values) or do operations asynchronously without realizing it. Now we retrieve frames and modify their content in turn.
If N is increased (N=1000000 or more), when the number of frames per second is not high, for example with artificial light and low exposure, everything seems ok, but after a while the video is lagged and the graphics impressed on it are blinking. With a higher frame rate the blinking appears from the beginning, even when the video is still fluid.
Is this because the mechanism of displaying images on the control (or refreshing or whatever else) is somehow asynchronous and when the picturebox is fetching its buffer of data it is modified in the meanwhile by the camera, deleting the graphics?
Or is there some other reason?
Why is the image lagged in that way, i.e. I would expect that the delay due to calculations only had the effect of skipping the frames received by the camera when the calculation are not done yet, and de facto only reducing the frame rate; or alternatively that all frames are received and the delay due to calculations brings the system to process images gotten minutes before, because the queue of images to process rises over time.
Instead, the observed behavior seems hybrid between the two: there is a delay of a few seconds, but this seems not increased much as the capturing process goes on.
B. buffercopy = true; copyBitmap = false; refresh = false;
Here I make a deep copy of the buffer into a second buffer, following the advice of the OpenCv documentation.
Nothing changes. The second buffer doesn’t change its address in memory during the run.
C. buffercopy = false; copyBitmap = true; refresh = false;
Now the (deep) copy of the bitmap is made allocating every time a new space in memory.
The blinking effect has gone, but the lagging keep arising after a certain time.
D. buffercopy = false; copyBitmap = false; refresh = true;
As before.
Please help me explain these results!

If I may be so frank, it is a bit tedious to understand all the details of your questions, but let me make a few points to help you analyse your results.
In case A, you say you perform calculations directly on the buffer. The documentation says you shouldn't do this, so if you do, you can expect undefined results. OpenCV assumes you won't touch it, so it might do stuff like suddenly delete that part of memory, let some other app process it, etc. It might look like it works, but you can never know for sure, so don't do it *slaps your wrist* In particular, if your processing takes a long time, the camera might overwrite the buffer while you're in the middle of processing it.
The way you should do it is to copy the buffer before doing anything. This will give you a piece of memory that is yours to do with whatever you wish. You can create a Bitmap that refers to this memory, and manually free the memory when you no longer need it.
If your processing rate (frames processed per second) is less than the number of frames captured per second by the camera, you have to expect some frames will be dropped. If you want to show a live view of the processed images, it will lag and there's no simple way around it. If it is vital that your application processes a fluid video (e.g. this might be necessary if you're tracking an object), then consider storing the video to disk so you don't have to process in real-time. You can also consider multithreading to process several frames at once, but the live view would have a latency.
By the way, is there any particular reason why you're not using EmguCV? It has abstractions for the camera and a system that raises an event whenever the camera has captured a new frame. This way, you don't need to continuously call cvQueryFrame on a background thread.

I think that you still have a problem with your UnmanageCopy method in that you only clone the image the first time this is called and you subsequently copy it. I believe that you need to do a cvCloneImage(f) every time as copy performs only a shallow copy, not a deep copy as you seem to think.

Get/set pixel from Graphics object GDI+

I'm trying to find an alternative solution to getting/setting pixels at certain position in a Graphics object.
Right now I'm using GDI functions:
[DllImport("gdi32.dll")]
public static extern int GetPixel(System.IntPtr hdc, int nXPos, int nYPos);
[DllImport("gdi32.dll")]
public static extern uint SetPixel(IntPtr hdc, int X, int Y, int crColor);
I couldn't find any for GDI+. GetPixel/SetPixel seems to be on Bitmap object only.
The alternatives from gdi32.dll work well when the Graphics is backed by the screen but when using Graphics with a bitmap it doesn't work anymore (GetPixel returns black, since this works on another bitmap not the actual one): http://support.microsoft.com/kb/311221
Some sample code:
private void ChangeImage(Graphics g)
{
IntPtr gDC = IntPtr.Zero;
try
{
gDC = g.GetHdc();
// get the pixel color
Color pixel = ColorTranslator.FromWin32(GetPixel(gDC, x, y));
//change pixel object and persist
SetPixel(gDC, x, y, ColorTranslator.ToWin32(pixel));
}
catch (Exception ex)
{
Trace.WriteLine(ex.Message);
}
finally
{
if (gDC != IntPtr.Zero)
g.ReleaseHdc(gDC);
}
}
Is there any way to parse the the Graphics object per pixel basis?
The Graphics object represents a surface where the user is free to add any objects including hand drawing. On top of this I need to apply filters (like blur or pixelation) on some screen parts so I need to access what has been painted already in the graphics.
I've also tried to find a way to persist the current graphics to a Bitmap and use GetPixel from the bitmap object but I've also couldn't find a way to save Graphics content.
Thx

No, it's not possible to read pixels from a Graphics object directly. You would need access to the underlying HDC or Bitmap object to do this.

Dim bmap As New Bitmap(550, 100)
Dim g As Graphics = Graphics.FromImage(bmap)
' Dont use the Graphics interface, use the BitMap interface
DIM col as color = bmap.GetPixel(x,y)

C# interop with CUDA C dll - redux

Asked a few questions about a project I was working on, got some good feedback and made some progress. The idea is to create an application that generates images of fractals, accelerated by CUDA. I am creating the ui in C# and having a DLL do the heavy lifting.
Basically, I am allocating a byte array in C#, passing that to the dll to fill with pixel data, and then using that to create a Bitmap and display that with a Windows Forms PictureBox in the ui. Previous questions have helped - was using dll to allocate memory before, now using consistent calling convention between dll and c#, but the code still gives an System.ArgumentException at "img = new Bitmap(...)
Relevant Code:
C++
extern "C" __declspec(dllexport) void __cdecl generateBitmap(void *bitmap)
{
int width = 1920;
int height = 1080;
int *dev_bmp;
gpuErrchk(cudaMalloc((void**)&dev_bmp, (3*width*height*sizeof(int))));
kernel<<<BLOCKS_PER_GRID, THREADS_PER_BLOCK>>>(dev_bmp, width, height);
gpuErrchk(cudaPeekAtLastError());
gpuErrchk(cudaDeviceSynchronize());
gpuErrchk(cudaMemcpy(bitmap, dev_bmp, (width*height*3), cudaMemcpyDeviceToHost));
cudaFree(dev_bmp);
}
c#
public unsafe class NativeMethods
{
[DllImport(#"C:\Users\Bill\Documents\Visual Studio 2012\Projects\FractalMaxUnmanaged\Debug\FractalMaxUnmanaged.dll", CallingConvention=CallingConvention.Cdecl)]
public static extern void generateBitmap(void *bitmap);
public static Bitmap create()
{
byte[] buf = new byte[1920 * 1080 * 3];
fixed (void* pBuffer = buf)
{
generateBitmap(pBuffer);
}
IntPtr unmanagedPtr = Marshal.AllocHGlobal(buf.Length);
Marshal.Copy(buf, 0, unmanagedPtr, buf.Length);
Bitmap img = new Bitmap(1920, 1080, 3, PixelFormat.Format24bppRgb, unmanagedPtr);
Marshal.FreeHGlobal(unmanagedPtr);
return img;
}
}
//...
private unsafe void mandlebrotButton_Click(object sender, EventArgs e)
{
FractalBox1.Image = (Image)NativeMethods.create();
}
What am I still doing wrong? As far as I can tell, all the parameters are invalid, but I get an invalid parameter exception in System.Drawing when I try to create the bitmap.

I am not sure what happens exactly in your case cause you didn't specify which parameter is invalid in the exception. I see that your stride must not be correct.
stride Type: System.Int32
Integer that specifies the byte offset between the beginning of one
scan line and the next. This is usually (but not necessarily) the
number of bytes in the pixel format (for example, 2 for 16 bits per
pixel) multiplied by the width of the bitmap. The value passed to this
parameter must be a multiple of four..
So your constructor should be like this:
Bitmap img = new Bitmap(1920, 1080, 1920 * 3, PixelFormat.Format24bppRgb, unmanagedPtr);

How do I prevent a memory leak with UIImage.FromImage(context.ToImage())?

I have the following function to convert a byte array of pixels into an image. However, I have a memory leak in the line:
unpackedImage = UIImage.FromImage(context.ToImage());
When I comment out the line above, the leak goes away. The leak is so bad that iOS is killing my app within about 30 seconds of startup. It's because of this line of code.
How do I prevent this memory leak? Is there a better way to do what I am trying to do?
public static void DrawCustomImage2(IntPtr buffer, int width, int height, int bytesPerRow, CGColorSpace colSpace, byte[] rawPixels, ref UIImage unpackedImage)
{
GCHandle pinnedArray = GCHandle.Alloc(rawPixels, GCHandleType.Pinned);
IntPtr pointer = pinnedArray.AddrOfPinnedObject();
// Set a grayscale drawing context using the image buffer
CGBitmapContext context = new CGBitmapContext(pointer, width, height, 8, bytesPerRow, colSpace, CGImageAlphaInfo.None);
// Turning off interpolation and Antialiasing is supposed to speed things up
context.InterpolationQuality = CGInterpolationQuality.None;
context.SetAllowsAntialiasing(false);
try
{
unpackedImage = UIImage.FromImage(context.ToImage()); // Convert the drawing context to an image and set it as the unpacked image
} finally
{
pinnedArray.Free();
if (context != null)
context.Dispose();
}
}
Here is the profiling screenshot (checked items all disappear when the critical line of code is commented out). You can see how the checked items (especially the Malloc) grow over time.
Here is the zoom-in view on the Malloc 1.50KB. You can see in the Extended Detail pane on the right that it is calling CGBitmapContextCreateImage and CGDataProviderCreateWithCopyOfData and then a malloc.
Here is the profiling screenshot with Rolf's suggestion. I ran the image loop twice. You can see that it cleans up the extra memory at the end of the first loop, but the system didn't clean it up fast enough the second time and iOS killed my app (you can see the low memory warning flags in the top right corner).

Do it like this:
using (var pool = new NSAutoreleasePool ()) {
using (var img = context.ToImage ()) {
unpackedImage = UIImage.FromImage (img);
}
}

Getting out of memory exception when loading high resolution image with IImagingFactory

I am trying to load high resolution image(like 3264x2448). For that I am using IImageFactory class in C#. IImageFactory is downloaded from the Microsoft site. If I try to load the high resolution image I am getting “out of memory” exception. Have a look at the below sample:
IImage GetIImage(string fileName)
{
Bitmap bitmap = null;
Graphics graphics = null;
IntPtr hdcDestination = IntPtr.Zero;
try {
IImage image = null;
IImagingFactory imagingFactory = ImagingFactory.GetImaging();
imagingFactory.CreateImageFromFile(fileName, out image);
bitmap = new Bitmap(width, height);
graphics = Graphics.FromImage(bitmap);
hdcDestination = graphics.GetHdc();
Rectangle dstRect = new Rectangle(0, 0, width, height);
image.Draw(_graphicsHDC, ref dstRect, IntPtr.Zero);
}
catch{}
}
Using the above source lines I was able to load the images but my effort to draw image fails, throwing an OutOfMemory Excep :(
Further, I tried the same by creating a Win32 mobile App, in which I could bot Load and Draw the desired images with the following lines.
void DrawImage(HDC *hdc, char *FileName, int width, int height)
{
IImagingFactory *pImgFactory = NULL;
IImage *pImage = NULL;
RECT rc = { 0, 0, width, height};
WCHAR Name[MAX_PATH] ={0};
mbstowcs (Name, FileName, strlen (FileName));
CoInitializeEx(NULL, COINIT_MULTITHREADED);
if (SUCCEEDED(CoCreateInstance (CLSID_ImagingFactory,
NULL,
CLSCTX_INPROC_SERVER,
IID_IImagingFactory,
(void **)&pImgFactory)))
{
// Load the image from the JPG file.
if (SUCCEEDED(pImgFactory->CreateImageFromFile(Name, &pImage)))
{
pImage->Draw(*hdc, &rc, NULL);
pImage->Release();
}
pImgFactory->Release();
}
CoUninitialize();}
I even tried creating a win32 dll and invoke it from C# app but I couldn't.
Can anyone help me out to resolve this issue?

Of course it does. You loaded an image and then are trying to display the whole thing. To do so, the CF has to turn it into a Bitmap for the display driver. 3264x2448x16bpp == nearly 16MB. What you should be doing is generating a thumbnail and drawing that to the rectangle.
And, as usual, the Smart Device Framework already has this done for you.

Though you don't want to use OpenNETCF, I will offer it anyway:
This seems to be exactly what you are experiencing.
Have to admit I didn't try this myself - just googled a bit.

you need to increase a memory limit for it ...
Got a similar problem with different language (PHP), once increased a memory limit to 25mb.
it solved a problem.