We Keep Coding

Drawing simultaneously on two controls in two windows without delay

In my application, there are 2 windows and both contain a PictureBox. The first (pb1) allows interaction and the image can be changed through click- and mouseMove-events. These events call pb1.Invalidate(); which works fine.
I want the second PictureBox (pb2) to redraw as well so I call pb2.Invalidate() from the paint-event of pb1. [Just for context, the second PictureBox shows nearly the same Image but on a bigger scale and some parts of the drawing will be left out in the future so I use the same Method in both paint events which decides what to draw and what not]
It works but it's "laggy" and I want it to be as smooth as the paint on the first PictureBox. I reduced the paint event just to a grid for test purposes.
Both windows are double buffered.
I tried replacing the picture boxes with SKGLControls from SkiaSharp (which should have better performance). The example code still uses the SkiaEvents so don't be confused if the problem occurs with both controls.
I tried to use .Update() or .Refresh() instead of .Invalidate() but i guess its to much to handle, the application just crashes..
Here is the method that is called by both OnPaint events
public void Update(SKPaintGLSurfaceEventArgs e, bool bigscreen)
{
SKCanvas canvas = e.Surface.Canvas;
canvas.Clear(SKColors.Beige);
//Zoom to specified area
SKMatrix matrix = SKMatrix.Identity;
if (!bigscreen)
{
matrix = matrix.PostConcat(SKMatrix.CreateScale(canvasSize / (float)zoomArea.Width, canvasSize / (float)zoomArea.Height));
}
else
{
matrix = matrix.PostConcat(SKMatrix.CreateScale(bigCanvasSize / (float)zoomArea.Width, bigCanvasSize / (float)zoomArea.Height));
}
matrix = matrix.PreConcat(SKMatrix.CreateTranslation(-zoomArea.X, -zoomArea.Y));
canvas.SetMatrix(matrix);
DrawGrid(canvas);
}
and the grid-draw method
private void DrawGrid(SKCanvas canvas)
{
using (SKPaint paint = new SKPaint() { IsAntialias = true,Color=SKColors.LightGray,StrokeWidth = 1})
{
canvas.DrawLine(0, 0, 0, gridCanvas.Height, paint); //Size gridCanvas is always the same at the moment and defines the space where the grid is drawn
canvas.DrawLine(0, 0, gridCanvas.Width, 0, paint);
for (int i = 0; i <= (gridCanvas.Width - gridoffsetX) / pxPerSquare; i++)
{
canvas.DrawLine(i * pxPerSquare + gridoffsetX, 0, i * pxPerSquare + gridoffsetX, gridCanvas.Height, paint);
}
for (int i = 0; i <= (gridCanvas.Height - gridoffsetY) / pxPerSquare; i++)
{
canvas.DrawLine(0, i * pxPerSquare + gridoffsetY, gridCanvas.Width, i * pxPerSquare + gridoffsetY, paint);
}
}
}
and finally the original Paint Event
private void Pb1_PaintSurface(object sender, SKPaintGLSurfaceEventArgs e)
{
win2.UpdateDrawing(); //Just calls .Invalidate() on pb2
painter.Update(e, false);
}
examplePicture
So my question is: Is there a way to make both controls draw at nearly the same time without delay, although I don't understand why the first PictureBox draws in real time and the second doesn't...
Thanks!

after searching for day i found this page right after posting, which helped me:
Onpaint events (invalidated) changing execution order after a period normal operation (runtime)

Continous data pass between forms

I have a MainForm that receives images from a camera (15-20 FPS) and extracts data from those images. I want to be able to plot those data, thus I need a Chart object that would update on each frame received.
Unfortunately, the API from the camera (Vimba) I am using is, I think, strangely written, and I can't draw the Chart in the same event that handles the reception of the frames and the extraction of the data. It just doesn't work (in the API, any instruction that involves a graphic object but isn't simply putting the received image in a Bitmap, or putting said bitmap into a PictureBox, doesn't run, I don't know why).
I have been fighting with this issue for a few days already, and one thing I haven't tried yet is passing the data extracted from the image to another Form, and draw the chart there. My problem is that I have no clue how to get the data to be sent to the new form on each frame received. I have already used a simple method to send data to another Form ONCE, but not continuously. Could someone explain to me how to achieve this?
Here is the event that runs when the computer receives a frame from the camera (the data I want to draw is stored in PixelColorCOuntReady) :
private void OnFrameReceived(Frame frame)
{
Bitmap myBitmap = null;
if (true == m_Acquiring)
{
mycamera.QueueFrame(frame);
}
frame.Fill(ref myBitmap);
pictureBoxLiveCamera.Image = myBitmap;
SaveBitmap = myBitmap.Clone(cloneRect, myBitmap.PixelFormat);
Array.Clear(PixelColorCount, 0, 256);
unsafe
{
int width = SaveBitmap.Width;
int height = SaveBitmap.Height;
int bytesPerPixel = 1;
int maxPointerLength = width * height * bytesPerPixel;
int stride = width * bytesPerPixel;
System.Drawing.Imaging.BitmapData bData = SaveBitmap.LockBits(new System.Drawing.Rectangle(0, 0, SaveBitmap.Width, SaveBitmap.Height), System.Drawing.Imaging.ImageLockMode.ReadWrite, SaveBitmap.PixelFormat);
byte* scan0 = (byte*)bData.Scan0.ToPointer();
byte B;
for (int i = 0; i < maxPointerLength; i++)
{
B = scan0[i];
PixelColorCount[B] += 1;
}
SaveBitmap.UnlockBits(bData);
}
PixelColorCountReady = PixelColorCount.DeepClone();
}

How to avoid an explicit call to GC.Collect() for this case that uses BitmapSource to continuously update a camera image

I need to include a camera image in my UI. The solution that I came up with for the View and ViewModel is given below. In the ViewModel I am using a BitmapSource which hold the camera image and that is continuously update, whenever the camera signals a new frame. To this BitmapSource I bind the View.
This code works find for the most part. However I have the issue that the application periodically consumes very large amounts of memory. When the video has not started up yet, the consumption is ~245MB. But when the video starts, it will quickly climb to ~1GB within ~4 seconds, which is when the Garbage-Collector kicks in and reduces that value back down to ~245MB. At this point the video will briefly stutter (I suspect due to CPU taxation). This happens periodically, every 4 seconds or so. Sometime, when the GC does not kick in after 4 seconds, memory usage can even reach 2GB and has also caused an out-of-memory exception.
The way I found to remedy this, is to explicitly call the GC each time a the frame is updated. See the two commented lines. When doing this, the memory will continue to hover at ~245MB after the video starts.
However this causes a significant increase in CPU usage from ~20% to ~35%.
I do not understand very well how the GC works, but I suspect, the reason that the GC kicks in so late, is that the thread that updates the BitmapSource is busy with updating the video (which runs at 25FPS) and therefore does not have time to run GC unless it explicitly told to do so.
So my question is: What is the reason for this behavior and is there a better way to achieve, what I am trying to do, while avoiding the explicit call to the GC?
I have tried wrapping this in a using-statement, but BitmapSource does not implement IDisponsable and from what I understand using is not created for this case, but for when you are accessing external/unmanaged resources.
Here is the code:
CameraImageView:
<Image Source="{Binding CameraImageSource}"/>
CameraImageViewModel:
public class CameraImageViewModel : ViewModelBase
{
private ICamera camera;
private UMat currentImage;
public BitmapSource CameraImageSource
{
get { return cameraImageSource; }
set
{
cameraImageSource = value;
RaisePropertyChanged("CameraImageSource");
}
}
private BitmapSource cameraImageSource;
public CameraImageViewModel(ICamera camera)
{
this.camera = camera;
camera.EventFrame += new EventHandler(UpdateCameraImage);
}
private void UpdateCameraImage(object s, EventArgs e)
{
camera.GetMatImage(out currentImage);
// commenting from here on downward, will also remove the described memory usage, but then we do not have an image anymore
BitmapSource tmpBitmap = ImageProcessing.UMatToBitmapSource(currentImage);
tmpBitmap.Freeze();
DispatcherHelper.CheckBeginInvokeOnUI(() => CameraImageSource = tmpBitmap);
//GC.Collect(); // without these lines I have the memory issue
//GC.WaitForPendingFinalizers();
}
}
ImageProcessing.UMatToBitmapSource:
public static BitmapSource UMatToBitmapSource(UMat image)
{
using (System.Drawing.Bitmap source = image.Bitmap)
{
return Convert(source);
}
}
/*
* REF for implementation of 'Convert': http://stackoverflow.com/questions/30727343/fast-converting-bitmap-to-bitmapsource-wpf/30729291#30729291
*/
public static BitmapSource Convert(System.Drawing.Bitmap bitmap)
{
var bitmapData = bitmap.LockBits(
new System.Drawing.Rectangle(0, 0, bitmap.Width, bitmap.Height),
System.Drawing.Imaging.ImageLockMode.ReadOnly, bitmap.PixelFormat);
var bitmapSource = BitmapSource.Create(
bitmapData.Width, bitmapData.Height, 96, 96, PixelFormats.Gray8, null,
bitmapData.Scan0, bitmapData.Stride * bitmapData.Height, bitmapData.Stride);
bitmap.UnlockBits(bitmapData);
return bitmapSource;
}

Replace creating new BitmapSource every UpdateCameraImage with with usage of single WriteableBitmap that you create once and update frequently. This way you will avoid creating copies of image in memory.
This code assumes that ImageWidth and ImageHeight do not change with time and are known in advance. If that is not the case, you will have to recreate the image dynamically when dimensions change.
Replace your cameraImageSource with:
private cameraImageSource = new WriteableBitmap(
ImageWidth,
ImageHeight,
96,
96,
PixelFormats.Gray8,
null);
Change your UpdateCameraImage to:
private void UpdateCameraImage(object s, EventArgs e)
{
camera.GetMatImage(out currentImage);
System.Drawing.Bitmap bitmap = currentImage.Bitmap;
var bitmapData = bitmap.LockBits(
new System.Drawing.Rectangle(0, 0, ImageWidth, ImageHeight),
System.Drawing.Imaging.ImageLockMode.ReadOnly, bitmap.PixelFormat);
cameraImageSource.CopyPixels(new Int32Rect(0, 0, ImageWidth,
ImageHeight), bitmapData.Scan0, bitmapData.Stride * bitmapData.Height, bitmapData.Stride);
bitmap.UnlockBits(bitmapData);
}
It's also possible that calling currentImage.Bitmap on UMat is not necessary. I don't know EmguCV library which you seem to be using, but UMat has also other properties like Ptr that could be passed directly to CopyPixels.

Changes to WriteableBitmap pixels don't update screen

I have a WriteableBitmap in a Windows Phone 8 application that is hooked up to an Image control. I'm looping through each row of the image and painting a row of pixels at a time asynchronously and then scheduling the next row for painting. However, it appears that changing the underlying pixel data does not fire a property changed so the control is not being updated. If I set the image source to a new WriteableBitmap created from the same pixels, the image updates fine but I'm doing a lot of excessive array copying.
void PaintImage(object state)
{
// get my height, width, row, etc. from the state
int[] bitmapData = new int[width];
// load the data for the row into the bitmap
Dispatcher.BeginInvoke(() =>
{
var bitmap = ImagePanel.Source as WriteableBitmap;
Array.Copy(bitmapData, 0, bitmap.Pixels, row * width, bitmapData.Length);
if (row < height - 1)
{
var newState = ... // create new state
ThreadPool.QueueUserWorkItem(PaintImage, newState);
}
});
}
If I add these lines after the Array.Copy above, the bitmap progressively is drawn to screen (although in reality it is just replacing the bitmap every time):
var newBitmap = new WriteableBitmap(width, height);
Array.Copy(bitmap.Pixels, newBitmap.Pixels, newBitmap.Pixels.Length);
ImagePanel.Source = newBitmap;
It seems like I need to manually have the WriteableBitmap fire some property changed notification so that the Image that has it. I'm guessing this problem will go away if I bind the image to a WriteableBitmap in a ViewModel?

Just add a dirty rectangle
_myBitmap.Lock();
_myBitmap.AddDirtyRect(new Int32Rect(0, 0, _myBitmap.PixelWidth, _myBitmap.PixelHeight));
_myBitmap.Unlock();
or if you are on a background thread
Application.Current.Dispatcher.InvokeAsync(() =>
{
_myBitmap.Lock();
_myBitmap.AddDirtyRect(new Int32Rect(0, 0, _myBitmap.PixelWidth, _myBitmap.PixelHeight));
_myBitmap.Unlock();
});

I think you should call Invalidate() to request for a redraw. Ref: http://msdn.microsoft.com/en-us/library/system.windows.media.imaging.writeablebitmap.invalidate(v=vs.95).aspx

How to implement pan/zoom on gigapixel bitmaps?

In my project, I'm using (uncompressed 16-bit grayscale) gigapixel images which come from a high resolution scanner for measurement purposes. Since these bitmaps can not be loaded in memory (mainly due to memory fragmentation) I'm using tiles (and tiled TIFF on disc). (see StackOverflow topic on this)
I need to implement panning/zooming in a way like Google Maps or DeepZoom. I have to apply image processing on the fly before presenting it on screen, so I can not use a precooked library which directly accesses an image file. For zooming I intend to keep a multi-resolution image in my file (pyramid storage). The most useful steps seem to be +200%, 50% and show all.
My code base is currently C# and .NET 3.5. Currently I assume Forms type, unless WPF gives me great advantage in this area. I have got a method which can return any (processed) part of the underlying image.
Specific issues:
hints or references on how to implement this pan/zoom with on-demand generation of image parts
any code which could be used as a basis (preferably commercial or LGPL/BSD like licenses)
can DeepZoom be used for this (i.e. is there a way that I can provide a function to provide a tile at the right resulution for the current zoom level?) ( I need to have pixel accurate addressing still)

This CodeProject article: Generate...DeepZoom Image Collection might be a useful read since it talks about generating a DeepZoom image source.
This MSDN article has a section Dynamic Deep Zoom: Supplying Image Pixels at Run Time and links to this Mandelbrot Explorer which 'kinda' sounds similar to what you're trying to do (ie. he is generating specific parts of the mandelbrot set on-demand; you want to retrieve specific parts of your gigapixel image on-demand).
I think the answer to "can DeepZoom be used for this?" is probably "Yes", however as it is only available in Silverlight you will have to do some tricks with an embedded web browser control if you need a WinForms/WPF client app.
Sorry I can't provide more specific answers - hope those links help.
p.s. I'm not sure if Silverlight supports TIFF images - that might be an issue unless you convert to another format.

I decided to try something myself. I came up with a straightforward GDI+ code, which uses the tiles I've already got. I just filter out the parts which are relevant for current clipping region. It works like magic! Please find my code below.
(Form settings double buffering for the best results)
protected override void OnPaint(PaintEventArgs e)
{
base.OnPaint(e);
Graphics dc = e.Graphics;
dc.ScaleTransform(1.0F, 1.0F);
Size scrollOffset = new Size(AutoScrollPosition);
int start_x = Math.Min(matrix_x_size,
(e.ClipRectangle.Left - scrollOffset.Width) / 256);
int start_y = Math.Min(matrix_y_size,
(e.ClipRectangle.Top - scrollOffset.Height) / 256);
int end_x = Math.Min(matrix_x_size,
(e.ClipRectangle.Right - scrollOffset.Width + 255) / 256);
int end_y = Math.Min(matrix_y_size,
(e.ClipRectangle.Bottom - scrollOffset.Height + 255) / 256);
// start * contain the first and last tile x/y which are on screen
// and which need to be redrawn.
// now iterate trough all tiles which need an update
for (int y = start_y; y < end_y; y++)
for (int x = start_x; x < end_x; x++)
{ // draw bitmap with gdi+ at calculated position.
dc.DrawImage(BmpMatrix[y, x],
new Point(x * 256 + scrollOffset.Width,
y * 256 + scrollOffset.Height));
}
}
To test it, I've created a matrix of 80x80 of 256 tiles (420 MPixel). Of course I'll have to add some deferred loading in real life. I can leave tiles out (empty) if they are not yet loaded. In fact, I've asked my client to stick 8 GByte in his machine so I don't have to bother about performance too much. Once loaded tiles can stay in memory.
public partial class Form1 : Form
{
bool dragging = false;
float Zoom = 1.0F;
Point lastMouse;
PointF viewPortCenter;
private readonly Brush solidYellowBrush = new SolidBrush(Color.Yellow);
private readonly Brush solidBlueBrush = new SolidBrush(Color.LightBlue);
const int matrix_x_size = 80;
const int matrix_y_size = 80;
private Bitmap[,] BmpMatrix = new Bitmap[matrix_x_size, matrix_y_size];
public Form1()
{
InitializeComponent();
Font font = new Font("Times New Roman", 10, FontStyle.Regular);
StringFormat strFormat = new StringFormat();
strFormat.Alignment = StringAlignment.Center;
strFormat.LineAlignment = StringAlignment.Center;
for (int y = 0; y < matrix_y_size; y++)
for (int x = 0; x < matrix_x_size; x++)
{
BmpMatrix[y, x] = new Bitmap(256, 256, PixelFormat.Format24bppRgb);
// BmpMatrix[y, x].Palette.Entries[0] = (x+y)%1==0?Color.Blue:Color.White;
using (Graphics g = Graphics.FromImage(BmpMatrix[y, x]))
{
g.FillRectangle(((x + y) % 2 == 0) ? solidBlueBrush : solidYellowBrush, new Rectangle(new Point(0, 0), new Size(256, 256)));
g.DrawString("hello world\n[" + x.ToString() + "," + y.ToString() + "]", new Font("Tahoma", 8), Brushes.Black,
new RectangleF(0, 0, 256, 256), strFormat);
g.DrawImage(BmpMatrix[y, x], Point.Empty);
}
}
BackColor = Color.White;
Size = new Size(300, 300);
Text = "Scroll Shapes Correct";
AutoScrollMinSize = new Size(256 * matrix_x_size, 256 * matrix_y_size);
}
Turned out this was the easy part. Getting async multithreaded i/o done in the background was a lot harder to acchieve. Still, I've got it working in the way described here. The issues to resolve were more .NET/Form multithreading related than to this topic.
In pseudo code it works like this:
after onPaint (and on Tick)
check if tiles on display need to be retrieved from disc
if so: post them to an async io queue
if not: check if tiles close to display area are already loaded
if not: post them to an async io/queue
check if bitmaps have arrived from io thread
if so: updat them on screen, and force repaint if visible
Result: I now have my own Custom control which uses roughly 50 MByte for very fast access to arbitrary size (tiled) TIFF files.

I guess you can address this issue following the steps below:
Image generation:
segment your image in multiple subimages (tiles) of a small resolution, for instace, 500x500. These images are depth 0
combine a series of tiles with depth 0 (4x4 or 6x6), resize the combination generating a new tile with 500x500 pixels in depth 1.
continue with this approach until get the entire image using only a few tiles.
Image visualization
Start from the highest depth
When user drags the image, load the tiles dynamically
When the user zoom a region of the image, decrease the depth, loading the tiles for that region in a higher resolution.
The final result is similar to Google Maps.