High CPU load when changing background image of Canvas containing overlay elements - c#

I am working on an Application that loads live video images from a camera and displays an overlay on top of said image. The Overlay does not change often so it can be considered as still. However it usually contains about 1,000 to 10,000 Lines.
When the video image is updated there is a notable impact to the CPU load depending on whether the overlay is visible or not. The overlay does neither get invalidated nor changed, just the image behind it is changing.
My setup is this:
<Canvas>
<Image/>
<Canvas>
<OverlayElement 1/>
<OverlayElement 2/>
<OverlayElement 3/>
<.../>
</Canvas>
</Canvas>
The Image's Source is a WriteableBitmap. Every time a new camera image (type byte[]) is available, the main Canvas' Dispatcher is invoked to write the image data by using WriteableBitmap.WritePixels().
The inner Canvas contains all Overlay Elements, being
- a contour (PolyLine)
- a circle (Path with EllipseGeometry) and
- a set of Rays (Path with one Figure containing LineSgements).
The number n of Points in the contour equals the number of line Segments in the last mentioned Path. n is usually around 1,000 - 3,000.
Depending on the count and length of Lines shown in the overlay the CPU load for showing a live image varies (increases if length or count go up) even if the overlay does not change. At some point this affects the frame rate and makes the program unusable. Line length is mostly correlated with line intersection, so maybe the Path is struggling to calculate it's fill area despite it is not painted?
So how could I improve the performance here?
What bugs me most is that even if the overlay does not change, the render time increases with it's primitive count. I would expect to have constant render time once the overlay has been drawn in it's last set state. What could I do to achieve that aside from rendering the whole overlay to a bitmap?
I am also open minded for suggestions on how to get the byte[] onto the screen more efficiently. Just keep in mind this problem is part of a bigger Application and i cannot change all paradigms concentrating on how to get the image drawn.
What I have tried so far:
Override the OnRender() method of the inner Canvas, drawing the overlay myself. This works fine but has the performance issue that brings me here ;)
Use Shapes (PolyLine, Ellipse, Path) as the inner Canvas' children to hold the overlay elements. This works, too. It is faster to redraw the overlay when it changes but on the other hand worsens the performance issue when updating the background image.
Like 2., but use Freeze() on Geometries wherever possible. Has no or little performance impact.
Thanks for your help in advance.

Related

Resizing System.Controls.Image with Nearest Neighbour resizing mode

I'm trying to make a game which uses a pixel art style. I've created my assets and used them as resources, which are added into the window as System.Controls.Image objects.
In the designer, they always turn out too small. As a result, when I resize them, they become somewhat blurred due to some sort of bicubic interpolation being applied onto them.
I have managed to avoid this by avoiding resizing; I resize in the designer to find out what size is appropriate, and then use nearest neighbour resizing on the original image (using an external program) to get the source file to that size. I then update the image in the project and remove any resizing, therefore leaving it at original size, i.e. interpolation-free.
As you can imagine, this is a rather tedious process. I looked into interpolation choices during resizing, but most answers I can find relate to System.Drawing.Image, not System.Controls.Image. I feel like any such solution (if adapted) would be horribly messy and involve multiple (and perhaps unecessary) conversions/casts.
Is there any way to get nearest neighbour resizing on System.Controls.Image?
To set the resize mode, you need to set the RenderOptions.BitmapScalingMode="NearestNeighbor" option for the visual tree. You can set this at the window level.
To address the larger issue, it seems that something is causing your images to be scaled in the first place:
Ensure you are setting the Stretch="None" option on the Image control,
Ensure that you are using SnapsToDeveicePixels or Layout rounding
Lastly if all else fails, explicitly set the width and height of the image.
I have also run into instances where the image file's DPI not being set to (I believe) 90, causes the renderer to apply scaling.

Visualising multi-layer image in WPF

My data structure has two fields:
* BackgroundImage (of type Bitmap/Image)
* Points (of type Point2D [])
My use case is as follows: a user can load an image into the app. After the image appears on user's screen, they might add points to it (by clicking a mouse button). The points should be visualized on the top of the image, but a user should be albe to reposition them if needed (e.g. drag'n'drop).
At the moment I solve the problem by doing the following every time the user adds / moves a point:
* clone the BackgroundImage
* draw all the points on the cloned image (using System.Drawing.Graphics)
* return the cloned image with the points (expose it as a property and bind to Image in WPF).
The time performance of this solution is ok, however it consumes a lot of memory, as everytime I end up copying whole image. I'm wondering if there's a better way of doing this (e.g. by using layers - then my BackgroundImage remains the same all the time while I keep modifying only the top layer).
My code is quite long, but if it's needed just let me know and I'll post it.
when it comes to memory consumption, there is nothing wrong with the aproach you described as long as you make sure the old instances of the image are not rooted anymore so that the GC can remove them.
However, during the timespans in which the cloned image is modified, occupation of memory might of course be up to double the lowest possible value when not cloning.
To reduce this memory consumption, the movable points can be implemented using UIElements. This could also help keeping the implementation simpler by using WPF hittesting for the "drag'n'drop" part.
Since UIElements require more memory than the points in the BitmapImage, the actual savings depend on the ratio of points in the BitmapImage to movable points.
To implement the points using UIElements, place the BitmapImage together with a Canvas in a Panel. Then use the Canvas as a container for the points and set their positions using the attached properties Canvas.Top and Canvas.Left.
To make the points appear in front of the BitmapImage, set the Panel.ZIndex of the Canvas.
But if you are seeing unreasonable memory consumption, you should use a memory profiler to take a closer look at what parts of the Process are actually taking up the space.

Comparing focus of 2 image

I'm trying to develop object detection algorithm. I plan to compare 2 image with different focus length. One image that correct focus on the object and one image that correct focus on background.
By reading about autofocus algorithm. I think it can done with contrast detection passive autofocus algorithm. It work on light intensity on the sensor.
But I don't sure that light intensity value from the image file has the same value as from the sensor. (it not a RAW image file. a jpeg image.) Is the light intensity value in jpeg image were the same as on the sensor? Can I use it to detect focus correctness with contrast detection? Is there a better way to detect which area of image were correct focus on the image?
I have tried to process the images a bit and I saw some progress. THis is what I did using opencv:
converted images to gray using cvtColor(I, Mgrey, CV_RGB2GRAY);
downsampled/decimated them a bit since they are huge (several Mb)
Took the sum of absolute horizontal and vertical gradients using http://docs.opencv.org/modules/imgproc/doc/filtering.html?highlight=sobel#cv.Sobel.
The result is below. The foreground when in focus does look brighter than background and vice versa.
You can probably try to match and subtract these images using translation from matchTemplate() on the original gray images; and then assemble pieces using the convex hull of the results as initialization mask for grab cut and plugging in color images. In case you aren’t familiar with the grab cut, chceck out my answer to this question.
But may be a simpler method will work here as well. You can try to apply a strong blur to your gradient images instead of precise matching and see what the difference give you in this case. The images below demonstrate the idea when I turned the difference in the binary masks.
It will be helpful to see your images. It I understood you correctly you try to separate background from foreground using focus (or blur) cue. Contrast in the image depends on focus but it also depend on the contrast of the target. So if the target is clouds you will never get sharp edges or high contrast. Finally jpeg image that use little compression should not affect the critical properties of your algorithm.
I would try to get a number of images at all possible focus lengths in a row and then build a graph of the contrast as a function of focal length (or even better focusing distance). The peak in this graph will give you the distance to the object regardless of object's own contrast. Note, however, that the accuracy of such visual cues goes down sharply with viewing distance.
This is what I expect you to obtain when measuring the sum of absolute gradient in a small window:
The next step for you will be to combine areas that are in focus with the areas that are solid color that is has no particular peak in the graph but none the less belong to the same object. Sometimes getting a convex hull of the focused areas can help to pinpoint the raw boundary of the object.

How to render a very large image or document that is drawn without any performance issues

I want to make a control that acts as a time ruler with a wave form for sound wav files like that in movie maker or in camtasia studio video editor
i have a method to parse the sound wav file
i have a method to draw the wave of a specified time period
the problem is that the duration is long how to render all of it??one time or when the user is scrolling to a point if it isn't rendered I'll render it? what after its all rendered ? it will be so long user will zoom in and so on how to do it without effecting the performance
If you're dealing with a big image and lots of data I would sugest this:
render/display what user is currently looking at
render rest of if in a background thread into a memory image
Once you have the whole image rendered in an image kept in memory, any scrolling and/or zooming would be just to position your display window.
Render your image in an unzoomed scale into a Bitmap object. When you need to display part of if use Bitmap.Clone to copy section of that bitmap into a picture displayed to user. A bit difficult part will be to render part of your base Bitmap which is currently demanded to be viewed.
This could be memory demanding. It is effectively a caching mechanism and as always in cases like that you get speed by using more memory.
Something similar to how google maps works.
We do not know how you are rendering your graphic, but I presume you are using unsafe lockbits to generate directly into a bitmap. As Maciej says, just render the part you need to display - when zoomed, either render again (should be fast). You can preempt by rendering current, zoom in 1, zoom out 1, left 1/2 a screen, right 1/2 screen etc in the background, but then you need to manage these (they can be rendered in parallel of course using background worker threads). The key to memory is to keep the bitmaps to a minimum (you have the algorithm of the graph and the renderer) and the key to speed is to keep it low level and prefetch what you can and parallelism.

GDI+ Offscreen Buffered Scrolling

I am creating a custom control using C# GDI+.
Quick explanation...the control will be say 500 pixels on screen but will contain perhaps 500000 pixels of information. So although i'm only showing 500px at a time i need to obviously scroll in the horizontal plane (left & right). The tricky part is that each 500px chunk of bitmap takes a while (between 100ms - 1000ms) to render.
So my plan is to maintain a 1500px bitmap in memory. i.e. the 500px visible part and 500px either side of the visible area and draw the off-screen parts asynchronously as the user scrolls.
I would like some feedback, advice, criticism or examples of code to help me achieve this. It seems fairly straight forward but after a few initial test attempts its proving more difficult than one would imagine.
Thanks.
The effectiveness of this approach depends on, amongst other things, the amount of moving around the user will do. If the user is making small movement, then stopping to consider the new information, this could work. However, if the user is whizzing back and forth, you'll still have an issue.
Does your application lend itself to gradually improving the quality of the image - i.e. providing a quick usable image and then improving it as the user stops to consider it?
I had a similar problem a few years back. As dommer mentions, if you're processing chunks of the image before displaying them you're best off showing something and improving it later. If you're having problems blitting the original image, you've got something wrong with your method. GDI+ is very particular about pixel depth (you want 32bpp with alpha).
In our case, we processed in 500px tiles and padded out a tile around the visible view Ff the user scrolled outside the area we'd processed we blitted bits of the original image with a dark semi-opaque rectangle super-imposed on them. These chunks were queued for processing. As we processed chunks of the image (centre-out) they semi-opaque rectangles would disappear.
It worked reasonably well, was very responsive and very fast. It's very fast to blit the original bitmap onto the screen, and in our case the processing was usually very close behind. The effect of the tiles getting lighter was actually quite pretty.
Draw only the visible area.
Build a method
Bitmap DrawGraph(leftMargin, rightMargin) {...}
and then in OnPaint() do
Bitmap bmp = new Bitmap(e.ClipRectangle.Width, e.ClipRectangle.Height);
bmp = DrawGraph(e.ClipRectangle.Left, e.ClipRectangle.Right);
e.Graphics.DrawImageUnscaled(bmp, e.ClipRectangle.Location);

Categories