We Keep Coding

Cut faraway objects based on depth map

I would like to do grabcut which uses a depth map that cuts away far objects, that is used in mixed reality application. So I would like to show just the front of what I see and the background as virtual reality scene.
The problem right now I tried to adapt so code and what I get is front which is cut but in black color, the mask actually.
I don't know where is the problem settle.
The input is a depth map from zed camera.
here is a picture of the behaviour:
My trial:
private void convertToGrayScaleValues(Mat mask)
{
int width = mask.rows();
int height = mask.cols();
byte[] buffer = new byte[width * height];
mask.get(0, 0, buffer);
for (int x = 0; x < width; x++)
{
for (int y = 0; y < height; y++)
{
int value = buffer[y * width + x];
if (value == Imgproc.GC_BGD)
{
buffer[y * width + x] = 0; // for sure background
}
else if (value == Imgproc.GC_PR_BGD)
{
buffer[y * width + x] = 85; // probably background
}
else if (value == Imgproc.GC_PR_FGD)
{
buffer[y * width + x] = (byte)170; // probably foreground
}
else
{
buffer[y * width + x] = (byte)255; // for sure foreground
}
}
}
mask.put(0, 0, buffer);
}
For Each depth frame from Camera:
Mat erodeElement = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(4, 4));
Mat dilateElement = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(7, 7));
depth.copyTo(maskFar);
Core.normalize(maskFar, maskFar, 0, 255, Core.NORM_MINMAX, CvType.CV_8U);
Imgproc.cvtColor(maskFar, maskFar, Imgproc.COLOR_BGR2GRAY);
Imgproc.threshold(maskFar, maskFar, 180, 255, Imgproc.THRESH_BINARY);
Imgproc.dilate(maskFar, maskFar, erodeElement);
Imgproc.erode(maskFar, maskFar, dilateElement);
Mat bgModel = new Mat();
Mat fgModel = new Mat();
Imgproc.grabCut(image, maskFar, new OpenCVForUnity.CoreModule.Rect(), bgModel, fgModel, 1, Imgproc.GC_INIT_WITH_MASK);
convertToGrayScaleValues(maskFar); // back to grayscale values
Imgproc.threshold(maskFar, maskFar, 180, 255, Imgproc.THRESH_TOZERO);
Mat foreground = new Mat(image.size(), CvType.CV_8UC4, new Scalar(0, 0, 0));
image.copyTo(foreground, maskFar);
Utils.fastMatToTexture2D(foreground, texture);

In this case, the graph cut on the depth image might not be the correct method to solve all of your issue.
If you insist the processing should be done in the depth image. To find everything that is not on the table and filter out the table part. You may first apply the disparity based approach for finding the object that's is not on the ground. Reference: https://github.com/windowsub0406/StereoVision
Then based on the V disparity output image, find the locally connected component that is grouped together. You may follow this link how to do this disparity map in OpenCV which is asking the similar way to find the objects that's not on the ground
If you are ok with RGB based approaches, then use any deep learning-based method to recognize the monitor should be the correct approaches. It can directly detect the mointer bounding box. By apply this bounding box to the depth image, you may have what you want. For deep learning based approaches, there are many available package such as Yolo series. You may find one that is suitable for you. reference: https://medium.com/#dvshah13/project-image-recognition-1d316d04cb4c

How to detect where the image content ends?

I receive images of the same size but with different amounts of information. Examples below (red borders are mine). The background is always white.
I am trying to detect where the information on the image ends - at what pixel height (and crop accordingly). In other words, find the first non-white pixel from the bottom.
Is there a better way to do this other than extract BitmapData out of Image object and loop through all the pixels?

Just to add a suggestion having looked over your images and your solution (below) and your method is fine but you may be able to improve efficiency.
The more you know about your image the better; you're confident the background is always white (according to your post, the code is a more generic utility but the following suggestion can still work); can you be confident on the furthest point in a non-white pixel will be found if the row is not empty?
For example; in your two pictures the furthest in non-white pixel on a row is about 60px in. If this is universally true for your data then you don't need to scan the whole line of the image, which would make your for loop:
for (int y = bitmap.Height - 1; y >= 0; y--) {
for (int x = 0; x < 60; x++) {
Color color = bitmap.GetPixel(x, y);
if (color.R != backColor.R || color.G != backColor.G || color.B != backColor.B) {
foundContentOnRow = y;
break;
}
}
}
(You could make it a parameter on the function so you can easily control it if needed).
Imagine for example that the first non-white row was 80px down. To find it currently you do 640 x 300 = 192,000 checks. If you could confidently say that you would know a row was blank within 100 pixels (an over-estimate based on the data presented) then this would be 100 * 300 = 30,000 checks per image.
If you always knew that the first 10 pixels of the image were always blank you could shave a little bit more off (say 3000 checks).
Musing on a setup where you knew that the first non-white pixel was between 10 and 60 pixels in (range of 50) you could find it at row 80 in 50 x 300 = 15,000 checks which is a good reduction.
Of course the downside about assumptions is that if things change your assumptions may not be valid, but if the data is going to remain fairly constant then it may be worthwhile, especially if you do this for a lot of images.

I've ended up using the following code to trim the image. Hopefully someone finds this useful.
class Program {
static void Main(string[] args) {
Image full = Image.FromFile("foo.png");
Image cropped = full.TrimOnBottom();
}
}
public static class ImageUtilities {
public static Image TrimOnBottom(this Image image, Color? backgroundColor = null, int margin = 30) {
var bitmap = (Bitmap)image;
int foundContentOnRow = -1;
// handle empty optional parameter
var backColor = backgroundColor ?? Color.White;
// scan the image from the bottom up, left to right
for (int y = bitmap.Height - 1; y >= 0; y--) {
for (int x = 0; x < bitmap.Width; x++) {
Color color = bitmap.GetPixel(x, y);
if (color.R != backColor.R || color.G != backColor.G || color.B != backColor.B) {
foundContentOnRow = y;
break;
}
}
// exit loop if content found
if (foundContentOnRow > -1) {
break;
}
}
if (foundContentOnRow > -1) {
int proposedHeight = foundContentOnRow + margin;
// only trim if proposed height smaller than existing image
if (proposedHeight < bitmap.Height) {
return CropImage(image, bitmap.Width, proposedHeight);
}
}
return image;
}
private static Image CropImage(Image image, int width, int height) {
Rectangle cropArea = new Rectangle(0, 0, width, height);
Bitmap bitmap = new Bitmap(image);
return bitmap.Clone(cropArea, bitmap.PixelFormat);
}
}

Find last drawn pixel of C# Metafile

I have a Metafile object. For reasons outside of my control, it has been provided much larger (thousands of times larger) than what would be required to fit the image drawn inside it.
For example, it could be 40 000 x 40 000, yet only contains "real" (non-transparent) pixels in an area 2000 x 1600.
Originally, this metafile was simply drawn to a control, and the control bounds limited the area to a reasonable size.
Now I am trying to split it into different chunks of dynamic size, depending on user input. What I want to do it count how many of those chunks will be there (in x and in y, even the splitting is into a two-dimensional grid of chunks).
I am aware that, technically, I could go the O(N²) way, and just check the pixels one by one to find the "real" bounds of the drawn image.
But this will be painfully slow.
I am looking for a way of getting the position (x,y) of the very last drawn pixel in the entire metafile, without iterating through every single one of them.
Since The DrawImage method is not painfully slow, at least not N² slow, I assume that the metafile object has some optimisations on the inside that would allow something like this. Just like the List object has a .Count Property that is much faster than actually counting the objects, is there some way of getting the practical bounds of a metafile?
The drawn content, in this scenario, will always be rectangular. I can safely assume that the last pixel will be the same, whether I loop in x then y, or in y then x.
How can I find the coordinates of this "last" pixel?

Finding the bounding rectangle of the non-transparent pixels for such a large image is indeed an interesting challenge.
The most direct approach would be tackling the WMF content but that is also by far the hardest to get right.
Let's instead render the image to a bitmap and look at the bitmap.
First the basic approach, then a few optimizations.
To get the bounds one need to find the left, top, right and bottom borders.
Here is a simple function to do that:
Rectangle getBounds(Bitmap bmp)
{
int l, r, t, b; l = t = r = b = 0;
for (int x = 0; x < bmp.Width - 1; x++)
for (int y = 0; y < bmp.Height - 1; y++)
if (bmp.GetPixel(x,y).A > 0) { l = x; goto l1; }
l1:
for (int x = bmp.Width - 1; x > l ; x--)
for (int y = 0; y < bmp.Height - 1; y++)
if (bmp.GetPixel(x,y).A > 0) { r = x; goto l2; }
l2:
for (int y = 0; y < bmp.Height - 1; y++)
for (int x = l; x < r; x++)
if (bmp.GetPixel(x,y).A > 0) { t = y; goto l3; }
l3:
for (int y = bmp.Height - 1; y > t; y--)
for (int x = l; x < r; x++)
if (bmp.GetPixel(x,y).A > 0) { b = y; goto l4; }
l4:
return Rectangle.FromLTRB(l,t,r,b);
}
Note that is optimizes the last, vertical loops a little to look only at the portion not already tested by the horizontal loops.
It uses GetPixel, which is painfully slow; but even Lockbits only gains 'only' about 10x or so. So we need to reduce the sheer numbers; we need to do that anyway, because 40k x 40k pixels is too large for a Bitmap.
Since WMF is usually filled with vector data we probably can scale it down a lot. Here is an example:
string fn = "D:\\_test18b.emf";
Image img = Image.FromFile(fn);
int w = img.Width;
int h = img.Height;
float scale = 100;
Rectangle rScaled = Rectangle.Empty;
using (Bitmap bmp = new Bitmap((int)(w / scale), (int)(h / scale)))
using (Graphics g = Graphics.FromImage(bmp))
{
g.ScaleTransform(1f/scale, 1f/scale);
g.Clear(Color.Transparent);
g.DrawImage(img, 0, 0);
rScaled = getBounds(bmp);
Rectangle rUnscaled = Rectangle.Round(
new RectangleF(rScaled.Left * scale, rScaled.Top * scale,
rScaled.Width * scale, rScaled.Height * scale ));
}
Note that to properly draw the wmf file one may need to adapt the resolutions. Here is an example i used for testing:
using (Graphics g2 = pictureBox.CreateGraphics())
{
float scaleX = g2.DpiX / img.HorizontalResolution / scale;
float scaleY = g2.DpiY / img.VerticalResolution / scale;
g2.ScaleTransform(scaleX, scaleY);
g2.DrawImage(img, 0, 0); // draw the original emf image.. (*)
g2.ResetTransform();
// g2.DrawImage(bmp, 0, 0); // .. it will look the same as (*)
g2.DrawRectangle(Pens.Black, rScaled);
}
I left this out but for fully controlling the rendering, it ought have been included in the snippet above as well..
This may or may not be good enough, depending on the accuracy needed.
To measure the bounds perfectly one can do this trick: Use the bounds from the scaled down test and measure unscaled but only a tiny stripe around the four bound numbers. When creating the render bitmap we move the origin accordingly.
Example for the right bound:
Rectangle rScaled2 = Rectangle.Empty;
int delta = 80;
int right = (int)(rScaled.Right * scale);
using (Bitmap bmp = new Bitmap((int)(delta * 2 ), (int)(h )))
using (Graphics g = Graphics.FromImage(bmp))
{
g.Clear(Color.Transparent);
g.DrawImage(img, - right - delta, 0);
rScaled2 = getBounds(bmp);
}
I could have optimized by not going over the full height but only the portion (plus delte) we already found..
Further optimization can be achieved if one can use knowledge about the data. If we know that the image data are connected we could use larger steps in the loops until a pixel is found and then trace back one step..

Excluding small chunks of pixels from Image .Net

I have black image with white lines. Is it possible to exclude chunks of whihte pixels, that are smaller than specific number? For example: change color of chunks of pixels that are made from less than 10 pixels from white to black.
Original Image:
Image on the output(small areas of white pixels are removed):
Right now I work with AForge library for C#, but C++ ways of solving this are also apreciated(Open CV, for example). And hint, on how this functionality might be called are also appreciated.

Without worrying to much about your details, it does seem trivially simple
Use bitmap in 32bits and use LockBits to get scanlines and direct pointer access to the array.
Scan every pixel with 2 for loops
Every time you find one that matches your target color, scan left right and up and down (X) Amount of pixels to determine if it matches your requirements,
If it does, leave the pixel, if not change it.
if you wanted more speed you could chuck this all in a parallel workload, also there is probably more you could do with a mask array to save you researching dead paths (just a thought)
Note, Obviously you can smarten this up a bit
Exmaple
// lock the array for direct access
var bitmapData = bitmap.LockBits(Bounds, ImageLockMode.ReadWrite, Bitmap.PixelFormat);
// get the pointer
var scan0Ptr = (int*)_bitmapData.Scan0;
// get the stride
var stride = _bitmapData.Stride / BytesPerPixel;
// local method
void Workload(Rectangle bounds)
{
// this is if synchronous, Bounds is just the full image rectangle
var rect = bounds ?? Bounds;
var white = Color.White.ToArgb();
var black = Color.Black.ToArgb();
// scan all x
for (var x = rect.Left; x < rect.Right; x++)
{
var pX = scan0Ptr + x;
// scan all y
for (var y = rect.Top; y < rect.Bottom; y++)
{
if (*(pX + y * stride ) != white)
{
// this will turn it to monochrome
// so add your threshold here, ie some more for loops
//*(pX + y * Stride) = black;
}
}
}
}
// unlock the bitmap
bitmap.UnlockBits(_bitmapData);
To parallel'ize it
You could use something like this to break your image up into smaller regions
public static List<Rectangle> GetSubRects(this Rectangle source, int size)
{
var rects = new List<Rectangle>();
for (var x = 0; x < size; x++)
{
var width = Convert.ToInt32(Math.Floor(source.Width / (double)size));
var xCal = 0;
if (x == size - 1)
{
xCal = source.Width - (width * size);
}
for (var y = 0; y < size; y++)
{
var height = Convert.ToInt32(Math.Floor(source.Height / (double)size));
var yCal = 0;
if (y == size - 1)
{
yCal = source.Height - (height * size) ;
}
rects.Add(new Rectangle(width * x, height * y, width+ xCal, height + yCal));
}
}
return rects;
}
And this
private static void DoWorkload(Rectangle bounds, ParallelOptions options, Action<Rectangle?> workload)
{
if (options == null)
{
workload(null);
}
else
{
var size = 5 // how many rects to work on, ie 5 x 5
Parallel.ForEach(bounds.GetSubRects(size), options, rect => workload(rect));
}
}
Usage
DoWorkload(Bounds, options, Workload);

Drawing zig-zag lines is much slower than drawing straight lines

While using a self-written graphing control I noticed that the painting of the graph was much slower while displaying noisy data than when it displayed clean data.
I dug further into and narrowed the problem down to its bare minimum difference: Drawing the same amount of lines with varying Y values versus drawing lines with the same Y value.
So for example I put together the following tests. I generate lists of points, one with random Y values, one with the same Y, and one with a Zig-Zag Y pattern.
private List<PointF> GenerateRandom(int n, int width, int height)
{
//Generate random pattern
Random rnd = new Random();
float stepwidth = Convert.ToSingle(width / n);
float mid = Convert.ToSingle(height / 2);
float lastx = 0;
float lasty = mid;
List<PointF> res = new List<PointF>();
res.Add(new PointF(lastx, lasty));
for (int i = 1; i <= n; i++)
{
var x = stepwidth * i;
var y = Convert.ToSingle(height * rnd.NextDouble());
res.Add(new PointF(x, y));
}
return res;
}
private List<PointF> GenerateUnity(int n, int width, int height)
{
//Generate points along a simple line
float stepwidth = Convert.ToSingle(width / n);
float mid = Convert.ToSingle(height / 2);
float lastx = 0;
float lasty = mid;
List<PointF> res = new List<PointF>();
res.Add(new PointF(lastx, lasty));
for (int i = 1; i <= n; i++)
{
var x = stepwidth * i;
var y = mid;
res.Add(new PointF(x, y));
}
return res;
}
private List<PointF> GenerateZigZag(int n, int width, int height)
{
//Generate an Up/Down List
float stepwidth = Convert.ToSingle(width / n);
float mid = Convert.ToSingle(height / 2);
float lastx = 0;
float lasty = mid;
List<PointF> res = new List<PointF>();
res.Add(new PointF(lastx, lasty));
var state = false;
for (int i = 1; i <= n; i++)
{
var x = stepwidth * i;
var y = mid - (state ? 50 : -50);
res.Add(new PointF(x, y));
state = !state;
}
return res;
}
I now draw each list of points a few times and compare how long it takes:
private void DoTheTest()
{
Bitmap bmp = new Bitmap(970, 512);
var random = GenerateRandom(2500, bmp.Width, bmp.Height).ToArray();
var unity = GenerateUnity(2500, bmp.Width, bmp.Height).ToArray();
var ZigZag = GenerateZigZag(2500, bmp.Width, bmp.Height).ToArray();
using (Graphics g = Graphics.FromImage(bmp))
{
var tUnity = BenchmarkDraw(g, 200, unity);
var tRandom = BenchmarkDraw(g, 200, random);
var tZigZag = BenchmarkDraw(g, 200, ZigZag);
MessageBox.Show(tUnity.ToString() + "\r\n" + tRandom.ToString() + "\r\n" + tZigZag.ToString());
}
}
private double BenchmarkDraw(Graphics g, int n, PointF[] Points)
{
var Times = new List<double>();
for (int i = 1; i <= n; i++)
{
g.Clear(Color.White);
System.DateTime d3 = DateTime.Now;
DrawLines(g, Points);
System.DateTime d4 = DateTime.Now;
Times.Add((d4 - d3).TotalMilliseconds);
}
return Times.Average();
}
private void DrawLines(Graphics g, PointF[] Points)
{
g.DrawLines(Pens.Black, Points);
}
I come up with the following durations per draw:
Straight Line: 0.095 ms
Zig-Zag Pattern: 3.24 ms
Random Pattern: 5.47 ms
So it seems to get progressively worse, the more change there is in the lines to be drawn, and that is also a real world effect I encountered in the control painting I mentioned in the beginning.
My questions are thus the following:
Why does it make a such a brutal difference, which lines are to be drawn?
How can I improve the drawing speed for the noisy data?

Three reasons come to mind:
Line Length : Depending on the actual numbers sloped lines may be longer by just a few pixels or a lot or even by some substantial factor. Looking at your code I suspect the latter..
Algorithm : Drawing sloped lines does take some algorithm to find the next pixels. Even fast drawing routines need to do some computations as opposed to vertical or horizontal lines, which run straight through the pixel arrays.
Anti-Aliasing : Unless you turn off anti-aliasing completely (with all the ugly consequences) the number of pixels to paint will also be around 2-3 times more as all those anti-aliasing pixels above and below the center lines must also be calculated and drawn. Not to forget calculating their colors!
The remedy for the latter part is obviously to turn off anti-aliasing, but the other problems are simply the way things are. So best don't worry and be happy about the speedy straight lines :-)

If you really have a lot of lines or your lines could be very long (a few time the size of the screen), or if you have a lot of almost 0 pixel line, you have to wrote code to reduce useless drawing of lines.
Well, here are some ideas:
If you write many lines at the same x, then you could replace those by a single line between min and max y at that x.
If your line goes way beyond the screen boundary, you should clip them.
If a line is completly outside of the visible area, you should skip it.
If a line have a 0 length, you should not write it.
If a line has a single pixel length, you should write only that pixel.
Obviously, the benefit depends a lot on how many lines you draw... And also the alternative might not give the exact same result...
In practice, it you draw a chart on a screen, then if you display only useful information, it should be pretty fast on modern hardware.
Well if you use style or colors, it might not be as trivial to optimize the displaying of the data.
Alternatively, they are some charting component that are optimized for display large data... The good one are generally expensive but it might still worth it. Often trials are available so you can get a good idea on how much you might increase the performance and then decide what to do.