We Keep Coding

Cut faraway objects based on depth map

I would like to do grabcut which uses a depth map that cuts away far objects, that is used in mixed reality application. So I would like to show just the front of what I see and the background as virtual reality scene.
The problem right now I tried to adapt so code and what I get is front which is cut but in black color, the mask actually.
I don't know where is the problem settle.
The input is a depth map from zed camera.
here is a picture of the behaviour:
My trial:
private void convertToGrayScaleValues(Mat mask)
{
int width = mask.rows();
int height = mask.cols();
byte[] buffer = new byte[width * height];
mask.get(0, 0, buffer);
for (int x = 0; x < width; x++)
{
for (int y = 0; y < height; y++)
{
int value = buffer[y * width + x];
if (value == Imgproc.GC_BGD)
{
buffer[y * width + x] = 0; // for sure background
}
else if (value == Imgproc.GC_PR_BGD)
{
buffer[y * width + x] = 85; // probably background
}
else if (value == Imgproc.GC_PR_FGD)
{
buffer[y * width + x] = (byte)170; // probably foreground
}
else
{
buffer[y * width + x] = (byte)255; // for sure foreground
}
}
}
mask.put(0, 0, buffer);
}
For Each depth frame from Camera:
Mat erodeElement = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(4, 4));
Mat dilateElement = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(7, 7));
depth.copyTo(maskFar);
Core.normalize(maskFar, maskFar, 0, 255, Core.NORM_MINMAX, CvType.CV_8U);
Imgproc.cvtColor(maskFar, maskFar, Imgproc.COLOR_BGR2GRAY);
Imgproc.threshold(maskFar, maskFar, 180, 255, Imgproc.THRESH_BINARY);
Imgproc.dilate(maskFar, maskFar, erodeElement);
Imgproc.erode(maskFar, maskFar, dilateElement);
Mat bgModel = new Mat();
Mat fgModel = new Mat();
Imgproc.grabCut(image, maskFar, new OpenCVForUnity.CoreModule.Rect(), bgModel, fgModel, 1, Imgproc.GC_INIT_WITH_MASK);
convertToGrayScaleValues(maskFar); // back to grayscale values
Imgproc.threshold(maskFar, maskFar, 180, 255, Imgproc.THRESH_TOZERO);
Mat foreground = new Mat(image.size(), CvType.CV_8UC4, new Scalar(0, 0, 0));
image.copyTo(foreground, maskFar);
Utils.fastMatToTexture2D(foreground, texture);

In this case, the graph cut on the depth image might not be the correct method to solve all of your issue.
If you insist the processing should be done in the depth image. To find everything that is not on the table and filter out the table part. You may first apply the disparity based approach for finding the object that's is not on the ground. Reference: https://github.com/windowsub0406/StereoVision
Then based on the V disparity output image, find the locally connected component that is grouped together. You may follow this link how to do this disparity map in OpenCV which is asking the similar way to find the objects that's not on the ground
If you are ok with RGB based approaches, then use any deep learning-based method to recognize the monitor should be the correct approaches. It can directly detect the mointer bounding box. By apply this bounding box to the depth image, you may have what you want. For deep learning based approaches, there are many available package such as Yolo series. You may find one that is suitable for you. reference: https://medium.com/#dvshah13/project-image-recognition-1d316d04cb4c

How to detect where the image content ends?

I receive images of the same size but with different amounts of information. Examples below (red borders are mine). The background is always white.
I am trying to detect where the information on the image ends - at what pixel height (and crop accordingly). In other words, find the first non-white pixel from the bottom.
Is there a better way to do this other than extract BitmapData out of Image object and loop through all the pixels?

Just to add a suggestion having looked over your images and your solution (below) and your method is fine but you may be able to improve efficiency.
The more you know about your image the better; you're confident the background is always white (according to your post, the code is a more generic utility but the following suggestion can still work); can you be confident on the furthest point in a non-white pixel will be found if the row is not empty?
For example; in your two pictures the furthest in non-white pixel on a row is about 60px in. If this is universally true for your data then you don't need to scan the whole line of the image, which would make your for loop:
for (int y = bitmap.Height - 1; y >= 0; y--) {
for (int x = 0; x < 60; x++) {
Color color = bitmap.GetPixel(x, y);
if (color.R != backColor.R || color.G != backColor.G || color.B != backColor.B) {
foundContentOnRow = y;
break;
}
}
}
(You could make it a parameter on the function so you can easily control it if needed).
Imagine for example that the first non-white row was 80px down. To find it currently you do 640 x 300 = 192,000 checks. If you could confidently say that you would know a row was blank within 100 pixels (an over-estimate based on the data presented) then this would be 100 * 300 = 30,000 checks per image.
If you always knew that the first 10 pixels of the image were always blank you could shave a little bit more off (say 3000 checks).
Musing on a setup where you knew that the first non-white pixel was between 10 and 60 pixels in (range of 50) you could find it at row 80 in 50 x 300 = 15,000 checks which is a good reduction.
Of course the downside about assumptions is that if things change your assumptions may not be valid, but if the data is going to remain fairly constant then it may be worthwhile, especially if you do this for a lot of images.

I've ended up using the following code to trim the image. Hopefully someone finds this useful.
class Program {
static void Main(string[] args) {
Image full = Image.FromFile("foo.png");
Image cropped = full.TrimOnBottom();
}
}
public static class ImageUtilities {
public static Image TrimOnBottom(this Image image, Color? backgroundColor = null, int margin = 30) {
var bitmap = (Bitmap)image;
int foundContentOnRow = -1;
// handle empty optional parameter
var backColor = backgroundColor ?? Color.White;
// scan the image from the bottom up, left to right
for (int y = bitmap.Height - 1; y >= 0; y--) {
for (int x = 0; x < bitmap.Width; x++) {
Color color = bitmap.GetPixel(x, y);
if (color.R != backColor.R || color.G != backColor.G || color.B != backColor.B) {
foundContentOnRow = y;
break;
}
}
// exit loop if content found
if (foundContentOnRow > -1) {
break;
}
}
if (foundContentOnRow > -1) {
int proposedHeight = foundContentOnRow + margin;
// only trim if proposed height smaller than existing image
if (proposedHeight < bitmap.Height) {
return CropImage(image, bitmap.Width, proposedHeight);
}
}
return image;
}
private static Image CropImage(Image image, int width, int height) {
Rectangle cropArea = new Rectangle(0, 0, width, height);
Bitmap bitmap = new Bitmap(image);
return bitmap.Clone(cropArea, bitmap.PixelFormat);
}
}

Excluding small chunks of pixels from Image .Net

I have black image with white lines. Is it possible to exclude chunks of whihte pixels, that are smaller than specific number? For example: change color of chunks of pixels that are made from less than 10 pixels from white to black.
Original Image:
Image on the output(small areas of white pixels are removed):
Right now I work with AForge library for C#, but C++ ways of solving this are also apreciated(Open CV, for example). And hint, on how this functionality might be called are also appreciated.

Without worrying to much about your details, it does seem trivially simple
Use bitmap in 32bits and use LockBits to get scanlines and direct pointer access to the array.
Scan every pixel with 2 for loops
Every time you find one that matches your target color, scan left right and up and down (X) Amount of pixels to determine if it matches your requirements,
If it does, leave the pixel, if not change it.
if you wanted more speed you could chuck this all in a parallel workload, also there is probably more you could do with a mask array to save you researching dead paths (just a thought)
Note, Obviously you can smarten this up a bit
Exmaple
// lock the array for direct access
var bitmapData = bitmap.LockBits(Bounds, ImageLockMode.ReadWrite, Bitmap.PixelFormat);
// get the pointer
var scan0Ptr = (int*)_bitmapData.Scan0;
// get the stride
var stride = _bitmapData.Stride / BytesPerPixel;
// local method
void Workload(Rectangle bounds)
{
// this is if synchronous, Bounds is just the full image rectangle
var rect = bounds ?? Bounds;
var white = Color.White.ToArgb();
var black = Color.Black.ToArgb();
// scan all x
for (var x = rect.Left; x < rect.Right; x++)
{
var pX = scan0Ptr + x;
// scan all y
for (var y = rect.Top; y < rect.Bottom; y++)
{
if (*(pX + y * stride ) != white)
{
// this will turn it to monochrome
// so add your threshold here, ie some more for loops
//*(pX + y * Stride) = black;
}
}
}
}
// unlock the bitmap
bitmap.UnlockBits(_bitmapData);
To parallel'ize it
You could use something like this to break your image up into smaller regions
public static List<Rectangle> GetSubRects(this Rectangle source, int size)
{
var rects = new List<Rectangle>();
for (var x = 0; x < size; x++)
{
var width = Convert.ToInt32(Math.Floor(source.Width / (double)size));
var xCal = 0;
if (x == size - 1)
{
xCal = source.Width - (width * size);
}
for (var y = 0; y < size; y++)
{
var height = Convert.ToInt32(Math.Floor(source.Height / (double)size));
var yCal = 0;
if (y == size - 1)
{
yCal = source.Height - (height * size) ;
}
rects.Add(new Rectangle(width * x, height * y, width+ xCal, height + yCal));
}
}
return rects;
}
And this
private static void DoWorkload(Rectangle bounds, ParallelOptions options, Action<Rectangle?> workload)
{
if (options == null)
{
workload(null);
}
else
{
var size = 5 // how many rects to work on, ie 5 x 5
Parallel.ForEach(bounds.GetSubRects(size), options, rect => workload(rect));
}
}
Usage
DoWorkload(Bounds, options, Workload);

How to draw thousands of tiles without killing FPS

I've looked everywhere for a workaround to this issue (I may just be blind to see the solutions lying around). My game currently renders the tilemap on the screen and will not render tiles that are not actually within the screen bounds. However, each tile is 16x16 pixels, that means 8100 tiles to draw if every pixel on the screen contains a tile at 1920x1080 resolution.
Drawing that many tiles every cycle really kills my FPS. If I run 800x600 resolution my FPS goes to ~20, and at 1920x1080 it runs at around 3-5 FPS. This really drives me nuts.
I've tried threading and using async tasks, but those just flicker the screen. Probably just me coding it incorrectly.
Here's the drawing code that I currently use.
// Get top-left tile-X
Vector topLeft = new Vector(Screen.Camera.X / 16 - 1,
Screen.Camera.Y / 16 - 1);
Vector bottomRight = new Vector(topLeft.X + (Screen.Width / 16) + 2,
topLeft.Y + (Screen.Height / 16) + 2);
// Iterate sections
foreach (WorldSection section in Sections)
{
// Continue if out of bounds
if (section.X + ((Screen.Width / 16) + 2) < (int)topLeft.X ||
section.X >= bottomRight.X)
continue;
// Draw all tiles within the screen range
for (int x = topLeft.X; x < bottomRight.X; x++)
for (int y = topLeft.Y; y < bottomRight.Y; y++)
if (section.Blocks[x - section.X, y] != '0')
DrawBlock(section.Blocks[x - section.X, y],
x + section.X, y);
}
There are between 8 and 12 sections. Each tile is represented by a char object in the two-dimensional array.
Draw block method:
public void DrawBlock(char block, int x int y)
{
// Get the source rectangle
Rectangle source = new Rectangle(Index(block) % Columns * FrameWidth,
Index(block) / Columns * FrameHeight, FrameWidth, FrameHeight);
// Get position
Vector2 position = new Vector2(x, y);
// Draw the block
Game.spriteBatch.Draw(Frameset, position * new Vector2(FrameWidth, FrameHeight) - Screen.Camera, source, Color.White);
}
The Index() method just returns the frame index of the tile corresponding to the char.
I'm wondering how I could make it possible to actually allow this much to be drawn at once without killing the framerate in this manner. Is the code I provided clearly not very optimized, or is it something specific I should be doing to make it possible to draw this many individual tiles without reducing performance?

Not sure if this is the best way to deal with the problem, but I've started to use RenderTarget2D to pre-render chunks of the world into textures. I have to load chunks within a given area around the actual screen bounds at a time, because loading all chunks at once will make it run out of memory.
When you get close to the bounds of the current pre-rendered area, it will re-process chunks based on your new position in the world. The processing takes roughly 100 milliseconds, so when loading new areas the player will feel a slight slowdown for this duration. I don't really like that, but at least the FPS is 60 now.
Here's my chunk processor:
public bool ProcessChunk(int x, int y)
{
// Create render target
using (RenderTarget2D target = new RenderTarget2D(Game.CurrentDevice, 16 * 48, 16 * 48,
false, SurfaceFormat.Color, DepthFormat.Depth24))
{
// Set render target
Game.CurrentDevice.SetRenderTarget(target);
// Clear back buffer
Game.CurrentDevice.Clear(Color.Black * 0f);
// Begin drawing
Game.spriteBatch.Begin(SpriteSortMode.Texture, BlendState.AlphaBlend);
// Get block coordinates
int bx = x * 48,
by = y * 48;
// Draw blocks
int count = 0;
foreach (WorldSection section in Sections)
{
// Continue if section is out of chunk bounds
if (section.X >= bx + 48) continue;
// Draw all tiles within the screen range
for (int ax = 0; ax < 48; ax++)
for (int ay = 0; ay < 48; ay++)
{
// Get the block character
char b = section.Blocks[ax + bx - section.X, ay + by];
// Draw the block unless it's an empty block
if (b != '0')
{
Processor.Blocks[b.ToString()].DrawBlock(new Vector2(ax, ay), true);
count++;
}
}
}
// End drawing
Game.spriteBatch.End();
// Clear target
target.GraphicsDevice.SetRenderTarget(null);
// Set texture
if (count > 0)
{
// Create texture
Chunks[x, y] = new Texture2D(Game.CurrentDevice, target.Width, target.Height, true, target.Format);
// Set data
Color[] data = new Color[target.Width * target.Height];
target.GetData<Color>(data);
Chunks[x, y].SetData<Color>(data);
// Return true
return true;
}
}
// Return false
return false;
}
If there are any suggestions on how this approach can be improved, I won't be sad to hear them!
Thanks for the help given here!

How can I introduce an overlay on an image

How can I manipulate images to add a semi-transparent 1x1 checked overlay like the second image in C#?

I was able to modify an answer I posted a while ago and create the overlay in code. After the overlay image is created, I use a TextureBrush to fill the area of the original image. The settings in the code below created the following image; you can change the size and colors to suit your needs.
// set the light and dark overlay colors
Color c1 = Color.FromArgb(80, Color.Silver);
Color c2 = Color.FromArgb(80, Color.DarkGray);
// set up the tile size - this will be 8x8 pixels, with each light/dark square being 4x4 pixels
int length = 8;
int halfLength = length / 2;
using (Bitmap overlay = new Bitmap(length, length, PixelFormat.Format32bppArgb))
{
// draw the overlay - this will be a 2 x 2 grid of squares,
// alternating between colors c1 and c2
for (int x = 0; x < length; x++)
{
for (int y = 0; y < length; y++)
{
if ((x < halfLength && y < halfLength) || (x >= halfLength && y >= halfLength))
overlay.SetPixel(x, y, c1);
else
overlay.SetPixel(x, y, c2);
}
}
// open the source image
using (Image image = Image.FromFile(#"C:\Users\Public\Pictures\Sample Pictures\homers_brain.jpg"))
using (Graphics graphics = Graphics.FromImage(image))
{
// create a brush from the overlay image, draw over the source image and save to a new image
using (Brush overlayBrush = new TextureBrush(overlay))
{
graphics.FillRectangle(overlayBrush, new Rectangle(new Point(0, 0), image.Size));
image.Save(#"C:\Users\Public\Pictures\Sample Pictures\homers_brain_overlay.jpg");
}
}
}

Load your original image in to a system.Drawing.Image, then create a graphics object from it. Load your 2nd image of the checker pattern you want to draw, and use the graphics object you created to repeatedly draw the checker image over the original image.
Untested Example
Image Original;
Image Overlay;
Original = new Bitmap(100, 100, System.Drawing.Imaging.PixelFormat.Format32bppArgb); //Load your real image here.
Overlay = new Bitmap(2, 2 ,System.Drawing.Imaging.PixelFormat.Format32bppArgb);//Load your 2x2 (or whatever size you want) overlay image here.
Graphics gr = Graphics.FromImage(Original);
for (int y = 0; y < Original.Height + Overlay.Height; y = y + Overlay.Height)
{
for (int x = 0; x < Original.Width + OverlayWidth; x = x + Overlay.Width)
{
gr.DrawImage(Overlay, x, y);
}
}
gr.Dispose();
After the code executes, Original will now contain the Original image with the overlay applied to it.