We Keep Coding

Cut faraway objects based on depth map

I would like to do grabcut which uses a depth map that cuts away far objects, that is used in mixed reality application. So I would like to show just the front of what I see and the background as virtual reality scene.
The problem right now I tried to adapt so code and what I get is front which is cut but in black color, the mask actually.
I don't know where is the problem settle.
The input is a depth map from zed camera.
here is a picture of the behaviour:
My trial:
private void convertToGrayScaleValues(Mat mask)
{
int width = mask.rows();
int height = mask.cols();
byte[] buffer = new byte[width * height];
mask.get(0, 0, buffer);
for (int x = 0; x < width; x++)
{
for (int y = 0; y < height; y++)
{
int value = buffer[y * width + x];
if (value == Imgproc.GC_BGD)
{
buffer[y * width + x] = 0; // for sure background
}
else if (value == Imgproc.GC_PR_BGD)
{
buffer[y * width + x] = 85; // probably background
}
else if (value == Imgproc.GC_PR_FGD)
{
buffer[y * width + x] = (byte)170; // probably foreground
}
else
{
buffer[y * width + x] = (byte)255; // for sure foreground
}
}
}
mask.put(0, 0, buffer);
}
For Each depth frame from Camera:
Mat erodeElement = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(4, 4));
Mat dilateElement = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(7, 7));
depth.copyTo(maskFar);
Core.normalize(maskFar, maskFar, 0, 255, Core.NORM_MINMAX, CvType.CV_8U);
Imgproc.cvtColor(maskFar, maskFar, Imgproc.COLOR_BGR2GRAY);
Imgproc.threshold(maskFar, maskFar, 180, 255, Imgproc.THRESH_BINARY);
Imgproc.dilate(maskFar, maskFar, erodeElement);
Imgproc.erode(maskFar, maskFar, dilateElement);
Mat bgModel = new Mat();
Mat fgModel = new Mat();
Imgproc.grabCut(image, maskFar, new OpenCVForUnity.CoreModule.Rect(), bgModel, fgModel, 1, Imgproc.GC_INIT_WITH_MASK);
convertToGrayScaleValues(maskFar); // back to grayscale values
Imgproc.threshold(maskFar, maskFar, 180, 255, Imgproc.THRESH_TOZERO);
Mat foreground = new Mat(image.size(), CvType.CV_8UC4, new Scalar(0, 0, 0));
image.copyTo(foreground, maskFar);
Utils.fastMatToTexture2D(foreground, texture);

In this case, the graph cut on the depth image might not be the correct method to solve all of your issue.
If you insist the processing should be done in the depth image. To find everything that is not on the table and filter out the table part. You may first apply the disparity based approach for finding the object that's is not on the ground. Reference: https://github.com/windowsub0406/StereoVision
Then based on the V disparity output image, find the locally connected component that is grouped together. You may follow this link how to do this disparity map in OpenCV which is asking the similar way to find the objects that's not on the ground
If you are ok with RGB based approaches, then use any deep learning-based method to recognize the monitor should be the correct approaches. It can directly detect the mointer bounding box. By apply this bounding box to the depth image, you may have what you want. For deep learning based approaches, there are many available package such as Yolo series. You may find one that is suitable for you. reference: https://medium.com/#dvshah13/project-image-recognition-1d316d04cb4c

Drawing zig-zag lines is much slower than drawing straight lines

While using a self-written graphing control I noticed that the painting of the graph was much slower while displaying noisy data than when it displayed clean data.
I dug further into and narrowed the problem down to its bare minimum difference: Drawing the same amount of lines with varying Y values versus drawing lines with the same Y value.
So for example I put together the following tests. I generate lists of points, one with random Y values, one with the same Y, and one with a Zig-Zag Y pattern.
private List<PointF> GenerateRandom(int n, int width, int height)
{
//Generate random pattern
Random rnd = new Random();
float stepwidth = Convert.ToSingle(width / n);
float mid = Convert.ToSingle(height / 2);
float lastx = 0;
float lasty = mid;
List<PointF> res = new List<PointF>();
res.Add(new PointF(lastx, lasty));
for (int i = 1; i <= n; i++)
{
var x = stepwidth * i;
var y = Convert.ToSingle(height * rnd.NextDouble());
res.Add(new PointF(x, y));
}
return res;
}
private List<PointF> GenerateUnity(int n, int width, int height)
{
//Generate points along a simple line
float stepwidth = Convert.ToSingle(width / n);
float mid = Convert.ToSingle(height / 2);
float lastx = 0;
float lasty = mid;
List<PointF> res = new List<PointF>();
res.Add(new PointF(lastx, lasty));
for (int i = 1; i <= n; i++)
{
var x = stepwidth * i;
var y = mid;
res.Add(new PointF(x, y));
}
return res;
}
private List<PointF> GenerateZigZag(int n, int width, int height)
{
//Generate an Up/Down List
float stepwidth = Convert.ToSingle(width / n);
float mid = Convert.ToSingle(height / 2);
float lastx = 0;
float lasty = mid;
List<PointF> res = new List<PointF>();
res.Add(new PointF(lastx, lasty));
var state = false;
for (int i = 1; i <= n; i++)
{
var x = stepwidth * i;
var y = mid - (state ? 50 : -50);
res.Add(new PointF(x, y));
state = !state;
}
return res;
}
I now draw each list of points a few times and compare how long it takes:
private void DoTheTest()
{
Bitmap bmp = new Bitmap(970, 512);
var random = GenerateRandom(2500, bmp.Width, bmp.Height).ToArray();
var unity = GenerateUnity(2500, bmp.Width, bmp.Height).ToArray();
var ZigZag = GenerateZigZag(2500, bmp.Width, bmp.Height).ToArray();
using (Graphics g = Graphics.FromImage(bmp))
{
var tUnity = BenchmarkDraw(g, 200, unity);
var tRandom = BenchmarkDraw(g, 200, random);
var tZigZag = BenchmarkDraw(g, 200, ZigZag);
MessageBox.Show(tUnity.ToString() + "\r\n" + tRandom.ToString() + "\r\n" + tZigZag.ToString());
}
}
private double BenchmarkDraw(Graphics g, int n, PointF[] Points)
{
var Times = new List<double>();
for (int i = 1; i <= n; i++)
{
g.Clear(Color.White);
System.DateTime d3 = DateTime.Now;
DrawLines(g, Points);
System.DateTime d4 = DateTime.Now;
Times.Add((d4 - d3).TotalMilliseconds);
}
return Times.Average();
}
private void DrawLines(Graphics g, PointF[] Points)
{
g.DrawLines(Pens.Black, Points);
}
I come up with the following durations per draw:
Straight Line: 0.095 ms
Zig-Zag Pattern: 3.24 ms
Random Pattern: 5.47 ms
So it seems to get progressively worse, the more change there is in the lines to be drawn, and that is also a real world effect I encountered in the control painting I mentioned in the beginning.
My questions are thus the following:
Why does it make a such a brutal difference, which lines are to be drawn?
How can I improve the drawing speed for the noisy data?

Three reasons come to mind:
Line Length : Depending on the actual numbers sloped lines may be longer by just a few pixels or a lot or even by some substantial factor. Looking at your code I suspect the latter..
Algorithm : Drawing sloped lines does take some algorithm to find the next pixels. Even fast drawing routines need to do some computations as opposed to vertical or horizontal lines, which run straight through the pixel arrays.
Anti-Aliasing : Unless you turn off anti-aliasing completely (with all the ugly consequences) the number of pixels to paint will also be around 2-3 times more as all those anti-aliasing pixels above and below the center lines must also be calculated and drawn. Not to forget calculating their colors!
The remedy for the latter part is obviously to turn off anti-aliasing, but the other problems are simply the way things are. So best don't worry and be happy about the speedy straight lines :-)

If you really have a lot of lines or your lines could be very long (a few time the size of the screen), or if you have a lot of almost 0 pixel line, you have to wrote code to reduce useless drawing of lines.
Well, here are some ideas:
If you write many lines at the same x, then you could replace those by a single line between min and max y at that x.
If your line goes way beyond the screen boundary, you should clip them.
If a line is completly outside of the visible area, you should skip it.
If a line have a 0 length, you should not write it.
If a line has a single pixel length, you should write only that pixel.
Obviously, the benefit depends a lot on how many lines you draw... And also the alternative might not give the exact same result...
In practice, it you draw a chart on a screen, then if you display only useful information, it should be pretty fast on modern hardware.
Well if you use style or colors, it might not be as trivial to optimize the displaying of the data.
Alternatively, they are some charting component that are optimized for display large data... The good one are generally expensive but it might still worth it. Often trials are available so you can get a good idea on how much you might increase the performance and then decide what to do.

Drawing sets of coordinates to pixels so they mutually scale

So I have a List<object> of longitude and latitude coordinates of two points, and I need to connect the line between them. The trick is to display all of the lines within a panel so that they are scaled within the panel's dimensions (converting coordinate numbers to match the pixels) and I almost got it. However I'm confounded by some unknown problem. The code is:
int canvasWidth = panel1.Width,
canvasHeight = panel1.Height;
var minX1 = tockeKoordinate.Min(x => x.startX);
var minX2 = tockeKoordinate.Min(x => x.endX);
var minX = Math.Min(minX1, minX2);
var maxX1 = tockeKoordinate.Max(x => x.startX);
var maxX2 = tockeKoordinate.Max(x => x.endX);
var maxX = Math.Max(maxX1, maxX2);
var maxY1 = tockeKoordinate.Max(x => x.startY);
var maxY2 = tockeKoordinate.Max(x => x.endY);
var maxY = Math.Max(maxY1, maxY2);
var minY1 = tockeKoordinate.Min(x => x.startY);
var minY2 = tockeKoordinate.Min(x => x.endY);
var minY = Math.Min(minY1, minY2);
double coordinatesWidth = Math.Abs(maxX - minX),
coordinatesHeight = Math.Abs(maxY - minY);
float coefWidth = (float)coordinatesWidth / canvasWidth,
coefHeight = (float)coordinatesHeight / canvasHeight;
Basically I check the List for minimum and maximum XY coordinates, so I know what the extreme values are. Then I use a coeficient value to recalculate the coords in pixels so that are within the panel. When I use this:
drawLine(Math.Abs((float)(line.startX - minX) / coefWidth),
Math.Abs((float)(line.startY - minY) / coefHeight),
Math.Abs((float)(line.endX - maxX) / coefWidth),
Math.Abs((float)(line.endY - maxY) / coefHeight));
which is in foreach loop that iterates trough all the elements from the List . The drawline() method is as follows:
private void drawLine(float startX, float startY, float endX, float endY)
{
PointF[] points =
{
new PointF(startX, startY),
new PointF(endX, endY),
};
g.DrawLine(myPen, points[0], points[1]);
}
WHen all of this is put together, I get this picture:
I know for a fact that the "lines" should be connected and form shapes, in this case they represent roads in a suburban area.
I figured that it treats every coordinate set like it is the only one and then scales it to the panel dimensions. Actually it should scale it in reference to all of the other coordinates
It should "zoom" them out and connect with each other, because that is the way I defined the panel dimensions and everything else.
EDIT: ToW's solution did the trick, with this line of code changed to use my List:
foreach (var line in tockeKoordinate)
{
gp.AddLine((float)(line.startX), (float)(line.startY), (float)(line.endX), (float)(line.endY));
gp.CloseFigure();
}
End result when working properly:

As far as I can see your best bet would be to add all those lines to a GraphicsPath.
After it is complete you can look at its bounding rectangle and compare it to the size your Panel offers.
Then you can calculate a scale for the Graphics object to draw with and also a translation.
Finally you draw the lines with Graphics.DrawPath.
All with just 2 division on your side :-)
Here is an example:
private void panel1_Paint(object sender, PaintEventArgs e)
{
Graphics G = e.Graphics;
Random R = new Random(13);
GraphicsPath gp = new GraphicsPath();
for (int i = 0; i < 23; i++)
{
gp.AddLine(R.Next(1234), R.Next(1234), R.Next(1234), R.Next(1234));
gp.CloseFigure(); // disconnect lines
}
RectangleF rect = gp.GetBounds();
float scale = Math.Min(1f * panel1.Width / rect.Width,
1f * panel1.Height / rect.Height);
using (Pen penUnscaled = new Pen(Color.Blue, 4f))
using (Pen penScaled = new Pen(Color.Red, 4f))
{
G.Clear(Color.White);
G.DrawPath(penUnscaled, gp);
G.ScaleTransform(scale, scale);
G.TranslateTransform(-rect.X, -rect.Y);
G.DrawPath(penScaled, gp);
}
}
A few notes:
The blue lines do not fit onto the panel
The red lines are scaled down to fit
The Pen is scaled along with the rest of the Graphics but won't go under 1f.
To create connected lines do add a PointF[] or, more convenient a List<PointF>.ToArray().
I really should have used panel1.ClientSize.Width instead of panel1.Width etc..; now it is off a tiny bit at the bottom; bad boy me ;-)

Extract colour gradient pixel information

I am attempting to control a 240 long line of RGB pixels (ws2812b) using an artnet to dmx controller and need to generate colour gradients down the length of the line of pixels.
I had the idea of using the C# built in graphics libraries to generate the colour gradients and then extract the individual pixel values and send these to the dmx controller.
Is it possible to extract individual interpolated values from a LinearGradientBrush or a LinearGradientBrush applied to a shape (line/rectangle etc)?

What you could do is let the brush draw a line on a bitmap and extract the pixels from that, but I believe that would be unnecessarily expensive and complicated. What would be better is simply lerping between the colours you want.
This can be achieved by writing a lerp method like so:
float Lerp(float from, float to, float amount)
{
return from + amount * (to - from);
}
and using this for the R G and B values of the colors you want to lerp between. For example:
Color Lerp(Color from, Color to, float amount)
{
return Color.FromArgb(
(int)Lerp(from.R, to.R, amount),
(int)Lerp(from.G, to.G, amount),
(int)Lerp(from.B, to.B, amount));
}
I hope this helps.
~Luca

Here is a function that takes a list of stop colors and returns a list of evenly interpolated colors:
List<Color> interpolateColors(List<Color> stopColors, int count)
{
List<Color> ColorList = new List<Color>();
using (Bitmap bmp = new Bitmap(count, 1))
using (Graphics G = Graphics.FromImage(bmp))
{
Rectangle bmpCRect = new Rectangle(Point.Empty, bmp.Size);
LinearGradientBrush br = new LinearGradientBrush
(bmpCRect, Color.Empty, Color.Empty, 0, false);
ColorBlend cb = new ColorBlend();
cb.Colors = stopColors.ToArray();
float[] Positions = new float[stopColors.Count];
for (int i = 0; i < stopColors.Count; i++)
Positions [i] = 1f * i / (stopColors.Count-1);
cb.Positions = Positions;
br.InterpolationColors = cb;
G.FillRectangle(br, bmpCRect);
for (int i = 0; i < count; i++) ColorList.Add(bmp.GetPixel(i, 0));
br.Dispose();
}
return ColorList;
}
You could call it as:
List<Color> ColorList = interpolateColors(
new List<Color>{Color.Red, Color.Blue, Color.Yellow}, 240);
240 and 740 colors. To get all distinct colors make sure they are not too many and not too close, as the maximum number of RGB hues between two colors is 256, so the second example may hit that limit..

Draw types of signals with changing amplitude and frequency on-the-fly

As title said:
I have form with 2 trackbars. One for frequency and one for amplitude. I set up timer for on-the-fly changing.
private void timer1_Tick(object sender, EventArgs e)
{
float amplitude, frequency;
amplitude = Convert.ToSingle(trackBar1.Value) / 100;
label1.Text = amplitude.ToString() + " V";
frequency = trackBar2.Value;
label2.Text = frequency.ToString() + " Hz";
}
I have also 4 radio-buttons to decide, which type of signal will be displayed (sine, square, triangle, sawthoot)
Now I have this implemented with ImageList (change image of signal).
How can I draw type of signal and regulate it with with trackbars? So it will be like in osciloscope.
Thanks for your answers and code.

Lets start by creating the different signal types, this is a function that creates one wavelength of amplitude 1:
private PointF[] CreateBaseSignal(SignalType signalType)
{
switch (signalType)
{
case SignalType.Sine:
const int oversampling = 32;
PointF[] signal = new PointF[oversampling];
for (int i = 0; i < signal.Length; i++)
{
signal[i].X = (float) i / oversampling;
signal[i].Y = Convert.ToSingle(Math.Sin((double) i / oversampling * 2 * Math.PI));
}
return signal;
case SignalType.Square:
return new PointF[]
{
new PointF(0.0f, -1.0f),
new PointF(0.5f, -1.0f),
new PointF(0.5f, 1.0f),
new PointF(1.0f, 1.0f),
};
case SignalType.Triangle:
return new PointF[]
{
new PointF(0.0f, -1.0f),
new PointF(0.5f, 1.0f),
};
case SignalType.Sawtooth:
return new PointF[]
{
new PointF(0.0f, -1.0f),
new PointF(1.0f, 1.0f),
};
default:
throw new ArgumentException("Invalid signal type", "signalType");
}
}
Then we create the actual signal with the selected amplitude and frequency:
private PointF[] CreateSignal(PointF[] baseSignal, float frequency, float amplitude)
{
PointF[] signal = new PointF[Convert.ToInt32(Math.Ceiling(baseSignal.Length * frequency))];
for(int i = 0; i < signal.Length; i++)
{
signal[i].X = baseSignal[i % baseSignal.Length].X / frequency + (i / baseSignal.Length) / frequency;
signal[i].Y = baseSignal[i % baseSignal.Length].Y * amplitude;
}
return signal;
}
Before attempting to plot this signal to a PictureBox, we scale the signal to fit the width and height:
private PointF[] ScaleSignal(PointF[] signal, int width, int height)
{
const float maximumAmplitude = 10.0f;
PointF[] scaledSignal = new PointF[signal.Length];
for(int i = 0; i < signal.Length; i++)
{
scaledSignal[i].X = signal[i].X * width;
scaledSignal[i].Y = signal[i].Y * height / 2 / maximumAmplitude;
}
return scaledSignal;
}
Using Graphics.DrawLine to plot the signal is way better than Bitmap.SetPixel, since the data points will be connected even at high frequencies. Bitmap.SetPixel is also very slow, you really need to use Bitmap.LockBits and unsafe code for manipulating single pixels to achieve any decent performance. Using Graphics.DrawLine, you also have control over line width, anti-aliasing etc.
Since we have stored the signal in a PointF array, we can use the simple Graphics.DrawLines method to plot the signal instead of iterating over the data points:
private void PlotSignal(PointF[] signal, PictureBox pictureBox)
{
Bitmap bmp = new Bitmap(pictureBox.ClientSize.Width, pictureBox.ClientSize.Height);
signal = ScaleSignal(signal, bmp.Width, bmp.Height); // Scale signal to fit image
using(Graphics gfx = Graphics.FromImage(bmp))
{
gfx.SmoothingMode = SmoothingMode.HighQuality;
gfx.TranslateTransform(0, bmp.Height / 2); // Move Y=0 to center of image
gfx.ScaleTransform(1, -1); // Make positive Y axis point upward
gfx.DrawLine(Pens.Black, 0, 0, bmp.Width, 0); // Draw zero axis
gfx.DrawLines(Pens.Blue, signal); // Draw signal
}
// Make sure the bitmap is disposed the next time around
Image old = pictureBox.Image;
pictureBox.Image = bmp;
if(old != null)
old.Dispose();
}
If you redraw the signal often, you probably want to reuse the the Bitmap and Graphics objects instead of creating new ones each time. Just remember to call Graphics.Clear between each redraw.
Putting everything together in one big statement:
PlotSignal(
CreateSignal(
CreateBaseSignal(signalType),
frequency,
amplitude),
thePictureBox);

If you're after a fast plotting library, I really like Dynamic Data Display
Dynamic Data Display
This is a WPF component, but for fast, smooth drawing applications I really think it is worthwhile to port to WPF sooner rathar than later. It feels like you're not too far into your project at the moment anyway.
Development for WPF seems to have stopped for this component (although it continues to be worked on for Silverlight). The documentation is terrible but the source code is available from the link above so you can extend it as needed (it's quite well written and very extensible) and the source is invaluable as a substitute for the near complete lack of any documentation.

Assuming you want to plot a sin wave on a picture box control, create a picture box control on your form then perform the following:
int width = pictureBox1.Width;
int height = pictureBox1.Height;
Bitmap b = new Bitmap(width, height);
for (int i = 0; i < width; i++)
{
int y = (int)((Math.Sin((double)i * 2.0 * Math.PI / width) + 1.0) * (height - 1) / 2.0);
b.SetPixel(i, y, System.Drawing.Color.Red);
}
pictureBox1.Image = b;