Manually Feed Point Cloud Data into Kinect Fusion Toolkit - c#

I'm trying to manually feed in data to Kinect Fusion's AlignPointCloud functionality. In the Kinect SDK Toolkit 1.8, there's an example of how to use Kinect Fusion. I'm using the toolkit provided in the samples to try to use Fusion to try to align two point clouds; however, I can't seem to get the AlignPointCloud method to ever converge successfully.
I'm sure that there is something I'm misunderstanding about how to copy data into the FusionPointCloudImageFrame.
What I'm currently trying to do (A trivial case, simply matching together two planes) fails:
float[] arr1 = new float[80 * 60 * 6];
float[] arr2 = new float[80 * 60 * 6];
for (int y = 0; y < 60; y++) {
for (int x = 0; x < 80; x++) {
int ind = y * 80 + x;
arr1[ind] = x / .1f; // X coordinate
arr1[ind + 1] = y / .1f; // Y coordinate
arr1[ind + 2] = 1; // Z coordinate
// Normals
arr1[ind + 3] = 0;
arr1[ind + 4] = 0;
arr1[ind + 5] = 1;
arr2[ind] = x / .1f; // X coordinate
arr2[ind + 1] = y / .1f; // Y coordinate
arr2[ind + 2] = 2; // Z coordinate
// Normals
arr1[ind + 3] = 0;
arr1[ind + 4] = 0;
arr1[ind + 5] = 1;
}
}
FusionPointCloudImageFrame pcl1 = new FusionPointCloudImageFrame(80, 60);
FusionPointCloudImageFrame pcl2 = new FusionPointCloudImageFrame(80, 60);
pcl1.CopyPixelDataFrom(arr1);
pcl2.CopyPixelDataFrom(arr2);
Matrix4 m = Matrix4.Identity;
bool success = FusionDepthProcessor.AlignPointClouds(pcl1, pcl2, 7, null, ref m);
// Does not converge, m is identity regardless of what it was before
What am I doing incorrectly, or what do I need to change to manually feed in data to match two point clouds? Also, can someone please explain to me the significance of a point cloud having a width and height? Each point has an x,y, and z value and doesn't need to be in any ordered way afaik, so why do we need to provide a width or height? If I'm reading the data from a .obj (wavefront) file, how can I determine the width and height?
Thanks!

At a quick glance, this example probably fails to converge because there is no texture to "fix" these planes: they can just slide around and match equally well. Try using some less trivial test data.
The "point cloud" is essentially just a depth image, so it can have a width and height as with a frame that originated from the Kinect camera.
If you are rendering from a mesh, you can choose an appropriate width and height, e.g. 640x480.

I don't know if this is still helpful, but there is a problem with your code, you are only storing the x coord and for the normals you're not storing them for the second array. For my part I use coordinates saved from a previous mesh from kinect fusion that I store in a file and then load them to compare them with the actual point cloud frame. However it seems to give me true each time so I don't know if alignpointcloud is used to limit transformation between two point cloud or can be used for object recognition.

Related

How to find the "best" position using newtons method for n samples

I am trying to find the best position for n sphere intersections using newtons method.
Sphere equation: (x - a)2 + (y - b)2 + (z - c)2 = r2
where a, b, c = center of sphere
where x, y, z = sample location
where r2 = radius squared
for a point to lie on any sphere, it needs to satisfy this equation. So, for a point to lie on (or close) to multiple spheres, it needs to satisfy this equation for multiple spheres.
I think we can make a cost function like this:
sum((x - a_i)2 + (y - b_i)2 + (z - c_i)2 - (r_i)2) == 0;
where a_i, b_i, c_i = center of i sphere
where x, y, z = sample location
where r_i = radius of i sphere
since dot(pos - center) should equal radii squared, if dot(pos - center) - radii squared == 0, it is the same function.
Newtons method works by using taking samples at intervals, and then uses the derivative of a function in order to move it. This equation looks like this:
s2 = s1 - (f(s1) / f`(s1))
where (s2) = second sample location (target)
f(s1) = first sample location's equation
f`(s1) = derivative of the sample
I have written the following function in C#, which solves the problem when I input 2 spheres:
// For maxIterations:
for (int i = 0; i < maxIterations; i++)
{
float3 vDist = float3.zero;
float errorFunction = 0f;
float3 derivative = float3.zero;
// For each sphere
for (int j = 0; j < centers.Length; j++)
{
// Calculate the sqDistance between the sample point and the sphere, and subtract it with the squared radii in order to get an error function
vDist = sampleLocation - centers[j];
errorFunction += math.dot(vDist, vDist) - (radii[j] * radii[j]);
// The derivative of the sphere is needed in order to move the sample
derivative += new float3((sampleLocation.x - centers[j].x) * (sampleLocation.x - centers[j].x), (sampleLocation.y - centers[j].y) * (sampleLocation.y - centers[j].y), (sampleLocation.z - centers[j].z) * (sampleLocation.z - centers[j].z));
}
// If the errorFunction gets within a certain threshold, the position is found
if (errorFunction >= -threshold && errorFunction <= threshold)
{
print("Found position: " + sampleLocation + " in " + i + ", error: " + errorFunction);
break;
}
// Move the sample based on the error function and the derivative of the sphere
sampleLocation = sampleLocation - (errorFunction / derivative);
}
However, when I input 3 spheres, it does not work anymore, I think because the equation becomes non-linear?
I found a good reference on stackoverflow:
Math - 3d positioning/multilateration
but I don't quite understand the answer given there.
Could someone try to help me extend this function to work with multiple spheres?

Enhance performance to paint image, is SIMD perhapse a solution?

I have no experience with SIMD, but have a method that is too slow. I know get 40fps, and I need more.
Does anyone know how I could make this paint method faster? Perhaps the SIMD instructions are a solution?
The sourceData is now a byte[] (videoBytes) but could use a pointer too.
public bool PaintFrame(IntPtr layerBuffer, ushort vStart, byte vScale)
{
for (ushort y = 0; y < height; y++)
{
ushort eff_y = (ushort)(vScale * (y - vStart) / 128);
var newY = tileHeight > 0 ? eff_y % tileHeight : 0;
uint y_add = (uint)(newY * tileWidth * bitsPerPixel >> 3);
for (int x = 0; x < width; x++)
{
var newX = tileWidth > 0 ? x % tileWidth : 0;
ushort x_add = (ushort)(newX * bitsPerPixel >> 3);
uint tile_offset = y_add + x_add;
byte color = videoBytes[tile_offset];
var colorIndex = BitsPerPxlCalculation(color, newX);
// Apply Palette Offset
if (paletteOffset > 0)
colorIndex += paletteOffset;
var place = x + eff_y * width;
Marshal.WriteByte(layerBuffer + place, colorIndex);
}
}
return true;
}
private void UpdateBitPerPixelMethod()
{
// Convert tile byte to indexed color
switch (bitsPerPixel)
{
case 1:
BitsPerPxlCalculation = (color, newX) => color;
break;
case 2:
BitsPerPxlCalculation = (color, newX) => (byte)(color >> 6 - ((newX & 3) << 1) & 3);
break;
case 4:
BitsPerPxlCalculation = (color, newX) => (byte)(color >> 4 - ((newX & 1) << 2) & 0xf);
break;
case 8:
BitsPerPxlCalculation = (color, newX) => color;
break;
}
}
More info
Depending on the settings, the bpp can be changed. The indexed colors and the palette colors are separatly stored. Here I have to recreate the image pixels indexes, so later on I use the palette and color indexes in WPF(Windows) or SDL(Linux, Mac) to display the image.
vStart is the ability to crop the image on top.
The UpdateBitPerPixelMethod() will not change during a frame rendering, only before. During the for, no settings data can be changed.
So I was hoping that some parts can be written with SIMD, because the procedure is the same for all pixels.
Hy,
your code is not the clearest to me. Are you trying to create a new matrix / image ? If yes create a new 2D allocation and calculate the entire image into it. Set it to 0 after you do not need the calculations anymore.
Replace the Marshal.WriteByte(layerBuffer + place, colorIndex);with a 2D image ( maybe this is the image ?).
Regarding the rest it is a problem because you have non uniform offsets in indexing and jumps. That will make developing a SIMD solution difficult (you need masking and stuff). My bet would be to calculate everything for all the indices and save it into individual 2D matrices, that are allocated once at the begining.
For example:
ushort eff_y = (ushort)(vScale * (y - vStart) / 128);
Is calculated per every image row. Now you could calculate it once as an array since I do not believe that the format size of the images changes during the run.
I dont know if vStart and vScale are defined as a constant at program start. You should do this for every calculation that uses constant, and just read the matrices later to calculate.
SIMD can help but only if you do every iteration you calculate the same thing and if you avoid branching and switch cases.
Addition 1
You have multiple problems and design considerations from my stand point.
First of all you need to get away from the idea SIMD is going to help in your case. You would need to remove all conditional statements. SIMD-s are not build to deal with conditional statements.
Your idea should be to split up the logic into manageable pieces so you can see witch piece of the code takes most time.
One big problem is the write byte in the marshal, this is automatically saying to the compiler that you handle only and exclusively 1 byte. I'm guessing that this creates on big bottle neck.
By code analysis I see in each loop you are doing checks. This must be restructured.
Assumption is the image get rarely cropped this would be a separation from the image calculations.
List<ushort> eff_y = new List<ushort>();
List<uint> y_add = new List<uint>();
for (ushort y = 0; y < height; y++)
{
eff_y.add((ushort)(vScale * (y - vStart) / 128));
var newY = tileHeight > 0 ? eff_y % tileHeight : 0;
y_add = (uint)(newY * tileWidth * bitsPerPixel >> 3);
}
So this can be precalculated and changed only when the cropping changes.
Now it gets realy tricky.
paletteOffset - the if statement makes only sense in paletteOffset can be negative, then zero it out and remove the if statement
bitsPerPixel - this looks like a fixed value for the rendering duration
so remove the UpdateBitPerPixelMethod and send in a parameter.
for (ushort y = 0; y < height; y++)
{
for (int x = 0; x < width; x++)
{
var newX = tileWidth > 0 ? x % tileWidth : 0; // conditional stetement
ushort x_add = (ushort)(newX * bitsPerPixel >> 3);
uint tile_offset = y_add + x_add;
byte color = videoBytes[tile_offset];
var colorIndex = BitsPerPxlCalculation(color, newX);
// Apply Palette Offset
if (paletteOffset > 0) // conditional stetement
colorIndex += paletteOffset;
var place = x + eff_y * width;
Marshal.WriteByte(layerBuffer + place, colorIndex);
}
}
This are only few things that need to be done before you try anything with the SIMD. But by that time the changes will give the compiler hints about what you want to do. This could improve the machine code execution. You need also to test the performance of your code to pinpoint the bottle neck it is very hard to assume or guess correctly by code.
Good luck

Coordinate system of panels in c#?

I'm very new to c# and running into a problem while trying to program and visualize the Mandelbrotset.
I have created a 400 by 400 panel and want to use this to graph the set. I want my graph to go from -2 to 2 on both axes so I'm using a scale of 0.01. When looking at my code, I think the paint method should work; the coordinates are converted in the correct way it seems. The problem is that the graph is not fully shown on the panel while running. (0,0) is somewhere in the lower right corner, removing much of the coordinates in that corner.
Below is the function used to draw the graph on the panel. Did I make a mistake in coding the coordinates this way? Or am I misunderstanding how coordinates work in a panel?
for (int xco =0; xco<400; xco++)
{
for (int yco=0; yco<400; yco++)
{
double x = (xco - 200) * scale;
double y = (yco - 200) * scale;
int mandel = 0;
double fx = x, fy = y;
double distance = 0;
while ((mandel<max) && (distance<2))
{
double fx1 = fx;
fx = fx * fx - fy * fy + x;
fy = 2 * fx1 * fy + y;
distance = Math.Sqrt(fx * fx + fy * fy);
mandel++;
}
if (mandel%2==1)
pea.Graphics.FillRectangle(Brushes.White, xco, yco, xco + 1, yco + 1);
else
pea.Graphics.FillRectangle(Brushes.Black, xco, yco, xco + 1, yco + 1);
}
}

Creating features(point) grid on polygon

I am working on GIS based desktop application using C#. I am using dotspatial library in this project.
Now I need to create a grid of features on polygon. This grid cell (rectangle) should be 20*20 Meter Square.
I have worked on it and able to create grid but facing issue regarding to cell size. Whenever polygon size changed cell size also reduced. My code.
// Polygon Width = 2335
// Polygon Height = 2054
int RowsCount = 111;
int ColumnsCount = 111;
var maxPointX = Polygon.Extent.MaxPointX;
var minPointX = Polygon.Extent.MinPointX;
var maxPointY = Polygon.Extent.MaxPointY;
var minPointY = Polygon.Extent.MinPointY;
var dXStep = (maxPointX - minPointX) / (ColumnsCount - 1);
var dYStep = (maxPointY - minPointY) / (RowsCount - 1);
var gridColumnsPoints = new double[1000000];
var gridRowPoints = new double[1000000];
// Calculate the coordinates of grid
var nextPointX = minPointX;
for (int i = 1; i <= ColumnsCount; i++)
{
gridColumnsPoints[i - 1] = nextPointX;
nextPointX = nextPointX + dXStep;
}
var nextPointY = minPointY;
for (int i = 1; i <= RowsCount; i++)
{
gridRowPoints[i - 1] = nextPointY;
nextPointY = nextPointY + dYStep;
}
Output
Now when I tried this code on small size of Polygon then grid cell size also decreased.
I know my approach is not correct, So I have searched on it and got some tools. like
https://gis.stackexchange.com/questions/79681/creating-spatially-projected-polygon-grid-with-arcmap
But I want to create it in C# and unable to found any algorithm or any other helping material.
Please share your knowledge. Thanks
I am not able to understand, if you want the grid cell size to be 20*20 meters, how does the size change from polygon to polygon. It should always be 20*20 meters.
In you code, where did you get the values for ColumnsCount and RowsCount?
Your dx and dy should always be 20 (if the spatial reference units are in meters) or you need to convert the 20 meters to appropriate length of units of the spatial reference.
Pseudo code for creating grid:
var xMax = Polygon.extent.xmax;
var xMin = Polygon.extent.xmin;
var yMax = Polygon.extent.ymax;
var yMin = Polygon.extent.ymin;
var gridCells = [];
var x = xMin, y = yMin;
while(x <= xMax){
var dx = x + 20;
while(y <= yMax){
var dy = y + 20;
var cell = new Extent(x, y, dx, dy);
gridCells.push(cell);
y = dy;
}
x = dx;
}
The problem is here:
var dXStep = (maxPointX - minPointX) / (ColumnsCount - 1);
var dYStep = (maxPointY - minPointY) / (RowsCount - 1);
because it makes the grid size dependent on the polygon, but it should be fixed to the scale of the view.
I'm not familiar with the dotspatial framwork, but you must operate in a coordinate system of a kind. You should align your grid to that coordinate system by calculating the first x pos to the left of the polygon in some distance from the polygons bounding box (max/min) and then step with the resolution of the coordinate system through to the max X of the polygon.

Find vertices in equilateral triangle mesh originating from a center vertex

I'd like to ask whether there is code out there or if you can give me some help in writing some (C#, but I guess the maths is the same everywhere).
I'd like to specify a center point from which an equilateral triangle mesh is created and get the vertex points of these triangles. The center point should not be a face center, but a vertex itself.
A further input would be the size of the triangles (i.e side length) and a radius to which triangle vertices are generated.
The reason behind it is that I want to create a mesh which is centered nicely on the screen/window center with as little code as possible. I just find mesh generation code, but not a "radial outward propagation" example.
In the end, I'd like to have the subsequently farther away vertices being displaced in a logarithmic fashion, but I guess that's just an easy addition once the mesh code is there.
Can anybody help me with that? Thanks!
You need to specify two things, a radius and the direction that the first triangle points.
The radius will be the distance from the initial point to the vertices of the first triangle. All triangles will have the same radius.
The direction is some specification in radians. I will assume that 0 means pointing to the right (PI would be point to the left).
Finding the vertices of the first triangle can be done like this (pseudo-code, not language specific):
float theta = 0; // The direction, 0 means pointing to the right
float thetaInc = TWO_PI/3; // 3 because you want a triangle
for (int i = 0; i < 3; i++) {
vertX[i] = initialPointX+cos(theta)*radius;
vertY[i] = initialPointY+sin(theta)*radius;
theta += thetaInc;
}
There are many ways to find the center points of the neighboring triangles. One way would be to use the same code but initialize theta = TWO_PI/6, replace radius with foo (see math below), assign new center points of neighboring triangles in the for loop, and then use the same code with an appropriately rotated direction (theta += PI) to find the vertices of those triangles.
Distance from one triangle center to another only knowing radius:
hypotenuse = sqrt(sq(radius)+sq(radius));
halfHypotenuse = hypotenuse/2.0;
Pythagorean theorem to find distance from center of triangle to center of an edge: foo = sqrt(sq(radius)-sq(halfHypotenuse));
Final distance = foo*2.0;
Code to find the center points of the neighboring triangles:
float[] nx = new float[3];
float[] ny = new float[3];
float theta = TWO_PI/6;
float hyp = sqrt(sq(radius)+sq(radius));
float halfHyp = hyp/2.0;
float foo = sqrt((sq(radius)-sq(halfHyp)))*2.0;
for (int i = 0; i < 3; i++) {
nx[i] = initialPointX+cos(theta)*foo;
ny[i] = initialPointY+sin(theta)*foo;
theta += thetaInc;
}
Thank you very much for your answer. I will play around with your code - the propagation part will come handy for sure.
In the meantime I have played around with hexagons instead of triangles and this codes works fairly alright for the same purpose.:
//populate array from the centre hex, going outwards the desired number of hex rings
for (int i = 0; i < numberOfHexagonRings; i++)
{
for (double j = -i; j <= i; j++)
for (double k = -i; k <= i; k++)
for (double l = -i; l <= i; l++)
if ((Math.Abs(j) + Math.Abs(k) + Math.Abs(l) == i * 2) && (j + k + l == 0))
{
positionX = (int)(screenCenterX + ((double)sideLength * (l / 2 + j)));
positionY = (int)(screenCenterY + (3/2 * ((double)sideLength / Math.Sqrt(3)) * l));

Categories