Trying to accurately measure 3D distance from a 2D image - c#

I am trying to extract out 3D distance in mm between two known points in a 2D image. I am using square AR markers in order to get the camera coordinates relative to the markers in the scene. The points are the corners of these markers.
An example is shown below:
The code is written in C# and I am using XNA. I am using AForge.net for the CoPlanar POSIT
The steps I take in order to work out the distance:
1. Mark corners on screen. Corners are represented in 2D vector form, Image centre is (0,0). Up is positive in the Y direction, right is positive in the X direction.
2. Use AForge.net Co-Planar POSIT algorithm to get pose of each marker:
float focalLength = 640; //Needed for POSIT
float halfCornerSize = 50; //Represents 1/2 an edge i.e. 50mm
AVector[] modelPoints = new AVector3[]
{
new AVector3( -halfCornerSize, 0, halfCornerSize ),
new AVector3( halfCornerSize, 0, halfCornerSize ),
new AVector3( halfCornerSize, 0, -halfCornerSize ),
new AVector3( -halfCornerSize, 0, -halfCornerSize ),
};
CoplanarPosit coPosit = new CoplanarPosit(modelPoints, focalLength);
coPosit.EstimatePose(cornersToEstimate, out marker1Rot, out marker1Trans);
3. Convert to XNA rotation/translation matrix (AForge uses OpenGL matrix form):
float yaw, pitch, roll;
marker1Rot.ExtractYawPitchRoll(out yaw, out pitch, out roll);
Matrix xnaRot = Matrix.CreateFromYawPitchRoll(-yaw, -pitch, roll);
Matrix xnaTranslation = Matrix.CreateTranslation(marker1Trans.X, marker1Trans.Y, -marker1Trans.Z);
Matrix transform = xnaRot * xnaTranslation;
4. Find 3D coordinates of the corners:
//Model corner points
cornerModel = new Vector3[]
{
new Vector3(halfCornerSize,0,-halfCornerSize),
new Vector3(-halfCornerSize,0,-halfCornerSize),
new Vector3(halfCornerSize,0,halfCornerSize),
new Vector3(-halfCornerSize,0,halfCornerSize)
};
Matrix markerTransform = Matrix.CreateTranslation(cornerModel[i].X, cornerModel[i].Y, cornerModel[i].Z);
cornerPositions3d1[i] = (markerTransform * transform).Translation;
//DEBUG: project corner onto screen - represented by brown dots
Vector3 t3 = viewPort.Project(markerTransform.Translation, projectionMatrix, viewMatrix, transform);
cornersProjected1[i].X = t3.X; cornersProjected1[i].Y = t3.Y;
5. Look at the 3D distance between two corners on a marker, this represents 100mm. Find the scaling factor needed to convert this 3D distance to 100mm. (I actually get the average scaling factor):
for (int i = 0; i < 4; i++)
{
//Distance scale;
distanceScale1 += (halfCornerSize * 2) / Vector3.Distance(cornerPositions3d1[i], cornerPositions3d1[(i + 1) % 4]);
}
distanceScale1 /= 4;
6. Finally I find the 3D distance between related corners and multiply by the scaling factor to get distance in mm:
for(int i = 0; i < 4; i++)
{
distance[i] = Vector3.Distance(cornerPositions3d1[i], cornerPositions3d2[i]) * scalingFactor;
}
The distances acquired are never truly correct. I used the cutting board as it allowed me easy calculation of what the distances should be. The above image calculated a distance of 147mm (expected 150mm) for corner 1 (red to purple). The image below shows 188mm (expected 200mm).
What is also worrying is the fact that when measuring the distance between marker corners sharing an edge on the same marker, the 3D distances obtained are never the same. Another thing I noticed is that the brown dots never seem to exactly match up with the colored dots. The colored dots are the coordinates used as input to the CoPlanar posit. The brown dots are the calculated positions from the center of the marker calculated via POSIT.
Does anyone have any idea what might be wrong here? I am pulling out my hair trying to figure it out. The code should be quite simple, I don't think I have made any obvious mistakes with the code. I am not great at maths so please point out where my basic maths might be wrong as well...

You are using way to many black boxes in your question. What is the focal length in the second step? Why go through ypr in step 3? How do you calibrate? I recommend to start over from scratch without using libraries that you do not understand.
Step 1: Create a camera model. Understand the errors, build a projection. If needed apply a 2d filter for lens distortion. This might be hard.
Step 2: Find you markers in 2d, after removing lens distortion. Make sure you know the error and that you get the center. Maybe over multiple frames.
Step 3: Un-project to 3d. After 1 and 2 this should be easy.
Step 4: ???
Step 5: Profit! (Measure distance in 3d and know your error)

I think you need to have 3D photo (two photo from a set of distance) so you can get the parallax distance from image differences

Related

How can I calculate an average terrain slope based on four points?

I'm programming a simple physics system for vehicles in Unity and I'm trying to figure out how to calculate the average slope based on four points in order to apply the gravitational acceleration. I can get the terrain height at the vehicle wheels positions like this:
Terrain.activeTerrain.SampleHeight(wheel.position);
How can I calculate the average terrain slope based on those four points? I thought about calculating direction vectors between different points, but I'm not sure how to combine them into a single slope vector.
(Assuming that all 4 wheels are always coplanar) I think you could take any 3 wheels, take the height sample
var wheels = new Vector3[3]{ wheel1.transform.position, wheel2.transform.position, wheel3.transform.position };
for(var i = 0; i < wheels.Length; i++)
{
var w = wheels[i];
w.y = Terrain.activeTerrain.SampleHeight(w)
wheels[i]= w;
}
and from that resulting 3 points generate a plane
var plane = new Plane(wheels[0], wheels[1], wheels[2]);
Now you can take any of these points and take a vector in straight up direction.
var upPoint = wheels[0] + Vector3.up;
Then you can map that vector onto the plane using closest point on plane.
var planePoint = plane.ClosestPointOnPlane(upPoint);
As a last step normalize the vector between that mapped point and your start wheel point and you should have the direction of the slope + the Y component tells you how steep it is
var planeVector = wheels[0] - planePoint;
var direction = planeVector.normalized;
Debug.Log($"Slope direction is {direction} with a steepness factor of {direction.y} or also {-planeVector.magnitude}");
where both planeVector.magnitude and direction.y should be moving between 0 (completely horizontal surface) and -1 (completely vertical surface)
Something like that I guess ^^
Wouldn't that be the average slope between the middle point of the back wheels and the middle point of the front wheels? You get the tangent of the angle by dividing the difference between those heights and their distance (as projected). Guess that's too simple?
Assuming 4 points (x1,y1), (x2,y2), (x3,y3) and (x4,y4). The middlepoints would be xB = (x1+x2)/2, yB=(y1+y2)/2 and xF=(x3+x4)/2 and yF=(y3+y4)/2. Then Tan A = (yF-yB)/(xF-xB)

How to find the area that two triangles are in contact

Given the positions of the vertices, and the surface normals. How can I calculate the area(which may be 0) that the 2 triangles are in contact? These triangles are also in 3D space so if they aren't lined up properly, just jammed into each other, the contact area should be 0.
(In C#)
This is not a trivial problem, so let's break it down into steps.
Check if the two triangles are coplanar, otherwise the area is 0.
Project the triangles onto a 2D surface
Calculate the intersection polygon
Calculate the area
1. Checking for coplanarity
For the two triangles to be coplanar, all vertices of one triangle must lie in the plane determined by the other one.
Using the algorithm described here we can check for every vertex whether that is the case, but due to the fact that floating point numbers are not perfectly precise, you will need to define some error theshold to determine what still counts as coplanar.
Assuming va and vb are the vectors of the triangles A and B respectively, the code could look something like this.
(Note: I have never worked with Unity and am writing all of the code from memory, so please excuse if it isn't 100% correct).
public static bool AreTrianglesCoplanar(Vector3[] va, Vector3[] vb) {
// make sure these are actually triangles
Debug.Assert(va.Length == 3);
Debug.Assert(vb.Length == 3);
// calculate the (scaled) normal of triangle A
var normal = Vector3.Cross(va[1] - va[0], va[2] - va[0]);
// iterate all vertices of triangle B
for(var vertex in vb) {
// calculate the dot product between the normal and the vector va[0] --> vertex
// the dot product will be 0 (or very small) if the angle between these vectors
// is a right angle (90°)
float dot = Vector3.Dot(normal, vertex - va[0]).
// the error threshold
const float epsilon = 0.001f;
// if the dot product is above the threshold, the vertex lies outside the plane
// in that case the two triangles are not coplanar
if(Math.Abs(dot) > epsilon)
return false;
}
return true;
}
2. Projecting the triangles into 2D space
We now know that all six vertices are in the same 2D plane embedded into 3D space, but all of our vertex coordinates are still three-dimensional. So the next step would be to project our points into a 2D coordinate system, such that their relative position is preserved.
This answer explains the math pretty well.
First, we need to find a set of three vectors forming an orthonormal basis (they must be orthoginal to each other and of length 1).
One of them is just the plane's normal vector, so we need two more vectors that are orthogonal to the first, and also orthogonal to each other.
By definition, all vectors in the plane defined by our triangles are orthogonal to the normal vector, so we can just pick one (for example the vector from va[0] to va[1]) and normalize it.
The third vector has to be orthogonal to both of the others, we can find such a vector by taking the cross product of the previous two.
We also need to choose a point in the plane as our origin point, for example va[0].
With all of these parameters, and using the formula from the linked amswer, we can determine our new projected (x, y) coordinates (t_1 and t_2 from the other answer). Note that -- because all of our points lie in the plane defining that normal vector -- the third coordinate (called s in the other answer) will always be (close to) zero.
public static void ProjectTo2DPlane(
Vector3[] va, Vector3[] vb
out Vector2[] vaProjected, out Vector2[] vbProjecte
) {
// calculate the three coordinate system axes
var normal = Vector3.Cross(va[1] - va[0], va[2] - va[0]).normalized;
var e1 = Vector3.Normalize(va[1] - va[0]);
var e2 = Vector3.Cross(normal, e1);
// select an origin point
var origin = va[0];
// projection function we will apply to every vertex
Vector2 ProjectVertex(Vector3 vertex) {
float s = Dot(normal, vertex - origin);
float t1 = Dot(e1, vertex - origin);
float t2 = Dot(e2, vertex - origin);
// sanity check: this should be close to zero
// (otherwise the point is outside the plane)
Debug.Assert(Math.Abs(s) < 0.001);
return new Vector2(t1, t2);
}
// project the vertices using Linq
vaProjected = va.Select(ProjectVertex).ToArray();
vbProjected = vb.Select(ProjectVertex).ToArray();
}
Sanity check:
The vertex va[0] should be projected to (0, 0).
The vertex va[1] should be projected to (*, 0), so somewhere on the new x axis.
3. / 4. Calculating the intersection area in 2D
This answer this answer mentions the algorithms necessary for the last step.
The Sutherland-Hodgman algorithm successively clips one triangles with each side of the other. The result of this will be either a triangle, a quadrilateral or an empty polygon.
Finally, the shoelace formula can be used to calculate the area of the resulting clipping polygon.
Bringing it all together
Assuming you implemented the two functions CalculateIntersecionPolygon and CalculatePolygonArea, the final intersection area could be calculated like this:
public static float CalculateIntersectionArea(Mesh triangleA, Mesh triangleB) {
var verticesA = triangleA.GetVertices();
var verticesB = triangleB.GetVertices();
if(!AreTrianglesCoplanar(verticesA, verticesB))
return 0f; // the triangles are not coplanar
ProjectTo2DPlane(verticesA, verticesB, out Vector2[] projectedA, out Vector2[] projectedB);
CalculateIntersecionPolygon(projectedA, projectedB, out List<Vector2> intersection);
if(intersection.Count == 0)
return 0f; // the triangles didn't overlap
return CalculatePolygonArea(intersection);
}

Smoothing noises with different amplitudes (Part 2)

Well, I'm continuing this question without answer (Smoothing random noises with different amplitudes) and I have another question.
I have opted to use the contour/shadow of a shape (Translating/transforming? list of points from its center with an offset/distance).
This contour/shadow is bigger than the current path. I used this repository (https://github.com/n-yoda/unity-vertex-effects) to recreate the shadow. And this works pretty well, except for one fact.
To know the height of all points (obtained by this shadow algorithm (Line 13 of ModifiedShadow.cs & Line 69 of CircleOutline.cs)) I get the distance of the current point to the center and I divide between the maximum distance to the center:
float dist = orig.Max(v => (v - Center).magnitude);
foreach Point in poly --> float d = 1f - (Center - p).magnitude / dist;
Where orig is the entire list of points obtained by the shadow algorithm.
D is the height of the shadow.
But the problem is obvious I get a perfect circle:
In red and black to see the contrast:
And this is not what I want:
As you can see this not a perfect gradient. Let's explain what's happening.
I use this library to generate noises: https://github.com/Auburns/FastNoise_CSharp
Note: If you want to know what I use to get noises with different amplitude: Smoothing random noises with different amplitudes (see first block of code), to see this in action, see this repo
Green background color represent noises with a mean height of -0.25 and an amplitude of 0.3
White background color represent noises with a mean height of 0 and an amplitude of 0.1
Red means 1 (total interpolation for noises corresponding to white pixels)
Black means 0 (total interpolation for noises corresponding to green pixels)
That's why we have this output:
Actually, I have tried comparing distances of each individual point to the center, but this output a weird and unexpected result.
Actually, I don't know what to try...
The problem is that the lerp percentage (e.g., from high/low or "red" to "black" in your visualization) is only a function of the point's distance from the center, which is divided by a constant (which happens to be the maximum distance of any point from the center). That's why it appears circular.
For instance, the centermost point on the left side of the polygon might be 300 pixels away from the center, while the centermost point on the right might be 5 pixels. Both need to be red, but basing it off of 0 distance from center = red won't have either be red, and basing it off the min distance from center = red will only have red on the right side.
The relevant minimum and maximum distances will change depending on where the point is
One alternative method is for each point: find the closest white pixel, and find the closest green pixel, (or, the closest shadow pixel that is adjacent to green/white, such as here). Then, choose your redness depending on how the distances compare between those two points and the current point.
Therefore, you could do this (pseudo-C#):
foreach pixel p in shadow_region {
// technically, closest shadow pixel which is adjacent to x Pixel:
float closestGreen_distance = +inf;
float closestWhite_distance = +inf;
// Possibly: find all shadow-adjacent pixels prior to the outer loop
// and cache them. Then, you only have to loop through those pixels.
foreach pixel p2 in shadow {
float p2Dist = (p-p2).magnitude;
if (p2 is adjacent to green) {
if (p2Dist < closestGreen_distance) {
closestGreen_distance = p2Dist;
}
}
if (p2 is adjacent to white) {
if (p2Dist < closestWhite_distance) {
closestWhite_distance = p2Dist;
}
}
}
float d = 1f - closestWhite_distance / (closestWhite_distance + closestGreen_distance)
}
Using the code you've posted in the comments, this might look like:
foreach (Point p in value)
{
float minOuterDistance = outerPoints.Min(p2 => (p - p2).magnitude);
float minInnerDistance = innerPoints.Min(p2 => (p - p2).magnitude);
float d = 1f - minInnerDistance / (minInnerDistance + minOuterDistance);
Color32? colorValue = func?.Invoke(p.x, p.y, d);
if (colorValue.HasValue)
target[F.P(p.x, p.y, width, height)] = colorValue.Value;
}
The above part was chosen for the solution. The below part, mentioned as another option, turned out to be unnecessary.
If you can't determine if a shadow pixel is adjacent to white/green, here's an alternative that only requires the calculation of the normals of each vertex in your pink (original) outline.
Create outer "yellow" vertices by going to each pink vertex and following its normal outward. Create inner "blue" vertices by going to each pink vertex and following its normal inward.
Then, when looping through each pixel in the shadow, loop through the yellow vertices to get your "closest to green" and through the blue to get "closest to white".
The problem is that since your shapes aren't fully convex, these projected blue and yellow outlines might be inside-out in some places, so you would need to deal with that somehow. I'm having trouble determining an exact method of dealing with that but here's what I have so far:
One step is to ignore any blues/yellows that have outward-normals that point towards the current shadow pixel.
However, if the current pixel is inside of a point where the yellow/blue shape is inside out, I'm not sure how to proceed. There might be something to ignoring blue/yellow vertexes that are closer to the closest pink vertex than they should be.
extremely rough pseudocode:
list yellow_vertex_list = new list
list blue_vertex_list = new list
foreach pink vertex p:
given float dist;
vertex yellowvertex = new vertex(p+normal*dist)
vertex bluevertex = new vertex(p-normal*dist)
yellow_vertex_list.add(yellowvertex)
blue_vertex_list.add(bluevertex)
create shadow
for each pixel p in shadow:
foreach vertex v in blue_vertex_list
if v.normal points towards v: break;
if v is the wrong side of inside-out region: break;
if v is closest so far:
closest_blue = v
closest_blue_dist = (v-p).magnitude
foreach vertex v in yellow_vertex_list
if v.normal points towards v break;
if v is the wrong side of inside-out region: break;
if v is closest so far:
closest_yellow = v
closest_yellow_dist = (v-p).magnitude
float d = 1f - closest_blue_dist / (closest_blue_dist + closest_yellow_dist)

Contrain camera to rectangle while tracking multiple objects

I'm making a 2D platformer that features a dynamic camera. The camera must track 4 players at once so that they're all on the screen. In addition the camera must not move beyond a predefined rectangle boundary. I've tried implementing it but I just can't seem to get the process of zooming the camera so that it's always close as possible to the four objects.
The general algorithm I have so far is
1. Define the viewing space by calculating a 2D axis aligned bounding box using the 4 object positions being tracked and use its center as a camera postion (or averaging)
2. Calculate an orthographic size by using the largest x OR y value using a vector from the camera's position to each object being tracked.
If the camera is beyond the camera's boundary calculate the excess amount and displace in the opposite direction.
This seems simple enough on paper but I can't seem to get a correct working implementation.
Why dont you just take the Average of the 4 players Position and use it as Camera Position, also check if the players are out of boundary and when they are, zoom out.
float x = 0;
float y = 0;
GameObject[] players = new GameObjects[5];
foreach(GameObject _ply in players)
{
x += _ply.transform.position.x;
y += _ply.transform.position.y;
}
x = x/players.Length;
y = y/players.Length;
foreach(GameObject _ply in players)
{
if(_ply.transform.position.x > (x + (Screen.Width / 2)))
//zoom out
if(_ply.transform.position.y > (y + (Screen.Height / 2)))
//zoom out
}
But you have to fix Zoomin.

C# code snippet calculating the surface and vertex normals

I need a C# code snippet calculating the surface and vertex normals. Kind of surface is triangulated 3D closed mesh. The required code snippet must be able to use a vertex set and triangleindices. These are ready to use at the moment. The surface of 3D mesh object is not smooth, so it needs to be smoothed.
Could you help me.
It sounds like you're trying to display your 3D mesh and apply a smooth shading appearance by interpolating surface normals, such as in Phong shading, and you need to calculate the normals first. This is different from smoothing the surface of the mesh itself, since that implies altering the positions of its vertices.
Surface normals can be calculated by getting the vector cross product of two edges of a triangle.
As far as code, I'm unaware of any C# examples, but here is one in C++ that should be easy to port. It is taken from the popular NeHe tutorials for OpenGL:
void calcNormal(float v[3][3], float out[3]) // Calculates Normal For A Quad Using 3 Points
{
float v1[3],v2[3]; // Vector 1 (x,y,z) & Vector 2 (x,y,z)
static const int x = 0; // Define X Coord
static const int y = 1; // Define Y Coord
static const int z = 2; // Define Z Coord
// Finds The Vector Between 2 Points By Subtracting
// The x,y,z Coordinates From One Point To Another.
// Calculate The Vector From Point 1 To Point 0
v1[x] = v[0][x] - v[1][x]; // Vector 1.x=Vertex[0].x-Vertex[1].x
v1[y] = v[0][y] - v[1][y]; // Vector 1.y=Vertex[0].y-Vertex[1].y
v1[z] = v[0][z] - v[1][z]; // Vector 1.z=Vertex[0].y-Vertex[1].z
// Calculate The Vector From Point 2 To Point 1
v2[x] = v[1][x] - v[2][x]; // Vector 2.x=Vertex[0].x-Vertex[1].x
v2[y] = v[1][y] - v[2][y]; // Vector 2.y=Vertex[0].y-Vertex[1].y
v2[z] = v[1][z] - v[2][z]; // Vector 2.z=Vertex[0].z-Vertex[1].z
// Compute The Cross Product To Give Us A Surface Normal
out[x] = v1[y]*v2[z] - v1[z]*v2[y]; // Cross Product For Y - Z
out[y] = v1[z]*v2[x] - v1[x]*v2[z]; // Cross Product For X - Z
out[z] = v1[x]*v2[y] - v1[y]*v2[x]; // Cross Product For X - Y
ReduceToUnit(out); // Normalize The Vectors
}
The normalization function ReduceToUnit() can be found there as well.
Note that this calculates a surface normal for a single triangle. Since you give no information about how your vertices and indices are stored, I will leave it up to you to derive the set of triangles you need to pass to this function.
EDIT: As an additional note, I think the "winding direction" of your triangles is significant. Winding in the wrong direction will cause the normal to point in the opposite direction as well.

Categories