We Keep Coding

Unable to get Yolov4-Tiny working with Windows.AI.MachineLearning APIs

I am implementing Yolov4-Tiny (onnx model found here) in Unity with the Windows ML APIs. I can load the model, and begin a session with no issue. I am using a VideoFrame (sized to 416x416) as the input and can access the two output Tensors. The problems arise when I begin to parse the output Tensors. With a confidence threshold of .5, I get between 700 and 1000 detections each frame, way way more than I expect. Also, the bboxes appear to be very small. The NMS and IOU functions below are nearly verbatim from here so I am not using anchors for the bounding boxes. I believe my issues is in the NMS and IOU functions, but I cannot locate the problem. My gut tells me I am manipulating the output Tensors incorrectly. Any ideas?
private List<DetectionResult> ParseResult(float[] boxes, float[] classes)
{
int c_values = 80;
int c_boxes = boxes.Length / 4;
int c_classNames = classes.Length / c_values;
float confidence_threshold = 0.5f;
List<DetectionResult> detections = new List<DetectionResult>();
for (int i_box = 0; i_box < c_classNames; i_box++)
{
float max_prob = 0.0f;
int label_index = -1;
for (int j_confidence = 0; j_confidence < c_values; j_confidence++)
{
int index = i_box * c_values + j_confidence;
if (Sigmoid(classes[index]) > max_prob)
{
max_prob = Sigmoid(classes[index]) ;
label_index = j_confidence;
}
}
if (max_prob > confidence_threshold)
{
//Debug.Log(_labels[label_index]);
List<float> bbox = new List<float>();
bbox.Add(boxes[(i_box * 4) + 0] * 416);
bbox.Add(boxes[(i_box * 4) + 1] * 416);
bbox.Add(boxes[(i_box * 4) + 2] * 416);
bbox.Add(boxes[(i_box * 4) + 3] * 416);
detections.Add(new DetectionResult()
{
label = _labels[label_index],
bbox = bbox,
prob = max_prob
});
}
}
private List<DetectionResult> NMS(IReadOnlyList<DetectionResult> detections,
float IOU_threshold = 0.45f,
float score_threshold = 0.3f)
{
List<DetectionResult> final_detections = new List<DetectionResult>();
for (int i = 0; i < detections.Count; i++)
{
int j = 0;
for (j = 0; j < final_detections.Count; j++)
{
if (ComputeIOU(final_detections[j], detections[i]) > IOU_threshold)
{
break;
}
}
if (j == final_detections.Count)
{
final_detections.Add(detections[i]);
}
}
return final_detections;
}
private float ComputeIOU(DetectionResult DRa, DetectionResult DRb)
{
float ay1 = DRa.bbox[0];
float ax1 = DRa.bbox[1];
float ay2 = DRa.bbox[2];
float ax2 = DRa.bbox[3];
float by1 = DRb.bbox[0];
float bx1 = DRb.bbox[1];
float by2 = DRb.bbox[2];
float bx2 = DRb.bbox[3];
// determine the coordinates of the intersection rectangle
float x_left = Math.Max(ax1, bx1);
float y_top = Math.Max(ay1, by1);
float x_right = Math.Min(ax2, bx2);
float y_bottom = Math.Min(ay2, by2);
if (x_right < x_left || y_bottom < y_top)
return 0;
float intersection_area = (x_right - x_left) * (y_bottom - y_top);
float bb1_area = (ax2 - ax1) * (ay2 - ay1);
float bb2_area = (bx2 - bx1) * (by2 - by1);
float iou = intersection_area / (bb1_area + bb2_area - intersection_area);
return iou;
}

Issues with Steering Behavior Seperation

The problem
I am trying to procedurally generate dungeon rooms with random X, Y sizes inside of a radius (r). However, even after I validate that the starting grid (origin of the "room") is not in the same position as other origins after running the separation function there are rooms still building inside of each other.
Solutions I have tried
I tried using math to calculate an optimal radius that will be able to fit the average of all the room sizes * amount of rooms. However, the separation should hypothetically work with any radius (though I want to keep them relatively close in order to keep hallways short).
Code
All my code is based on one tile. This means that all calculations are using one tile, and will remain one tile until the very end, then I scale them up.
private void GenerateRooms(int amount)
{
// init sizes
Vector2[] room_sizes = new Vector2[amount];
for (int i = 0; i < amount; i++)
{
room_sizes[i] = new Vector2(Random.Range(minimum_room_height, maximum_room_height), Random.Range(minimum_room_width, maximum_room_width));
}
float biggest_room = calculations.CalculateBiggest(room_sizes);
Vector2[] room_points = new Vector2[amount];
Vector2[] used_points = new Vector2[amount];
float radius = calculations.CalculateAverage(room_sizes) * amount;
for (int i = 0; i < amount; i++)
{
do {
Vector2 test_point = new Vector2(Random.Range(-radius, radius), Random.Range(-radius, radius));
foreach (Vector2 point in used_points) {
if (test_point == point) {
continue;
} else {
room_points[i] = test_point;
used_points[i] = test_point;
break;
}
}
} while (Vector2.Distance(Vector2.zero, room_points[i]) < radius);
}
for (int i = 0; i < amount; i++)
{
//Vector2 origin = room_points[i];
Vector3 position = calculations.computeSeperate(room_points, room_points[i], biggest_room);
//position = new Vector3(position.x + origin.x, position.y + origin.y, 0);
Vector3Int location = tile_map.WorldToCell(position);
tile_map.SetTile(location, tile);
calculations.scaleUpRooms(position, room_sizes[i].x, room_sizes[i].y, tile_map, tile);
}
}
The above is code for calling all the functions and validating the points. Here are the important functions (calculation functions):
public Vector2 computeSeperate(Vector2[] point_array, Vector2 target_point, float minimum_distance)
{
int neighbor_count = 0;
for (int i = 0; i < point_array.Length; i++)
{
if (point_array[i] != target_point)
{
if (Vector2.Distance(target_point, point_array[i]) < minimum_distance * 2)
{
target_point.x += point_array[i].x - target_point.x;
target_point.y += point_array[i].y - target_point.y;
neighbor_count++;
}
}
}
if (neighbor_count == 0)
{
return target_point;
} else
{
target_point.x /= neighbor_count;
target_point.y /= neighbor_count;
target_point.x *= -1;
target_point.y *= -1;
target_point.Normalize();
return target_point;
}
}
public void scaleUpRooms(Vector2 base_point, float scale_x, float scale_y, Tilemap tile_map, Tile tile) // ex: 5x5
{
List<Vector2> Calculate(Vector2 size)
{
List<Vector2> results = new List<Vector2>();
for (int i = 0; i < size.y; i++)
for (int o = 0; o < size.x; o++)
results.Add(new Vector2(o, i) + (new Vector2(size.x % 2 != 0 ? .5f : 1, size.y % 2 != 0 ? .5f : 1) - (size / 2)));
string st = "";
for (int i = 0; i < results.Count; i++)
st += "\n" + results[i].ToString();
return results;
}
Vector2 desired_scale = new Vector2(scale_x, scale_y);
List<Vector2> Offsets = Calculate(desired_scale);
for (int i = 0; i < Offsets.Count; i++)
{
Vector3 position = base_point + Offsets[i];
Vector3Int location = tile_map.WorldToCell(position);
tile_map.SetTile(location, tile);
}
}

How can I reduce the gaps between objects?

At the top :
[Range(10, 100)]
public int gap = 10;
private int oldGapValue = 0;
And in Update :
private void Update()
{
if(gap > oldGapValue)
{
for(int i = 0; i < numberOfUnits; i++)
{
units[i].transform.position = (i + 1f) * new Vector3(gap, 0, 0);
}
oldGapValue = gap;
}
else
{
}
This working for adding gaps. but in the else part I want to reduce the gaps between the objects.
I tried this but this is not working good, now it's working fine when reducing the gaps but now when adding gaps it's not working good it's adding gaps but also double the units objects until I stop adding gaps it's all stuttering.
if(gap > oldGapValue)
{
for(int i = 0; i < numberOfUnits; i++)
{
units[i].transform.position = (i + 1f) * new Vector3(gap, 0, 0);
}
oldGapValue = gap;
}
else
{
for (int i = 0; i < numberOfUnits; i++)
{
units[i].transform.position = (i + 1f) * new Vector3(-gap, 0, 0);
}
}

It looks like you're trying to space objects, by the value of gap, every single frame because of your logic.
Might I suggest a different approach and only change the object spacing when the value of gap actually changes. As an example, try this code:
[SerializeField]
[Range ( 10f, 100f )]
private float _gap = 10f;
private float _oldGap = 10f;
public float gap
{
get => _gap;
set
{
_oldGap = _gap;
Debug.Log ( $"New Gap Value {_gap}" );
for ( int i = 0; i < units.Length; i++ )
{
units [ i ].transform.position = new Vector3 ( (gap * i) + 1f, 0, 0 );
}
}
}
private void OnValidate ( )
{
if ( _gap != _oldGap )
gap = _gap;
}

When calculating the space for each direction when the grid size is multiply by 1.5 the directions are wrong why?

If the grid is 10x10 or 23x7 it's working fine but when the grid have 1.5 spaces between the cubes the directions sometimes are wrong.
This is the grid script:
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
public class GridGenerator : MonoBehaviour
{
public GameObject gridBlock;
public int gridWidth = 10;
public int gridHeight = 10;
public GameObject[] allBlocks;
private GameObject[] wallsParents = new GameObject[4];
void Start()
{
wallsParents[0] = GameObject.Find("Top Wall");
wallsParents[1] = GameObject.Find("Left Wall");
wallsParents[2] = GameObject.Find("Right Wall");
wallsParents[3] = GameObject.Find("Bottom Wall");
GenerateGrid();
allBlocks = GameObject.FindGameObjectsWithTag("Blocks");
var findpath = GetComponent<PathFinder>();
findpath.FindPath();
}
public void AutoGenerateGrid()
{
allBlocks = GameObject.FindGameObjectsWithTag("Blocks");
for (int i = 0; i < allBlocks.Length; i++)
{
DestroyImmediate(allBlocks[i]);
}
var end = GameObject.FindGameObjectWithTag("End");
DestroyImmediate(end);
GenerateGrid();
allBlocks = GameObject.FindGameObjectsWithTag("Blocks");
var findpath = GetComponent<PathFinder>();
findpath.FindPath();
}
public void GenerateGrid()
{
for (int x = 0; x < gridWidth; x++)
{
for (int z = 0; z < gridHeight; z++)
{
GameObject block = Instantiate(gridBlock, Vector3.zero, gridBlock.transform.rotation) as GameObject;
block.transform.parent = transform;
block.transform.name = "Block";
block.transform.tag = "Blocks";
block.transform.localPosition = new Vector3(x * 1.5f, 0, z * 1.5f);
block.GetComponent<Renderer>().material.color = new Color(241, 255, 0, 255);
if (x == 0)//TOP
{
block.transform.parent = wallsParents[0].transform;
block.transform.name = "TopWall";
block.transform.tag = "Blocks";
}
else if (z == 0)//LEFT
{
block.transform.parent = wallsParents[1].transform;
block.transform.name = "LeftWall";
block.transform.tag = "Blocks";
}
else if (z == gridHeight - 1)//RIGHT
{
block.transform.parent = wallsParents[2].transform;
block.transform.name = "RightWall";
block.transform.tag = "Blocks";
}
else if (x == gridWidth - 1)//BOTTOM
{
block.transform.parent = wallsParents[3].transform;
block.transform.name = "BottomWall";
block.transform.tag = "Blocks";
}
}
}
}
}
On this line i'm adding the spaces between the cubes:
block.transform.localPosition = new Vector3(x * 1.5f, 0, z * 1.5f);
Then in another script i'm trying to find what directions next are possible to move to.
private void Directions()
{
GridGenerator gridgenerator = GetComponent<GridGenerator>();
Vector3 playerPosition;
playerPosition = player.localPosition;
if (playerPosition.x > 0)
{
// can go left
possibleDirections[0] = "Can go left";
}
else
{
possibleDirections[0] = "Can't go left";
}
if (playerPosition.x + 1 < gridgenerator.gridWidth * 1.5f)
{
// can go right
possibleDirections[1] = "Can go right";
}
else
{
possibleDirections[1] = "Can't go right";
}
if (playerPosition.z > 0)
{
// can go backward
possibleDirections[2] = "Can go backward";
}
else
{
possibleDirections[2] = "Can't go backward";
}
if (playerPosition.z + 1 < gridgenerator.gridHeight * 1.5f)
{
// can go backward
possibleDirections[3] = "Can go forward";
}
else
{
possibleDirections[3] = "Can't go forward";
}
}
possibleDirections is array string type
When the grid size is 10x10 without spaces between cubes this two lines:
if (playerPosition.x + 1 < gridgenerator.gridWidth * 1.5f)
if (playerPosition.z + 1 < gridgenerator.gridHeight * 1.5f)
Was:
if (playerPosition.x + 1 < gridgenerator.gridWidth)
if (playerPosition.z + 1 < gridgenerator.gridHeight)
But when i added the spaces between the cubes i tried to add to the gridgenerator.gridWidth and gridgenerator.gridHeight the * 1.5
But it didn't work so i tried also:
if (playerPosition.x + 1 < gridgenerator.gridWidth * (1 + 1.5))
if (playerPosition.z + 1 < gridgenerator.gridHeight * (1 + 1.5))
1 is the cube width and 1.5 is the space. But this is not working good either.
In the screenshot the player is in the top left corner facing up(forward)
He can't move forward but in the inspector it says "Can go forward" And should be "Can't go forward"
It only happens when there are spaces between the cubes.

This line is wrong:
if (playerPosition.x + 1 < gridgenerator.gridWidth * 1.5f)
Your gridWidth variable stores the number of cubes, not their collective spacing. You have 10 cubes representing move spaces, determining the out-of-bounds this value should remain constant (it's still only 10 cubes, even if they're spaced with a half-block worth of space between them).
You need to convert from the player's scene location (transform.position.x) to a board space location (likely dividing by the same multiplier used to space the cubes out).
Alternatively, the "this makes my soul cry" solution of doing this:
if (playerPosition.x + 1.5f < gridgenerator.gridWidth * 1.5f)
Because the next cube is 1.5 scene units away, not 1. And this makes my soul cry because it makes your code full of hard-coded 1.5f multipliers and offsets rather than keeping such things to a single, fixed, constant value stored Elsewhere and used sparingly.
Related:
possibleDirections[0] = "Can go left";
Why are you using stringly typed things? There are values called booleans for a reason...

Deskew scanned images

I am working in OMR project and we are using C#. When we come to scan the answer sheets, the images are skewed. How can we deskew them?

VB.Net Code for this is available here, however since you asked for C# here is a C# translation of their Deskew class (note: Binarize (strictly not necessary, but works much better) and Rotate are exercises left to the user).
public class Deskew
{
// Representation of a line in the image.
private class HougLine
{
// Count of points in the line.
public int Count;
// Index in Matrix.
public int Index;
// The line is represented as all x,y that solve y*cos(alpha)-x*sin(alpha)=d
public double Alpha;
}
// The Bitmap
Bitmap _internalBmp;
// The range of angles to search for lines
const double ALPHA_START = -20;
const double ALPHA_STEP = 0.2;
const int STEPS = 40 * 5;
const double STEP = 1;
// Precalculation of sin and cos.
double[] _sinA;
double[] _cosA;
// Range of d
double _min;
int _count;
// Count of points that fit in a line.
int[] _hMatrix;
public Bitmap DeskewImage(Bitmap image, int type, int binarizeThreshold)
{
Size oldSize = image.Size;
_internalBmp = BitmapFunctions.Resize(image, new Size(1000, 1000), true, image.PixelFormat);
Binarize(_internalBmp, binarizeThreshold);
return Rotate(image, GetSkewAngle());
}
// Calculate the skew angle of the image cBmp.
private double GetSkewAngle()
{
// Hough Transformation
Calc();
// Top 20 of the detected lines in the image.
HougLine[] hl = GetTop(20);
// Average angle of the lines
double sum = 0;
int count = 0;
for (int i = 0; i <= 19; i++)
{
sum += hl[i].Alpha;
count += 1;
}
return sum / count;
}
// Calculate the Count lines in the image with most points.
private HougLine[] GetTop(int count)
{
HougLine[] hl = new HougLine[count];
for (int i = 0; i <= count - 1; i++)
{
hl[i] = new HougLine();
}
for (int i = 0; i <= _hMatrix.Length - 1; i++)
{
if (_hMatrix[i] > hl[count - 1].Count)
{
hl[count - 1].Count = _hMatrix[i];
hl[count - 1].Index = i;
int j = count - 1;
while (j > 0 && hl[j].Count > hl[j - 1].Count)
{
HougLine tmp = hl[j];
hl[j] = hl[j - 1];
hl[j - 1] = tmp;
j -= 1;
}
}
}
for (int i = 0; i <= count - 1; i++)
{
int dIndex = hl[i].Index / STEPS;
int alphaIndex = hl[i].Index - dIndex * STEPS;
hl[i].Alpha = GetAlpha(alphaIndex);
//hl[i].D = dIndex + _min;
}
return hl;
}
// Hough Transforamtion:
private void Calc()
{
int hMin = _internalBmp.Height / 4;
int hMax = _internalBmp.Height * 3 / 4;
Init();
for (int y = hMin; y <= hMax; y++)
{
for (int x = 1; x <= _internalBmp.Width - 2; x++)
{
// Only lower edges are considered.
if (IsBlack(x, y))
{
if (!IsBlack(x, y + 1))
{
Calc(x, y);
}
}
}
}
}
// Calculate all lines through the point (x,y).
private void Calc(int x, int y)
{
int alpha;
for (alpha = 0; alpha <= STEPS - 1; alpha++)
{
double d = y * _cosA[alpha] - x * _sinA[alpha];
int calculatedIndex = (int)CalcDIndex(d);
int index = calculatedIndex * STEPS + alpha;
try
{
_hMatrix[index] += 1;
}
catch (Exception ex)
{
System.Diagnostics.Debug.WriteLine(ex.ToString());
}
}
}
private double CalcDIndex(double d)
{
return Convert.ToInt32(d - _min);
}
private bool IsBlack(int x, int y)
{
Color c = _internalBmp.GetPixel(x, y);
double luminance = (c.R * 0.299) + (c.G * 0.587) + (c.B * 0.114);
return luminance < 140;
}
private void Init()
{
// Precalculation of sin and cos.
_cosA = new double[STEPS];
_sinA = new double[STEPS];
for (int i = 0; i < STEPS; i++)
{
double angle = GetAlpha(i) * Math.PI / 180.0;
_sinA[i] = Math.Sin(angle);
_cosA[i] = Math.Cos(angle);
}
// Range of d:
_min = -_internalBmp.Width;
_count = (int)(2 * (_internalBmp.Width + _internalBmp.Height) / STEP);
_hMatrix = new int[_count * STEPS];
}
private static double GetAlpha(int index)
{
return ALPHA_START + index * ALPHA_STEP;
}
}

Scanned document are always skewed for an average [-10;+10] degrees angle.
It's easy to deskew them using the Hough transform, like Lou Franco said. This transform detects lines on your image for several angles. You just have to select the corresponding one to your document horizontal lines, then rotate it.
try to isolate the pixel corresponding to your document horizontal lines (for instance, black pixels that have a white pixel at their bottom).
Run Hough transform. Do not forget to use 'unsafe' mode in C# to fasten the process of your whole image by using a pointor.
Rotate your document in the opposite angle found.
Works like a charm on binary documents (easily extendable to grey level ones)

Disclaimer: I work at Atalasoft, DotImage Document Imaging can do this with a couple of lines of code.
Deskew is a term of art that describes what you are trying to do. As Ben Voigt said, it's technically rotation, not skew -- however, you will find algorithms under automatic deskew if you search.
The normal way to do this is to do a hough transform to look for the prevalent lines in the image. With normal documents, many of them will be orthogonal to the sides of the paper.

Are you sure it's "skew" rather than "rotation" (rotation preserves angles, skew doesn't).
Use some sort of registration mark (in at least two places) which you can recognize even when rotated.
Find the coordinates of these marks and calculate the rotation angle.
Apply a rotation transformation matrix to the image.