We Keep Coding

How to get Tensorflow PoseNet working with Xamarin forms/Xam.Android

I've tried with java/Kotlin android and then using it as a .jar but cant get it to work. I have this code in Java, and i need to try to replicate it using C# for Xamarin/Xamarin.Android
JAVA METHOD
fun estimateSinglePose(bitmap: Bitmap): Person {
var t1: Long = SystemClock.elapsedRealtimeNanos()
val inputArray = arrayOf(initInputArray(bitmap))
var t2: Long = SystemClock.elapsedRealtimeNanos()
Log.i("posenet", String.format("Scaling to [-1,1] took %.2f ms", 1.0f *
(t2 - t1) / 1_000_000))
val outputMap = initOutputMap(interpreter!!)
t1 = SystemClock.elapsedRealtimeNanos()
interpreter!!.runForMultipleInputsOutputs(inputArray, outputMap)
t2 = SystemClock.elapsedRealtimeNanos()
Log.i("posenet", String.format("Interpreter took %.2f ms", 1.0f * (t2 - t1) / 1_000_000))
val heatmaps = outputMap[0] as Array<Array<Array<FloatArray>>>
val offsets = outputMap[1] as Array<Array<Array<FloatArray>>>
val height = heatmaps[0].size
val width = heatmaps[0][0].size
val numKeypoints = heatmaps[0][0][0].size
// Finds the (row, col) locations of where the keypoints are most likely to be.
val keypointPositions = Array(numKeypoints) { Pair(0, 0) }
for (keypoint in 0 until numKeypoints) {
var maxVal = heatmaps[0][0][0][keypoint]
var maxRow = 0
var maxCol = 0
for (row in 0 until height) {
for (col in 0 until width) {
heatmaps[0][row][col][keypoint] = sigmoid(heatmaps[0][row][col][keypoint])
if (heatmaps[0][row][col][keypoint] > maxVal) {
maxVal = heatmaps[0][row][col][keypoint]
maxRow = row
maxCol = col
}
}
}
keypointPositions[keypoint] = Pair(maxRow, maxCol)
}
// Calculating the x and y coordinates of the keypoints with offset adjustment.
val xCoords = IntArray(numKeypoints)
val yCoords = IntArray(numKeypoints)
val confidenceScores = FloatArray(numKeypoints)
keypointPositions.forEachIndexed { idx, position ->
val positionY = keypointPositions[idx].first
val positionX = keypointPositions[idx].second
yCoords[idx] = (
position.first / (height - 1).toFloat() * bitmap.height +
offsets[0][positionY][positionX][idx]
).toInt()
xCoords[idx] = (
position.second / (width - 1).toFloat() * bitmap.width +
offsets[0][positionY]
[positionX][idx + numKeypoints]
).toInt()
confidenceScores[idx] = heatmaps[0][positionY][positionX][idx]
}
val person = Person()
val keypointList = Array(numKeypoints) { KeyPoint() }
var totalScore = 0.0f
enumValues<BodyPart>().forEachIndexed { idx, it ->
keypointList[idx].bodyPart = it
keypointList[idx].position.x = xCoords[idx]
keypointList[idx].position.y = yCoords[idx]
keypointList[idx].score = confidenceScores[idx]
totalScore += confidenceScores[idx]
}
person.keyPoints = keypointList.toList()
person.score = totalScore / numKeypoints
return person
}
I have got the PoseNet model running in a C# method but this method is written for object detection and not returning Keypoints, i don't know how to adapt it to get the keypoints
C# Method
public void Recognize(int[] colors)
{
if (!initialized)
{
throw new Exception("Initialize TensorflowLiteService first");
}
MessagingCenter.Instance.Send(this, nameof(AR.InputTensorMessage), new InputTensorMessage()
{
Colors = colors,
});
using (var op = new BuildinOpResolver())
{
using (var interpreter = new Interpreter(model, op))
{
InvokeInterpreter(colors, interpreter);
}
}
}
private void InvokeInterpreter(int[] colors, Interpreter interpreter)
{
if (useNumThreads)
{
interpreter.SetNumThreads(Environment.ProcessorCount);
}
var allocateTensorStatus = interpreter.AllocateTensors();
if (allocateTensorStatus == Status.Error)
{
throw new Exception("Failed to allocate tensor");
}
var input = interpreter.GetInput();
using (var inputTensor = interpreter.GetTensor(input[0]))
{
CopyColorsToTensor(inputTensor.DataPointer, colors);
var watchInvoke = Stopwatch.StartNew();
interpreter.Invoke();
watchInvoke.Stop();
Console.WriteLine($"InterpreterInvoke: {watchInvoke.ElapsedMilliseconds}ms");
}
var output = interpreter.GetOutput();
var outputIndex = output[0];
var outputTensors = new Tensor[output.Length];
for (var i = 0; i < output.Length; i++)
{
outputTensors[i] = interpreter.GetTensor(outputIndex + i);
}
var detection_boxes_out = outputTensors[0].GetData() as float[];
var detection_classes_out = outputTensors[1].GetData() as float[];
var detection_scores_out = outputTensors[2].GetData() as float[];
var num_detections_out = outputTensors[3].GetData() as float[];
var numDetections = num_detections_out[0];
LogDetectionResults(detection_classes_out, detection_scores_out, detection_boxes_out, (int)numDetections);
}

Curve fitting to a 3D polynomial with variable powers

I have 3 data sets. 2 for polynomial itself (let's call them x and y) and 1 for the function value (it's gonna be z).
Polynomial looks something like this (assuming the power of both dimensions is 3):
z = a00 + a01*x + a02*x^2 + a03*x^3 + a10*y + a11*y*x + a12*y*x^2 ... etc
I need to be able to set the power of each dimension when preparing for approximation of values of "a".
I don't really get how CurveFitting functions work with Math.NET Numerics, but i've tried Fit.LinearMultiDim and MultipleRegression.QR. I'm having trouble with initializing the Func delegate
var zs = new double[]
{
//values here
};
var x = new double[]
{
//values here
};
var y = new double[]
{
//values here. But the amounts are equal
};
var design = Matrix<double>.Build.DenseOfRowArrays(Generate.Map2(x, y,(t, w) =>
{
var list = new List<double>(); //Can i get this working?
for (int i = 0; i <= 3; i++)
{
for (int j = 0; j <= 3; j++)
{
list.Add(Math.Pow(t, j)*Math.Pow(w, i));
}
}
return list.ToArray();
}));
double[] p = MultipleRegression.QR(design, Vector<double>.Build.Dense(zs)).ToArray();
So ideally i need to be able to compose the function with some sort of loop that accounts for the max power of both variables.
UPD: The function is always above zero on any axis

I think i got it working.
Here's the code:
List<Func<double[], double>> ps = new List<Func<double[],double>>();
for (int i = 0; i <= polynomialOrderFirstVal; i++)
{
for (int j = 0; j <= polynomialOrderSecVal; j++)
{
var orderFirst = j;
var orderSecond = i;
ps.Add(d => Math.Pow(d[0], orderFirst) * Math.Pow(d[1],orderSecond));
}
}
var psArr = ps.ToArray();
double[] coefs = Fit.LinearMultiDim(x, y, psArr);

Extracting pdf images in a correct order iTextSharp

I'm trying to extract images from a PDF File, but I really need to have it at the correct order to get the correct image.
static void Main(string[] args)
{
string filename = "D:\\910723575_marca_coletiva.pdf";
PdfReader pdfReader = new PdfReader(filename);
var imagemList = ExtraiImagens(pdfReader);
// converter byte[] para um bmp
List<Bitmap> bmpSrcList = new List<Bitmap>();
IList<byte[]> imagensProcessadas = new List<byte[]>();
foreach (var imagem in imagemList)
{
System.Drawing.ImageConverter converter = new System.Drawing.ImageConverter();
try
{
Image img = (Image)converter.ConvertFrom(imagem);
ConsoleWriteImage(img);
imagensProcessadas.Add(imagem);
}
catch (Exception)
{
continue;
}
}
System.Console.ReadLine();
}
public static void ConsoleWriteImage(Image img)
{
int sMax = 39;
decimal percent = Math.Min(decimal.Divide(sMax, img.Width), decimal.Divide(sMax, img.Height));
Size resSize = new Size((int)(img.Width * percent), (int)(img.Height * percent));
Func<System.Drawing.Color, int> ToConsoleColor = c =>
{
int index = (c.R > 128 | c.G > 128 | c.B > 128) ? 8 : 0;
index |= (c.R > 64) ? 4 : 0;
index |= (c.G > 64) ? 2 : 0;
index |= (c.B > 64) ? 1 : 0;
return index;
};
Bitmap bmpMin = new Bitmap(img, resSize.Width, resSize.Height);
Bitmap bmpMax = new Bitmap(img, resSize.Width * 2, resSize.Height * 2);
for (int i = 0; i < resSize.Height; i++)
{
for (int j = 0; j < resSize.Width; j++)
{
Console.ForegroundColor = (ConsoleColor)ToConsoleColor(bmpMin.GetPixel(j, i));
Console.Write("██");
}
Console.BackgroundColor = ConsoleColor.Black;
Console.Write(" ");
for (int j = 0; j < resSize.Width; j++)
{
Console.ForegroundColor = (ConsoleColor)ToConsoleColor(bmpMax.GetPixel(j * 2, i * 2));
Console.BackgroundColor = (ConsoleColor)ToConsoleColor(bmpMax.GetPixel(j * 2, i * 2 + 1));
Console.Write("▀");
Console.ForegroundColor = (ConsoleColor)ToConsoleColor(bmpMax.GetPixel(j * 2 + 1, i * 2));
Console.BackgroundColor = (ConsoleColor)ToConsoleColor(bmpMax.GetPixel(j * 2 + 1, i * 2 + 1));
Console.Write("▀");
}
System.Console.WriteLine();
}
}
public static IList<byte[]> ExtraiImagens(PdfReader pdfReader)
{
var data = new byte[] { };
IList<byte[]> imagensList = new List<byte[]>();
for (int numPag = 1; numPag <= 3; numPag++)
//for (int numPag = 1; numPag <= pdfReader.NumberOfPages; numPag++)
{
var pg = pdfReader.GetPageN(numPag);
var res = PdfReader.GetPdfObject(pg.Get(PdfName.RESOURCES)) as PdfDictionary;
var xobj = PdfReader.GetPdfObject(res.Get(PdfName.XOBJECT)) as PdfDictionary;
if (xobj == null) continue;
var keys = xobj.Keys;
if (keys == null) continue;
PdfObject obj = null;
PdfDictionary tg = null;
for (int key = 0; key < keys.Count; key++)
{
obj = xobj.Get(keys.ElementAt(key));
if (!obj.IsIndirect()) continue;
tg = PdfReader.GetPdfObject(obj) as PdfDictionary;
obj = xobj.Get(keys.ElementAt(key));
if (!obj.IsIndirect()) continue;
tg = PdfReader.GetPdfObject(obj) as PdfDictionary;
var type = PdfReader.GetPdfObject(tg.Get(PdfName.SUBTYPE)) as PdfName;
if (!PdfName.IMAGE.Equals(type)) continue;
int XrefIndex = (obj as PRIndirectReference).Number;
var pdfStream = pdfReader.GetPdfObject(XrefIndex) as PRStream;
data = PdfReader.GetStreamBytesRaw(pdfStream);
imagensList.Add(PdfReader.GetStreamBytesRaw(pdfStream));
}
}
return imagensList;
}
}
The method ConsoleWriteImage is only to print the image at the console and I used it to study the behavior of the order that iTextSharp was retrieving it for me , based on my code.
Any help ?

Unfortunately the OP has not explained what the correct order is - this is not self-explanatory because there might be certain aspects of a PDF which are not obvious for a program, merely for a human reader viewing the rendered PDF.
At least, though, it is likely that the OP wants to get his images on a page-by-page basis. This obviously is not what his current code provides: His code scans the whole base of objects inside the PDF for image objects, so he will get image objects, but the order may be completely random; in particular he may even get images contained in the PDF but not used on any of its pages...
To retrieve images on a page-by-page order (and only images actually used), one should use the parser framework, e.g.
PdfReader reader = new PdfReader(pdf);
PdfReaderContentParser parser = new PdfReaderContentParser(reader);
MyImageRenderListener listener = new MyImageRenderListener();
for (int i = 1; i <= reader.NumberOfPages; i++) {
parser.ProcessContent(i, listener);
}
// Process images in the List listener.MyImages
// with names in listener.ImageNames
(Excerpt from the ExtractImages.cs iTextSharp example)
where MyImageRenderListener is defined to collect images:
public class MyImageRenderListener : IRenderListener {
/** the byte array of the extracted images */
private List<byte[]> _myImages;
public List<byte[]> MyImages {
get { return _myImages; }
}
/** the file names of the extracted images */
private List<string> _imageNames;
public List<string> ImageNames {
get { return _imageNames; }
}
public MyImageRenderListener() {
_myImages = new List<byte[]>();
_imageNames = new List<string>();
}
[...]
public void RenderImage(ImageRenderInfo renderInfo) {
try {
PdfImageObject image = renderInfo.GetImage();
if (image == null || image.GetImageBytesType() == PdfImageObject.ImageBytesType.JBIG2)
return;
_imageNames.Add(string.Format("Image{0}.{1}", renderInfo.GetRef().Number, image.GetFileType() ) );
_myImages.Add(image.GetImageAsBytes());
}
catch
{
}
}
[...]
}
(Excerpt from MyImageRenderListener.cs iTextSharp example)
The ImageRenderInfo renderInfo furthermore also contains information on location and orientation of the image on the page in question which might help to deduce the correct order the OP is after.

c# feed a method with x and y values from a chart (polynomial calculation)

I have got this method to get a polynomial with my desired degree:
public static double[] Polyfit(double[] x, double[] y, int degree)
{
// Vandermonde matrix
var v = new DenseMatrix(x.Length, degree + 1);
for (int i = 0; i < v.RowCount; i++)
for (int j = 0; j <= degree; j++) v[i, j] = Math.Pow(x[i], j);
var yv = new DenseVector(y).ToColumnMatrix();
QR qr = v.QR();
// Math.Net doesn't have an "economy" QR, so:
// cut R short to square upper triangle, then recompute Q
var r = qr.R.SubMatrix(0, degree + 1, 0, degree + 1);
var q = v.Multiply(r.Inverse());
var p = r.Inverse().Multiply(q.TransposeThisAndMultiply(yv));
Console.WriteLine(p.Column(0).ToString());
return p.Column(0).ToArray();
}
How can I feed the method above with values from my chart (x and y)?
chart.Series[0].Points.... ?

I think you need this:
chart1.Series[0].YValueMembers
chart1.Series[0].XValueMember

The Points property is a getter, so you cannot set a new instance of DataPointCollection to it. You should however be able to access methods on the current DataPointCollection.
You could try something along the lines of:
chart.Series[0].Points.AddXY(double, double)
You would then iterate the array(s) and set the points manually.
MSDN DataPointCollection for more information.

A working solution is:
////generate polynomial of degree 4 fiting to the points
double[] arrayX = new double[chart.Series[0].Points.Count()];
double[] arrayY = new double[chart.Series[0].Points.Count()];
double[] arrayResult = { };
for (int i = 0; i < chart.Series[0].Points.Count(); i++)
{
arrayX[i] = chart.Series[0].Points[i].XValue;
arrayY[i] = chart.Series[0].Points[i].YValues[0];
}
arrayResult = Polyfit(arrayX, arrayY, 4);
foreach (double element in arrayResult)
{
MessageBox.Show(element.ToString());
}
double functionVarE = arrayResult[0];
double functionVarD = arrayResult[1];
double functionVarC = arrayResult[2];
double functionVarB = arrayResult[3];
double functionVarA = arrayResult[4];
double equationVar = 0;
//prepare the function series in the graph
if (chart.Series.IndexOf("function") < 0)
chart.Series.Add("function");
chart.Series[2].Points.Clear();
chart.Series[2].ChartType = SeriesChartType.Line;
for (int x = -500; x < 1000; x++) //hardcoding
{
equationVar = functionVarA * (Math.Pow(x, 4)) + functionVarB * (Math.Pow(x, 3)) + functionVarC * (Math.Pow(x, 2)) + functionVarD * x + functionVarE;
chart.Series[2].Points.AddXY(Convert.ToDouble(x), equationVar);
}
This is a working solution I coded. If you see any improvement feel free to tell me!

Combining two double[] arrays into double[,]

I frequently have two arrays that I need to combine to one matrix (same lenght and type). I was wondering whether there is a linq way that is more elegant than:
var result = new double[dt.Count, 2];
for (int i = 0; i < dt.Count; i++)
{
result[i, 0] = dts[i];
result[i, 1] = dt[i];
}
I tried
var result = dts.zip(dt, (a,b) => new{a,b})
and:
var result = dts.Concat(dt).ToArray()
But neither do what I would like to do...

There is nothing in the framework, but here is a general solution (that works for 2 or more arrays):
public static class ArrayConvert
{
public static T[,] To2DArray<T>(params T[][] arrays)
{
if (arrays == null) throw new ArgumentNullException("arrays");
foreach (var a in arrays)
{
if (a == null) throw new ArgumentException("can not contain null arrays");
if (a.Length != arrays[0].Length) throw new ArgumentException("input arrays should have the same length");
}
var height = arrays.Length;
var width = arrays[0].Length;
var result = new T[width, height];
for (int i = 0; i < height; i++)
for (int j = 0; j < width; j++)
{
result[i, j] = arrays[i][j];
}
return result;
}
}
Which can then be used as follows:
var convertedArray = ArrayConvert.To2DArray(new[]{1,2,3}, new[]{4,5,6}, new[]{7,8,9});

ok then use this
class Program {
static void Main(string[] args) {
double[,] x = { { 1, 2, 3 }, { 4, 5, 6 } };
double[,] y = { { 7, 8, 9 }, { 10, 11, 12 } };
var xy = new StitchMatrix<int>(x, y);
Console.WriteLine("0,0=" + xy[0, 0]); // 1
Console.WriteLine("1,1=" + xy[1, 1]); // 5
Console.WriteLine("1,2=" + xy[1, 2]); // 6
Console.WriteLine("2,2=" + xy[2, 2]); // 9
Console.WriteLine("3,2=" + xy[3, 2]); // 12
}
}
class StitchMatrix<T> {
private T[][,] _matrices;
private double[] _lengths;
public StitchMatrix(params T[][,] matrices) {
// TODO: check they're all same size
_matrices = matrices;
// call uperbound once for speed
_lengths = _matrices.Select(m => m.GetUpperBound(0)).ToArray();
}
public T this[double x, double y] {
get {
// find the right matrix
double iMatrix = 0;
while (_lengths[iMatrix] < x) {
x -= (_lengths[iMatrix] + 1);
iMatrix++;
}
// return value at cell
return _matrices[iMatrix][x, y];
}
}
}

Here is another solution. I "prepare" the input for LINQ processing. Don't sure that this is elegant, but it is LINQ:
// the input
double[] dts = { 1, 2, 3, 4, 5 };
double[] dt = { 10, 20, 30, 40, 50 };
// list of lists, for iterating the input with LINQ
double[][] combined = { dts, dt };
var indexes = Enumerable.Range(0, dt.Length);
var subIndexes = Enumerable.Range(0, 2);
// the output
var result = new double[dt.Length, 2];
var sss = from i in indexes
from j in subIndexes
select result[i, j] = combined[j][i];
// just to activate the LINQ iterator
sss.ToList();

I suggest against doing it directly in LINQ. You can write a generic method to do it for you, something like:
public static T[,] To2DArray<T>(this T[][] arr)
{
if (arr.Length == 0)
{
return new T[,]{};
}
int standardLength = arr[0].Length;
foreach (var x in arr)
{
if (x.Length != standardLength)
{
throw new ArgumentException("Arrays must have all the same length");
}
}
T[,] solution = new T[arr.Length, standardLength];
for (int i = 0; i < arr.Length; i++)
{
for (int j = 0; j < standardLength; j++)
{
solution[i, j] = arr[i][j];
}
}
return solution;
}

I know that wasnt the question but the most elegant answer is to use f#:
let combinearrays (arr1:array<'a>) (arr2:array<'a>) =
let rws = arr1|> Array.length
Array2D.init rws 2 (fun i j -> match j with |0 -> arr1.[i] |1 -> arr2.[i])
From John see here

Here is the solution Convert two 1d Arrays in one 2D Array which I developed for my own use.
(from a1 in array1.Select((n,index)=>new{Index=index,c1=n}).ToList()
join a2 in array2.Select((n,index)=>new {Index=index,c2=n}).ToList() on a1.Index equals a2.Index
select new {c1,c2}
).ToArray()