We Keep Coding

Detecting square in image

At a school we are preparing artwork which we have scanned and want automatically crop to the correct size. The kids (attempt) to draw within a rectangle:
I want to detect the inner rectangle borders, so I have applied a few filters with accord.net:
var newImage = new Bitmap(#"C:\Temp\temp.jpg");
var g = Graphics.FromImage(newImage);
var pen = new Pen(Color.Purple, 10);
var grayScaleFilter = new Grayscale(1, 0, 0);
var image = grayScaleFilter.Apply(newImage);
image.Save(#"C:\temp\grey.jpg");
var skewChecker = new DocumentSkewChecker();
var angle = skewChecker.GetSkewAngle(image);
var rotationFilter = new RotateBilinear(-angle);
rotationFilter.FillColor = Color.White;
var rotatedImage = rotationFilter.Apply(image);
rotatedImage.Save(#"C:\Temp\rotated.jpg");
var thresholdFilter = new IterativeThreshold(10, 128);
thresholdFilter.ApplyInPlace(rotatedImage);
rotatedImage.Save(#"C:\temp\threshold.jpg");
var invertFilter = new Invert();
invertFilter.ApplyInPlace(rotatedImage);
rotatedImage.Save(#"C:\temp\inverted.jpg");
var bc = new BlobCounter
{
BackgroundThreshold = Color.Black,
FilterBlobs = true,
MinWidth = 1000,
MinHeight = 1000
};
bc.ProcessImage(rotatedImage);
foreach (var rect in bc.GetObjectsRectangles())
{
g.DrawRectangle(pen, rect);
}
newImage.Save(#"C:\Temp\test.jpg");
This produces the following inverted image that the BlobCounter uses as input:
But the result of the blobcounter isn't super accurate, the purple lines indicate what the BC has detected.
Would there be a better alternative to the BlobCounter in accord.net or are there other C# library better suited for this kind of computer vision?

Here is a simple solution while I was bored on my lunch break.
Basically it just scans all the dimensions from outside to inside for a given color threshold (black), then takes the most prominent result.
Given
[MethodImpl(MethodImplOptions.AggressiveInlining)]
public static unsafe bool IsValid(int* scan0Ptr, int x, int y,int stride, double thresh)
{
var c = *(scan0Ptr + x + y * stride);
var r = ((c >> 16) & 255);
var g = ((c >> 8) & 255);
var b = ((c >> 0) & 255);
// compare it against the threshold
return r * r + g * g + b * b < thresh;
}
private static int GetBest(IEnumerable<int> array)
=> array.Where(x => x != 0)
.GroupBy(i => i)
.OrderByDescending(grp => grp.Count())
.Select(grp => grp.Key)
.First();
Example
private static unsafe Rectangle ConvertImage(string path, Color source, double threshold)
{
var thresh = threshold * threshold;
using var bmp = new Bitmap(path);
// lock the array for direct access
var bitmapData = bmp.LockBits(new Rectangle(0, 0, bmp.Width, bmp.Height), ImageLockMode.ReadWrite, PixelFormat.Format32bppPArgb);
int left, top, bottom, right;
try
{
// get the pointer
var scan0Ptr = (int*)bitmapData.Scan0;
// get the stride
var stride = bitmapData.Stride / 4;
var array = new int[bmp.Height];
for (var y = 0; y < bmp.Height; y++)
for (var x = 0; x < bmp.Width; x++)
if (IsValid(scan0Ptr, x, y, stride, thresh))
{
array[y] = x;
break;
}
left = GetBest(array);
array = new int[bmp.Height];
for (var y = 0; y < bmp.Height; y++)
for (var x = bmp.Width-1; x > 0; x--)
if (IsValid(scan0Ptr, x, y, stride, thresh))
{
array[y] = x;
break;
}
right = GetBest(array);
array = new int[bmp.Width];
for (var x = 0; x < bmp.Width; x++)
for (var y = 0; y < bmp.Height; y++)
if (IsValid(scan0Ptr, x, y, stride, thresh))
{
array[x] = y;
break;
}
top = GetBest(array);
array = new int[bmp.Width];
for (var x = 0; x < bmp.Width; x++)
for (var y = bmp.Height-1; y > 0; y--)
if (IsValid(scan0Ptr, x, y, stride, thresh))
{
array[x] = y;
break;
}
bottom = GetBest(array);
}
finally
{
// unlock the bitmap
bmp.UnlockBits(bitmapData);
}
return new Rectangle(left,top,right-left,bottom-top);
}
Usage
var fileName = #"D:\7548p.jpg";
var rect = ConvertImage(fileName, Color.Black, 50);
using var src = new Bitmap(fileName);
using var target = new Bitmap(rect.Width, rect.Height);
using var g = Graphics.FromImage(target);
g.DrawImage(src, new Rectangle(0, 0, target.Width, target.Height), rect, GraphicsUnit.Pixel);
target.Save(#"D:\Test.Bmp");
Output
Notes :
This is not meant to be bulletproof or the best solution. Just a fast simple one.
There are many approaches to this, even machine learning ones that are likely better and more robust.
There is a lot of code repetition here, basically I just copied, pasted and tweaked for each side
I have just picked an arbitrary threshold that seems to work. Play with it
Getting the most common occurrence for the side is likely not the best approach, maybe you would want to bucket the results.
You could probably sanity limit the amount a side needs to scan in.

Using fastai (pytorch) in c# code, how to normalize Bitmap with mean and std?

A few days ago i switched from tensorflow to fastai for my c# Project. But now i am facing a problem with my normalisation. For both i use an onnx pipeline to load the model and the data.
var onnxPipeline = mLContext.Transforms.ResizeImages(resizing: ImageResizingEstimator.ResizingKind.Fill, outputColumnName: inputName,
imageWidth: ImageSettings.imageWidth, imageHeight: ImageSettings.imageHeight,
inputColumnName: nameof(ImageInputData.Image))
.Append(mLContext.Transforms.ExtractPixels(outputColumnName: inputName, interleavePixelColors: true, scaleImage: 1 / 255f))
.Append(mLContext.Transforms.ApplyOnnxModel(outputColumnName: outputName, inputColumnName: inputName, modelFile: onnxModelPath));
var emptyData = mLContext.Data.LoadFromEnumerable(new List<ImageInputData>());
var onnxModel = onnxPipeline.Fit(emptyData);
with
class ImageInputData
{
[ImageType(ImageSettings.imageHeight, ImageSettings.imageWidth)]
public Bitmap Image { get; set; }
public ImageInputData(byte[] image)
{
using (var ms = new MemoryStream(image))
{
Image = new Bitmap(ms);
}
}
public ImageInputData(Bitmap image)
{
Image = image;
}
}
After using fastai i learned, that the models get better accuracy if the data is normalized with a specific mean and standard deviation (because i used the resnet34 model it should be means { 0.485, 0.456, 0.406 } stds = { 0.229, 0.224, 0.225 } respectively).
So the pixelvalues (for each color ofc.) have to be transformed with those values to match the trainings images. But how can i achive this in C#?
What i tried so far is:
int imageSize = 256;
double[] means = new double[] { 0.485, 0.456, 0.406 }; // used in fastai model
double[] stds = new double[] { 0.229, 0.224, 0.225 };
Bitmap bitmapImage = inputBitmap;
Image image = bitmapImage;
Color[] pixels = new Color[imageSize * imageSize];
for (int x = 0; x < bitmapImage.Width; x++)
{
for (int y = 0; y < bitmapImage.Height; y++)
{
Color pixel = bitmapImage.GetPixel(x, y);
pixels[x + y] = pixel;
double red = (pixel.R - (means[0] * 255)) / (stds[0] * 255); // *255 to scale the mean and std values to the Bitmap
double gre = (pixel.G - (means[1] * 255)) / (stds[1] * 255);
double blu = (pixel.B - (means[2] * 255)) / (stds[2] * 255);
Color pixel_n = Color.FromArgb(pixel.A, (int)red, (int)gre, (int)blu);
bitmapImage.SetPixel(x, y, pixel_n);
}
}
Ofcourse its not working, because the Colorvalues can`t be negative (which i realised only later).
But how can i achive this normalisation between -1 and 1 for my model in C# with the onnx-model?
Is there a different way to feed the model or to handle the normalisation?
Any help would be appreciated!

One way to solve this problem is to switch from an onnx pipeline to an onnx Inferencesession, which is in my view simpler and better to understand:
public List<double> UseOnnxSession(Bitmap image, string onnxModelPath)
{
double[] means = new double[] { 0.485, 0.456, 0.406 };
double[] stds = new double[] { 0.229, 0.224, 0.225 };
using (var session = new InferenceSession(onnxModelPath))
{
List<double> scores = new List<double>();
Tensor<float> t1 = ConvertImageToFloatData(image, means, stds);
List<float> fl = new List<float>();
var inputMeta = session.InputMetadata;
var inputs = new List<NamedOnnxValue>()
{
NamedOnnxValue.CreateFromTensor<float>("input_1", t1)
};
using (var results = session.Run(inputs))
{
foreach (var r in results)
{
var x = r.AsTensor<float>().First();
var y = r.AsTensor<float>().Last();
var softmaxScore = Softmax(new double[] { x, y });
scores.Add(softmaxScore[0]);
scores.Add(softmaxScore[1]);
}
}
return scores;
}
}
// Create your Tensor and add transformations as you need.
public static Tensor<float> ConvertImageToFloatData(Bitmap image, double[] means, double[] std)
{
Tensor<float> data = new DenseTensor<float>(new[] { 1, 3, image.Width, image.Height });
for (int x = 0; x < image.Width; x++)
{
for (int y = 0; y < image.Height; y++)
{
Color color = image.GetPixel(x, y);
var red = (color.R - (float)means[0] * 255) / ((float)std[0] * 255);
var gre = (color.G - (float)means[1] * 255) / ((float)std[1] * 255);
var blu = (color.B - (float)means[2] * 255) / ((float)std[2] * 255);
data[0, 0, x, y] = red;
data[0, 1, x, y] = gre;
data[0, 2, x, y] = blu;
}
}
return data;
}
Also i have to use my own Softmax method on these scores to get the real probabilities out of my model:
public double[] Softmax(double[] values)
{
double[] ret = new double[values.Length];
double maxExp = values.Select(Math.Exp).Sum();
for (int i = 0; i < values.Length; i++)
{
ret[i] = Math.Round((Math.Exp(values[i]) / maxExp), 4);
}
return ret;
}
Hope this helps someone who has a similar Problem.

Synchronization within nested Parallel.For loops

I wanted to turn a regular for loop into a Parallel.For loop.
This-
for (int i = 0; i < bitmapImage.Width; i++)
{
for (int x = 0; x < bitmapImage.Height; x++)
{
System.Drawing.Color oc = bitmapImage.GetPixel(i, x);
int gray = (int)((oc.R * 0.3) + (oc.G * 0.59) + (oc.B * 0.11));
System.Drawing.Color nc = System.Drawing.Color.FromArgb(oc.A, gray, gray, gray);
bitmapImage.SetPixel(i, x, nc);
}
}
Into this-
Parallel.For(0, bitmapImage.Width - 1, i =>
{
Parallel.For(0, bitmapImage.Height - 1, x =>
{
System.Drawing.Color oc = bitmapImage.GetPixel(i, x);
int gray = (int)((oc.R * 0.3) + (oc.G * 0.59) + (oc.B * 0.11));
System.Drawing.Color nc = System.Drawing.Color.FromArgb(oc.A, gray, gray, gray);
bitmapImage.SetPixel(i, x, nc);
});
});
It fails with message-
Object is currently in use elsewhere.
at below line since multiple threads trying to access the non-thread safe reasources. Any idea how I can make this work?
System.Drawing.Color oc = bitmapImage.GetPixel(i, x);

It's not a clean solution, seeing what you would like to achieve. It would be better to get all the pixels in one shot, and then process them in the parallel for.
An alternative that I personally used, and improved the performance dramatically, is doing this conversion using unsafe functions to output a grayscale image.
public static byte[] MakeGrayScaleRev(byte[] source, ref Bitmap bmp,int Hei,int Wid)
{
int bytesPerPixel = 4;
byte[] bytesBig = new byte[Wid * Hei]; //create array to contain bitmap data with padding
unsafe
{
int ic = 0, oc = 0, x = 0;
//Convert the pixel to it's luminance using the formula:
// L = .299*R + .587*G + .114*B
//Note that ic is the input column and oc is the output column
for (int ind = 0, i = 0; ind < 4 * Hei * Wid; ind += 4, i++)
{
int g = (int)
((source[ind] / 255.0f) *
(0.301f * source[ind + 1] +
0.587f * source[ind + 2] +
0.114f * source[ind + 3]));
bytesBig[i] = (byte)g;
}
}
try
{
bmp = new Bitmap(Wid, Hei, PixelFormat.Format8bppIndexed);
bmp.Palette = GetGrayScalePalette();
Rectangle dimension = new Rectangle(0, 0, Wid, Hei);
BitmapData picData = bmp.LockBits(dimension, ImageLockMode.ReadWrite, bmp.PixelFormat);
IntPtr pixelStartAddress = picData.Scan0;
Marshal.Copy(forpictures, 0, pixelStartAddress, forpictures.Length);
bmp.UnlockBits(picData);
return bytesBig;
}
catch (Exception ex)
{
Console.WriteLine(ex.StackTrace);
return null;
}
}
It gets the bytearray of all the pixels of the input image, its height and width and output the computed grayscale array, and in ref Bitmap bmp the output grayscale bitmap.

C# scanning image code improvement

I'm working on a screen sharing app, which runs a loop and grab fast screenshots using GDI methods . example here
Of course I also use a flood fill algorithm to find the changes areas between 2 images (previous screenshot and current).
I use another small trick - I downscale the snapshot resolution in 10, because processing 1920*1080=2073600 pixels very constantly is not very efficient.
However when I find the rectangle bounds - I apply it on the original full size bitmap and I just multiply by 10 the dimension (including top, left, width, height).
This is the scanning code:
unsafe bool ArePixelsEqual(byte* p1, byte* p2, int bytesPerPixel)
{
for (int i = 0; i < bytesPerPixel; ++i)
if (p1[i] != p2[i])
return false;
return true;
}
private unsafe List<Rectangle> CodeImage(Bitmap bmp, Bitmap bmp2)
{
List<Rectangle> rec = new List<Rectangle>();
var bmData1 = bmp.LockBits(new System.Drawing.Rectangle(0, 0, bmp.Width, bmp.Height), System.Drawing.Imaging.ImageLockMode.ReadOnly, bmp.PixelFormat);
var bmData2 = bmp2.LockBits(new System.Drawing.Rectangle(0, 0, bmp.Width, bmp.Height), System.Drawing.Imaging.ImageLockMode.ReadOnly, bmp2.PixelFormat);
int bytesPerPixel = 4;
IntPtr scan01 = bmData1.Scan0;
IntPtr scan02 = bmData2.Scan0;
int stride1 = bmData1.Stride;
int stride2 = bmData2.Stride;
int nWidth = bmp.Width;
int nHeight = bmp.Height;
bool[] visited = new bool[nWidth * nHeight];
byte* base1 = (byte*)scan01.ToPointer();
byte* base2 = (byte*)scan02.ToPointer();
for (int y = 0; y < nHeight; y ++)
{
byte* p1 = base1;
byte* p2 = base2;
for (int x = 0; x < nWidth; ++x)
{
if (!ArePixelsEqual(p1, p2, bytesPerPixel) && !(visited[x + nWidth * y]))
{
// fill the different area
int minX = x;
int maxX = x;
int minY = y;
int maxY = y;
var pt = new Point(x, y);
Stack<Point> toBeProcessed = new Stack<Point>();
visited[x + nWidth * y] = true;
toBeProcessed.Push(pt);
while (toBeProcessed.Count > 0)
{
var process = toBeProcessed.Pop();
var ptr1 = (byte*)scan01.ToPointer() + process.Y * stride1 + process.X * bytesPerPixel;
var ptr2 = (byte*)scan02.ToPointer() + process.Y * stride2 + process.X * bytesPerPixel;
//Check pixel equality
if (ArePixelsEqual(ptr1, ptr2, bytesPerPixel))
continue;
//This pixel is different
//Update the rectangle
if (process.X < minX) minX = process.X;
if (process.X > maxX) maxX = process.X;
if (process.Y < minY) minY = process.Y;
if (process.Y > maxY) maxY = process.Y;
Point n; int idx;
//Put neighbors in stack
if (process.X - 1 >= 0)
{
n = new Point(process.X - 1, process.Y); idx = n.X + nWidth * n.Y;
if (!visited[idx]) { visited[idx] = true; toBeProcessed.Push(n); }
}
if (process.X + 1 < nWidth)
{
n = new Point(process.X + 1, process.Y); idx = n.X + nWidth * n.Y;
if (!visited[idx]) { visited[idx] = true; toBeProcessed.Push(n); }
}
if (process.Y - 1 >= 0)
{
n = new Point(process.X, process.Y - 1); idx = n.X + nWidth * n.Y;
if (!visited[idx]) { visited[idx] = true; toBeProcessed.Push(n); }
}
if (process.Y + 1 < nHeight)
{
n = new Point(process.X, process.Y + 1); idx = n.X + nWidth * n.Y;
if (!visited[idx]) { visited[idx] = true; toBeProcessed.Push(n); }
}
}
//finaly set a rectangle.
Rectangle r = new Rectangle(minX * 10, minY * 10, (maxX - minX + 1) * 10, (maxY - minY + 1) * 10);
rec.Add(r);
//got the rectangle now i'll do whatever i want with that.
//notify i scaled everything by x10 becuse i want to apply the changes on the originl 1920x1080 image.
}
p1 += bytesPerPixel;
p2 += bytesPerPixel;
}
base1 += stride1;
base2 += stride2;
}
bmp.UnlockBits(bmData1);
bmp2.UnlockBits(bmData2);
return rec;
}
This is my call:
private void Start()
{
full1 = GetDesktopImage();//the first,intial screen.
while (true)
{
full2 = GetDesktopImage();
a = new Bitmap(full1, 192, 108);//resizing for faster processing the images.
b = new Bitmap(full2, 192, 108); // resizing for faster processing the images.
CodeImage(a, b);
count++; // counter for the performance.
full1 = full2; // assign old to current bitmap.
}
}
However, after all the tricks and techniques I used, the algorithm runs quite slow... on my machine - Intel i5 4670k 3.4ghz - it runs only 20 times (at the maximum! It might get lower)! It maybe sounds fast (don't forget I have to send each changed area over the network after), but I'm looking to achieve more processed image per second. I think the main bottleneck is in the resizing of the 2 images - but I just thought it would be even faster after resizing - because it would have to loop through less pixels... 192*108=200,000 only..
I would appreciate any help, any improvement. Thanks.

Draw wave file using picturebox

I want to know the error in the following code.I want to draw the values of array that contains wave file samples.in the form i put panel and inside it picturebox.
private void button1_Click(object sender, EventArgs e)
{
string ss = "test.wav";
double[] xxwav = prepare(ss);
int xmin = 300; int ymin = 250; int xmax = 1024; int ymax = 450;
int xpmin = 0; int xpmax = xxwav.Length; int ypmin = 32767; int ypmax = -32768;
double a = (double)((xmax - xmin)) /(double) (xpmax - xpmin);
double b = (double)(xpmin - (a * xmin));
double c = (double)((ymax - ymin) /(double) (ypmax - ypmin));
double d = (double)(ypmin - (c * ymin));
double xp1,yp1,xp2,yp2;
Pen redPen = new Pen(Color.Red, 1);
Bitmap bmp = new Bitmap(40000, 500);
Graphics g = Graphics.FromImage(bmp);
PointF p1;
PointF p2;
for (int i = 1; i < xxwav.Length; i++)
{
xp1 = a * (i-1) + b;
yp1 = c * xxwav[i-1] + d;
xp2=a * i + b;
yp2=c * xxwav[i] + d;
p1 =new PointF ((float)xp1,(float)yp1);
p2 =new PointF ((float)xp2,(float)yp2);
g.DrawLine(redPen, p1, p2);
}
pictureBox1.Image = bmp;
MessageBox.Show("complete");
}
public static Double[] prepare(String wavePath)
{
Double[] data;
byte[] wave;
byte[] sR = new byte[4];
System.IO.FileStream WaveFile = System.IO.File.OpenRead(wavePath);
wave = new byte[WaveFile.Length];
data = new Double[(wave.Length - 44) / 4];//shifting the headers out of the PCM data;
WaveFile.Read(wave, 0, Convert.ToInt32(WaveFile.Length));//read the wave file into the wave variable
/***********Converting and PCM accounting***************/
for (int i = 0; i < data.Length; i++)
{
data[i] = BitConverter.ToInt16(wave, i * 2) / 32768.0;
}
//65536.0.0=2^n, n=bits per sample;
return data;
}

Your code worked for me only after I fiddled with your transformations and scaling parameters.
I have replaced your code with the scaling and transformation methods available in the System.Drawing namespace. This did gave me a view of one of my wav files. You only have to replace the private void button1_Click(object sender, EventArgs e) implementation.
var xxwav = prepare(wavFile);
// determine max and min
var max = (from v in xxwav
select v).Max();
var min = (from v in xxwav
select v).Min();
// what is our Y-axis scale
var mid = (max - min);
Pen redPen = new Pen(Color.Red, 1);
Bitmap bmp = new Bitmap(this.pictureBox1.Size.Width, this.pictureBox1.Size.Height);
Graphics g = Graphics.FromImage(bmp);
// x / y position (y-axis to the middle)
g.TranslateTransform(
0
, this.pictureBox1.Size.Height / 2);
// scaling according to picturebox size
g.ScaleTransform(
(float)this.pictureBox1.Size.Width / (float)xxwav.Length
, (float)this.pictureBox1.Size.Height / ((float)mid));
//first point
var prev = new PointF(0, (float)xxwav[0]);
// iterate over next points
for (int i = 1; i < xxwav.Length; i++)
{
var next = new PointF((float) i , (float) xxwav[i] );
g.DrawLine(redPen, prev, next);
prev = next;
}
pictureBox1.Image = bmp;