We Keep Coding

emgucv: pan card improper skew detection in C#

I am having three image of pan card for testing skew of image using emgucv and c#.
1st image which is on top Detected 180 degree working properly.
2nd image which is in middle Detected 90 dgree should detected as 180 degree.
3rd image Detected 180 degree should detected as 90 degree.
One observation I am having that i wanted to share here is when i crop unwanted part of image from up and down side of pan card using paint brush, it gives me expected result using below mention code.
Now i wanted to understand how i can remove the unwanted part using programming.
I have played with contour and roi but I am not able to figure out how to fit the same. I am not able to understand whether emgucv itself selects contour or I have to do something.
Please suggest any suitable code example.
Please check code below for angle detection and please help me. Thanks in advance.
imgInput = new Image<Bgr, byte>(impath);
Image<Gray, Byte> img2 = imgInput.Convert<Gray, Byte>();
Bitmap imgs;
Image<Gray, byte> imgout = imgInput.Convert<Gray, byte>().Not().ThresholdBinary(new Gray(50), new Gray(125));
VectorOfVectorOfPoint contours = new VectorOfVectorOfPoint();
Emgu.CV.Mat hier = new Emgu.CV.Mat();
var blurredImage = imgInput.SmoothGaussian(5, 5, 0 , 0);
CvInvoke.AdaptiveThreshold(imgout, imgout, 255, Emgu.CV.CvEnum.AdaptiveThresholdType.GaussianC, Emgu.CV.CvEnum.ThresholdType.Binary, 5, 45);
CvInvoke.FindContours(imgout, contours, hier, Emgu.CV.CvEnum.RetrType.External, Emgu.CV.CvEnum.ChainApproxMethod.ChainApproxSimple);
if (contours.Size >= 1)
{
for (int i = 0; i <= contours.Size; i++)
{
Rectangle rect = CvInvoke.BoundingRectangle(contours[i]);
RotatedRect box = CvInvoke.MinAreaRect(contours[i]);
PointF[] Vertices = box.GetVertices();
PointF point = box.Center;
PointF edge1 = new PointF(Vertices[1].X - Vertices[0].X, Vertices[1].Y - Vertices[0].Y);
PointF edge2 = new PointF(Vertices[2].X - Vertices[1].X, Vertices[2].Y - Vertices[1].Y);
double r = edge1.X + edge1.Y;
double edge1Magnitude = Math.Sqrt(Math.Pow(edge1.X, 2) + Math.Pow(edge1.Y, 2));
double edge2Magnitude = Math.Sqrt(Math.Pow(edge2.X, 2) + Math.Pow(edge2.Y, 2));
PointF primaryEdge = edge1Magnitude > edge2Magnitude ? edge1 : edge2;
double primaryMagnitude = edge1Magnitude > edge2Magnitude ? edge1Magnitude : edge2Magnitude;
PointF reference = new PointF(1, 0);
double refMagnitude = 1;
double thetaRads = Math.Acos(((primaryEdge.X * reference.X) + (primaryEdge.Y * reference.Y)) / (primaryMagnitude * refMagnitude));
double thetaDeg = thetaRads * 180 / Math.PI;
imgInput = imgInput.Rotate(thetaDeg, new Bgr());
imgout = imgout.Rotate(box.Angle, new Gray());
Bitmap bmp = imgout.Bitmap;
break;
}
}

The Problem
Let us start with the problem before the solution:
Your Code
When you submit code, asking for help, at least make some effort to "clean" it. Help people help you! There's so many lines of code here that do nothing. You declare variables that are never used. Add some comments that let people know what it is that you think your code should do.
Bitmap imgs;
var blurredImage = imgInput.SmoothGaussian(5, 5, 0, 0);
Rectangle rect = CvInvoke.BoundingRectangle(contours[i]);
PointF point = box.Center;
double r = edge1.X + edge1.Y;
// Etc
Adaptive Thresholding
The following line of code produces the following images:
CvInvoke.AdaptiveThreshold(imgout, imgout, 255, Emgu.CV.CvEnum.AdaptiveThresholdType.GaussianC, Emgu.CV.CvEnum.ThresholdType.Binary, 5, 45);
Image 1
Image 2
Image 3
Clearly this is not what you're aiming for since the primary contour, the card edge, is completely lost. As a tip, you can always use the following code to display images at runtime to help you with debugging.
CvInvoke.NamedWindow("Output");
CvInvoke.Imshow("Output", imgout);
CvInvoke.WaitKey();
The Soltuion
Since your in example images the card is primarily a similar Value (in the HSV sense) to the background. I do not think simple gray scale thresholding is the correct approach in this case. I purpose the following:
Algorithm
Use Canny Edge Detection to extract the edges in the image.
Dilate the edges so as the card content combines.
Use Contour Detection to filter for the combined edges with the largest bounding.
Fit this primary contour with a rotated rectangle in order to extract the corner points.
Use the corner points to define a transformation matrix to be applied using WarpAffine.
Warp and crop the image.
The Code
You may wish to experiment with the parameters of the Canny Detection and Dilation.
// Working Images
Image<Bgr, byte> imgInput = new Image<Bgr, byte>("Test1.jpg");
Image<Gray, byte> imgEdges = new Image<Gray, byte>(imgInput.Size);
Image<Gray, byte> imgDilatedEdges = new Image<Gray, byte>(imgInput.Size);
Image<Bgr, byte> imgOutput;
// 1. Edge Detection
CvInvoke.Canny(imgInput, imgEdges, 25, 80);
// 2. Dilation
CvInvoke.Dilate(
imgEdges,
imgDilatedEdges,
CvInvoke.GetStructuringElement(
ElementShape.Rectangle,
new Size(3, 3),
new Point(-1, -1)),
new Point(-1, -1),
5,
BorderType.Default,
new MCvScalar(0));
// 3. Contours Detection
VectorOfVectorOfPoint inputContours = new VectorOfVectorOfPoint();
Mat hierarchy = new Mat();
CvInvoke.FindContours(
imgDilatedEdges,
inputContours,
hierarchy,
RetrType.External,
ChainApproxMethod.ChainApproxSimple);
VectorOfPoint primaryContour = (from contour in inputContours.ToList()
orderby contour.GetArea() descending
select contour).FirstOrDefault();
// 4. Corner Point Extraction
RotatedRect bounding = CvInvoke.MinAreaRect(primaryContour);
PointF topLeft = (from point in bounding.GetVertices()
orderby Math.Sqrt(Math.Pow(point.X, 2) + Math.Pow(point.Y, 2))
select point).FirstOrDefault();
PointF topRight = (from point in bounding.GetVertices()
orderby Math.Sqrt(Math.Pow(imgInput.Width - point.X, 2) + Math.Pow(point.Y, 2))
select point).FirstOrDefault();
PointF botLeft = (from point in bounding.GetVertices()
orderby Math.Sqrt(Math.Pow(point.X, 2) + Math.Pow(imgInput.Height - point.Y, 2))
select point).FirstOrDefault();
PointF botRight = (from point in bounding.GetVertices()
orderby Math.Sqrt(Math.Pow(imgInput.Width - point.X, 2) + Math.Pow(imgInput.Height - point.Y, 2))
select point).FirstOrDefault();
double boundingWidth = Math.Sqrt(Math.Pow(topRight.X - topLeft.X, 2) + Math.Pow(topRight.Y - topLeft.Y, 2));
double boundingHeight = Math.Sqrt(Math.Pow(botLeft.X - topLeft.X, 2) + Math.Pow(botLeft.Y - topLeft.Y, 2));
bool isLandscape = boundingWidth > boundingHeight;
// 5. Define warp crieria as triangles
PointF[] srcTriangle = new PointF[3];
PointF[] dstTriangle = new PointF[3];
Rectangle ROI;
if (isLandscape)
{
srcTriangle[0] = botLeft;
srcTriangle[1] = topLeft;
srcTriangle[2] = topRight;
dstTriangle[0] = new PointF(0, (float)boundingHeight);
dstTriangle[1] = new PointF(0, 0);
dstTriangle[2] = new PointF((float)boundingWidth, 0);
ROI = new Rectangle(0, 0, (int)boundingWidth, (int)boundingHeight);
}
else
{
srcTriangle[0] = topLeft;
srcTriangle[1] = topRight;
srcTriangle[2] = botRight;
dstTriangle[0] = new PointF(0, (float)boundingWidth);
dstTriangle[1] = new PointF(0, 0);
dstTriangle[2] = new PointF((float)boundingHeight, 0);
ROI = new Rectangle(0, 0, (int)boundingHeight, (int)boundingWidth);
}
Mat warpMat = new Mat(2, 3, DepthType.Cv32F, 1);
warpMat = CvInvoke.GetAffineTransform(srcTriangle, dstTriangle);
// 6. Apply the warp and crop
CvInvoke.WarpAffine(imgInput, imgInput, warpMat, imgInput.Size);
imgOutput = imgInput.Copy(ROI);
imgOutput.Save("Output1.bmp");
Two extension methods are used:
static List<VectorOfPoint> ToList(this VectorOfVectorOfPoint vectorOfVectorOfPoint)
{
List<VectorOfPoint> result = new List<VectorOfPoint>();
for (int contour = 0; contour < vectorOfVectorOfPoint.Size; contour++)
{
result.Add(vectorOfVectorOfPoint[contour]);
}
return result;
}
static double GetArea(this VectorOfPoint contour)
{
RotatedRect bounding = CvInvoke.MinAreaRect(contour);
return bounding.Size.Width * bounding.Size.Height;
}
Outputs
Meta Example

Emgu CV Image sharpening and controur detection

I am working on a project where I need to identify dots from IR lasers on a surface. I use for that a camera with IR filter
Some input images:
There can be several dots, too. So I tried to sharpen this image from webcam and then use FindContours method of Emgu CV.
There is my code:
public static Image<Gray, byte> Sharpen(Image<Gray, byte> image, int w, int h, double sigma1, double sigma2, int k)
{
w = (w % 2 == 0) ? w - 1 : w;
h = (h % 2 == 0) ? h - 1 : h;
//apply gaussian smoothing using w, h and sigma
var gaussianSmooth = image.SmoothGaussian(w, h, sigma1, sigma2);
//obtain the mask by subtracting the gaussian smoothed image from the original one
var mask = image - gaussianSmooth;
//add a weighted value k to the obtained mask
mask *= k;
//sum with the original image
image += mask;
return image;
}
private void ProcessFrame(object sender, EventArgs arg)
{
Mat frame = new Mat();
if (_capture.Retrieve(frame, CameraDevice))
{
Image<Bgr, byte> original = frame.ToImage<Bgr, byte>();
Image<Gray, byte> img = Sharpen(frame.ToImage<Gray, byte>(), 100, 100, 100, 100, 30);
Image<Gray, byte> thresh = new Image<Gray, byte>(img.Size);
CvInvoke.PyrDown(img, thresh);
CvInvoke.PyrUp(thresh, thresh);
Image<Gray, byte> mask = new Image<Gray, byte>(thresh.Size);
Image<Gray, byte> cannyImg = thresh.Canny(10, 50);
VectorOfVectorOfPoint contours = new VectorOfVectorOfPoint();
Mat hierarchy = new Mat();
CvInvoke.FindContours(
cannyImg,
contours,
hierarchy,
RetrType.External,
ChainApproxMethod.ChainApproxSimple
);
Image<Bgr, byte> resultImage = img.Copy().Convert<Bgr, byte>();
int contCount = contours.Size;
for (int i = 0; i < contCount; i++)
{
using (VectorOfPoint contour = contours[i])
{
resultImage.Draw(CvInvoke.BoundingRectangle(contour), new Bgr(255, 0, 0), 5);
}
}
captureBox.Image = original.Bitmap;
cvBox.Image = resultImage.Bitmap;
}
}
Example of result image:
So it almost all the time works as I expect it to, but framerate is very low. I'm getting like 10-15 fps with resolution of 640x480. I need to be able to do the same thing for 1920x1080 with at least 30 fps. It's my first time with OpenCV and Emgu.CV. What can I do to make it perform better?

I solved this just setting the threshold, so that image turns black and white only. By adjusting the threshold I was able to achieve the same results if not better in terms of clarity, but also performance drastically improved since there is not heavy processing going on
Here is a snippet with ARCore library instead on EmguCV
var bitmap = eventArgs.Frame;
var filter = new Grayscale(0.2125, 0.7154, 0.0721);
var grayImage = filter.Apply(bitmap);
var thresholdFilter = new Threshold(CurrentThreshold);
thresholdFilter.ApplyInPlace(grayImage);
var blobCounter = new BlobCounter();
blobCounter.ProcessImage(grayImage);
var rectangles = blobCounter.GetObjectsRectangles();

Using Earth's Mover Distance with Emgu CV

I am working on a project in which I try to compare two images in C#. I'm using EmguCV (a C# wrapper for OpenCV). I've tested some funcitons which work (compareHist for example).
I am now trying to use the implementation of Earth's Mover Distance. As I use color images, I build a 2d Histogram based on the HSV image. I then build the corresponding signature (as described in the docs, and explained here).
The problem is that I always obtain a NaN as the output of my code. Since I'm new to C# and EmguCV, I've tried to make the same steps, but this time in Python, and it works, EMD returns a number without any error.
I've spent a lot of time on this problem trying to change the type of the histogram (between the OpenCV Mat and EmguCV image), looking at the histograms values to verify if they are right,... But I don't find what I'm doing wrong.
About the code:
I have a "Comparator" class which just contain 2 images:
class Comparator
{
public Image<Bgr, Byte> RefImage;
public Image<Bgr, Byte> TestImage;
public Comparator(string TestPath, string RefPath)
{
Image<Bgr, Byte> TestImagetemp = new Image<Bgr, Byte>(TestPath);
Image<Bgr, Byte> RefImagetemp = new Image<Bgr, Byte>(RefPath);
int newCols = Math.Min(TestImagetemp.Cols, RefImagetemp.Cols);
int newRows = Math.Min(RefImagetemp.Rows, TestImagetemp.Rows);
Rectangle roi = new Rectangle(0, 0, newCols, newRows);
this.RefImage = crop(RefImagetemp, roi);
this.TestImage = crop(TestImagetemp, roi);
string DiffPath = "C:\\Users\\EPIERSO\\Docs\\testdiff";
this.TestImage.Save(DiffPath + "testavant.png");
}
Here is the method used for computing the histogram:
public static Mat CalcHistHSV(Image<Bgr,Byte> image)
{
int[] histbins = new int[] { 30, 32 };
float[] ranges = new float[] { 0.0f, 180.0f, 0.0f, 256.0f };
Mat hist = new Mat();
VectorOfMat vm = new VectorOfMat();
Image<Hsv,float> imghsv = image.Convert<Hsv, float>();
vm.Push(imghsv);
CvInvoke.CalcHist(vm, new int[] { 0, 1 }, null, hist, histbins, ranges, false);
return hist;
}
And this is the method used for comparing with EMD:
public bool EMDCompare()
{
int hbins = 30;
int sbins = 32;
Mat histref = CalcHistHSV(RefImage);
Mat histtest = CalcHistHSV(TestImage);
//Computing the signatures
Mat sigref = new Mat(hbins*sbins,3,Emgu.CV.CvEnum.DepthType.Cv32F,1);
Mat sigtest = new Mat(hbins*sbins,3, Emgu.CV.CvEnum.DepthType.Cv32F, 1);
for (int h = 0; h<hbins; h++)
{
for (int s = 0; s < sbins; s++)
{
var bin = MatExtension.GetValue(histref,h,s);
MatExtension.SetValue(sigref, h * sbins + s, 0, bin);
MatExtension.SetValue(sigref, h * sbins + s, 1, h);
MatExtension.SetValue(sigref, h * sbins + s, 2, s);
var bin2 = MatExtension.GetValue(histtest, h, s);
MatExtension.SetValue(sigtest, h * sbins + s, 0, bin2);
MatExtension.SetValue(sigtest, h * sbins + s, 1, h);
MatExtension.SetValue(sigtest, h * sbins + s, 2, s);
}
}
float emd = CvInvoke.EMD(sigref, sigtest, DistType.L2);
return ((1 - emd) > 0.7);
}
For modifying Mat values, I use an extension named MatExtension, found here: How can I get and set pixel values of an EmguCV Mat image?
This is the equivalent Python code: https://pastebin.com/drhvNMNs

calcHist OpenCV for C# error

I'm trying to calculate an histogram from an image with OpenCV for C# (EmguCV).
This is my code:
VectorOfMat bgr_planes = new VectorOfMat();
CvInvoke.Split(myImage, bgr_planes);
int[] dim = new int[] { 0 };
int[] histSize = new int[] { 256 };
float[] range = new float[] { 0f, 255f };
bool accumulate = false;
Mat b_hist = new Mat();
Mat g_hist = new Mat();
Mat r_hist = new Mat();
CvInvoke.CalcHist(bgr_planes, dim, new Mat(), b_hist, histSize, range, accumulate);
CvInvoke.CalcHist(bgr_planes, dim, new Mat(), g_hist, histSize, range, accumulate);
CvInvoke.CalcHist(bgr_planes, dim, new Mat(), r_hist, histSize, range, accumulate);`
But I do not get Data b_hist, g_hist and r_hist. I can't address the specific channels bgr_planes[0], because those are of type Mat and it throws an error.
How can I adjust the parameters dim, histSize and range to get a color histogram?
Thank you so much for your help!

I solved it myself (of course right after posting the question...):
Here is what I changed:
Mat x = bgr_planes[0];
Mat[] bVals = new Mat[1] { x };
VectorOfMat colorChannelB = new VectorOfMat();
colorChannelB.Push(bVals);
This separates one single color channel of the bgr_planes. In VectorOfMat colorChannelB is now only one color channel. Do that separately for every channel and keep the parameters.

Imgproc.FindContours return empty contours

I use OpenCV 2.4.11 for Xamarin.Android with OpenCvBinding. I'm trying to find the largest color area in image.
static public Tuple<Bitmap,double> GetArea(Bitmap srcBitmap)
{
Mat mat = new Mat();
Mat gray = new Mat();
Mat mat2 = new Mat();
double max = 0;
Mat Hierarchy = new Mat();
List<MatOfPoint> contours = new List<MatOfPoint>();
Utils.BitmapToMat(srcBitmap, mat);
Imgproc.CvtColor(mat, gray, Imgproc.ColorRgba2gray);
Imgproc.AdaptiveThreshold(gray, mat2, 255, Imgproc.AdaptiveThreshGaussianC, Imgproc.ThreshBinaryInv,1111,0);
Imgproc.FindContours(mat2, contours, Hierarchy, Imgproc.RetrTree, Imgproc.ChainApproxSimple);
foreach (MatOfPoint contour in contours)
{ // never goes here
if (max < Imgproc.ContourArea(contour)) max = Imgproc.ContourArea(contour);
}
Utils.MatToBitmap(mat2,srcBitmap);
return new Tuple<Bitmap, double>(srcBitmap,max);
}
Input Image
If I comment the line with FindContours, I'll get an excellent picture for searching contours.
Threshholded image
FindContours returns correct image(Reputation doesn't allow to add another link), but(!!) list of contours standing empty. So i can't get the area of these contures.
I would be glad of any help. Thanks!

use IList contours = new JavaList();