We Keep Coding

Emgucv crop detected shape automatically

I have an application which is going to be used to crop blank spaces from scanned documents for example this image. What I want to do is extract only the card and remove all the white/blank area. I'm using Emgucv FindContours to do this and at the moment I'm able to find the card contour and some noise captured by the scanner in the image as you can see below.
My question is how can I crop the largest contour found or how to extract it by removing other contours and blanks/whitespaces? Or maybe it is possible with the contour index?
Edit: Maybe another possible solution is if is possible to draw the contour to another pictureBox.
Here is the code that I'm using:
Image<Bgr, byte> imgInput;
Image<Bgr, byte> imgCrop;
private void abrirToolStripMenuItem_Click(object sender, EventArgs e)
{
try
{
OpenFileDialog dialog = new OpenFileDialog();
if (dialog.ShowDialog() ==DialogResult.OK)
{
imgInput = new Image<Bgr, byte>(dialog.FileName);
pictureBox1.Image = imgInput.Bitmap;
imgCrop = imgInput;
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
private void shapeToolStripMenuItem_Click(object sender, EventArgs e)
{
if (imgCrop == null)
{
return;
}
try
{
var temp = imgCrop.SmoothGaussian(5).Convert<Gray, byte>().ThresholdBinaryInv(new Gray(230), new Gray(255));
VectorOfVectorOfPoint contours = new VectorOfVectorOfPoint();
Mat m = new Mat();
CvInvoke.FindContours(temp, contours, m, Emgu.CV.CvEnum.RetrType.External, Emgu.CV.CvEnum.ChainApproxMethod.ChainApproxSimple);
for (int i = 0; i < contours.Size; i++)
{
double perimeter = CvInvoke.ArcLength(contours[i], true);
VectorOfPoint approx = new VectorOfPoint();
CvInvoke.ApproxPolyDP(contours[i], approx, 0.04 * perimeter, true);
CvInvoke.DrawContours(imgCrop, contours, i, new MCvScalar(0, 0, 255), 2);
pictureBox2.Image = imgCrop.Bitmap;
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}

I'll give you my answer in C++, but the same operations should be available in Emgu CV.
I propose the following approach: Segment (that is – separate) the target object using the HSV color space. Calculate a binary mask for the object of interest. Get the biggest blob in the binary mask, this should be the card. Compute the bounding box of the card. Crop the card out of the input image
Ok, first get (or read) the input image. Apply a median blur filter, it will help in getting rid of that high-frequency noise (the little grey blobs) that you see on the input. The main parameter to adjust is the size of the kernel (or filter aperture) be careful, though – a high value will result in an aggressive effect and will likely destroy your image:
//read input image:
std::string imageName = "C://opencvImages//yoshiButNotYoshi.png";
cv::Mat imageInput = cv::imread( imageName );
//apply a median blur filter, the size of the kernel is 5 x 5:
cv::Mat blurredImage;
cv::medianBlur ( imageInput, blurredImage, 5 );
This is the result of the blur filter (The embedded image is resized):
Next, segment the image. Exploit the fact that the background is white, and everything else (the object of interest, mainly) has some color information. You can use the HSV color space. First, convert the BGR image into HSV:
//BGR to HSV conversion:
cv::Mat hsvImg;
cv::cvtColor( blurredImage, hsvImg, CV_RGB2HSV );
The HSV color space encodes color information differently than the typical BGR/RGB color space. Its advantage over other color models pretty much depends on the application, but in general, it is more robust while working with hue gradients. I'll try to get an HSV-based binary mask for the object of interest.
In a binary mask, everything you are interested on the input image is colored in white, everything else in black (or vice versa). You can obtain this mask using the inRange function. However, you must specify the color ranges that will be rendered in white (or black) in the output mask. For your image, and using the HSV color model those values are:
cv::Scalar minColor( 0, 0, 100 ); //the lower range of colors
cv::Scalar maxColor( 0, 0, 255 ); //the upper range of colors
Now, get the binary mask:
//prepare the binary mask:
cv::Mat binaryMask;
//create the binary mask using the specified range of color
cv::inRange( hsvImg, minColor, maxColor, binaryMask );
//invert the mask:
binaryMask = 255 - binaryMask;
You get this image:
Now, you can get rid of some of the noise (that survived the blur filter) via morphological filtering. Morphological filters are, essentially, logical rules applied on binary (or gray) images. They take a "neighborhood" of pixels in the input and apply logical functions to get an output. They are quite handy while cleaning up binary images. I'll apply a series of logical filters to achieve just that.
I'll first erode the image and then dilate it using 3 iterations. The structuring element is a rectangle of size 3 x 3:
//apply some morphology the clean the binary mask a little bit:
cv::Mat SE = cv::getStructuringElement( cv::MORPH_RECT, cv::Size(3, 3) );
int morphIterations = 3;
cv::morphologyEx( binaryMask, binaryMask, cv::MORPH_ERODE, SE, cv::Point(-1,-1), morphIterations );
cv::morphologyEx( binaryMask, binaryMask, cv::MORPH_DILATE, SE, cv::Point(-1,-1), morphIterations );
You get this output. Check out how the noisy blobs are mostly gone:
Now, comes the cool part. You can loop through all the contours in this image and get the biggest of them all. That's a typical operation that I constantly perform, so, I've written a function that does that. It is called findBiggestBlob. I'll present the function later. Check out the result you get after finding and extracting the biggest blob:
//find the biggest blob in the binary image:
cv::Mat biggestBlob = findBiggestBlob( binaryMask );
You get this:
Now, you can get the bounding box of the biggest blob using boundingRect:
//Get the bounding box of the biggest blob:
cv::Rect bBox = cv::boundingRect( biggestBlob );
Let's draw the bounding box on the input image:
cv::Mat imageClone = imageInput.clone();
cv::rectangle( imageClone, bBox, cv::Scalar(255,0,0), 2 );
Finally, let's crop the card out of the input image:
cv::Mat croppedImage = imageInput( bBox );
This is the cropped output:
This is the code for the findBiggestBlob function. The idea is just to compute all the contours in the binary input, calculate their area and store the contour with the largest area of the bunch:
//Function to get the largest blob in a binary image:
cv::Mat findBiggestBlob( cv::Mat &inputImage ){
cv::Mat biggestBlob = inputImage.clone();
int largest_area = 0;
int largest_contour_index = 0;
std::vector< std::vector<cv::Point> > contours; // Vector for storing contour
std::vector< cv::Vec4i > hierarchy;
// Find the contours in the image
cv::findContours( biggestBlob, contours, hierarchy, CV_RETR_CCOMP, CV_CHAIN_APPROX_SIMPLE );
for( int i = 0; i < (int)contours.size(); i++ ) {
//Find the area of the contour
double a = cv::contourArea( contours[i], false);
//Store the index of largest contour:
if( a > largest_area ){
largest_area = a;
largest_contour_index = i;
}
}
//Once you get the biggest blob, paint it black:
cv::Mat tempMat = biggestBlob.clone();
cv::drawContours( tempMat, contours, largest_contour_index, cv::Scalar(0),
CV_FILLED, 8, hierarchy );
//Erase the smaller blobs:
biggestBlob = biggestBlob - tempMat;
tempMat.release();
return biggestBlob;
}

How to apply barrel distortion (lens correction) with SharpDX?

i've made a small application to grap screenshots from any windowed game and send it to the iPhone to creat an virtual reality app, like oculus rift (see https://github.com/gagagu/VR-Streamer-Windows-Server for more info).
The images will be captured with SharpDX and everything is working fine.
Now i want to implement such like lens correction (barrel distortion) and i'm looking for the fastest way to realize it. I'm looking many internet sites with informations about barrel distortion and i think the fastest way is to use a shader for it, but i'm very new to sharpdx (and no knowledge about shaders) and i don't know how to implement a shader to my code. The most tutorials applys a shader to an object (like a cube) but not to a captured image and so i don't know how to do it.
[STAThread]
public System.Drawing.Bitmap Capture()
{
isInCapture = true;
try
{
// init
bool captureDone = false;
bitmap = new System.Drawing.Bitmap(captureRect.Width, captureRect.Height, PixelFormat.Format32bppArgb);
// the capture needs some time
for (int i = 0; !captureDone; i++)
{
try
{
//capture
duplicatedOutput.AcquireNextFrame(-1, out duplicateFrameInformation, out screenResource);
// only for wait
if (i > 0)
{
using (var screenTexture2D = screenResource.QueryInterface<Texture2D>())
device.ImmediateContext.CopyResource(screenTexture2D, screenTexture);
mapSource = device.ImmediateContext.MapSubresource(screenTexture, 0, MapMode.Read, MapFlags.None);
mapDest = bitmap.LockBits(new System.Drawing.Rectangle(0, 0, captureRect.Width, captureRect.Height),
ImageLockMode.WriteOnly, bitmap.PixelFormat);
sourcePtr = mapSource.DataPointer;
destPtr = mapDest.Scan0;
// set x position offset to rect.x
int rowPitch = mapSource.RowPitch - offsetX;
// set pointer to y position
sourcePtr = IntPtr.Add(sourcePtr, mapSource.RowPitch * captureRect.Y);
for (int y = 0; y < captureRect.Height; y++) // needs to speed up!!
{
// set pointer to x position
sourcePtr = IntPtr.Add(sourcePtr, offsetX);
// copy pixel to bmp
Utilities.CopyMemory(destPtr, sourcePtr, pWidth);
// incement pointert to next line
sourcePtr = IntPtr.Add(sourcePtr, rowPitch);
destPtr = IntPtr.Add(destPtr, mapDest.Stride);
}
bitmap.UnlockBits(mapDest);
device.ImmediateContext.UnmapSubresource(screenTexture, 0);
captureDone = true;
}
screenResource.Dispose();
duplicatedOutput.ReleaseFrame();
}
catch//(Exception ex) // catch (SharpDXException e)
{
//if (e.ResultCode.Code != SharpDX.DXGI.ResultCode.WaitTimeout.Result.Code)
//{
// // throw e;
//}
return new Bitmap(captureRect.Width, captureRect.Height, PixelFormat.Format32bppArgb);
}
}
}
catch
{
return new Bitmap(captureRect.Width, captureRect.Height, PixelFormat.Format32bppArgb);
}
isInCapture = false;
return bitmap;
}
It would be really great to get a little start assist from someone who willing to help.
I've found some shaders on inet but it is written for opengl (https://github.com/dghost/glslRiftDistort/tree/master/libovr-0.4.x/glsl110). Can i use the also for directx (sharpdx)?
Thanks forward for any help!

Now I've never used DirectX myself, but I suppose you'll need to use HLSL instead of GLSL (which should be fairly similar though). The idea is that you'll have to load your "screenshot" into a texture buffer, as an input to your fragment shader (pixel shader). Fragment shaders are deceptively easy to understand, it's just a piece of code (written in GLSL or HLSL) looking very much like a subset of C to which a few math functions has been added (vector and matrices manipulation mostly) executed for every single pixel to be rendered.
The code should be fairly simple, you'll take the current pixel position, apply the barrel distortion transformation to it's coordinates, then look up that coordinate in your screenshot texture. The transformation should look something like that :
vec2 uv;
/// Barrel Distortion ///
float d=length(uv);
float z = sqrt(1.0 - d * d);
float r = atan(d, z) / 3.14159;
float phi = atan(uv.y, uv.x);
uv = vec2(r*cos(phi)+.5,r*sin(phi)+.5);
Here's a shadertoy link if you wanna play with it and figure out how it works
I have no idea how HLSL handles texture filtering (which pixel you'll get when using floating point values for coordinates), but I'd put my money on bilinear filtering, which may very well give an unpleasant pixelyness to your output. You'll have to look at better filtering methods once you get the distortion working. Shouldn't be anything too complicated, familiarize yourself with HLSL syntax, find how to load your screenshot into a texture in DirectX and get rolling.
Edit : I said barrel distortion but the code is actually for the fisheye effect. Of course both are pretty much identical, the barrel distortion being only on one axis. I believe what you need is the fisheye effect though, it's what is commonly used for HMDs if I'm not mistaken.

How to render an image with a color-keyed green mask in C#?

Trying to figure out the most elegant way to render an image inside of a specific color of mask in C# (via System.Drawing or equivalent that will work in both desktop and ASP.NET applications).
The mask image will contain green keys where the image should be 'painted'.
(Desired Result image below is not perfect, hand lasso'd...)

There are various techniques for this:
Scan pixel data and build a mask image (as already suggested by itsme86 and Moby Disk)
A variant of scanning that builds a clipping region from the mask and uses that when drawing (refer to this article by Bob Powell)
Use color keys to mask in the Graphics.DrawImage call.
I'll focus on the third option.
Assuming that the image color that you want to eliminate from your mask is Color.Lime, we can use ImageAttributes.SetColorKey to stop any of that color from being drawn during a call to Graphics.DrawImage like this:
using (Image background = Bitmap.FromFile("tree.png"))
using (Image masksource = Bitmap.FromFile("mask.png"))
using (var imgattr = new ImageAttributes())
{
// set color key to Lime
imgattr.SetColorKey(Color.Lime, Color.Lime);
// Draw non-lime portions of mask onto original
using (var g = Graphics.FromImage(background))
{
g.DrawImage(
masksource,
new Rectangle(0, 0, masksource.Width, masksource.Height),
0, 0, masksource.Width, masksource.Height,
GraphicsUnit.Pixel, imgattr
);
}
// Do something with the composited image here...
background.Save("Composited.png");
}
And the results:
You can use the same technique (with color key on Color.Fuchsia) if you want to put those bits of tree into another image.

You want something like this:
Bitmap original = new Bitmap(#"tree.jpg");
Bitmap mask = new Bitmap(#"mask.jpg");
int width = original.Width;
int height = original.Height;
// This is the color that will be replaced in the mask
Color key = Color.FromArgb(0,255,0);
// Processing one pixel at a time is slow, but easy to understand
for (int y = 0; y < height; y++)
{
for (int x = 0; x < width; x++)
{
// Is this pixel "green" ?
if (mask.GetPixel(x,y) == key)
{
// Copy the pixel color from the original
Color c = original.GetPixel(x,y);
// Into the mask
mask.SetPixel(x,y,c);
}
}
}

You could probably read in the mask and translate it into an image that has the alpha channel set to 0 when the pixel is green and the alpha channel set to 0xFF when the pixel is any other color. Then you could draw the mask image over the original image.

How to Create an Infrared Filter Effect in C#

I recently came across some very cool pictures from a google image search for infrared, and I'd like to try out some pixel manipulation to create this effect. I do have experience with pixel manipulation, usually with just going from start to end of an image, extracting the argb values, manipulating them, then recreating the pixel and placing it back into the image or a copy. I suppose my question is, how can I find out how to manipulate the rgb values to create such an effect from a regular image?
I would probably be using something of the following to extract argb values from the pixel data while looping through the image's pixels
for (int x = 0; x < width; ++x, ++index)
{
uint currentPixel = sourcePixels[index]; // get the current pixel
uint alpha = (currentPixel & 0xff000000) >> 24; // alpha component
uint red = (currentPixel & 0x00ff0000) >> 16; // red color component
uint green = (currentPixel & 0x0000ff00) >> 8; // green color component
uint blue = currentPixel & 0x000000ff; // blue color component
//Modify pixel values
uint newPixel = (alpha << 24) | (red << 16) | (green << 8) | blue; // reassembling each component back into a pixel
targetPixels[index] = newPixel; // assign the newPixel to the equivalent location in the output image
}
Edit: Sample Picture Below
OR

Unfortunately, the effect of infrared shooting is difficult for reproducing without understanding how different objects on the photo reflect or absorb infrared light. Therein lays the problem that doesn’t allow us to create the universal infrared filter. Although there are some workarounds. We know that leafs and grass usually reflect infrared light more than other objects. Thus, the main objective should be manipulation with a green color (or other color if you want some additional effects).
AForge.Imaging is an open-source .NET library, which might be a good starting point. It provides a variety of filters and you can easily check how each of them was implemented. You can also check examples which come within the project. Another option is to look at the project on Codeproject. I wrote a bit of code in order to show how to use some filters.
public static class ImageProcessing
{
public static Bitmap Process(Bitmap image, IFilter filter)
{
return filter.Apply(image);
}
public static Bitmap Process(string path, IFilter filter)
{
var image = (Bitmap)Image.FromFile(path);
return filter.Apply(image);
}
public static Bitmap Process(string path, IEnumerable<IFilter> filters)
{
var image = (Bitmap)Image.FromFile(path);
foreach (var filter in filters)
{
Bitmap tempImage = filter.Apply(image);
image.Dispose();
image = tempImage;
}
return image;
}
}
Original image (test.jpg)
Application of hue modifier
ImageProcessing.Process("test.jpg", new HueModifier(30))
.Save("result_1.jpg");
Result of hue modifier (result_1.jpg)
Application of saturation correction
ImageProcessing.Process("test.jpg", new SaturationCorrection(0.35f))
.Save("result_2.jpg");
Result of saturation correction (result_2.jpg)
Application of filter chain
ImageProcessing.Process("test.jpg"
,new List<IFilter>() {
new BrightnessCorrection(),
new SaturationCorrection(0.1f),
new HueModifier(300)})
.Save("result_3.jpg");
Result of filter chain (result_3.jpg)

Finding an Image Inside Another Image

I'm trying to build an application that solves a puzzle (trying to develop a graph algorithm), and I don't want to enter sample input by hand all the time.
Edit: I'm not trying to build a game. I'm trying to build an agent which plays the game "SpellSeeker"
Say I have an image (see attachment) on the screen with numbers in it, and I know the locations of the boxes, and I have the exact images for these numbers. What I want to do is simply tell which image (number) is on the corresponding box.
So I guess I need to implement
bool isImageInsideImage(Bitmap numberImage,Bitmap Portion_Of_ScreenCap) or something like that.
What I've tried is (using AForge libraries)
public static bool Contains(this Bitmap template, Bitmap bmp)
{
const Int32 divisor = 4;
const Int32 epsilon = 10;
ExhaustiveTemplateMatching etm = new ExhaustiveTemplateMatching(0.9f);
TemplateMatch[] tm = etm.ProcessImage(
new ResizeNearestNeighbor(template.Width / divisor, template.Height / divisor).Apply(template),
new ResizeNearestNeighbor(bmp.Width / divisor, bmp.Height / divisor).Apply(bmp)
);
if (tm.Length == 1)
{
Rectangle tempRect = tm[0].Rectangle;
if (Math.Abs(bmp.Width / divisor - tempRect.Width) < epsilon
&&
Math.Abs(bmp.Height / divisor - tempRect.Height) < epsilon)
{
return true;
}
}
return false;
}
But it returns false when searching for a black dot in this image.
How can I implement this?

I'm answering my question since I've found the solution:
this worked out for me:
System.Drawing.Bitmap sourceImage = (Bitmap)Bitmap.FromFile(#"C:\SavedBMPs\1.jpg");
System.Drawing.Bitmap template = (Bitmap)Bitmap.FromFile(#"C:\SavedBMPs\2.jpg");
// create template matching algorithm's instance
// (set similarity threshold to 92.5%)
ExhaustiveTemplateMatching tm = new ExhaustiveTemplateMatching(0.921f);
// find all matchings with specified above similarity
TemplateMatch[] matchings = tm.ProcessImage(sourceImage, template);
// highlight found matchings
BitmapData data = sourceImage.LockBits(
new Rectangle(0, 0, sourceImage.Width, sourceImage.Height),
ImageLockMode.ReadWrite, sourceImage.PixelFormat);
foreach (TemplateMatch m in matchings)
{
Drawing.Rectangle(data, m.Rectangle, Color.White);
MessageBox.Show(m.Rectangle.Location.ToString());
// do something else with matching
}
sourceImage.UnlockBits(data);
The only problem was it was finding all (58) boxes for said game. But changing the value 0.921f to 0.98 made it perfect, i.e. it finds only the specified number's image (template)
Edit: I actually have to enter different similarity thresholds for different pictures. I found the optimized values by trying, in the end I have a function like
float getSimilarityThreshold(int number)

A better approach is to build a custom class which holds all the information you need instead of relying on the image itself.
For example:
public class MyTile
{
public Bitmap TileBitmap;
public Location CurrentPosition;
public int Value;
}
This way you can "move around" the tile class and read the value from the Value field instead of analyzing the image. You just draw whatever image the class hold to the position it's currently holding.
You tiles can be held in an array like:
private list<MyTile> MyTiles = new list<MyTile>();
Extend class as needed (and remember to Dispose those images when they are no longer needed).
if you really want to see if there is an image inside the image, you can check out this extension I wrote for another post (although in VB code):
Vb.Net Check If Image Existing In Another Image