We Keep Coding

How to detect the bullet holes on the target using open cv in c#

I am trying to identify the holes in the target and score them accordingly. I have tried to find contours and it does a lot of work but it did not give me the 100% result. Sometimes it gives me an accurate result and sometimes it misses some bullets. I do not know how to do it. I am new in open CV and image processing. May be its due to the live streaming of the camera and light frequency. Kindly help me to solve this problem.
Details of my target
top is 6 feet from the ground surface
camera is 1 feet from the ground level
Target image
Image With Holes
Gray Scale Image
Here is my code to get video from the camera:
private void button1_Click(object sender, EventArgs e)
{
if (capture == null)
{
Cursor.Current = Cursors.WaitCursor;
//capture = new Capture(0);
capture = new Capture("rtsp://admin:admin123#192.168.1.64:554/live.avi");
capture.ImageGrabbed += Capture_ImageGrabbed;
capture.Start();
Cursor.Current = Cursors.Default;
}
index = 0;
if (index < panlist.Count)
{
panlist[++index].BringToFront();
}
CamPnelList[0].BackColor = Color.Red;
Rifle = true;
}
private void Capture_ImageGrabbed(object sender, EventArgs e)
{
try
{
Mat m = new Mat();
capture.Retrieve(m);
imginpt = m.ToImage<Gray, byte>();
RecImg = m.ToImage<Rgb, byte>();
if (rec.X != 0 && rec.Y != 0 && CamPnelList[0].BackColor == Color.LightGreen)
{
imginpt.ROI = rec;
RecImg.ROI = rec;
imgout1 = new Image<Gray, byte>(imginpt.Width, imginpt.Height, new Gray(0));
imgout1 = imginpt.Convert<Gray, byte>().ThresholdBinary(new Gray(100), new Gray(255));
imginpt.ROI = Rectangle.Empty;
tempimg1 = imgout1.CopyBlank();
imgout1.CopyTo(tempimg1);
cam1pictureBox.Image = imgout1.Bitmap;
//Application.DoEvents();
}
else
{
cam1pictureBox.Image = imginpt.Bitmap;
}
//System.Threading.Thread.Sleep(50);
}
catch (Exception x)
{
// MessageBox.Show(x.ToString());
}
}
Here is how I am extracting contours:
contoursimg1 = new Image<Gray, byte>(tempimg1.Width, tempimg1.Height, new Gray(0));
Emgu.CV.Util.VectorOfVectorOfPoint contours = new Emgu.CV.Util.VectorOfVectorOfPoint();
Mat Hier = new Mat();
CvInvoke.FindContours(tempimg1, contours, Hier, Emgu.CV.CvEnum.RetrType.Tree, Emgu.CV.CvEnum.ChainApproxMethod.ChainApproxSimple);
CvInvoke.DrawContours(contoursimg1, contours, -1, new MCvScalar(255, 0, 0));

I've done a few similar projects using video as a source and when the target objects have been small but a fairly well defined size the I've taken the difference between frames and used blob detection which is a good fast algorithm to use when dealing with real-time video. I noticed the perspective seems to have changed a little between your two example shots so rather than do that I tried the following code:
const int blobSizeMin = 1;
const int blobSizeMax = 5;
var white = new Bgr(255, 255, 255).MCvScalar;
Mat frame = CvInvoke.Imread(#"e:\temp\Frame.jpg", ImreadModes.Grayscale);
Mat mask = CvInvoke.Imread(#"e:\temp\Mask.jpg", ImreadModes.Grayscale);
frame.CopyTo(frame = new Mat(), mask);
CvInvoke.BitwiseNot(frame, frame);
CvInvoke.Threshold(frame, frame, 128, 255, ThresholdType.ToZero);
var blobs = new Emgu.CV.Cvb.CvBlobs();
var blobDetector = new Emgu.CV.Cvb.CvBlobDetector();
Image<Gray, Byte> img = frame.ToImage<Gray, Byte>();
blobDetector.Detect(img, blobs);
int bulletNumber = 0;
foreach (var blob in blobs.Values)
{
if (blob.BoundingBox.Width >= blobSizeMin && blob.BoundingBox.Width <= blobSizeMax
&& blob.BoundingBox.Height >= blobSizeMin && blob.BoundingBox.Height <= blobSizeMax)
{
bulletNumber++;
Point textPos = new Point((int) blob.Centroid.X - 1, (int) blob.Centroid.Y - 1);
CvInvoke.PutText(frame, bulletNumber.ToString(), textPos, FontFace.HersheyPlain,
fontScale: 1, color: white);
}
}
CvInvoke.Imwrite(#"e:\temp\Out.png", frame);
It inverts the frame so the holes are white, discards values below 50% and then does blob detection only taking notice of blobs with a size between one and five pixels. That was close to working but picked up a few extra points at the top left & right and bottom-left that look pretty similar to bullet holes to the eye as well. What I've done in the past that works well when you mount the camera in a fixed location is to have a black & white mask image to remove anything outside the area of interest:
Mask.jpg
Once that was added I detected a total of 21 bullet holes which looks correct:
Out.png
But assuming you're detecting the shots in real-time I think you should have good luck with looking at the difference between frames and that should remove the need to use a mask image. Take a look at the CvInvoke.Subtract method, from some existing code you can use something like the following:
CvInvoke.Subtract(frame, lastFrame, diff);
CvInvoke.CvtColor(diff, gray, ColorConversion.Bgr2Gray);
CvInvoke.Threshold(gray, gray, detectThreshold, 255, ThresholdType.ToZero);

Rasterize wpf textblock into a bitmap via drawingcontext

My program is sort of copy version of MS paint and Pickpick.
and one of features is rasterizing the selected object such as textblock or shape.
Regarding the selectable object, in order to resize and move with adorner,
it has 1 ContentControl which comprise 1 textblock + 1 shape.
ContentControl (able to resize, rotate, move)
└─> Textblock (bold, italic, V-align, H-align, word wrap...)
└─> Shape (can be a triangle, rectangle etc...)
It was not hard to convert to draw the shape with drawing context instead of render at Canvas.
var SH = CC.GetShape();
var TB = CC.GetTextBlock();
var visual = new DrawingVisual();
Geometry geo = null;
System.Windows.Media.Pen pen = null;
System.Windows.Media.Brush brush = null;
if (SH != null)
{
geo = SH.RenderedGeometry; // shape to geo
if (geo == null)
return;
pen = new System.Windows.Media.Pen(SH.Stroke, SH.StrokeThickness);
brush = SH.Fill;
}
using (var dc = visual.RenderOpen())
{
// Draw the background first
dc.DrawImage(first, new Rect(0, 0, first.Width, first.Height));
dc.PushTransform(new TranslateTransform(left, top));
// Draw the shape
if (SH != null && geo != null)
dc.DrawGeometry(brush, pen, geo);
}
But while drawing Textblock with drawing context,
I've referred below link to calculate the position of Textblock
Vertical alignment with DrawingContext.DrawText
but the problem is when the Textblock has multiline or word wrapped.
screenshot of my program
if (TB.Text.Equals(string.Empty) == false)
{
var typeface = new Typeface(CC.txtSetting.fontFamily,
CC.txtSetting.fontStyle,
CC.txtSetting.fontWeight,
FontStretches.Normal);
var formattedText = new FormattedText(TB.Text
, CultureInfo.CurrentCulture
, FlowDirection.LeftToRight
, typeface
, CC.txtSetting.fontSize
, new SolidColorBrush(CC.txtSetting.fontColor));
double centerX = CC.ActualWidth / 2;
double centerY = CC.ActualHeight / 2;
double txtPositionX = 0.0f;
double txtPositionY = 0.0f;
if (TB.TextAlignment == TextAlignment.Left)
{
txtPositionX = 1.0f;
}
else if (TB.TextAlignment == TextAlignment.Center)
{
txtPositionX = centerX - formattedText.WidthIncludingTrailingWhitespace / 2;
}
else if (TB.TextAlignment == TextAlignment.Right)
{
txtPositionX = CC.Width -
formattedText.WidthIncludingTrailingWhitespace - 1.0f;
}
if (TB.VerticalAlignment == VerticalAlignment.Top)
{
txtPositionY = 1.0f;
}
else if (TB.VerticalAlignment == VerticalAlignment.Center)
{
txtPositionY = centerY - formattedText.Height / 2;
}
else if (TB.VerticalAlignment == VerticalAlignment.Bottom)
{
txtPositionY = CC.Height - formattedText.Height - 1.0f;
}
var ptLocation = new System.Windows.Point(txtPositionX, txtPositionY);
dc.DrawText(formattedText, ptLocation);
}
Additionally, the textblock is wrapped by ContentControl so depending on user change the property of textblock, it will vary so much.
I guess it seems not possible to convert every variable.
So, I'm thinking alternative ways to draw.
Draw with GDI+ instead of drawing with drawing context. (still uncertain)
Use drawing context while the user is editing the text. (so it'll be the same before rasterizing and vice-versa)
Any way to directly convert/capture the Textblock into an image or Geometry? (it would be the best way if it's possible.)
For example, to get a shader effect applied image source, I did like this. so.. probably there's the way.
How can I get the object of effect-applied source
You can also refer to this program from http://ngwin.com/picpick
screenshot of picpick
Any better ideas? Thank you in advance.

I made it!
I could capture the particular control with RenderTargetBimap. Since ContentControl is a part of Visual Element.
CustomControl is inherited control from ContentControl.
public static BitmapSource ControlToBitmap(CustomControl control)
{
int W = (int)control.ActualWidth;
int H = (int)control.ActualHeight;
RenderTargetBitmap renderBitmap = new RenderTargetBitmap(
W, H,
96d, 96d, PixelFormats.Pbgra32);
// needed otherwise the image output is black
control.Measure(new System.Windows.Size(W, H));
control.Arrange(new Rect(new System.Windows.Size(W, H)));
renderBitmap.Render(control);
var BS = RenderTargetBitmapToBitmap(renderBitmap);
return BS;
}
Additionally, I had to deal with the angle. Because I couldn't capture the angled control directly. but my idea is
back up the angle value first.
And restore the control to be non-rotated(RotateTransform = 0.0)
Capture a non-rotated control to a bitmap.
Then rotate the captured bitmap again.
Combine both bitmaps into one.
public static void OverlayControl(ImageSource first, CustomControl CC)
{
if (CC == null)
return;
var visual = new DrawingVisual();
double left = Canvas.GetLeft(CC);
double top = Canvas.GetTop(CC);
// Get control's angle.
double rotationInDegrees = 0.0f;
RotateTransform rotation = CC.RenderTransform as RotateTransform;
if (rotation != null) // Make sure the transform is actually a RotateTransform
{
rotationInDegrees = rotation.Angle; // back up this to temp var.
rotation.Angle = 0.0f; // Set this to 0.0 to capture properly.
}
var second = ControlToBitmap(CC);
using (var dc = visual.RenderOpen())
{
// Draw the background image frist.
dc.DrawImage(first, new Rect(0, 0, first.Width, first.Height));
// Push angle if the control has rotated.
if (rotationInDegrees != 0.0f)
dc.PushTransform(new RotateTransform(rotationInDegrees, left + (CC.Width / 2), top + (CC.Height / 2)));
// transfrom as much as control moved from the origin.
dc.PushTransform(new TranslateTransform(left, top));
// Draw the second image. (captured image from the control)
dc.DrawImage(second, new Rect(0, 0, second.Width, second.Height));
// pop transforms
dc.Pop();
}
var rtb = new RenderTargetBitmap((int)first.Width, (int)first.Height,
96, 96, PixelFormats.Default);
rtb.Render(visual);
// Set as a one combined image.
MainWindow.VM.RenderedImage = rtb;
}
Now, everything seems alright.

Eye detection using OpenCVSharp in Unity (fps issues)

I'm currently working on a project involving integrating OpenCVSharp into Unity, to allow eye tracking within a game environment. I've managed to get OpenCVSharp integrated into the Unity editor and currently have eye-detection (not tracking) working within a game. It can find your eyes within a webcam image, then display where its currently detected them on a texture, which I display within the scene.
However its causing a HUGE fps drop, mainly because every frame its converting a webcam texture into an IPLimage so that OpenCV can handle it. It then has to convert it back to a 2Dtexture to be displayed within the scene, after its done all the eye detection. So understandably its too much for the CPU to handle. (As far as I can tell its only using 1 core on my CPU).
Is there a way to do all the eye detection without converting the texture to an IPLimage? Or any other way to fix the fps drop. Some things that I've tried include:
Limiting the frames that it updates on. However this just causes it
to run smoothly, then stutter horribly on the frame that it has to
update.
Looking at threading, but as far as I'm aware Unity doesn't allow it.
As far as I can tell its only using 1 core on my CPU which seems a bit silly. If there was a way to change this it could fix the issue?
Tried different resolutions on the camera, however the resolution that the game can actually run smoothly at, is too small for the eye's to actually be detected, let alone tracked.
I've included the code below, of if you would prefer to look at it in a code editor here is a link to the C# File. Any suggestions or help would be greatly appreciated!
For reference I used code from here (eye detection using opencvsharp).
using UnityEngine;
using System.Collections;
using System;
using System.IO;
using OpenCvSharp;
//using System.Xml;
//using OpenCvSharp.Extensions;
//using System.Windows.Media;
//using System.Windows.Media.Imaging;
public class CaptureScript : MonoBehaviour
{
public GameObject planeObj;
public WebCamTexture webcamTexture; //Texture retrieved from the webcam
public Texture2D texImage; //Texture to apply to plane
public string deviceName;
private int devId = 1;
private int imWidth = 640; //camera width
private int imHeight = 360; //camera height
private string errorMsg = "No errors found!";
static IplImage matrix; //Ipl image of the converted webcam texture
CvColor[] colors = new CvColor[]
{
new CvColor(0,0,255),
new CvColor(0,128,255),
new CvColor(0,255,255),
new CvColor(0,255,0),
new CvColor(255,128,0),
new CvColor(255,255,0),
new CvColor(255,0,0),
new CvColor(255,0,255),
};
const double Scale = 1.25;
const double ScaleFactor = 2.5;
const int MinNeighbors = 2;
// Use this for initialization
void Start ()
{
//Webcam initialisation
WebCamDevice[] devices = WebCamTexture.devices;
Debug.Log ("num:" + devices.Length);
for (int i=0; i<devices.Length; i++) {
print (devices [i].name);
if (devices [i].name.CompareTo (deviceName) == 1) {
devId = i;
}
}
if (devId >= 0) {
planeObj = GameObject.Find ("Plane");
texImage = new Texture2D (imWidth, imHeight, TextureFormat.RGB24, false);
webcamTexture = new WebCamTexture (devices [devId].name, imWidth, imHeight, 30);
webcamTexture.Play ();
matrix = new IplImage (imWidth, imHeight, BitDepth.U8, 3);
}
}
void Update ()
{
if (devId >= 0)
{
//Convert webcam texture to iplimage
Texture2DtoIplImage();
/*DO IMAGE MANIPULATION HERE*/
//do eye detection on iplimage
EyeDetection();
/*END IMAGE MANIPULATION*/
if (webcamTexture.didUpdateThisFrame)
{
//convert iplimage to texture
IplImageToTexture2D();
}
}
else
{
Debug.Log ("Can't find camera!");
}
}
void EyeDetection()
{
using(IplImage smallImg = new IplImage(new CvSize(Cv.Round (imWidth/Scale), Cv.Round(imHeight/Scale)),BitDepth.U8, 1))
{
using(IplImage gray = new IplImage(matrix.Size, BitDepth.U8, 1))
{
Cv.CvtColor (matrix, gray, ColorConversion.BgrToGray);
Cv.Resize(gray, smallImg, Interpolation.Linear);
Cv.EqualizeHist(smallImg, smallImg);
}
using(CvHaarClassifierCascade cascade = CvHaarClassifierCascade.FromFile (#"C:\Users\User\Documents\opencv\sources\data\haarcascades\haarcascade_eye.xml"))
using(CvMemStorage storage = new CvMemStorage())
{
storage.Clear ();
CvSeq<CvAvgComp> eyes = Cv.HaarDetectObjects(smallImg, cascade, storage, ScaleFactor, MinNeighbors, 0, new CvSize(30, 30));
for(int i = 0; i < eyes.Total; i++)
{
CvRect r = eyes[i].Value.Rect;
CvPoint center = new CvPoint{ X = Cv.Round ((r.X + r.Width * 0.5) * Scale), Y = Cv.Round((r.Y + r.Height * 0.5) * Scale) };
int radius = Cv.Round((r.Width + r.Height) * 0.25 * Scale);
matrix.Circle (center, radius, colors[i % 8], 3, LineType.AntiAlias, 0);
}
}
}
}
void OnGUI ()
{
GUI.Label (new Rect (200, 200, 100, 90), errorMsg);
}
void IplImageToTexture2D ()
{
int jBackwards = imHeight;
for (int i = 0; i < imHeight; i++) {
for (int j = 0; j < imWidth; j++) {
float b = (float)matrix [i, j].Val0;
float g = (float)matrix [i, j].Val1;
float r = (float)matrix [i, j].Val2;
Color color = new Color (r / 255.0f, g / 255.0f, b / 255.0f);
jBackwards = imHeight - i - 1; // notice it is jBackward and i
texImage.SetPixel (j, jBackwards, color);
}
}
texImage.Apply ();
planeObj.renderer.material.mainTexture = texImage;
}
void Texture2DtoIplImage ()
{
int jBackwards = imHeight;
for (int v=0; v<imHeight; ++v) {
for (int u=0; u<imWidth; ++u) {
CvScalar col = new CvScalar ();
col.Val0 = (double)webcamTexture.GetPixel (u, v).b * 255;
col.Val1 = (double)webcamTexture.GetPixel (u, v).g * 255;
col.Val2 = (double)webcamTexture.GetPixel (u, v).r * 255;
jBackwards = imHeight - v - 1;
matrix.Set2D (jBackwards, u, col);
//matrix [jBackwards, u] = col;
}
}
}
}

You can move these out of the per frame update loop :
using(CvHaarClassifierCascade cascade = CvHaarClassifierCascade.FromFile (#"C:\Users\User\Documents\opencv\sources\data\haarcascades\haarcascade_eye.xml"))
using(CvMemStorage storage = new CvMemStorage())
No reason to be building the recognizer graph each frame.
Threading is the logical way to go moving forward if you want real speed updates, unity itself is not threaded, but you can fold in other threads if your careful.
Do the texture -> ipl image on the main thread then trigger an event to fire off your thread.
The thread can do all the CV work, probably construct the tex2d and then push back to main to render.

You should also be able to gain some performance improvements if you use:
Color32[] pixels;
pixels = new Color32[webcamTexture.width * webcamTexture.height];
webcamTexture.GetPixels32(pixels);
The Unity doco suggests that this can be quite a bit faster than calling "GetPixels" (and certainly faster than calling GetPixel for each pixel), and then you don't need to scale each RGB channel against 255 manually.

Image-map click events in zoomable image

I've got a large zoomable* image (loaded once using canvas nested in a ViewportControl) and I want to implement the following functionality:
The image should have several clickable areas attached that move along the image when the user zooms in/out, pinch, moves around etc.
Think about an image of a map from which I should get click events from specific areas.
I haven't figured out whether these areas should be re-attached when the user zooms or if I could attach them once.
Any ideas?
*I used the pinch and zoom code from this sample: http://code.msdn.microsoft.com/wpapps/Image-Recipes-0c0b8fee
edit:
I'm trying to make a single temp button work using the DeltaScale property of the zoom functionality.
These are two main function:
void OnManipulationDelta(object sender, ManipulationDeltaEventArgs e)
{
if (e.PinchManipulation != null)
{
e.Handled = true;
if (!_pinching)
{
_pinching = true;
Point center = e.PinchManipulation.Original.Center;
_relativeMidpoint = new Point(center.X / TestImage.ActualWidth, center.Y / TestImage.ActualHeight);
var xform = TestImage.TransformToVisual(viewport);
_screenMidpoint = xform.Transform(center);
}
_scale = _originalScale * e.PinchManipulation.CumulativeScale;
// sample double variables for storing the changes of scaling
// initial values of mytop & myleft are the initial actual spot of tempbtn on the image
mytop = e.PinchManipulation.DeltaScale * mytop;
myleft = e.PinchManipulation.DeltaScale * myleft;
CoerceScale(false);
ResizeImage(false);
}
else if (_pinching)
{
_pinching = false;
_originalScale = _scale = _coercedScale;
}
}
void OnManipulationCompleted(object sender, ManipulationCompletedEventArgs e)
{
_pinching = false;
_scale = _coercedScale;
Canvas.SetTop(tempbtn, mytop);
Canvas.SetLeft(tempbtn, myleft);
}
The resizing function for the main image in the background:
void ResizeImage(bool center)
{
if (_coercedScale != 0 && _bitmap != null)
{
double newWidth = canvas.Width = Math.Round(_bitmap.PixelWidth * _coercedScale);
double newHeight = canvas.Height = Math.Round(_bitmap.PixelHeight * _coercedScale);
xform.ScaleX = xform.ScaleY = _coercedScale;
viewport.Bounds = new Rect(0, 0, newWidth, newHeight);
if (center)
{
viewport.SetViewportOrigin(
new Point(
Math.Round((newWidth - viewport.ActualWidth) / 2),
Math.Round((newHeight - viewport.ActualHeight) / 2)
));
}
else
{
Point newImgMid = new Point(newWidth * _relativeMidpoint.X, newHeight * _relativeMidpoint.Y);
Point origin = new Point(newImgMid.X - _screenMidpoint.X, newImgMid.Y - _screenMidpoint.Y);
viewport.SetViewportOrigin(origin);
}
}
}
The problem is that my button always stands a little bit (around 10px, gets more as I zoom) to the top & left of the actual place I want it positioned. Something causes this deviation but I can't find out where's the problem. Any guesses/suggestions?

XNA Get Mesh value of Y at value of X Reply

I am having trouble trying to get a loaded meshes value of Y at a value of X in order to perform some very limited version of detection. Essentially, I am going to get the value of X of a camera and detect if the Y value of my mesh is 5 for example. If so.. there is a wall there.
I load my model with this:
landscape = Content.Load("landscape");
I draw the model with this:
foreach (ModelMesh mesh in landscape.Meshes)
{
if (mesh.Name != "Billboards")
{
foreach (BasicEffect effect in mesh.Effects)
{
effect.View = view;
effect.Projection = projection;
effect.LightingEnabled = true;
effect.DirectionalLight0.Enabled = true;
effect.DirectionalLight0.Direction = lightDirection;
effect.DirectionalLight0.DiffuseColor = lightColor;
//if (flashEnabled == true)
//{
effect.DirectionalLight1.Enabled = flashEnabled;
effect.DirectionalLight1.Direction = cameraFront;
effect.DirectionalLight1.DiffuseColor = lightColor;
effect.DirectionalLight1.SpecularColor = colorFlashLight.ToVector3();
//}
effect.AmbientLightColor = ambientLightColor;
effect.FogEnabled = fogEnabled;
effect.FogColor = color.ToVector3();
effect.FogStart = 9.75f;
effect.FogEnd = 10.25f;
}
device.BlendState = BlendState.Opaque;
device.DepthStencilState = DepthStencilState.Default;
device.RasterizerState = RasterizerState.CullCounterClockwise;
mesh.Draw();
}
So moving on, in my update or input functions I would run an evaluative function to determine if LandscapeVertexY#CameraX is greater than a value of 5.
Any help?

If i understand your question correctly you are trying to retrieve the rendered position of a vertex with by using a specified camera.
In this case you need to project the position of the vertex manually in your coordinate space by using matrix multiplication. This could be done by multiplying the position with your model-view-projection matrix. ( It can be done with the XNA matrix and vectors ).