We Keep Coding

How to capture frames from a webcam with SharpDX

I'm trying to implement a webcam capture app which should take still frames, display them on the screen and save to the disk.
Since I'm using SharpDX already to capture the screen, I thought it would be nice to use that library. I was not sure if SharpDX had any video capture capabilities, so I started searching and found parts of what it looks like a webcam capture prototype:
var attributes = new MediaAttributes(1);
attributes.Set<Guid>(CaptureDeviceAttributeKeys.SourceType, CaptureDeviceAttributeKeys.SourceTypeVideoCapture.Guid);
var activates = MediaFactory.EnumDeviceSources(attributes);
var dic = new Dictionary<string, Activate>();
foreach (var activate in activates)
{
var uid = activate.Get(CaptureDeviceAttributeKeys.SourceTypeVidcapSymbolicLink);
dic.Add(uid, activate);
}
var camera = dic.First().Value;
It outputs camera with a strange uid. I'm not sure if it's correct.
What I am supposed to do after this?
Edit
I got this code kind of working. I still don't understand why the output is strange.
var attributes = new MediaAttributes(1);
attributes.Set(CaptureDeviceAttributeKeys.SourceType.Guid, CaptureDeviceAttributeKeys.SourceTypeVideoCapture.Guid);
var mediaSource = MediaFactory.EnumDeviceSources(attributes)[0].ActivateObject<MediaSource>();
mediaSource.CreatePresentationDescriptor(out var presentationDescriptor);
var reader = new SourceReader(mediaSource);
var mediaTypeIndex = 0;
int width, height;
using (var mt = reader.GetNativeMediaType(0, mediaTypeIndex))
{
UnpackLong(mt.Get(MediaTypeAttributeKeys.FrameSize), out width, out height);
UnpackLong(mt.Get(MediaTypeAttributeKeys.FrameRate), out var frameRateNumerator, out var frameRateDenominator);
UnpackLong(mt.Get(MediaTypeAttributeKeys.PixelAspectRatio), out var aspectRatioNumerator, out var aspectRatioDenominator);
}
var sample = reader.ReadSample(SourceReaderIndex.AnyStream, SourceReaderControlFlags.None, out var readStreamIndex, out var readFlags, out var timestamp);
if (sample == null)
sample = reader.ReadSample(SourceReaderIndex.AnyStream, SourceReaderControlFlags.None, out readStreamIndex, out readFlags, out timestamp);
var sourceBuffer = sample.GetBufferByIndex(0); // sample.ConvertToContiguousBuffer();
var sourcePointer = sourceBuffer.Lock(out var maxLength, out var currentLength);
var data = new byte[sample.TotalLength];
Marshal.Copy(sourcePointer, data, 0, sample.TotalLength);
var newData = new byte[width * 4 * height];
var partWidth = width / 4;
var partHeight = height / 3;
for (var i = 0; i < sample.TotalLength; i += 4)
{
//X8R8B8G8 -> BGRA = 4
newData[i] = data[i + 3];
newData[i + 1] = data[i + 2];
newData[i + 2] = data[i + 1];
newData[i + 3] = 255; //data[i];
}
//var source = BitmapSource.Create(width, height, 96, 96, PixelFormats.Bgra32, null, data, ((width * 24 + 31) / 32) * 4);
var source = BitmapSource.Create(width, height, 96, 96, PixelFormats.Bgra32, null, newData, width * 4);
sourceBuffer.Unlock();
sourceBuffer.Dispose();
The output image is this (I was showing a color spectrum to my webcam):
The image is repeating 4 times, each part has a grayscale image and a color version with half the height.
Two thirds of the image is transparent.

your output is NV12, here's some sample code to convert nv12 to rgb
unsafe private static void TransformImage_NV12(IntPtr pDest, int lDestStride, IntPtr pSrc, int lSrcStride, int dwWidthInPixels, int dwHeightInPixels)
{
uint imageWidth = (uint)dwWidthInPixels;
uint widthHalf = imageWidth / 2;
uint imageHeight = (uint)dwHeightInPixels;
byte* nv12Data = (byte*)pSrc;
byte* rgbData = (byte*)pDest;
uint dataSize = imageWidth * imageHeight * 3;
for (uint y = 0; y < imageHeight; y++)
{
for (uint x = 0; x < imageWidth; x++)
{
uint xEven = x & 0xFFFFFFFE;
uint yEven = y & 0xFFFFFFFE;
uint yIndex = y * imageWidth + x;
uint cIndex = imageWidth * imageHeight + yEven * widthHalf + xEven;
byte yy = nv12Data[yIndex];
byte cr = nv12Data[cIndex + 0];
byte cb = nv12Data[cIndex + 1];
uint outputIndex = (dataSize - (y * imageWidth + x) * 3) - 3;
rgbData[outputIndex + 0] = (byte)Math.Min(Math.Max((yy + 1.402 * (cr - 128)), 0), 255);
rgbData[outputIndex + 1] = (byte)Math.Min(Math.Max((yy - 0.344 * (cb - 128) - 0.714 * (cr - 128)), 0), 255);
rgbData[outputIndex + 2] = (byte)Math.Min(Math.Max((yy + 1.772 * (cb - 128)), 0), 255);
}
}
}

Bitmap Alpha Channel

I'm working on a simple drawing application where I can 'paint' on top of an existing image. I've made a little headway, but I've noticed a weird issue with the alpha channel of the bitmap I'm displaying that doesn't seem quite right. My draw brush function looks like this:
public unsafe void BitmapDrawBrush(double _x, double _y, double _radius, double _falloff, double _strength)
{
if (DisplayBmp == null)
{
DisplayBmp = new Bitmap(ImageWidth, ImageHeight, PixelFormat.Format32bppArgb);
}
const int pixelSize = 4; // 32 bits per pixel
Bitmap target = new Bitmap(DisplayBmp.Width, DisplayBmp.Height, PixelFormat.Format32bppArgb);
BitmapData sourceData = null, targetData = null;
try
{
sourceData = DisplayBmp.LockBits(new Rectangle(0, 0, DisplayBmp.Width, DisplayBmp.Height),ImageLockMode.ReadOnly, PixelFormat.Format32bppArgb);
targetData = target.LockBits(new Rectangle(0, 0, target.Width, target.Height),ImageLockMode.WriteOnly, PixelFormat.Format32bppArgb);
for (int y = 0; y < DisplayBmp.Height; ++y)
{
byte* sourceRow = (byte*)sourceData.Scan0 + (y * sourceData.Stride);
byte* targetRow = (byte*)targetData.Scan0 + (y * targetData.Stride);
for (int x = 0; x < DisplayBmp.Width; ++x)
{
byte b = sourceRow[x * pixelSize + 0];
byte g = sourceRow[x * pixelSize + 1];
byte r = sourceRow[x * pixelSize + 2];
byte a = sourceRow[x * pixelSize + 3];
double nx = x / (double)ImageWidth;
double ny = y/(double)ImageHeight;
double xDist = nx - _x;
double yDist = ny - _y;
if ((xDist * xDist) + (yDist * yDist) <= (_radius * _radius))
{
double pxDist = 1.0 - (((xDist * xDist) + (yDist * yDist)) / (_radius * _radius));
r = (byte)(255 * pxDist);
g = (byte)(255 * pxDist);
b = (byte)(255 * pxDist);
a = (byte)(255 * pxDist * _strength); // <-the alpha channel value
}
targetRow[x * pixelSize + 0] = b;
targetRow[x * pixelSize + 1] = g;
targetRow[x * pixelSize + 2] = r;
targetRow[x * pixelSize + 3] = a;
}
}
}
finally
{
if (sourceData != null)
DisplayBmp.UnlockBits(sourceData);
if (targetData != null)
target.UnlockBits(targetData);
}
DisplayBmp = target;
UpdateBitmap();
}
Yet, when I run my application and place a few brush strokes (actually the draw brush function is only called on mouse down at the moment so I'm only drawing dots really), you see that the alpha channel doesn't seem to be fading toward the edge. What I would expect is that as the 'dot' gets more black, so too does the alpha channel diminish. See below. Any ideas as to why this is happening?

C# scanning image code improvement

I'm working on a screen sharing app, which runs a loop and grab fast screenshots using GDI methods . example here
Of course I also use a flood fill algorithm to find the changes areas between 2 images (previous screenshot and current).
I use another small trick - I downscale the snapshot resolution in 10, because processing 1920*1080=2073600 pixels very constantly is not very efficient.
However when I find the rectangle bounds - I apply it on the original full size bitmap and I just multiply by 10 the dimension (including top, left, width, height).
This is the scanning code:
unsafe bool ArePixelsEqual(byte* p1, byte* p2, int bytesPerPixel)
{
for (int i = 0; i < bytesPerPixel; ++i)
if (p1[i] != p2[i])
return false;
return true;
}
private unsafe List<Rectangle> CodeImage(Bitmap bmp, Bitmap bmp2)
{
List<Rectangle> rec = new List<Rectangle>();
var bmData1 = bmp.LockBits(new System.Drawing.Rectangle(0, 0, bmp.Width, bmp.Height), System.Drawing.Imaging.ImageLockMode.ReadOnly, bmp.PixelFormat);
var bmData2 = bmp2.LockBits(new System.Drawing.Rectangle(0, 0, bmp.Width, bmp.Height), System.Drawing.Imaging.ImageLockMode.ReadOnly, bmp2.PixelFormat);
int bytesPerPixel = 4;
IntPtr scan01 = bmData1.Scan0;
IntPtr scan02 = bmData2.Scan0;
int stride1 = bmData1.Stride;
int stride2 = bmData2.Stride;
int nWidth = bmp.Width;
int nHeight = bmp.Height;
bool[] visited = new bool[nWidth * nHeight];
byte* base1 = (byte*)scan01.ToPointer();
byte* base2 = (byte*)scan02.ToPointer();
for (int y = 0; y < nHeight; y ++)
{
byte* p1 = base1;
byte* p2 = base2;
for (int x = 0; x < nWidth; ++x)
{
if (!ArePixelsEqual(p1, p2, bytesPerPixel) && !(visited[x + nWidth * y]))
{
// fill the different area
int minX = x;
int maxX = x;
int minY = y;
int maxY = y;
var pt = new Point(x, y);
Stack<Point> toBeProcessed = new Stack<Point>();
visited[x + nWidth * y] = true;
toBeProcessed.Push(pt);
while (toBeProcessed.Count > 0)
{
var process = toBeProcessed.Pop();
var ptr1 = (byte*)scan01.ToPointer() + process.Y * stride1 + process.X * bytesPerPixel;
var ptr2 = (byte*)scan02.ToPointer() + process.Y * stride2 + process.X * bytesPerPixel;
//Check pixel equality
if (ArePixelsEqual(ptr1, ptr2, bytesPerPixel))
continue;
//This pixel is different
//Update the rectangle
if (process.X < minX) minX = process.X;
if (process.X > maxX) maxX = process.X;
if (process.Y < minY) minY = process.Y;
if (process.Y > maxY) maxY = process.Y;
Point n; int idx;
//Put neighbors in stack
if (process.X - 1 >= 0)
{
n = new Point(process.X - 1, process.Y); idx = n.X + nWidth * n.Y;
if (!visited[idx]) { visited[idx] = true; toBeProcessed.Push(n); }
}
if (process.X + 1 < nWidth)
{
n = new Point(process.X + 1, process.Y); idx = n.X + nWidth * n.Y;
if (!visited[idx]) { visited[idx] = true; toBeProcessed.Push(n); }
}
if (process.Y - 1 >= 0)
{
n = new Point(process.X, process.Y - 1); idx = n.X + nWidth * n.Y;
if (!visited[idx]) { visited[idx] = true; toBeProcessed.Push(n); }
}
if (process.Y + 1 < nHeight)
{
n = new Point(process.X, process.Y + 1); idx = n.X + nWidth * n.Y;
if (!visited[idx]) { visited[idx] = true; toBeProcessed.Push(n); }
}
}
//finaly set a rectangle.
Rectangle r = new Rectangle(minX * 10, minY * 10, (maxX - minX + 1) * 10, (maxY - minY + 1) * 10);
rec.Add(r);
//got the rectangle now i'll do whatever i want with that.
//notify i scaled everything by x10 becuse i want to apply the changes on the originl 1920x1080 image.
}
p1 += bytesPerPixel;
p2 += bytesPerPixel;
}
base1 += stride1;
base2 += stride2;
}
bmp.UnlockBits(bmData1);
bmp2.UnlockBits(bmData2);
return rec;
}
This is my call:
private void Start()
{
full1 = GetDesktopImage();//the first,intial screen.
while (true)
{
full2 = GetDesktopImage();
a = new Bitmap(full1, 192, 108);//resizing for faster processing the images.
b = new Bitmap(full2, 192, 108); // resizing for faster processing the images.
CodeImage(a, b);
count++; // counter for the performance.
full1 = full2; // assign old to current bitmap.
}
}
However, after all the tricks and techniques I used, the algorithm runs quite slow... on my machine - Intel i5 4670k 3.4ghz - it runs only 20 times (at the maximum! It might get lower)! It maybe sounds fast (don't forget I have to send each changed area over the network after), but I'm looking to achieve more processed image per second. I think the main bottleneck is in the resizing of the 2 images - but I just thought it would be even faster after resizing - because it would have to loop through less pixels... 192*108=200,000 only..
I would appreciate any help, any improvement. Thanks.

Image Distortion with Lock Bits

I'm having a problem with writing to files using lock bits. I'm working on an edge detection software which has a strange distortion effect with most images. I've tried to isolate the problem, and it seems very random. It is not associated with format, but rather the only images that seem to work are pictures made for desktop wallpapers, and I don't really know why. I only switched to writing to files using lockbits recently, so I am sure the problem is with that (there were no problems when I was reading with lockbits and writing with set pixel). Here's a screenshot of the effect:
As you can see, the edge detection works, but the image is distorted horizontally, making the image into a parallelogram.
Here's a code snippet of the method that handles all this (in C#):
private void analyze()
{
//When the analyze button is pressed
percentageInt = float.Parse(textBox1.Text);
float scale = 1;
if (comboBox1.SelectedItem == "Auto")
{
scale = pic.Width / pictureBox1.Width;
}
else if (comboBox1.SelectedItem == "1/2")
{
scale = 2;
}
else if (comboBox1.SelectedItem == "1/4")
{
scale = 4;
}
else if (comboBox1.SelectedItem == "Original")
{
scale = 1;
}
else
{
scale = pic.Width / pictureBox1.Width;
}
int tempWidth = 1;
int tempHeight = 1;
if (scale >= 1)
{
tempWidth = (int)Math.Floor(pic.Width / scale);
tempHeight = (int)Math.Floor(pic.Height / scale);
}
else
{
tempWidth = pic.Width;
tempHeight = pic.Height;
}
width = pic.Width;
height = pic.Height;
edgeData = new Boolean[pic.Width, pic.Height];
img = (Bitmap)resizeImage(pic, new Size(tempWidth, tempHeight));
pic2 = new Bitmap(tempWidth, tempHeight);
Bitmap img2 = (Bitmap)pic2;
Color[] pixels = null;
BitmapData data = img.LockBits(new Rectangle(0, 0, img.Width, img.Height),
ImageLockMode.ReadWrite, PixelFormat.Format24bppRgb);
int size = Math.Abs(data.Stride) * img.Height;
Byte[] bytes = new byte[size];
int scaledPercent = (int)(Math.Round(percentageInt * 255));
Debug.WriteLine("percent " + scaledPercent);
unsafe
{
Debug.WriteLine("Woah there, unsafe stuff");
byte* prevLine = (byte*)data.Scan0;
byte* currLine = prevLine + data.Stride;
byte* nextLine = currLine + data.Stride;
for (int y = 1; y < img.Height - 1; y++)
{
byte* pp = prevLine + 3;
byte* cp = currLine + 3;
byte* np = nextLine + 3;
for (int x = 1; x < img.Width - 1; x++)
{
if (IsEdgeOptimized(pp, cp, np, scaledPercent))
{
edgeData[x, y] = true;
//Debug.WriteLine("x " + x + "y " + y);
//img2.SetPixel(x, y, Color.Black);
//bytes[(y * img.Width + x) * 3 + 2] = 255;
}
else
{
bytes[(y * img.Width + x) * 3] = 255;
bytes[(y * img.Width + x) * 3 + 1] = 255;
bytes[(y * img.Width + x) * 3 + 2] = 255;
//img2.SetPixel(x, y, Color.White);
}
pp += 3; cp += 3; np += 3;
}
prevLine = currLine;
currLine = nextLine;
nextLine += data.Stride;
}
}
System.Runtime.InteropServices.Marshal.Copy(bytes, 0, data.Scan0, size);
img.UnlockBits(data);
pictureBox2.Image = img;
} // end analyze
So what is causing the problem, and how can I fix it? If you need more details, feel free to comment.

You're initializing your bytes buffer with stride x height bytes:
int size = Math.Abs(data.Stride) * img.Height;
Byte[] bytes = new byte[size];
But then using the width (instead of stride) when you write to it:
bytes[(y * img.Width + x) * 3] = 255;

How to find rectangle of difference between two images

I have two images the same size. What is the best way to find the rectangle in which they differ. Obviously I could go through the image 4 times in different directions, but i'm wondering if there's an easier way.
Example:

A naive approach would be to start at the origin, and work line by line, column by column. Compare each pixel, keeping note of the topmost, leftmost, rightmost, and bottommost, from which you can calculate your rectangle. There will be cases where this single pass approach would be faster (i.e. where there is a very small differing area)

If you want a single rectangle, use int.MaxValue for the threshold.
var diff = new ImageDiffUtil(filename1, filename2);
var diffRectangles = diff.GetDiffRectangles(int.MaxValue);
If you want multiple rectangles, use a smaller threshold.
var diff = new ImageDiffUtil(filename1, filename2);
var diffRectangles = diff.GetDiffRectangles(8);
ImageDiffUtil.cs
using System;
using System.Collections.Generic;
using System.Drawing;
using System.Linq;
namespace diff_images
{
public class ImageDiffUtil
{
Bitmap image1;
Bitmap image2;
public ImageDiffUtil(string filename1, string filename2)
{
image1 = Image.FromFile(filename1) as Bitmap;
image2 = Image.FromFile(filename2) as Bitmap;
}
public IList<Point> GetDiffPixels()
{
var widthRange = Enumerable.Range(0, image1.Width);
var heightRange = Enumerable.Range(0, image1.Height);
var result = widthRange
.SelectMany(x => heightRange, (x, y) => new Point(x, y))
.Select(point => new
{
Point = point,
Pixel1 = image1.GetPixel(point.X, point.Y),
Pixel2 = image2.GetPixel(point.X, point.Y)
})
.Where(pair => pair.Pixel1 != pair.Pixel2)
.Select(pair => pair.Point)
.ToList();
return result;
}
public IEnumerable<Rectangle> GetDiffRectangles(double distanceThreshold)
{
var result = new List<Rectangle>();
var differentPixels = GetDiffPixels();
while (differentPixels.Count > 0)
{
var cluster = new List<Point>()
{
differentPixels[0]
};
differentPixels.RemoveAt(0);
while (true)
{
var left = cluster.Min(p => p.X);
var right = cluster.Max(p => p.X);
var top = cluster.Min(p => p.Y);
var bottom = cluster.Max(p => p.Y);
var width = Math.Max(right - left, 1);
var height = Math.Max(bottom - top, 1);
var clusterBox = new Rectangle(left, top, width, height);
var proximal = differentPixels
.Where(point => GetDistance(clusterBox, point) <= distanceThreshold)
.ToList();
proximal.ForEach(point => differentPixels.Remove(point));
if (proximal.Count == 0)
{
result.Add(clusterBox);
break;
}
else
{
cluster.AddRange(proximal);
}
};
}
return result;
}
static double GetDistance(Rectangle rect, Point p)
{
var dx = Math.Max(rect.Left - p.X, 0);
dx = Math.Max(dx, p.X - rect.Right);
var dy = Math.Max(rect.Top - p.Y, 0);
dy = Math.Max(dy, p.Y - rect.Bottom);
return Math.Sqrt(dx * dx + dy * dy);
}
}
}
Form1.cs
using System.Drawing;
using System.Linq;
using System.Windows.Forms;
namespace diff_images
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
var filename1 = #"Gelatin1.PNG";
var filename2 = #"Gelatin2.PNG";
var diff = new ImageDiffUtil(filename1, filename2);
var diffRectangles = diff.GetDiffRectangles(8);
var img3 = Image.FromFile(filename2);
Pen redPen = new Pen(Color.Red, 1);
var padding = 3;
using (var graphics = Graphics.FromImage(img3))
{
diffRectangles
.ToList()
.ForEach(rect =>
{
var largerRect = new Rectangle(rect.X - padding, rect.Y - padding, rect.Width + padding * 2, rect.Height + padding * 2);
graphics.DrawRectangle(redPen, largerRect);
});
}
var pb1 = new PictureBox()
{
Image = Image.FromFile(filename1),
Left = 8,
Top = 8,
SizeMode = PictureBoxSizeMode.AutoSize
};
var pb2 = new PictureBox()
{
Image = Image.FromFile(filename2),
Left = pb1.Left + pb1.Width + 16,
Top = 8,
SizeMode = PictureBoxSizeMode.AutoSize
};
var pb3 = new PictureBox()
{
Image = img3,
Left = pb2.Left + pb2.Width + 16,
Top = 8,
SizeMode = PictureBoxSizeMode.AutoSize
};
Controls.Add(pb1);
Controls.Add(pb2);
Controls.Add(pb3);
}
}
}

Image processing like this is expensive, there are a lot of bits to look at. In real applications, you almost always need to filter the image to get rid of artifacts induced by imperfect image captures.
A common library used for this kind of bit whacking is OpenCV, it takes advantage of dedicated CPU instructions available to make this fast. There are several .NET wrappers available for it, Emgu is one of them.

I don't think there is an easier way.
In fact doing this will just be a (very) few lines of code, so unless you find a library that does that for you directly you won't find a shorter way.

Idea:
Consider an image as a 2D Array with each Array element as a pixel of the image. Hence, I would say Image Differencing is nothing but 2D Array Differencing.
Idea is to just scan through the array elements width-wise and find the place where there is a difference in pixel values. If example [x, y] co-ordinates of both 2D Array are different then our rectangle finding logic starts. Later on the rectangles would be used to patch the last updated Frame Buffer.
We need to scan through the boundaries of the rectangles for differences and if any difference is found in the boundary of rectangle, then the boundary will be increased width-wise or height-wise depending upon the type of scan made.
Consider I scanned width-wise of 2D Array and I found a location where there exist a co-ordinate which is different in both the 2D Arrays, I will create a rectangle with the starting position as [x-1, y-1] and with the width and height as 2 and 2 respectively. Please note that width and height refers to the number of pixels.
eg: Rect Info:
X = 20
Y = 35
W = 26
H = 23
i.e width of the rectangle starts from co-ordinate [20, 35] -> [20, 35 + 26 - 1]. Maybe when you find the code you may be able to understand it better.
Also there are possibilities that there are smaller rectangles inside a bigger rectangle you have found, thus we need to remove the smaller rectangles from our reference because they mean nothing to us except that they occupu my precious space !!
The above logic would be helpful in the case of VNC Server Implementation where there would be a need of rectangles that denotes differences in the image that is currently taken. Those rectangles could be sent in the network to the VNC Client which can patch the rectangles in the local copy of Frame Buffer it possesses thereby displaying it on the VNC Client Display Board.
P.S.:
I will be attaching the code in which I implemented my own algorithm. I would request viewers to comment for any mistakes or performance tuning. I would also request viewers to comment about any better algorithm that would make life simpler.
Code:
Class Rect:
public class Rect {
public int x; // Array Index
public int y; // Array Index
public int w; // Number of hops along the Horizontal
public int h; // Number of hops along the Vertical
#Override
public boolean equals(Object obj) {
Rect rect = (Rect) obj;
if(rect.x == this.x && rect.y == this.y && rect.w == this.w && rect.h == this.h) {
return true;
}
return false;
}
}
Class Image Difference:
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;
import java.util.LinkedList;
import javax.imageio.ImageIO;
public class ImageDifference {
long start = 0, end = 0;
public LinkedList<Rect> differenceImage(int[][] baseFrame, int[][] screenShot, int xOffset, int yOffset, int width, int height) {
// Code starts here
int xRover = 0;
int yRover = 0;
int index = 0;
int limit = 0;
int rover = 0;
boolean isRectChanged = false;
boolean shouldSkip = false;
LinkedList<Rect> rectangles = new LinkedList<Rect>();
Rect rect = null;
start = System.nanoTime();
// xRover - Rovers over the height of 2D Array
// yRover - Rovers over the width of 2D Array
int verticalLimit = xOffset + height;
int horizontalLimit = yOffset + width;
for(xRover = xOffset; xRover < verticalLimit; xRover += 1) {
for(yRover = yOffset; yRover < horizontalLimit; yRover += 1) {
if(baseFrame[xRover][yRover] != screenShot[xRover][yRover]) {
// Skip over the already processed Rectangles
for(Rect itrRect : rectangles) {
if(( (xRover < itrRect.x + itrRect.h) && (xRover >= itrRect.x) ) && ( (yRover < itrRect.y + itrRect.w) && (yRover >= itrRect.y) )) {
shouldSkip = true;
yRover = itrRect.y + itrRect.w - 1;
break;
} // End if(( (xRover < itrRect.x + itrRect.h) && (xRover >= itrRect.x) ) && ( (yRover < itrRect.y + itrRect.w) && (yRover >= itrRect.y) ))
} // End for(Rect itrRect : rectangles)
if(shouldSkip) {
shouldSkip = false;
// Need to come out of the if condition as below that is why "continue" has been provided
// if(( (xRover <= (itrRect.x + itrRect.h)) && (xRover >= itrRect.x) ) && ( (yRover <= (itrRect.y + itrRect.w)) && (yRover >= itrRect.y) ))
continue;
} // End if(shouldSkip)
rect = new Rect();
rect.x = ((xRover - 1) < xOffset) ? xOffset : (xRover - 1);
rect.y = ((yRover - 1) < yOffset) ? yOffset : (yRover - 1);
rect.w = 2;
rect.h = 2;
/* Boolean variable used to re-scan the currently found rectangle
for any change due to previous scanning of boundaries */
isRectChanged = true;
while(isRectChanged) {
isRectChanged = false;
index = 0;
/* I */
/* Scanning of left-side boundary of rectangle */
index = rect.x;
limit = rect.x + rect.h;
while(index < limit && rect.y != yOffset) {
if(baseFrame[index][rect.y] != screenShot[index][rect.y]) {
isRectChanged = true;
rect.y = rect.y - 1;
rect.w = rect.w + 1;
index = rect.x;
continue;
} // End if(baseFrame[index][rect.y] != screenShot[index][rect.y])
index = index + 1;;
} // End while(index < limit && rect.y != yOffset)
/* II */
/* Scanning of bottom boundary of rectangle */
index = rect.y;
limit = rect.y + rect.w;
while( (index < limit) && (rect.x + rect.h != verticalLimit) ) {
rover = rect.x + rect.h - 1;
if(baseFrame[rover][index] != screenShot[rover][index]) {
isRectChanged = true;
rect.h = rect.h + 1;
index = rect.y;
continue;
} // End if(baseFrame[rover][index] != screenShot[rover][index])
index = index + 1;
} // End while( (index < limit) && (rect.x + rect.h != verticalLimit) )
/* III */
/* Scanning of right-side boundary of rectangle */
index = rect.x;
limit = rect.x + rect.h;
while( (index < limit) && (rect.y + rect.w != horizontalLimit) ) {
rover = rect.y + rect.w - 1;
if(baseFrame[index][rover] != screenShot[index][rover]) {
isRectChanged = true;
rect.w = rect.w + 1;
index = rect.x;
continue;
} // End if(baseFrame[index][rover] != screenShot[index][rover])
index = index + 1;
} // End while( (index < limit) && (rect.y + rect.w != horizontalLimit) )
} // while(isRectChanged)
// Remove those rectangles that come inside "rect" rectangle.
int idx = 0;
while(idx < rectangles.size()) {
Rect r = rectangles.get(idx);
if( ( (rect.x <= r.x) && (rect.x + rect.h >= r.x + r.h) ) && ( (rect.y <= r.y) && (rect.y + rect.w >= r.y + r.w) ) ) {
rectangles.remove(r);
} else {
idx += 1;
} // End if( ( (rect.x <= r.x) && (rect.x + rect.h >= r.x + r.h) ) && ( (rect.y <= r.y) && (rect.y + rect.w >= r.y + r.w) ) )
} // End while(idx < rectangles.size())
// Giving a head start to the yRover when a rectangle is found
rectangles.addFirst(rect);
yRover = rect.y + rect.w - 1;
rect = null;
} // End if(baseFrame[xRover][yRover] != screenShot[xRover][yRover])
} // End for(yRover = yOffset; yRover < horizontalLimit; yRover += 1)
} // End for(xRover = xOffset; xRover < verticalLimit; xRover += 1)
end = System.nanoTime();
return rectangles;
}
public static void main(String[] args) throws IOException {
LinkedList<Rect> rectangles = null;
// Buffering the Base image and Screen Shot Image
BufferedImage screenShotImg = ImageIO.read(new File("screenShotImg.png"));
BufferedImage baseImg = ImageIO.read(new File("baseImg.png"));
int width = baseImg.getWidth();
int height = baseImg.getHeight();
int xOffset = 0;
int yOffset = 0;
int length = baseImg.getWidth() * baseImg.getHeight();
// Creating 2 Two Dimensional Arrays for Image Processing
int[][] baseFrame = new int[height][width];
int[][] screenShot = new int[height][width];
// Creating 2 Single Dimensional Arrays to retrieve the Pixel Values
int[] baseImgPix = new int[length];
int[] screenShotImgPix = new int[length];
// Reading the Pixels from the Buffered Image
baseImg.getRGB(0, 0, baseImg.getWidth(), baseImg.getHeight(), baseImgPix, 0, baseImg.getWidth());
screenShotImg.getRGB(0, 0, screenShotImg.getWidth(), screenShotImg.getHeight(), screenShotImgPix, 0, screenShotImg.getWidth());
// Transporting the Single Dimensional Arrays to Two Dimensional Array
long start = System.nanoTime();
for(int row = 0; row < height; row++) {
System.arraycopy(baseImgPix, (row * width), baseFrame[row], 0, width);
System.arraycopy(screenShotImgPix, (row * width), screenShot[row], 0, width);
}
long end = System.nanoTime();
System.out.println("Array Copy : " + ((double)(end - start) / 1000000));
// Finding Differences between the Base Image and ScreenShot Image
ImageDifference imDiff = new ImageDifference();
rectangles = imDiff.differenceImage(baseFrame, screenShot, xOffset, yOffset, width, height);
// Displaying the rectangles found
int index = 0;
for(Rect rect : rectangles) {
System.out.println("\nRect info : " + (++index));
System.out.println("X : " + rect.x);
System.out.println("Y : " + rect.y);
System.out.println("W : " + rect.w);
System.out.println("H : " + rect.h);
// Creating Bounding Box
for(int i = rect.y; i < rect.y + rect.w; i++) {
screenShotImgPix[ ( rect.x * width) + i ] = 0xFFFF0000;
screenShotImgPix[ ((rect.x + rect.h - 1) * width) + i ] = 0xFFFF0000;
}
for(int j = rect.x; j < rect.x + rect.h; j++) {
screenShotImgPix[ (j * width) + rect.y ] = 0xFFFF0000;
screenShotImgPix[ (j * width) + (rect.y + rect.w - 1) ] = 0xFFFF0000;
}
}
// Creating the Resultant Image
screenShotImg.setRGB(0, 0, width, height, screenShotImgPix, 0, width);
ImageIO.write(screenShotImg, "PNG", new File("result.png"));
double d = ((double)(imDiff.end - imDiff.start) / 1000000);
System.out.println("\nTotal Time : " + d + " ms" + " Array Copy : " + ((double)(end - start) / 1000000) + " ms");
}
}
Description:
There would be a function named
public LinkedList<Rect> differenceImage(int[][] baseFrame, int[][] screenShot, int width, int height)
which does the job of finding differences in the images and return a linkedlist of objects. The objects are nothing but the rectangles.
There is main function which does the job of testing the algorithm.
There are 2 sample images passed into the code in main function, they are nothing but the "baseFrame" and "screenShot" thereby creating the resultant image named "result".
I don't possess the desired reputation to post the resultant image which would be very interesting.
There is a blog which would provide the output
Image Difference

I don't think there can be anything better than exhaustively searching from each side in turn for the first point of difference in that direction. Unless, that is, you know a fact that in some way constrains the set of points of difference.

So here comes the easy way if you know how to use Lockbit :)
Bitmap originalBMP = new Bitmap(pictureBox1.ImageLocation);
Bitmap changedBMP = new Bitmap(pictureBox2.ImageLocation);
int width = Math.Min(originalBMP.Width, changedBMP.Width),
height = Math.Min(originalBMP.Height, changedBMP.Height),
xMin = int.MaxValue,
xMax = int.MinValue,
yMin = int.MaxValue,
yMax = int.MinValue;
var originalLock = originalBMP.LockBits(new Rectangle(0, 0, width, height), ImageLockMode.ReadWrite, originalBMP.PixelFormat);
var changedLock = changedBMP.LockBits(new Rectangle(0, 0, width, height), ImageLockMode.ReadWrite, changedBMP.PixelFormat);
for (int y = 0; y < height; y++)
{
for (int x = 0; x < width; x++)
{
//generate the address of the colour pixel
int pixelIdxOrg = y * originalLock.Stride + (x * 4);
int pixelIdxCh = y * changedLock.Stride + (x * 4);
if (( Marshal.ReadByte(originalLock.Scan0, pixelIdxOrg + 2)!= Marshal.ReadByte(changedLock.Scan0, pixelIdxCh + 2))
|| (Marshal.ReadByte(originalLock.Scan0, pixelIdxOrg + 1) != Marshal.ReadByte(changedLock.Scan0, pixelIdxCh + 1))
|| (Marshal.ReadByte(originalLock.Scan0, pixelIdxOrg) != Marshal.ReadByte(changedLock.Scan0, pixelIdxCh))
)
{
xMin = Math.Min(xMin, x);
xMax = Math.Max(xMax, x);
yMin = Math.Min(yMin, y);
yMax = Math.Max(yMax, y);
}
}
}
originalBMP.UnlockBits(originalLock);
changedBMP.UnlockBits(changedLock);
var result = changedBMP.Clone(new Rectangle(xMin, yMin, xMax - xMin, yMax - yMin), changedBMP.PixelFormat);
pictureBox3.Image = result;
disclaim it looks like your 2 pictures contains more differences than we can see with the naked eye so the result will be wider than you expect but you can add a tolerance so it wil fit even if the rest isn't 100% identical
to speed things up you will maybe able to us Parallel.For but do it only for the outer loop