We Keep Coding

Cut faraway objects based on depth map

I would like to do grabcut which uses a depth map that cuts away far objects, that is used in mixed reality application. So I would like to show just the front of what I see and the background as virtual reality scene.
The problem right now I tried to adapt so code and what I get is front which is cut but in black color, the mask actually.
I don't know where is the problem settle.
The input is a depth map from zed camera.
here is a picture of the behaviour:
My trial:
private void convertToGrayScaleValues(Mat mask)
{
int width = mask.rows();
int height = mask.cols();
byte[] buffer = new byte[width * height];
mask.get(0, 0, buffer);
for (int x = 0; x < width; x++)
{
for (int y = 0; y < height; y++)
{
int value = buffer[y * width + x];
if (value == Imgproc.GC_BGD)
{
buffer[y * width + x] = 0; // for sure background
}
else if (value == Imgproc.GC_PR_BGD)
{
buffer[y * width + x] = 85; // probably background
}
else if (value == Imgproc.GC_PR_FGD)
{
buffer[y * width + x] = (byte)170; // probably foreground
}
else
{
buffer[y * width + x] = (byte)255; // for sure foreground
}
}
}
mask.put(0, 0, buffer);
}
For Each depth frame from Camera:
Mat erodeElement = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(4, 4));
Mat dilateElement = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(7, 7));
depth.copyTo(maskFar);
Core.normalize(maskFar, maskFar, 0, 255, Core.NORM_MINMAX, CvType.CV_8U);
Imgproc.cvtColor(maskFar, maskFar, Imgproc.COLOR_BGR2GRAY);
Imgproc.threshold(maskFar, maskFar, 180, 255, Imgproc.THRESH_BINARY);
Imgproc.dilate(maskFar, maskFar, erodeElement);
Imgproc.erode(maskFar, maskFar, dilateElement);
Mat bgModel = new Mat();
Mat fgModel = new Mat();
Imgproc.grabCut(image, maskFar, new OpenCVForUnity.CoreModule.Rect(), bgModel, fgModel, 1, Imgproc.GC_INIT_WITH_MASK);
convertToGrayScaleValues(maskFar); // back to grayscale values
Imgproc.threshold(maskFar, maskFar, 180, 255, Imgproc.THRESH_TOZERO);
Mat foreground = new Mat(image.size(), CvType.CV_8UC4, new Scalar(0, 0, 0));
image.copyTo(foreground, maskFar);
Utils.fastMatToTexture2D(foreground, texture);

In this case, the graph cut on the depth image might not be the correct method to solve all of your issue.
If you insist the processing should be done in the depth image. To find everything that is not on the table and filter out the table part. You may first apply the disparity based approach for finding the object that's is not on the ground. Reference: https://github.com/windowsub0406/StereoVision
Then based on the V disparity output image, find the locally connected component that is grouped together. You may follow this link how to do this disparity map in OpenCV which is asking the similar way to find the objects that's not on the ground
If you are ok with RGB based approaches, then use any deep learning-based method to recognize the monitor should be the correct approaches. It can directly detect the mointer bounding box. By apply this bounding box to the depth image, you may have what you want. For deep learning based approaches, there are many available package such as Yolo series. You may find one that is suitable for you. reference: https://medium.com/#dvshah13/project-image-recognition-1d316d04cb4c

StartCoroutine() to fix targetTexture.ReadPixels error

As the title suggests I have a problem with the error occurring at the row
targetTexture.ReadPixels(new Rect(0, 0, cameraResolution.width, cameraResolution.height), 0, 0);
Error:
ReadPixels was called to read pixels from system frame buffer, while
not inside drawing frame. UnityEngine.Texture2D:ReadPixels(Rect,
Int32, Int32)
As I have understood from other posts one way to solve this issue is to make a Ienumerator method which yield return new WaitForSeconds or something, and call it like: StartCoroutine(methodname) so that the frames gets to load in time so that there will be pixels to read-ish.
What I don't get is where in the following code this method would make the most sense. Which part does not get to load in time?
PhotoCapture photoCaptureObject = null;
Texture2D targetTexture = null;
public string path = "";
CameraParameters cameraParameters = new CameraParameters();
private void Awake()
{
var cameraResolution = PhotoCapture.SupportedResolutions.OrderByDescending((res) => res.width * res.height).First();
targetTexture = new Texture2D(cameraResolution.width, cameraResolution.height);
// Create a PhotoCapture object
PhotoCapture.CreateAsync(false, captureObject =>
{
photoCaptureObject = captureObject;
cameraParameters.hologramOpacity = 0.0f;
cameraParameters.cameraResolutionWidth = cameraResolution.width;
cameraParameters.cameraResolutionHeight = cameraResolution.height;
cameraParameters.pixelFormat = CapturePixelFormat.BGRA32;
});
}
private void Update()
{
// if not initialized yet don't take input
if (photoCaptureObject == null) return;
if (Input.GetKey("k") || Input.GetKey("k"))
{
Debug.Log("k was pressed");
VuforiaBehaviour.Instance.gameObject.SetActive(false);
// Activate the camera
photoCaptureObject.StartPhotoModeAsync(cameraParameters, result =>
{
if (result.success)
{
// Take a picture
photoCaptureObject.TakePhotoAsync(OnCapturedPhotoToMemory);
}
else
{
Debug.LogError("Couldn't start photo mode!", this);
}
});
}
}
private static string FileName(int width, int height)
{
return $"screen_{width}x{height}_{DateTime.Now:yyyy-MM-dd_HH-mm-ss}.png";
}
private void OnCapturedPhotoToMemory(PhotoCapture.PhotoCaptureResult result, PhotoCaptureFrame photoCaptureFrame)
{
// Copy the raw image data into the target texture
photoCaptureFrame.UploadImageDataToTexture(targetTexture);
Resolution cameraResolution = PhotoCapture.SupportedResolutions.OrderByDescending((res) => res.width * res.height).First();
targetTexture.ReadPixels(new Rect(0, 0, cameraResolution.width, cameraResolution.height), 0, 0);
targetTexture.Apply();
byte[] bytes = targetTexture.EncodeToPNG();
string filename = FileName(Convert.ToInt32(targetTexture.width), Convert.ToInt32(targetTexture.height));
//save to folder under assets
File.WriteAllBytes(Application.streamingAssetsPath + "/Snapshots/" + filename, bytes);
Debug.Log("The picture was uploaded");
// Deactivate the camera
photoCaptureObject.StopPhotoModeAsync(OnStoppedPhotoMode);
}
private void OnStoppedPhotoMode(PhotoCapture.PhotoCaptureResult result)
{
// Shutdown the photo capture resource
VuforiaBehaviour.Instance.gameObject.SetActive(true);
photoCaptureObject.Dispose();
photoCaptureObject = null;
}
Sorry if this counts as a duplicate to this for example.
Edit
And this one might be useful when I get to that point.
Is it so that I don't need these three lines at all?
Resolution cameraResolution = PhotoCapture.SupportedResolutions.OrderByDescending((res) => res.width * res.height).First();
targetTexture.ReadPixels(new Rect(0, 0, cameraResolution.width, cameraResolution.height), 0, 0);
targetTexture.Apply();
As written in the comments the difference between using these three lines and not is that the photo saved has a black background + the AR-GUI. Without the second line of code above is a photo with the AR-GUI but with the background is a live stream of my computer webcam. And really I don't wanna see the computer webcam but what the HoloLens sees.

Your three lines
Resolution cameraResolution = PhotoCapture.SupportedResolutions.OrderByDescending((res) => res.width * res.height).First();
targetTexture.ReadPixels(new Rect(0, 0, cameraResolution.width, cameraResolution.height), 0, 0);
targetTexture.Apply();
make not much sense to me. Texture2D.ReadPixels is for creating a Screenshot so you would overwrite the texture you just received from PhotoCapture with a screenshot? (Also with incorrect dimensions since camera resolution very probably != screen resolution.)
That's also the reason for
As written in the comments the difference between using these three lines and not is that the photo saved has a black background + the AR-GUI.
After doing
photoCaptureFrame.UploadImageDataToTexture(targetTexture);
you already have the Texture2D received from the PhotoCapture in the targetTexture.
I think you probably confused it with Texture2D.GetPixels which is used to get the pixel data of a given Texture2D.
I would like to crop the captured photo from the center in the end and am thinking that maybe that is possible with this code row? Beginning the new rect at other pixels than 0, 0)
What you actually want is cropping the received Texture2D from the center as you mentioned in the comments. You can do that using GetPixels(int x, int y, int blockWidth, int blockHeight, int miplevel) which is used to cut out a certain area of a given Texture2D
public static Texture2D CropAroundCenter(Texture2D input, Vector2Int newSize)
{
if(input.width < newSize.x || input.height < newSize.y)
{
Debug.LogError("You can't cut out an area of an image which is bigger than the image itself!", this);
return null;
}
// get the pixel coordinate of the center of the input texture
var center = new Vector2Int(input.width / 2, input.height / 2);
// Get pixels around center
// Get Pixels starts width 0,0 in the bottom left corner
// so as the name says, center.x,center.y would get the pixel in the center
// we want to start getting pixels from center - half of the newSize
//
// than from starting there we want to read newSize pixels in both dimensions
var pixels = input.GetPixels(center.x - newSize.x / 2, center.y - newSize.y / 2, newSize.x, newSize.y, 0);
// Create a new texture with newSize
var output = new Texture2D(newSize.x, newSize.y);
output.SetPixels(pixels);
output.Apply();
return output;
}
for (hopefully) better understanding this is an illustration what that GetPixels overload with the given values does here:
and than use it in
private void OnCapturedPhotoToMemory(PhotoCapture.PhotoCaptureResult result, PhotoCaptureFrame photoCaptureFrame)
{
// Copy the raw image data into the target texture
photoCaptureFrame.UploadImageDataToTexture(targetTexture);
// for example take only half of the textures width and height
targetTexture = CropAroundCenter(targetTexture, new Vector2Int(targetTexture.width / 2, targetTexture.height / 2);
byte[] bytes = targetTexture.EncodeToPNG();
string filename = FileName(Convert.ToInt32(targetTexture.width), Convert.ToInt32(targetTexture.height));
//save to folder under assets
File.WriteAllBytes(Application.streamingAssetsPath + "/Snapshots/" + filename, bytes);
Debug.Log("The picture was uploaded");
// Deactivate the camera
photoCaptureObject.StopPhotoModeAsync(OnStoppedPhotoMode);
}
Or you could make it an extension method in an apart static class like
public static class Texture2DExtensions
{
public static void CropAroundCenter(this Texture2D input, Vector2Int newSize)
{
if (input.width < newSize.x || input.height < newSize.y)
{
Debug.LogError("You can't cut out an area of an image which is bigger than the image itself!");
return;
}
// get the pixel coordinate of the center of the input texture
var center = new Vector2Int(input.width / 2, input.height / 2);
// Get pixels around center
// Get Pixels starts width 0,0 in the bottom left corner
// so as the name says, center.x,center.y would get the pixel in the center
// we want to start getting pixels from center - half of the newSize
//
// than from starting there we want to read newSize pixels in both dimensions
var pixels = input.GetPixels(center.x - newSize.x / 2, center.y - newSize.y / 2, newSize.x, newSize.y, 0);
// Resize the texture (creating a new one didn't work)
input.Resize(newSize.x, newSize.y);
input.SetPixels(pixels);
input.Apply(true);
}
}
and use it instead like
targetTexture.CropAroundCenter(new Vector2Int(targetTexture.width / 2, targetTexture.height / 2));
Note:
UploadImageDataToTexture: You may only use this method if you specified the BGRA32 format in your CameraParameters.
Luckily you use that anyway ;)
Keep in mind that this operation will happen on the main thread and therefore be slow.
However the only alternative would be CopyRawImageDataIntoBuffer and generate the texture in another thread or external, so I'ld say it is ok to stay with UploadImageDataToTexture ;)
and
The captured image will also appear flipped on the HoloLens. You can reorient the image by using a custom shader.
by flipped they actually mean that the Y-Axis of the texture is upside down. X-Axis is correct.
For flipping the Texture vertically you can use a second extension method:
public static class Texture2DExtensions
{
public static void CropAroundCenter(){....}
public static void FlipVertically(this Texture2D texture)
{
var pixels = texture.GetPixels();
var flippedPixels = new Color[pixels.Length];
// These for loops are for running through each individual pixel and
// write them with inverted Y coordinates into the flippedPixels
for (var x = 0; x < texture.width; x++)
{
for (var y = 0; y < texture.height; y++)
{
var pixelIndex = x + y * texture.width;
var flippedIndex = x + (texture.height - 1 - y) * texture.width;
flippedPixels[flippedIndex] = pixels[pixelIndex];
}
}
texture.SetPixels(flippedPixels);
texture.Apply();
}
}
and use it like
targetTexture.FlipVertically();
Result: (I used FlipVertically and cropp to the half of size every second for this example and a given Texture but it should work the same for a taken picture.)
Image source: http://developer.vuforia.com/sites/default/files/sample-apps/targets/imagetargets_targets.pdf
Update
To your button problem:
Don't use
if (Input.GetKey("k") || Input.GetKey("k"))
First of all you are checking the exact same condition twice. And than GetKey fires every frame while the key stays pressed. Instead rather use
if (Input.GetKeyDown("k"))
which fires only a single time. I guess there was an issue with Vuforia and PhotoCapture since your original version fired so often and maybe you had some concurrent PhotoCapture processes...

Save Camera image from unity ARCore

This is in relation to this post Save AcquireCameraImageBytes() from Unity ARCore to storage as an image
I tried the steps mentioned by
#JordanRobinson I am having a similar issue of seeing just a gray square. I keep re-reading his update, and I am not clear how step 2 (creating a texture reader) ties to step 3. I added the update function to call Frame.CameraImage.AcquireCameraImageBytes. I think missing something.
I feel I am close as it is saving an image (just a gray nothing image :-) Any help you can offer will be greatly appreciated
Here is my code
private Texture2D m_TextureRender;
private TextureReader m_CachedTextureReader;
void Start ()
{
m_CachedTextureReader = GetComponent<TextureReader>();
m_CachedTextureReader.OnImageAvailableCallback += OnImageAvailable;
QuitOnConnectionErrors ();
}
void Update () {
Screen.sleepTimeout = SleepTimeout.NeverSleep;
using (var image = Frame.CameraImage.AcquireCameraImageBytes())
{
if (!image.IsAvailable)
{
return;
}
OnImageAvailable(TextureReaderApi.ImageFormatType.ImageFormatColor,
image.Width, image.Height, image.Y, 0);
}
}
private void OnImageAvailable(TextureReaderApi.ImageFormatType format, int width, int height, System.IntPtr pixelBuffer, int bufferSize)
{
if (format != TextureReaderApi.ImageFormatType.ImageFormatColor)
{
Debug.Log("No edge detected due to incorrect image format.");
return;
}
if (m_TextureRender == null || m_EdgeDetectionResultImage == null || m_TextureRender.width != width || m_TextureRender.height != height)
{
m_TextureRender = new Texture2D(width, height, TextureFormat.RGBA32, false, false);
m_EdgeDetectionResultImage = new byte[width * height * 4];
m_TextureRender.width = width;
m_TextureRender.height = height;
}
System.Runtime.InteropServices.Marshal.Copy(pixelBuffer, m_EdgeDetectionResultImage, 0, bufferSize);
// Update the rendering texture with the sampled image.
m_TextureRender.LoadRawTextureData(m_EdgeDetectionResultImage);
m_TextureRender.Apply();
var encodedJpg = m_TextureRender.EncodeToJPG();
var path = Application.persistentDataPath;
File.WriteAllBytes(path + "/test2.jpg", encodedJpg);
}

Is there a fast way to manipulate and buffer a screen in Windows Forms?

I am working on a game for learning purposes, I want to make it only with the .NET-Framework and a Windows Forms project in C#.
I want to get the 'screen' (Something that can be displayed on the window) as an int[]. Modify the array and reapply the altered array to the 'screen' in a buffered manner (So that it doesn't flicker).
I am currently using a Panel, which I draw a Bitmap on with Graphics. The Bitmap is converted to an int[] which I then can modify and reapply to the Bitmap and redraw. It works, but is very slow, especially because I have to scale up the image every frame because my game is only 300x160 and the screen 900x500.
Build up:
// Renders 1 frame
private void Render()
{
// Buffer setup
_bufferedContext = BufferedGraphicsManager.Current;
_buffer = _bufferedContext.Allocate(panel_canvas.CreateGraphics(), new Rectangle(0, 0, _scaledWidth, _scaledHeight));
_screen.clear();
// Get position of player on map
_xScroll = _player._xMap - _screen._width / 2;
_yScroll = _player._yMap - _screen._height / 2;
// Indirectly modifies the int[] '_pixels'
_level.render(_xScroll, _yScroll, _screen);
_player.render(_screen);
// Converts the int[] into a Bitmap (unsafe method is faster)
unsafe
{
fixed (int* intPtr = &_screen._pixels[0])
{
_screenImage = new Bitmap(_trueWidth, _trueHeight, _trueWidth * 4, PixelFormat.Format32bppRgb, new IntPtr(intPtr));
}
}
// Draw generated image on buffer
Graphics g = _buffer.Graphics;
g.InterpolationMode = System.Drawing.Drawing2D.InterpolationMode.NearestNeighbor;
g.DrawImage(_screenImage, new Rectangle(0, 0, 900, 506));
// Update panel buffered
_buffer.Render();
}
Is there a faster way without external libraries to make this work?

I'm not to sure about the unsafe code , But I do know about the buffered graphics manager. I think you should create a class for it instead of creating a new one every time.As well as having all of your sprites widths and heights be determined at the load instead of scaling them. That sped up my small game engine a good bit.
class Spritebatch
{
private Graphics Gfx;
private BufferedGraphics bfgfx;
private BufferedGraphicsContext cntxt = BufferedGraphicsManager.Current;
public Spritebatch(Size clientsize, Graphics gfx)
{
cntxt.MaximumBuffer = new Size(clientsize.Width + 1, clientsize.Height + 1);
bfgfx = cntxt.Allocate(gfx, new Rectangle(Point.Empty, clientsize));
Gfx = gfx;
}
public void Begin()
{
bfgfx.Graphics.Clear(Color.Black);
}
public void Draw(Sprite s)
{
bfgfx.Graphics.DrawImageUnscaled(s.Texture, new Rectangle(s.toRec.X - s.rotationOffset.Width,s.toRec.Y - s.rotationOffset.Height,s.toRec.Width,s.toRec.Height));
}
public void drawImage(Bitmap b, Rectangle rec)
{
bfgfx.Graphics.DrawImageUnscaled(b, rec);
}
public void drawImageClipped(Bitmap b, Rectangle rec)
{
bfgfx.Graphics.DrawImageUnscaledAndClipped(b, rec);
}
public void drawRectangle(Pen p, Rectangle rec)
{
bfgfx.Graphics.DrawRectangle(p, rec);
}
public void End()
{
bfgfx.Render(Gfx);
}
}
This is a example of what I used. It's set up to mimic the Spritebatch in Xna. Drawing the images Unscaled will really increase the speed of it.Also creating one instance of the buffered graphics and Context will be faster then creating a new one every time you have to render. So I would advise you to change the line g.DrawImage(_screenImage, new Rectangle(0, 0, 900, 506)); to DrawImageUnscaled(_screenImage, new Rectangle(0, 0, 900, 506));
Edited : Example of how to scale code on sprite load
public Sprite(Bitmap texture, float x, float y, int width, int height)
{
//texture is the image you originally start with.
Bitmap b = new Bitmap(width, height);
// Create a bitmap with the desired width and height
using (Graphics g = Graphics.FromImage(b))
{
g.DrawImage(texture, 0, 0, width, height);
}
// get the graphics from the new image and draw the old image to it
//scaling it to the proper width and height
Texture = b;
//set Texture which is the final picture to the sprite.
//Uppercase Texture is different from lowercase

Scaling of the image is expensive enough, even when is done without any interpolation. To speed up the things, you should minimize memory allocations: when you create brand new Bitmap every frame, it leads to object creation and pixmap buffer allocation. This fact negates all the benefits you get from BufferedGraphics. I advise you to do the following:
Create the Bitmap instance of required size (equal to screen size) only once, outside of Render method.
Use direct access to bitmap data through LockBits method, and try to implement the scaling be hand using nearest pixel.
Of course, using some sort of hardware acceleration for scaling operation is the most preferred option (for example, in opengl all images are usually drawn using textured rectangles, and rendering such rectangles implicitly involves the process of "scaling" when texture sampling is performed).

I'm wondering why do you call this "very slow", because I did some tests and the performance doesn't seem bad. Also have you measured the performance of your rendering code into int[] '_pixels' (unfortunately you haven't provided that code) separately from the bitmap operations, because it might be the slow part.
About your concrete question. As others mentioned, using preallocated buffered graphics and bitmap objects would speed up it a bit.
But do you really need that int[] buffer? BufferedGraphics is already backed internally with a bitmap, so what really happens is:
(1) You fill the int[] buffer
(2) int[] buffer is copied to the new/preallocated Bitmap
(3) Bitmap from step 2 is copied (applying scale) to the BufferedGraphics internal bitmap (via DrawImage)
(4) BufferedGraphics internal bitmap is copied to the screen (via Render)
As you can see, there are a lot of copy operations. The intended usage of BufferedGraphics is:
(1) You fill the BufferedGraphics internal bitmap via drawing methods of the BufferedGraphics.Graphics property. If setup, the Graphics will do the scaling (as well other transformations) for you.
(2) BufferedGraphics internal bitmap is copied to the screen (via Render)
I don't know what your drawing code is doing, but if you can afford it, this definitely should provide the best performance.
Here is my quick and dirty test in case you are interested in:
using System;
using System.Diagnostics;
using System.Drawing;
using System.Drawing.Drawing2D;
using System.Drawing.Imaging;
using System.Threading;
using System.Windows.Forms;
namespace Test
{
enum RenderMode { NewBitmap, PreallocatedBitmap, Graphics }
class Screen
{
Control canvas;
public Rectangle area;
int[,] pixels;
BitmapData info;
Bitmap bitmap;
BufferedGraphics buffer;
float scaleX, scaleY;
public RenderMode mode = RenderMode.NewBitmap;
public Screen(Control canvas, Size size)
{
this.canvas = canvas;
var bounds = canvas.DisplayRectangle;
scaleX = (float)bounds.Width / size.Width;
scaleY = (float)bounds.Height / size.Height;
area.Size = size;
info = new BitmapData { Width = size.Width, Height = size.Height, PixelFormat = PixelFormat.Format32bppRgb, Stride = size.Width * 4 };
pixels = new int[size.Height, size.Width];
bitmap = new Bitmap(size.Width, size.Height, info.PixelFormat);
buffer = BufferedGraphicsManager.Current.Allocate(canvas.CreateGraphics(), bounds);
buffer.Graphics.InterpolationMode = InterpolationMode.NearestNeighbor;
ApplyMode();
}
public void ApplyMode()
{
buffer.Graphics.ResetTransform();
if (mode == RenderMode.Graphics)
buffer.Graphics.ScaleTransform(scaleX, scaleY);
}
public void FillRectangle(Color color, Rectangle rect)
{
if (mode == RenderMode.Graphics)
{
using (var brush = new SolidBrush(color))
buffer.Graphics.FillRectangle(brush, rect);
}
else
{
rect.Intersect(area);
if (rect.IsEmpty) return;
int colorData = color.ToArgb();
var pixels = this.pixels;
for (int y = rect.Y; y < rect.Bottom; y++)
for (int x = rect.X; x < rect.Right; x++)
pixels[y, x] = colorData;
}
}
public unsafe void Render()
{
if (mode == RenderMode.NewBitmap)
{
var bounds = canvas.DisplayRectangle;
using (var buffer = BufferedGraphicsManager.Current.Allocate(canvas.CreateGraphics(), bounds))
{
Bitmap bitmap;
fixed (int* pixels = &this.pixels[0, 0])
bitmap = new Bitmap(info.Width, info.Height, info.Stride, info.PixelFormat, new IntPtr(pixels));
buffer.Graphics.InterpolationMode = InterpolationMode.NearestNeighbor;
buffer.Graphics.DrawImage(bitmap, bounds);
buffer.Render();
}
}
else
{
if (mode == RenderMode.PreallocatedBitmap)
{
fixed (int* pixels = &this.pixels[0, 0])
{
info.Scan0 = new IntPtr(pixels); info.Reserved = 0;
bitmap.LockBits(area, ImageLockMode.WriteOnly | ImageLockMode.UserInputBuffer, info.PixelFormat, info);
bitmap.UnlockBits(info);
}
buffer.Graphics.DrawImage(bitmap, canvas.DisplayRectangle);
}
buffer.Render();
}
}
}
class Game
{
[STAThread]
public static void Main()
{
Application.EnableVisualStyles();
Application.SetCompatibleTextRenderingDefault(false);
var game = new Game();
game.Run();
}
Form form;
Control canvas;
Screen screen;
Level level;
Player player;
private Game()
{
form = new Form();
canvas = new Control { Parent = form, Bounds = new Rectangle(0, 0, 900, 506) };
form.ClientSize = canvas.Size;
screen = new Screen(canvas, new Size(300, 160));
level = new Level { game = this };
player = new Player { game = this };
}
private void Run()
{
bool toggleModeRequest = false;
canvas.MouseClick += (sender, e) => toggleModeRequest = true;
var worker = new Thread(() =>
{
int frameCount = 0;
Stopwatch drawT = new Stopwatch(), applyT = new Stopwatch(), advanceT = Stopwatch.StartNew(), renderT = Stopwatch.StartNew(), infoT = Stopwatch.StartNew();
while (true)
{
if (advanceT.ElapsedMilliseconds >= 3)
{
level.Advance(); player.Advance();
advanceT.Restart();
}
if (renderT.ElapsedMilliseconds >= 8)
{
frameCount++;
drawT.Start(); level.Render(); player.Render(); drawT.Stop();
applyT.Start(); screen.Render(); applyT.Stop();
renderT.Restart();
}
if (infoT.ElapsedMilliseconds >= 1000)
{
double drawS = drawT.ElapsedMilliseconds / 1000.0, applyS = applyT.ElapsedMilliseconds / 1000.0, totalS = drawS + applyS;
var info = string.Format("Render using {0} - Frames:{1:n0} FPS:{2:n0} Draw:{3:p2} Apply:{4:p2}",
screen.mode, frameCount, frameCount / totalS, drawS / totalS, applyS / totalS);
form.BeginInvoke(new Action(() => form.Text = info));
infoT.Restart();
}
if (toggleModeRequest)
{
toggleModeRequest = false;
screen.mode = (RenderMode)(((int)screen.mode + 1) % 3);
screen.ApplyMode();
frameCount = 0; drawT.Reset(); applyT.Reset();
}
}
});
worker.IsBackground = true;
worker.Start();
Application.Run(form);
}
class Level
{
public Game game;
public int pos = 0; bool right = true;
public void Advance() { Game.Advance(ref pos, ref right, 0, game.screen.area.Right - 1); }
public void Render()
{
game.screen.FillRectangle(Color.SaddleBrown, new Rectangle(0, 0, pos, game.screen.area.Height));
game.screen.FillRectangle(Color.DarkGreen, new Rectangle(pos, 0, game.screen.area.Right, game.screen.area.Height));
}
}
class Player
{
public Game game;
public int x = 0, y = 0;
public bool right = true, down = true;
public void Advance()
{
Game.Advance(ref x, ref right, game.level.pos, game.screen.area.Right - 5, 2);
Game.Advance(ref y, ref down, 0, game.screen.area.Bottom - 1, 2);
}
public void Render() { game.screen.FillRectangle(Color.Yellow, new Rectangle(x, y, 4, 4)); }
}
static void Advance(ref int pos, ref bool forward, int minPos, int maxPos, int delta = 1)
{
if (forward) { pos += delta; if (pos < minPos) pos = minPos; else if (pos > maxPos) { pos = maxPos; forward = false; } }
else { pos -= delta; if (pos > maxPos) pos = maxPos; else if (pos < minPos) { pos = minPos; forward = true; } }
}
}
}

Two image compare using win32 api c#

i have two images and i want to compare two image and want to get difference. i search google and found a link from where i copy paste the code for image comparison using win32 api.
so this is the url
http://blog.bobcravens.com/2009/04/create-a-remote-desktop-viewer-using-c-and-wcf/
here i am pasting the code.
private void button1_Click(object sender, EventArgs e)
{
Bitmap _prevBitmap = new Bitmap(#"d:\prev.jpg");
Bitmap _newBitmap = new Bitmap(#"d:\current.jpg");
Rectangle bounds = GetBoundingBoxForChanges(_prevBitmap, _newBitmap);
if (bounds == Rectangle.Empty)
{
}
Bitmap diff = new Bitmap(bounds.Width, bounds.Height);
Graphics g = Graphics.FromImage(diff);
g.DrawImage(_newBitmap, 0, 0, bounds, GraphicsUnit.Pixel);
g.Dispose();
// Set the current bitmap as the previous to prepare
// for the next screen capture.
//
diff.Save(#"d:\diff.bmp");
//return diff;
}
private Rectangle GetBoundingBoxForChanges(Bitmap _prevBitmap, Bitmap _newBitmap)
{
// The search algorithm starts by looking
// for the top and left bounds. The search
// starts in the upper-left corner and scans
// left to right and then top to bottom. It uses
// an adaptive approach on the pixels it
// searches. Another pass is looks for the
// lower and right bounds. The search starts
// in the lower-right corner and scans right
// to left and then bottom to top. Again, an
// adaptive approach on the search area is used.
//
// Note: The GetPixel member of the Bitmap class
// is too slow for this purpose. This is a good
// case of using unsafe code to access pointers
// to increase the speed.
//
// Validate the images are the same shape and type.
//
if (_prevBitmap.Width != _newBitmap.Width ||
_prevBitmap.Height != _newBitmap.Height ||
_prevBitmap.PixelFormat != _newBitmap.PixelFormat)
{
// Not the same shape...can't do the search.
//
return Rectangle.Empty;
}
// Init the search parameters.
//
int width = _newBitmap.Width;
int height = _newBitmap.Height;
int left = width;
int right = 0;
int top = height;
int bottom = 0;
BitmapData bmNewData = null;
BitmapData bmPrevData = null;
try
{
// Lock the bits into memory.
//
bmNewData = _newBitmap.LockBits(
new Rectangle(0, 0, _newBitmap.Width, _newBitmap.Height),
ImageLockMode.ReadOnly, _newBitmap.PixelFormat);
bmPrevData = _prevBitmap.LockBits(
new Rectangle(0, 0, _prevBitmap.Width, _prevBitmap.Height),
ImageLockMode.ReadOnly, _prevBitmap.PixelFormat);
// The images are ARGB (4 bytes)
//
int numBytesPerPixel = 4;
// Get the number of integers (4 bytes) in each row
// of the image.
//
int strideNew = bmNewData.Stride / numBytesPerPixel;
int stridePrev = bmPrevData.Stride / numBytesPerPixel;
// Get a pointer to the first pixel.
//
// Note: Another speed up implemented is that I don't
// need the ARGB elements. I am only trying to detect
// change. So this algorithm reads the 4 bytes as an
// integer and compares the two numbers.
//
System.IntPtr scanNew0 = bmNewData.Scan0;
System.IntPtr scanPrev0 = bmPrevData.Scan0;
// Enter the unsafe code.
//
unsafe
{
// Cast the safe pointers into unsafe pointers.
//
int* pNew = (int*)(void*)scanNew0;
int* pPrev = (int*)(void*)scanPrev0;
// First Pass - Find the left and top bounds
// of the minimum bounding rectangle. Adapt the
// number of pixels scanned from left to right so
// we only scan up to the current bound. We also
// initialize the bottom & right. This helps optimize
// the second pass.
//
// For all rows of pixels (top to bottom)
//
for (int y = 0; y < _newBitmap.Height; ++y)
{
// For pixels up to the current bound (left to right)
//
for (int x = 0; x < left; ++x)
{
// Use pointer arithmetic to index the
// next pixel in this row.
//
if ((pNew + x)[0] != (pPrev + x)[0])
{
// Found a change.
//
if (x < left)
{
left = x;
}
if (x > right)
{
right = x;
}
if (y < top)
{
top = y;
}
if (y > bottom)
{
bottom = y;
}
}
}
// Move the pointers to the next row.
//
pNew += strideNew;
pPrev += stridePrev;
}
// If we did not find any changed pixels
// then no need to do a second pass.
//
if (left != width)
{
// Second Pass - The first pass found at
// least one different pixel and has set
// the left & top bounds. In addition, the
// right & bottom bounds have been initialized.
// Adapt the number of pixels scanned from right
// to left so we only scan up to the current bound.
// In addition, there is no need to scan past
// the top bound.
//
// Set the pointers to the first element of the
// bottom row.
//
pNew = (int*)(void*)scanNew0;
pPrev = (int*)(void*)scanPrev0;
pNew += (_newBitmap.Height - 1) * strideNew;
pPrev += (_prevBitmap.Height - 1) * stridePrev;
// For each row (bottom to top)
//
for (int y = _newBitmap.Height - 1; y > top; y--)
{
// For each column (right to left)
//
for (int x = _newBitmap.Width - 1; x > right; x--)
{
// Use pointer arithmetic to index the
// next pixel in this row.
//
if ((pNew + x)[0] != (pPrev + x)[0])
{
// Found a change.
//
if (x > right)
{
right = x;
}
if (y > bottom)
{
bottom = y;
}
}
}
// Move up one row.
//
pNew -= strideNew;
pPrev -= stridePrev;
}
}
}
}
catch (Exception ex)
{
int xxx = 0;
}
finally
{
// Unlock the bits of the image.
//
if (bmNewData != null)
{
_newBitmap.UnlockBits(bmNewData);
}
if (bmPrevData != null)
{
_prevBitmap.UnlockBits(bmPrevData);
}
}
// Validate we found a bounding box. If not
// return an empty rectangle.
//
int diffImgWidth = right - left + 1;
int diffImgHeight = bottom - top + 1;
if (diffImgHeight < 0 || diffImgWidth < 0)
{
// Nothing changed
return Rectangle.Empty;
}
// Return the bounding box.
//
return new Rectangle(left, top, diffImgWidth, diffImgHeight);
}
when GetBoundingBoxForChanges() call then i am getting error and error message is Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
error occur at this code if ((pNew + x)[0] != (pPrev + x)[0])
so i am not being able to find out the reason. how to fix this error. please guide. thanks

bmNewData = _newBitmap.LockBits(...., _newBitmap.PixelFormat);
This algorithm implicitly assumes that a pixel has 4 bytes and can be addressed with an int*. It however fails to provide that guarantee. Asking for _newBitmap.PixelFormat in LockBits() is not sufficient, that just asks for the same format that the original image used. You'll get a hard crash if the images are 24bpp for example, very common.
Explicitly ask for 32bppArgb instead.

Instead of the Win32 API you could use a managed image processing library such as AForge.NET. In the documentation look for the AForge.Imaging.Filters.Difference class. It works with Bitmap objects so you will have to make minimal changes to your program.
Bitmap overlayImage;
Bitmap sourceImage;
//ToDo: Load the two images.
// Create filter.
Difference filter = new Difference(overlayImage);
// Apply the filter and return a new bitmap that is the difference between the source and overlay images.
Bitmap resultImage = filter.Apply(sourceImage);
// If you don't want a new image the you can apply the filter directly to the source image.
filter.ApplyInPlace(sourceImage);

This is the mechanism that I use to calculate image differences in C#. Note that it need to be compiled with the unsafe directive. Hope it helps:
using System;
using System.Collections.Generic;
using System.Text;
using System.Drawing;
using System.Drawing.Imaging;
using log4net;
namespace ImageDiff
{
public class ImageDifferences
{
private static ILog mLog = LogManager.GetLogger("ImageDifferences");
public static unsafe Bitmap PixelDiff(Image a, Image b)
{
if (!a.Size.Equals(b.Size)) return null;
if (!(a is Bitmap) || !(b is Bitmap)) return null;
return PixelDiff(a as Bitmap, b as Bitmap);
}
public static unsafe Bitmap PixelDiff(Bitmap a, Bitmap b)
{
Bitmap output = new Bitmap(
Math.Max(a.Width, b.Width),
Math.Max(a.Height, b.Height),
PixelFormat.Format32bppArgb);
Rectangle recta = new Rectangle(Point.Empty, a.Size);
Rectangle rectb = new Rectangle(Point.Empty, b.Size);
Rectangle rectOutput = new Rectangle(Point.Empty, output.Size);
BitmapData aData = a.LockBits(recta, ImageLockMode.ReadOnly, PixelFormat.Format32bppArgb);
BitmapData bData = b.LockBits(rectb, ImageLockMode.ReadOnly, PixelFormat.Format32bppArgb);
BitmapData outputData = output.LockBits(rectOutput, ImageLockMode.ReadWrite, PixelFormat.Format32bppArgb);
try
{
byte* aPtr = (byte*)aData.Scan0;
byte* bPtr = (byte*)bData.Scan0;
byte* outputPtr = (byte*)outputData.Scan0;
int len = aData.Stride * aData.Height;
for (int i = 0; i < len; i++)
{
// For alpha use the average of both images (otherwise pixels with the same alpha won't be visible)
if ((i + 1) % 4 == 0)
*outputPtr = (byte)((*aPtr + *bPtr) / 2);
else
*outputPtr = (byte)~(*aPtr ^ *bPtr);
outputPtr++;
aPtr++;
bPtr++;
}
return output;
}
catch (Exception ex)
{
mLog.Error("Error calculating image differences: " + ex.Message);
return null;
}
finally
{
a.UnlockBits(aData);
b.UnlockBits(bData);
output.UnlockBits(outputData);
}
}
}
}