We Keep Coding

searching image in another image, only return a partial result

The error with the following code is, it only find, partially, the first image itself.
Let me be more clear, I took a screenshot of an image, cropped it and putted it on the desktop. Then I saved the actual original image in full size (not cropped) on desktop as well.
When I run the code, it correctly detects all the coordinates of the cropped image on the desktop, however, when I open the full size image and hover it, it won't detect anything and return null, while it should be able to detect the pixels location inside the full size image as well.
I'm gonna post a video of what I mean, to be more clear:
https://www.youtube.com/watch?v=Ha3eGxWcAF8
As you see, it does not detect the pixels on the main image.
What's wrong on there?
Why it detects only the loaded cropped image, instead of a position on the screenshot based on the actual full fize image?
Code below:
public static Point? Find(Bitmap first, Bitmap second)
{
if (null == first || null == second)
{
return null;
}
if (first.Width < second.Width || first.Height < second.Height)
{
return null;
}
var firstArray = GetPixelArray(first);
var secondArray = GetPixelArray(second);
foreach (var firstLineMatchPoint in FindMatch(firstArray.Take(first.Height - second.Height), secondArray[0]))
{
if (IssecondPresentAtLocation(firstArray, secondArray, firstLineMatchPoint, 1))
{
return firstLineMatchPoint;
}
}
return null;
}
private static int[][] GetPixelArray(Bitmap bitmap)
{
var result = new int[bitmap.Height][];
var bitmapData = bitmap.LockBits(new Rectangle(0, 0, bitmap.Width, bitmap.Height), ImageLockMode.ReadOnly,
PixelFormat.Format32bppArgb);
for (int y = 0; y < bitmap.Height; ++y)
{
result[y] = new int[bitmap.Width];
Marshal.Copy(bitmapData.Scan0 + y * bitmapData.Stride, result[y], 0, result[y].Length);
}
bitmap.UnlockBits(bitmapData);
return result;
}
private static IEnumerable<Point> FindMatch(IEnumerable<int[]> firstLines, int[] secondLine)
{
var y = 0;
foreach (var firstLine in firstLines)
{
for (int x = 0, n = firstLine.Length - secondLine.Length; x < n; ++x)
{
if (ContainSameElements(firstLine, x, secondLine, 0, secondLine.Length))
{
yield return new Point(x, y);
}
}
y += 1;
}
}
private static bool ContainSameElements(int[] first, int firstStart, int[] second, int secondStart, int length)
{
for (int i = 0; i < length; ++i)
{
if (first[i + firstStart] != second[i + secondStart])
{
return false;
}
}
return true;
}
private static bool IssecondPresentAtLocation(int[][] first, int[][] second, Point point, int alreadyVerified)
{
//we already know that "alreadyVerified" lines already match, so skip them
for (int y = alreadyVerified; y < second.Length; ++y)
{
if (!ContainSameElements(first[y + point.Y], point.X, second[y], 0, second.Length))
{
return false;
}
}
return true;
}
static void Main(string[] args)
{
int counter = 1;
while (true)
{
Console.WriteLine("Executing pixel search");
Bitmap bitmap = new Bitmap(Screen.PrimaryScreen.Bounds.Width, Screen.PrimaryScreen.Bounds.Height);
Graphics graphics = Graphics.FromImage(bitmap);
graphics.CopyFromScreen(0, 0, 0, 0, bitmap.Size);
Bitmap bitmap2 = (Bitmap)Image.FromFile(#"path_to_desktop\cropped.jpg", true);
if (Find(bitmap, bitmap2) == null)
{
Console.WriteLine("Not found.");
}
else {
Console.WriteLine($"{Find(bitmap, bitmap2)}.");
}
System.Threading.Thread.Sleep(500); // loop to detect new position
counter++;
}
}

Edit 7.11.2020: I published a Sample Project and Video as a result of this question:
Sample:
https://github.com/DataJuggler/SubImageCreator
Could be called SubImageSearcher also.
Video:
How To Search Images Using C# And DataJuggler.PixelDatabase
https://youtu.be/JKc7QtdaxWY
Search for a sub image in a larger image using C#.
Nuget Updates
DataJuggler.PixelDatabase .Net Core
DataJuggler.PixelDatabase.Net .Net Framework
// Return a sub image after a PixelDatabase is loaded
Bitmap subImage = CreateSubImage(Point topLeft, Rectangle size);
// Search for a sub image inside a larger image
SearchResult result = PixelDatabase.SearchForSubImage(bitmap, searchDepth);
Search Result contains a Point of the top left and a score. A score of zero is a perfect match.
The sample project draws a little yellow rectangle on the Top Left hand corner if the sub image is found inside the larger image.
--
End Edit
Are the images something you can share? If not a sample image might give something to work with. If yes, tell me what you are trying to find and maybe I can find "where the pixels are you are searching for".
Another thing to watch out for is if either of the images were stretched, any difference like that could distort exact pixel colors and locations.
Are you using .Net Framework or .Net Core?
I have a Nuget package that might help you with this:
Nuget:
DataJuggler.PixelDatabase .Net Core
DataJuggler.PixelDatabase.Net .Net Framework
Both packages were updated yesterday.
Showing a .Net Core example
using DataJuggler.PixelDatabase;
// local
DataJuggler.PixelDatabase.PixelDatabase pixelDatabase = null;
// Load the pixelDatabase (you can also pass in a Bitmap or an Image also)
pixelDatabase = PixelDatabaseLoader.LoadPixelDatabase(FullImagePath, null);
// get information about a pixel at the coordinates given
PixelInformation pixel = PixelDatabase.GetPixel(x, y);
The pixel information object contains the Red Green and Blue colors, plus the X, Y and a lot of read only properties such as Total, BlueRed, MinMaxDifference, etc., that make querying and updating Bitmaps pretty simple.
Another thing you could do is if you know of a certain color you are looking for:
// color
Color color = Color.FromArgb(128, 55, 92);
// get the pixels that match a certain color
List<PixelInformation> pixels = PixelDatabase.GetPixels(color);
That might help you locate certain groups of pixels in the image.
The same code that powers the Nuget packages is also used on my site:
https://PixelDatabase.Net
The site and the Nuget packages use a language called Bitmap Query Language, or BQL for short. This is very similar to SQL for databases if you have learned that yet.
There are lots of help and videos, but if you do any Pixel querying, I think you will find it worth the price of free.
The full source code is on Git Hub:
https://github.com/DataJuggler/PixelDatabase.Net .Net Framework
https://github.com/DataJuggler/PixelDatabase .Net Core
And Help is located here:
https://pixeldatabase.net/Help
And lots of videos here:
https://www.youtube.com/playlist?list=PLKrW5tXCPiX2PxrLPszDzlcEZwQG-Qb8r

GameCard value recognition gives wrong result with AForge.NET

Actually I'm using AForge.NET to recognize suit and value from a gamecard.
Here is a snippet where I decide which suit/value is there:
public static CardTemplateIdentifier GetBestMatchingIdentifier(this List<CardTemplateIdentifier> templates, Bitmap bitmap)
{
float maxSimilar = 0f;
CardTemplateIdentifier result = null;
foreach (var template in templates)
{
// Identify similarity between template and bmp
ExhaustiveTemplateMatching tm = new ExhaustiveTemplateMatching(0);
TemplateMatch[] matchings = tm.ProcessImage(bitmap, template.Sample);
// If the currently tested template fits better than the best one so far,
// set the value as the identified card value
if (matchings.Length > 0 && matchings[0].Similarity > maxSimilar)
{
maxSimilar = matchings[0].Similarity;
result = template;
}
}
return result;
}
CardTemplateIdentifier contains a list with all possible cards and their comparism sample.
...
Now If trying to recognize the images, I can not rely on this being reliably recognized: Sometimes the scaling of the images differs from the one I made the sample. Or the font has got less thicknes, see example below:
I think I'm on the wrong way to detect the values from the images. Is there a better or convience way how to solve this problem?

Morphological Operations On Image

I am currently doing a project in which I am trying to identify humans based on hands vascular pattern in C# using Emgu CV.
The gray-scale image of the hand was first processed using the Adaptive Threshold function.
Now I want to create a mask of the image using the morphological operations.
The purpose is to remove the noise from the image.
This is the adaptive-thresholded image:
Kindly guide me which function should I use and how to use.

The code here is in C++. It shouldn't be difficult to port to C#, since it's mostly OpenCV functions calls. You can use that as guideline. Sorry about that.
You can apply a open operation with a small kernel to remove most of the noise:
Mat1b opened;
Mat kernel = getStructuringElement(MORPH_ELLIPSE, Size(3, 3));
morphologyEx(thresholded, opened, MORPH_OPEN, kernel);
As you can see, some noise is still present, and you can't remove it with other morphological operations. You can simply consider the largest blob as the correct one (in green here):
Then you can floodfill the inside of the hand (in gray here):
And set to 0 all values in original image where the corresponding mask in not the same color of the inside of the image:
This is the full code (again, it's C++):
#include <opencv2/opencv.hpp>
using namespace cv;
int main(int, char**)
{
// Load grayscale image
Mat1b thresholded = imread("path_to_image", IMREAD_GRAYSCALE);
// Get rid of JPEG compression artifacts
thresholded = thresholded > 100;
// Needed so findContours handles borders contours correctly
Mat1b bin;
copyMakeBorder(thresholded, bin, 1,1,1,1, BORDER_CONSTANT, 0);
// Apply morphological operation "close"
Mat1b closed;
Mat kernel = getStructuringElement(MORPH_ELLIPSE, Size(3, 3));
morphologyEx(bin, closed, MORPH_OPEN, kernel);
// Find contours
vector<vector<Point>> contours;
findContours(bin.clone(), contours, RETR_EXTERNAL, CHAIN_APPROX_NONE, Point(-1,-1)); // Point(-1,-1) accounts for previous copyMakeBorder
// Keep largest contour
int size_largest = 0;
int idx_largest = -1;
for (int i = 0; i < contours.size(); ++i)
{
if (contours[i].size() > size_largest)
{
size_largest = contours[i].size();
idx_largest = i;
}
}
Mat3b dbg;
cvtColor(closed, dbg, COLOR_GRAY2BGR);
// Black initialized mask
Mat1b mask(thresholded.rows, thresholded.cols, uchar(0));
if (idx_largest >= 0)
{
drawContours(dbg, contours, idx_largest, Scalar(0, 255, 0), CV_FILLED);
// Draw filled polygin on mask
drawContours(mask, contours, idx_largest, Scalar(255), 1);
}
// Get a point inside the contour
Moments m = moments(contours[idx_largest]);
Point2f inside(m.m10 / m.m00, m.m01 / m.m00);
floodFill(mask, inside, Scalar(127));
Mat3b result;
cvtColor(thresholded, result, COLOR_GRAY2BGR);
result.setTo(Scalar(0), mask != 127);
imshow("Closed", closed);
imshow("Contour", dbg);
imshow("Result", result);
waitKey();
return 0;
}

How to implement pan/zoom on gigapixel bitmaps?

In my project, I'm using (uncompressed 16-bit grayscale) gigapixel images which come from a high resolution scanner for measurement purposes. Since these bitmaps can not be loaded in memory (mainly due to memory fragmentation) I'm using tiles (and tiled TIFF on disc). (see StackOverflow topic on this)
I need to implement panning/zooming in a way like Google Maps or DeepZoom. I have to apply image processing on the fly before presenting it on screen, so I can not use a precooked library which directly accesses an image file. For zooming I intend to keep a multi-resolution image in my file (pyramid storage). The most useful steps seem to be +200%, 50% and show all.
My code base is currently C# and .NET 3.5. Currently I assume Forms type, unless WPF gives me great advantage in this area. I have got a method which can return any (processed) part of the underlying image.
Specific issues:
hints or references on how to implement this pan/zoom with on-demand generation of image parts
any code which could be used as a basis (preferably commercial or LGPL/BSD like licenses)
can DeepZoom be used for this (i.e. is there a way that I can provide a function to provide a tile at the right resulution for the current zoom level?) ( I need to have pixel accurate addressing still)

This CodeProject article: Generate...DeepZoom Image Collection might be a useful read since it talks about generating a DeepZoom image source.
This MSDN article has a section Dynamic Deep Zoom: Supplying Image Pixels at Run Time and links to this Mandelbrot Explorer which 'kinda' sounds similar to what you're trying to do (ie. he is generating specific parts of the mandelbrot set on-demand; you want to retrieve specific parts of your gigapixel image on-demand).
I think the answer to "can DeepZoom be used for this?" is probably "Yes", however as it is only available in Silverlight you will have to do some tricks with an embedded web browser control if you need a WinForms/WPF client app.
Sorry I can't provide more specific answers - hope those links help.
p.s. I'm not sure if Silverlight supports TIFF images - that might be an issue unless you convert to another format.

I decided to try something myself. I came up with a straightforward GDI+ code, which uses the tiles I've already got. I just filter out the parts which are relevant for current clipping region. It works like magic! Please find my code below.
(Form settings double buffering for the best results)
protected override void OnPaint(PaintEventArgs e)
{
base.OnPaint(e);
Graphics dc = e.Graphics;
dc.ScaleTransform(1.0F, 1.0F);
Size scrollOffset = new Size(AutoScrollPosition);
int start_x = Math.Min(matrix_x_size,
(e.ClipRectangle.Left - scrollOffset.Width) / 256);
int start_y = Math.Min(matrix_y_size,
(e.ClipRectangle.Top - scrollOffset.Height) / 256);
int end_x = Math.Min(matrix_x_size,
(e.ClipRectangle.Right - scrollOffset.Width + 255) / 256);
int end_y = Math.Min(matrix_y_size,
(e.ClipRectangle.Bottom - scrollOffset.Height + 255) / 256);
// start * contain the first and last tile x/y which are on screen
// and which need to be redrawn.
// now iterate trough all tiles which need an update
for (int y = start_y; y < end_y; y++)
for (int x = start_x; x < end_x; x++)
{ // draw bitmap with gdi+ at calculated position.
dc.DrawImage(BmpMatrix[y, x],
new Point(x * 256 + scrollOffset.Width,
y * 256 + scrollOffset.Height));
}
}
To test it, I've created a matrix of 80x80 of 256 tiles (420 MPixel). Of course I'll have to add some deferred loading in real life. I can leave tiles out (empty) if they are not yet loaded. In fact, I've asked my client to stick 8 GByte in his machine so I don't have to bother about performance too much. Once loaded tiles can stay in memory.
public partial class Form1 : Form
{
bool dragging = false;
float Zoom = 1.0F;
Point lastMouse;
PointF viewPortCenter;
private readonly Brush solidYellowBrush = new SolidBrush(Color.Yellow);
private readonly Brush solidBlueBrush = new SolidBrush(Color.LightBlue);
const int matrix_x_size = 80;
const int matrix_y_size = 80;
private Bitmap[,] BmpMatrix = new Bitmap[matrix_x_size, matrix_y_size];
public Form1()
{
InitializeComponent();
Font font = new Font("Times New Roman", 10, FontStyle.Regular);
StringFormat strFormat = new StringFormat();
strFormat.Alignment = StringAlignment.Center;
strFormat.LineAlignment = StringAlignment.Center;
for (int y = 0; y < matrix_y_size; y++)
for (int x = 0; x < matrix_x_size; x++)
{
BmpMatrix[y, x] = new Bitmap(256, 256, PixelFormat.Format24bppRgb);
// BmpMatrix[y, x].Palette.Entries[0] = (x+y)%1==0?Color.Blue:Color.White;
using (Graphics g = Graphics.FromImage(BmpMatrix[y, x]))
{
g.FillRectangle(((x + y) % 2 == 0) ? solidBlueBrush : solidYellowBrush, new Rectangle(new Point(0, 0), new Size(256, 256)));
g.DrawString("hello world\n[" + x.ToString() + "," + y.ToString() + "]", new Font("Tahoma", 8), Brushes.Black,
new RectangleF(0, 0, 256, 256), strFormat);
g.DrawImage(BmpMatrix[y, x], Point.Empty);
}
}
BackColor = Color.White;
Size = new Size(300, 300);
Text = "Scroll Shapes Correct";
AutoScrollMinSize = new Size(256 * matrix_x_size, 256 * matrix_y_size);
}
Turned out this was the easy part. Getting async multithreaded i/o done in the background was a lot harder to acchieve. Still, I've got it working in the way described here. The issues to resolve were more .NET/Form multithreading related than to this topic.
In pseudo code it works like this:
after onPaint (and on Tick)
check if tiles on display need to be retrieved from disc
if so: post them to an async io queue
if not: check if tiles close to display area are already loaded
if not: post them to an async io/queue
check if bitmaps have arrived from io thread
if so: updat them on screen, and force repaint if visible
Result: I now have my own Custom control which uses roughly 50 MByte for very fast access to arbitrary size (tiled) TIFF files.

I guess you can address this issue following the steps below:
Image generation:
segment your image in multiple subimages (tiles) of a small resolution, for instace, 500x500. These images are depth 0
combine a series of tiles with depth 0 (4x4 or 6x6), resize the combination generating a new tile with 500x500 pixels in depth 1.
continue with this approach until get the entire image using only a few tiles.
Image visualization
Start from the highest depth
When user drags the image, load the tiles dynamically
When the user zoom a region of the image, decrease the depth, loading the tiles for that region in a higher resolution.
The final result is similar to Google Maps.

C# Image.Clone Out of Memory Exception

Why am I getting an out of memory exception?
So this dies in C# on the first time through:
splitBitmaps.Add(neededImage.Clone(rectDimensions, neededImage.PixelFormat));
Where splitBitmaps is a List<BitMap> BUT this works in VB for at least 4 iterations:
arlSplitBitmaps.Add(Image.Clone(rectDimensions, Image.PixelFormat))
Where arlSplitBitmaps is a simple array list. (And yes I've tried arraylist in c#)
This is the fullsection:
for (Int32 splitIndex = 0; splitIndex <= numberOfResultingImages - 1; splitIndex++)
{
Rectangle rectDimensions;
if (splitIndex < numberOfResultingImages - 1)
{
rectDimensions = new Rectangle(splitImageWidth * splitIndex, 0,
splitImageWidth, splitImageHeight);
}
else
{
rectDimensions = new Rectangle(splitImageWidth * splitIndex, 0,
sourceImageWidth - (splitImageWidth * splitIndex), splitImageHeight);
}
splitBitmaps.Add(neededImage.Clone(rectDimensions, neededImage.PixelFormat));
}
neededImage is a Bitmap by the way.
I can't find any useful answers on the intarweb, especially not why it works just fine in VB.
Update:
I actually found a reason (sort of) for this working but forgot to post it. It has to do with converting the image to a bitmap instead of just trying to clone the raw image if I remember.

Clone() may also throw an Out of memory exception when the coordinates specified in the Rectangle are outside the bounds of the bitmap. It will not clip them automatically for you.

I found that I was using Image.Clone to crop a bitmap and the width took the crop outside the bounds of the original image. This causes an Out of Memory error. Seems a bit strange but can beworth knowing.

I got this too when I tried to use the Clone() method to change the pixel format of a bitmap. If memory serves, I was trying to convert a 24 bpp bitmap to an 8 bit indexed format, naively hoping that the Bitmap class would magically handle the palette creation and so on. Obviously not :-/

This is a reach, but I've often found that if pulling images directly from disk that it's better to copy them to a new bitmap and dispose of the disk-bound image. I've seen great improvement in memory consumption when doing so.
Dave M. is on the money too... make sure to dispose when finished.

I struggled to figure this out recently - the answers above are correct. Key to solving this issue is to ensure the rectangle is actually within the boundaries of the image. See example of how I solved this.
In a nutshell, checked to if the area that was being cloned was outside the area of the image.
int totalWidth = rect.Left + rect.Width; //think -the same as Right property
int allowableWidth = localImage.Width - rect.Left;
int finalWidth = 0;
if (totalWidth > allowableWidth){
finalWidth = allowableWidth;
} else {
finalWidth = totalWidth;
}
rect.Width = finalWidth;
int totalHeight = rect.Top + rect.Height; //think same as Bottom property
int allowableHeight = localImage.Height - rect.Top;
int finalHeight = 0;
if (totalHeight > allowableHeight){
finalHeight = allowableHeight;
} else {
finalHeight = totalHeight;
}
rect.Height = finalHeight;
cropped = ((Bitmap)localImage).Clone(rect, System.Drawing.Imaging.PixelFormat.DontCare);

Make sure that you're calling .Dispose() properly on your images, otherwise unmanaged resources won't be freed up. I wonder how many images are you actually creating here -- hundreds? Thousands?