We Keep Coding

Find a smaller image within another big one and fast

Anyway to make this thing go faster ? coz right now it's like 6 seconds on the sourceImage the size of 1024x768 and template 50x50 around. This is using AForge, if anyone knows other faster and rather simple ways please submit.
The task i'm trying to do is to find a smaller image within a screenshot. And preferably fast my limit is 1 second. The image i'm looking for is a red rectangle simple image and the screenshot is more complex.
System.Drawing.Bitmap sourceImage = (Bitmap)Bitmap.FromFile(#"C:\SavedBMPs\1.jpg");
System.Drawing.Bitmap template = (Bitmap)Bitmap.FromFile(#"C:\SavedBMPs\2.jpg");
// create template matching algorithm's instance
// (set similarity threshold to 92.5%)
ExhaustiveTemplateMatching tm = new ExhaustiveTemplateMatching(0.921f);
// find all matchings with specified above similarity
TemplateMatch[] matchings = tm.ProcessImage(sourceImage, template);
// highlight found matchings
BitmapData data = sourceImage.LockBits(
new Rectangle(0, 0, sourceImage.Width, sourceImage.Height),
ImageLockMode.ReadWrite, sourceImage.PixelFormat);
foreach (TemplateMatch m in matchings)
{
Drawing.Rectangle(data, m.Rectangle, Color.White);
MessageBox.Show(m.Rectangle.Location.ToString());
// do something else with matching
}
sourceImage.UnlockBits(data);

http://opencv.willowgarage.com/wiki/FastMatchTemplate - here you can find interesting idea for speeding up the template matching using two steps, first try to match downsampled images and when found match the original ones with smaller search region.
Also there is opencv implementation of template matching in matchTemplate function. This function is ported to GPU which can get significant speed up.
See the following
http://opencv.willowgarage.com/documentation/cpp/object_detection.html - matchTemplate function.
http://opencv.willowgarage.com/wiki/OpenCV_GPU - about OpenCV functionality ported to GPU.

Verify image sequence

Problem
Problem shaping
Image sequence position and size are fixed and known beforehand (it's not scaled). It will be quite short, maximum of 20 frames and in a closed loop. I want to verify (event driven by button click), that I have seen it before.
Lets say I have some image sequence, like:
http://img514.imageshack.us/img514/5440/60372aeba8595eda.gif
If seen, I want to see the ID associated with it, if not - it will be analyzed and added as new instance of image sequence, that has been seen. I have though about this quite a while, and I admit, this might be a hard problem. I seem to be having hard time of putting this all together, can someone assist (in C#)?
Limitations and uses
I am not trying to recreate copyright detection system, like content id system Youtube has implemented (Margaret Gould Stewart at TED ( link )). The image sequence can be thought about like a (.gif) file, but it is not and there is no direct way to get binary. Similar method could be used, to avoid duplicates in "image sharing database", but it is not what I am trying to do.
My effort
Gaussian blur
Mathematica function to generate Gaussian blur kernels:
getKernel[L_] := Transpose[{L}].{L}/(Total[Total[Transpose[{L}].{L}]])
getVKernel[L_] := L/Total[L]
Turns out, that it is much more efficient to use 2 passes of vector kernel, then matrix kernel. Thy are based on Pascal triangle uneven rows:
{1d/4, 1d/2, 1d/4}
{1d/16, 1d/4, 3d/8, 1d/4, 1d/16}
{1d/64, 3d/32, 15d/64, 5d/16, 15d/64, 3d/32, 1d/64}
Data input, hashing, grayscaleing and lightboxing
Example of source bits, that might be useful:
Lightbox around the known rectangle: FrameX
Using MD5CryptoServiceProvider to get md5 hash of the content inside known rectangle atm.
Using ColorMatrix to grayscale image
Source example
Source example (GUI; code):
Get current content inside defined rectangle.
private Bitmap getContentBitmap() {
Rectangle r = f.r;
Bitmap hc = new Bitmap(r.Width, r.Height);
using (Graphics gf = Graphics.FromImage(hc)) {
gf.CopyFromScreen(r.Left, r.Top, 0, 0, //
new Size(r.Width, r.Height), CopyPixelOperation.SourceCopy);
}
return hc;
}
Get md5 hash of bitmap.
private byte[] getBitmapHash(Bitmap hc) {
return md5.ComputeHash(c.ConvertTo(hc, typeof(byte[])) as byte[]);
}
Get grayscale of the image.
public static Bitmap getGrayscale(Bitmap hc){
Bitmap result = new Bitmap(hc.Width, hc.Height);
ColorMatrix colorMatrix = new ColorMatrix(new float[][]{
new float[]{0.5f,0.5f,0.5f,0,0}, new float[]{0.5f,0.5f,0.5f,0,0},
new float[]{0.5f,0.5f,0.5f,0,0}, new float[]{0,0,0,1,0,0},
new float[]{0,0,0,0,1,0}, new float[]{0,0,0,0,0,1}});
using (Graphics g = Graphics.FromImage(result)) {
ImageAttributes attributes = new ImageAttributes();
attributes.SetColorMatrix(colorMatrix);
g.DrawImage(hc, new Rectangle(0, 0, hc.Width, hc.Height),
0, 0, hc.Width, hc.Height, GraphicsUnit.Pixel, attributes);
}
return result;
}

I think you have a few issues with this:
Not all image sequences [videos] are equal [but many are similar]
Where is your data coming from?
How will you repesent the data related to your viewings?
Size of the data
Issue #1:
Many images can differ slightly by compression, water marking, missing frames, and adding clips. I would suggest sampling the video. For example you may want to consider sub-sampling small sections of the images in the video. Additionally, to avoid noisy images and issues with lossely compression algorithms. You may want to consider grayscaling the frames sampled, and doing a gaussian blur. [Guassian because its "more natural" (short answer)] Once you have enough sub samples to where you have a good confidence of similarity to the video then store it in a database. With the samples you can hash them, or store them to do a % similarity later.
Issue #2
Your datasource is going to influence the tool kits, and libraries that you use.
I would suggest keeping this simple [keep it with gifs and create a custom viewer, dont' try to write a browser plugin while developing your logic]
Issue #3
Using something like Postgres [if there are a lot of large sized objects] or SQLLite is highly suggested for indexing, storing, and recalling past meta data.
Issue #4
The size of the data will have a huge determination on recall, sampling, querying the database, etc.
Overall advice: Don't bite off more than you can handle at this stage. Start small and then grow.
Also take a look at Computer Vision algorithms for more help on the object representation/recall.

The question itself is sure very interesting and challenging, however there are many practical issues as stated by #monksy.
The opportunist pragmatic in me would take a step back, look at the big picture and see if there is another way to solve the problem. For example, if you are building some kind of "image sharing community" and want to avoid duplicates in the database, you could do a simple md5 on the file (animated gifs on the web are usually always the same, it's rare that people modify them).
Another example: if you are analyzing scientific samples (like meteo sequences) it may be easier to directly embed some kind of hash in every file when generating them.

This depends on wether you only want to know wether you've seen an absolutely identical movie again, or you also want to identify movies that are very similar but have been changed a bit (made lighter, have a watermark added, compression changed, etc.)
In the first case, just take any type of hash of the file and use that (because the file will be identical on the binary level.
In the second case (which I think is what you want) you have an interesting image processing problem on your hands. You could find yourself at the front-lines of image processing science with this if you'd want. If that is the case I suggest you start reading about SURF and OpenCV, and continue on from that.
If you want to match very similar, but not identical videos, and don't want to go the ultra-robus scientific route then I'd suggest the following process:
Do the gaussian blur you already do.
Divide each image into a few equally sized rectangles (you'd have to test for the best number, but I'd suggest you start with 9.
For each rectangle in each frame compute the full-colour histogram, then find the most occurring colour in that rectangle. This gives you 9*20 = 180 numbers. This is the "fingerprint" of this movie.
Find the most similar fingerprint in your database, if it is similar enough you already know about it, otherwise you don't.
Step 4 is a bit vague because I'm not really into this field. You are currently using an MD5 hash as a sort of fingerprint, but this is unsuitable in this case because slight differences in the input of a good cryptographic hashing function produce very large differences in the hash. This will mean that two very similar frames will have a totally different MD5 hash, so from the hash you'd never know they were similar.
As long as speed of database lookups is not an issue I'd just go for the sum of square differences as a measure of fingerprint similarity, and set a threshold on that to identify equal movies. However, this is not very fast for huge datasets, and in those cases you'd probably need to transform your fingerprint to something that will allow you to find similar fingerprints faster. One thing you could do here is start by selecting all known movies with very similar average colour for the entire video, then from that select the movies that have very similar average colour in each frame, and in the ones that remain at that point do the full rectangle-by-rectangle fingerprint match. But I'm sure there are even faster options for matching 180 numbers.

Perhaps you can find a way to get a binary copy of the image data of each frame in a variable. Hash that data (md5?) and store each of the hashes. Then you can see if you've ever seen that hash before. If you haven't, it's a new frame.

Faking thermal image camera with aforge or any other .net solution , emgucv, tbeta

I am simulating a thermal camera effect. I have a webcam at a party pointed at people in front of a wall. I went with background subtraction technique and using Aforge blobcounter I get blobs that I want to fill with gradient coloring. My problem = GetBlobsEdgePoints doesn't return sorted point cloud so I can't use it with, for example, PathGradientBrush from GDI+ to simply draw gradients.
I'm looking for simple,fast, algorithm to trace blobs into path (can make mistakes).
A way to track blobs received by blobcounter.
A suggestion for some other way to simulate the effect.
I took a quick look at Emgu.CV.VideoSurveillance but didn't get it to work (examples are for v1.5 and I went with v2+) but I gave up because people say it's slow on forums.
thanks for reading.
sample code of aforge background removal
Bitmap bmp =(Bitmap)e.VideoFrame.Clone();
if (backGroundFrame == null)
{
backGroundFrame = (Bitmap)e.VideoFrame.Clone();
difference.OverlayImage = backGroundFrame;
}
difference.ApplyInPlace(bmp);
bmp = grayscale.Apply(bmp);
threshold.ApplyInPlace(bmp);

Well, could you post some sample image of the result of GetBlobsEdgePoints, then it might be easier to understand what types if image processing algorithms are needed.
1) You may try a greedy algorithm, first pick a point at random, mark that point as "taken", pick the closest point not marked as "taken" and so on.
You need to find suitable termination conditions. If there can be several disjunct paths you need to find out a definition of how far away points need to be to be part of disjunct paths.
3) If you have a static background you can try to create a difference between two time shifted images, like 200ms apart. Just do a pixel by pixel difference and use abs(diff) as index in your heat color map. That will give more like an edge glow effect of moving objects.

This is the direction i'm going to take (looks best for now):
Define a set of points on the blob by my own logic (color of skin blobs should be warmer etc..)
draw gradients around those points
GraphicsPath gp=new GraphicsPath();
var rect = new Rectangle(CircumferencePoint.X - radius, CircumferencePoint.Y - radius, radius*2, radius*2);
gp.AddEllipse(rect);
GradientShaper = new PathGradientBrush(gp);
GradientShaper.CenterColor = Color.White;
GradientShaper.SurroundColors = surroundingColors;
drawBmp.FillPath(GradientShaper,gp);
mask those gradients with blob shape
blobCounter.ExtractBlobsImage(bmp,blob,true);
mask.OverlayImage = blob.Image;
mask.ApplyInPlace(rslt);
colorize with color remapping
tnx for the help #Albin

How to check if an image is a scaled version of another image

I am looking for an EASY way to check if an image is a scaled version of another image. It does not have to be very fast, it just should be "fairly" accurate. And written in .NET. And for free.
I know, wishful thinking :-)
I am pretty sure, even without having tried it, that converting the bigger image to the smaller scale and comparing checksums is not working (especially if the smaller version was done with another software then .NET).
The next approach would be to scale down and compare pixels. But first of all, it seems like a really bad idea running a loop over all pixels with a bool comparison results, I am sure there will be some pixels off by a bit or so...
Any library coming to mind? Way back in the university we had some MPEG7 classes, so I am thinking about using a combination of "statistics" like tone distribution, brightness, etc..
Any ideas or links for that topic?
Thanks,
Chris

I think this is going to be your best solution. First check the aspect ratio. Then scale the images to the smaller of the 2 if they're not the same size. Finally, do a hash comparison of the 2 images. This is a lot faster than doing a pixel compare. I found the hash compare method in a post from someone else and just adapted the answer here to fit. I was trying to think of the best way to do this myself for a project where I'm going to have to compare over 5200 images. After I read a few of the posts here I realized I already had everything I needed for it and figured I'd share.
public class CompareImages2
{
public enum CompareResult
{
ciCompareOk,
ciPixelMismatch,
ciAspectMismatch
};
public static CompareResult Compare(Bitmap bmp1, Bitmap bmp2)
{
CompareResult cr = CompareResult.ciCompareOk;
//Test to see if we have the same size of image
if (bmp1.Size.Height / bmp1.Size.Width == bmp2.Size.Height / bmp2.Size.Width)
{
if (bmp1.Size != bmp2.Size)
{
if (bmp1.Size.Height > bmp2.Size.Height)
{
bmp1 = (new Bitmap(bmp1, bmp2.Size));
}
else if (bmp1.Size.Height < bmp2.Size.Height)
{
bmp2 = (new Bitmap(bmp2, bmp1.Size));
}
}
//Convert each image to a byte array
System.Drawing.ImageConverter ic = new System.Drawing.ImageConverter();
byte[] btImage1 = new byte[1];
btImage1 = (byte[])ic.ConvertTo(bmp1, btImage1.GetType());
byte[] btImage2 = new byte[1];
btImage2 = (byte[])ic.ConvertTo(bmp2, btImage2.GetType());
//Compute a hash for each image
SHA256Managed shaM = new SHA256Managed();
byte[] hash1 = shaM.ComputeHash(btImage1);
byte[] hash2 = shaM.ComputeHash(btImage2);
//Compare the hash values
for (int i = 0; i < hash1.Length && i < hash2.Length && cr == CompareResult.ciCompareOk; i++)
{
if (hash1[i] != hash2[i])
cr = CompareResult.ciPixelMismatch;
}
}
else cr = CompareResult.ciAspectMismatch;
return cr;
}
}

One idea to achieve this:
If the image is 10x10, and your original is 40x40
Loop each pixel in the 10x10, then retrieve the 4 pixels representative of that looped pixel.
So for each pixel in the smaller image, find the corresponding scaled amount of pixels in the larger image.
You can then take the average colour of the 4 pixels, and compare with the pixel in the smaller image. You can specify error bounds, IE -10% or +10% bounds are considered a match, others are considered a failure.
Build up a count of matches and failures and use the bounds to determine if it is considered a match or not.
I think this might perform better than scaling the image to the same size and doing a 1pixel:1pixel comparison as I'm not sure how resizing algorithms necesserially work and you might lose some detail which will give less accurate results. Or if there might be different ways and methods of resizing images. But, again I don't know how the resize might work depends on how you go about doing it.

Just scale the larger image back to the size of the smaller one, then compare each pixel by taking the absolute value of the difference in each of the red, green and blue components.
You can then set a threshold for deciding how close you need to be to count it as a match, e.g. if 95%+ of the pixels are within 5% of the colour value, you have a match.
The fuzzy match is necessary because you may have scaling artefacts / anti-aliasing effects.

You'll have to loop over the pixels at some point or another.
Something that is easy to implement yet quite powerful is to calculate the difference between individual color components (RGB) for each pixel, find the average, and see if it crosses a certain threshold. It's certainly not the best method, but for a quick check it should do.

I'd have said roughly what Tom Gullen except I'd just scale down the bigger image to the smaller before comparing (otherwise you're just going to have hard maths if you are comparing a 25x25 with a 30x30 or something).
The other thing I might consider depending on image sizes is to scale them both down to a smaller image. ie if you have one that is 4000x4000 and another that is 3000x3000 then you can scale them both down to 200x200 and compare them at that size.
As others have said you would then need to do a check with a threshold (preferably on colour components) and decide what tolerances work best. I'd suggest this is probably best done by trial and error.

The easiest way is just to scale the biggest image to the smaller images size and compare color difference. Since you don't know if the scaling is cubic or linear (or something else) you have to accept a small difference.
Don't forget to take the absolute value of each pixel difference. ;)

Having absolutely no authority or experience in this area I'm going to make a stab at helping you.
I'd start with the aspect ratio matching by some tolerance, unless you're comparing cropped sections of images, which will makes things a bit harder.
I'd then scan the pixels for regions of similarity, no exactness, again a tolerance level is needed. Then when an area is similar, run along in a straight line comparing one to the other, and find another similarly coloured area. Black & white's gonna be harder.
If you get a hit, you'll have two areas in a line with patches of likeness. With two points you have a reference of length between them and so now you can see what the scaling might be. You could also scale the images first, but this doesn't account for cropped sections where aspects don't match.
Now choose a random point in the source image and get the colour info. Then using the scale factor, find that same random point on the other image and see if the colour checks out. Do it a few times with random points. If many turn up similar it's likely a copy.
You might then want to mark it for further, more CPU intensive, inspection. Either a pixel by pixel comparison or something else.
I know Microsoft (Photosynth) use filters like "outline" (the sort of stuff in Photoshop) to remove the image colours and leave just squrly lines which leave just the 'components' of the picture for matching (they match boundaries and overlap).
For speed, I'd break the problem down into chunks and really think about how humans decide two photos are similar. For non-speed, exhaustively comparing colour will probably get you there.
The process in short:
If you hole punched a sheet of paper randomly 4 times, then put it over two photos, just by seeing the colours coming through you could tell if they were likely a copy and need further inspection.

How do I compare two images & recognize the pattern in an image?

How do I compare two images & recognize the pattern in an image irrespective of its size and pattern size, and using .Net C#? Also, which algorithms are used for doing so from Image Processing?

See Scale-invariant feature transform, template matching, and Hough transform. A quick and inaccurate guess may be to make a histogram of color and compare it. If the image is complicated enough, you might be able to distinguish between several sets of images.
To make the matter simple, assume we have three buckets for R, G, and B. A completely white image would have (100%, 100%, 100%) for (R, G, B). A completely red image would have (100%, 0%, 0%). A complicated image might have something like (23%, 53%, 34%). If you take the distance between the points in that (R, G, B) space, you can compare which one is "closer".

look up pattern recognition. I known very little about it other than the name.
Warning: If that is what you want, it is one of the hardest "real world" programming problems known.

I am no expert in image recognition by I once stummbeled upon the AForge library which is written in C# and does image recognition. Maybe it can help...

Techniques for image matching and image recognition can be very different. For the first task, you may make use of SIFT or hand craft your own distance function, based on RGB or otherwise. For recognition, there a vast amount of machine learning techniques that you can use, more popular techniques involves Adaboost, SVM and other hybrid neural networks method. There are no lack of related research papers in this field. Google is your friend.

Jinmala, you've asked a question here that is extremely broad. There are literally thousands of papers in the literature about these topics. There is no correct answer, and there are many unsolved issues in the comparison of images, so you really probably can't hope for a simple solution that just works (unless your situation is quite simple and constrained)
If you narrow things down, I might be able to help.

You might be looking for this
System.Drawing.Bitmap sourceImage = (Bitmap)Bitmap.FromFile(#"C:\SavedBMPs\1.jpg");
System.Drawing.Bitmap template = (Bitmap)Bitmap.FromFile(#"C:\SavedBMPs\2.jpg");
// create template matching algorithm's instance
// (set similarity threshold to 92.5%)
ExhaustiveTemplateMatching tm = new ExhaustiveTemplateMatching(0.921f);
// find all matchings with specified above similarity
TemplateMatch[] matchings = tm.ProcessImage(sourceImage, template);
// highlight found matchings
BitmapData data = sourceImage.LockBits(
new Rectangle(0, 0, sourceImage.Width, sourceImage.Height),
ImageLockMode.ReadWrite, sourceImage.PixelFormat);
foreach (TemplateMatch m in matchings)
{
Drawing.Rectangle(data, m.Rectangle, Color.White);
MessageBox.Show(m.Rectangle.Location.ToString());
// do something else with matching
}
sourceImage.UnlockBits(data);
I warn you it is quite slow takes around 6 seconds to process image of 1024x768 finding in it pciture with the size of 50x50.enter code here

template matching, you can do this with EmguCV ,OpendotnetCV,Aforge.net

Scale-invariant feature transform (SIFT) might be what you're looking for. It's not simple to understand or implement, however.