We Keep Coding

Improve performance of per-pixel image multiplication with mask and thresholding

I am looking for suggestions to make this algorithm run faster. Using C# and EmguCV (a managed wrapper over OpenCV), I am processing images in real-time, adjusting the gain of pixels that are not fully saturated (basically a flat-field correction with a threshold). The code works correctly, but the CPU usage is relatively high due to frame-rate and image size.
I have dabbled a bit with using UMat to perform the same operations, but performance has been worse. I also tried a version with integer math (i.e., doing integer multiplication and division instead of a FP multiply), but that was slower than leaving it in floating point.
I have profiled the code and optimized it some by doing things like reusing my Mat objects, pointing Mats to existing blocks of memory instead allocating new memory, etc. Suggestions using OpenCV would also be welcome, and I will find/use the EmguCV equivalents.
var ushortImage = new Mat(_frameH, _frameW, DepthType.Cv16U, 1, (IntPtr)pixelsPtr, _frameW * 2);
//get mask of all the pixels that are at max-value
CvInvoke.InRange(ushortImage, _lowerRange, _upperRange, _byteMask);
//convert input to float for multiplication
ushortImage.ConvertTo(_floatFrame, DepthType.Cv32F);
//multiply each pixel by it's gain value
CvInvoke.Multiply(_floatFrame, gainMat, _floatFrame);
//threshold to allowable range
CvInvoke.Threshold(_floatFrame, _floatFrame, _frameMaxPixelValue, _frameMaxPixelValue, ThresholdType.Trunc);
//convert back to 16-bit
_floatFrame.ConvertTo(ushortImage, DepthType.Cv16U);
//restore the max-intensity pixels
ushortImage.SetTo(new MCvScalar(_frameMaxPixelValue), _byteMask);
Here is the profiling info from dotTrace. Nothing really stands out as being the bottleneck, so I'm hoping there's a way to combine some operations into a single function call. Even a 10% improvement would be helpful because I'm trying to basically double the throughout of the application.

Downsample a double array to another length

I have a data source that provides many (4096) double values in an array. These are measured with high resolution and are the result of a FFT. For visualisation purposes, they need to be reduced. (Reapplying the FFT on the raw signal is not possible here.) I could simply average each n samples and have a resulting array of length / n values. To allow more flexible selection of the number of resulting values, I need interpolation, though.
I've looked up some basic information about this on Wikipedia. I am already familiar with 2D downsampling/interpolation from a user prespective in raster image editors. Now I need this in 1D in C# code. Think of it as changing (reducing) the raster image size of a 1D barcode image, or resampling an audio wave file.
One library I've found recommended is Math.NET Numerics. This is already used for other tasks in my application, so I could easily use that. There's the CubicSpline class in there but I have no idea how to use it.
Q: What would be an approach to reduce the number of samples in a double[] to an arbitrary number using interpolation?
I'm not interested in finding a single double value between two others. I need to combine multiple source values into each a single output value, while not losing any information (single frequency bins with an extreme level) at the boundaries of the groups, and without aliasing effects or rounding because of different group sizes if the numbers aren't divisible.
Maybe the use of bitmap image functions and a 1*n bitmap size is a good solution instead of dealing with the math directly? This would involve a lot of data conversion, though, which reduces performance and probably also precision. Or some library from the autio processing field?

Determine if an image is blurred

I saw a lot a topic about this, I understood the theory but I'm not able to code this.
I have some pictures and I want to determine if they are blurred or not. I found a library (aforge.dll) and I used it to compte a FFT for an image.
As an example, there is two images i'm working on :
My code is in c# :
public Bitmap PerformFFT(Bitmap Picture)
{
//Loade Image
ComplexImage output = ComplexImage.FromBitmap(Picture);
// Perform FFT
output.ForwardFourierTransform();
// return image
return = output.ToBitmap();
}
How can I determine if the image is blurred ? I am not very comfortable with the theory, I need concret example. I saw this post, but I have no idea how to do that.
EDIT:
I'll clarify my question. When I have a 2D array of complex ComplexImage output (image FFT), what is the C# code (or pseudo code) I can use to determine if image is blurred ?

The concept of "blurred" is subjective. How much power at high frequencies indicates it's not blurry? Note that a blurry image of a complex scene has more power at high frequencies than a sharp image of a very simple scene. For example a sharp picture of a completely uniform scene has no high frequencies whatsoever. Thus it is impossible to define a unique blurriness measure.
What is possible is to compare two images of the same scene, and determine which one is more blurry (or identically, which one is sharper). This is what is used in automatic focussing. I don't know how exactly what process commercial cameras use, but in microscopy, images are taken at a series of focal depths, and compared.
One of the classical comparison methods doesn't involve Fourier transforms at all. One computes the local variance (for each pixel, take a small window around it and compute the variance for those values), and averages it across the image. The image with the highest variance has the best focus.
Comparing high vs low frequencies as in MBo's answer would be comparable to computing the Laplace filtered image, and averaging its absolute values (because it can return negative values). The Laplace filter is a high-pass filter, meaning that low frequencies are removed. Since the power in the high frequencies gives a relative measure of sharpness, this statistic does too (again relative, it is to be compared only to images of the same scene, taken under identical circumstances).

Blurred image has FFT result with smaller magnitude in high-frequency regions. Array elements with low indexes (near Result[0][0]) represent low-frequency region.
So divide resulting array by some criteria, sum magnitudes in both regions and compare them. For example, select a quarter of result array (of size M) with index<M/2 and indexy<M/2
For series of more and more blurred image (for the same initial image) you should see higher and higher ratio Sum(Low)/Sum(High)
Result is square array NxN. It has central symmetry (F(x,y)=F(-x,-y) because source is pure real), so it is enough to treat top half of array with y<N/2.
Low-frequency components are located near top-left and top-right corners of array (smallest values of y, smallest and highest values of x). So sum magnitudes of array elements in ranges
for y in range 0..N/2
for x in range 0..N
amp = magnitude(y,x)
if (y<N/4) and ((x<N/4)or (x>=3*N/4))
low = low + amp
else
high = high + amp
Note that your picture shows jumbled array pieces - this is standard practice to show zero component in the center.

Simple brightness but slow

I have a picture, and I get every pixels and multiply the RGB for a number. I also need to take care when R * number > 255. When this happen, r = 255.
It's taking about 10s+ to complete a 1024x768 image. The common softwares that do brightness, takes less than 1s to do it. Any ideias to improve my strategy? Thanks.

I had a similar problem:
How to use ColorMatrix in .NET to change Brightness, Color, Saturation, Hue
For brightness alone, colormatrix will work fine. If you want to start using contrast, etc, you will need to use some other solution. It seems to be SetPixel is the slowest part. See this solution for doing this quickly:
http://www.codeproject.com/KB/GDI-plus/csharpgraphicfilters11.aspx

Using a ColorMatrix would probably be the best way to go. Here's an article to get you on your way: http://www.c-sharpcorner.com/UploadFile/mahesh/Transformations0512192005050129AM/Transformations05.aspx

when I did some simple image manipulation on multi megabyte images I significantly improved performance using unsafe code and pointer manipulation to get at the to the raw bytes.
This might get you in the right direction http://wcode.net/2009/08/unsafe-in-c-and-image-processing/

well this site helped me a lot:
http://blogs.msdn.com/b/llobo/archive/2007/03/08/bitmapsource-bitmap-interop.aspx

How to check if an image is a scaled version of another image

I am looking for an EASY way to check if an image is a scaled version of another image. It does not have to be very fast, it just should be "fairly" accurate. And written in .NET. And for free.
I know, wishful thinking :-)
I am pretty sure, even without having tried it, that converting the bigger image to the smaller scale and comparing checksums is not working (especially if the smaller version was done with another software then .NET).
The next approach would be to scale down and compare pixels. But first of all, it seems like a really bad idea running a loop over all pixels with a bool comparison results, I am sure there will be some pixels off by a bit or so...
Any library coming to mind? Way back in the university we had some MPEG7 classes, so I am thinking about using a combination of "statistics" like tone distribution, brightness, etc..
Any ideas or links for that topic?
Thanks,
Chris

I think this is going to be your best solution. First check the aspect ratio. Then scale the images to the smaller of the 2 if they're not the same size. Finally, do a hash comparison of the 2 images. This is a lot faster than doing a pixel compare. I found the hash compare method in a post from someone else and just adapted the answer here to fit. I was trying to think of the best way to do this myself for a project where I'm going to have to compare over 5200 images. After I read a few of the posts here I realized I already had everything I needed for it and figured I'd share.
public class CompareImages2
{
public enum CompareResult
{
ciCompareOk,
ciPixelMismatch,
ciAspectMismatch
};
public static CompareResult Compare(Bitmap bmp1, Bitmap bmp2)
{
CompareResult cr = CompareResult.ciCompareOk;
//Test to see if we have the same size of image
if (bmp1.Size.Height / bmp1.Size.Width == bmp2.Size.Height / bmp2.Size.Width)
{
if (bmp1.Size != bmp2.Size)
{
if (bmp1.Size.Height > bmp2.Size.Height)
{
bmp1 = (new Bitmap(bmp1, bmp2.Size));
}
else if (bmp1.Size.Height < bmp2.Size.Height)
{
bmp2 = (new Bitmap(bmp2, bmp1.Size));
}
}
//Convert each image to a byte array
System.Drawing.ImageConverter ic = new System.Drawing.ImageConverter();
byte[] btImage1 = new byte[1];
btImage1 = (byte[])ic.ConvertTo(bmp1, btImage1.GetType());
byte[] btImage2 = new byte[1];
btImage2 = (byte[])ic.ConvertTo(bmp2, btImage2.GetType());
//Compute a hash for each image
SHA256Managed shaM = new SHA256Managed();
byte[] hash1 = shaM.ComputeHash(btImage1);
byte[] hash2 = shaM.ComputeHash(btImage2);
//Compare the hash values
for (int i = 0; i < hash1.Length && i < hash2.Length && cr == CompareResult.ciCompareOk; i++)
{
if (hash1[i] != hash2[i])
cr = CompareResult.ciPixelMismatch;
}
}
else cr = CompareResult.ciAspectMismatch;
return cr;
}
}

One idea to achieve this:
If the image is 10x10, and your original is 40x40
Loop each pixel in the 10x10, then retrieve the 4 pixels representative of that looped pixel.
So for each pixel in the smaller image, find the corresponding scaled amount of pixels in the larger image.
You can then take the average colour of the 4 pixels, and compare with the pixel in the smaller image. You can specify error bounds, IE -10% or +10% bounds are considered a match, others are considered a failure.
Build up a count of matches and failures and use the bounds to determine if it is considered a match or not.
I think this might perform better than scaling the image to the same size and doing a 1pixel:1pixel comparison as I'm not sure how resizing algorithms necesserially work and you might lose some detail which will give less accurate results. Or if there might be different ways and methods of resizing images. But, again I don't know how the resize might work depends on how you go about doing it.

Just scale the larger image back to the size of the smaller one, then compare each pixel by taking the absolute value of the difference in each of the red, green and blue components.
You can then set a threshold for deciding how close you need to be to count it as a match, e.g. if 95%+ of the pixels are within 5% of the colour value, you have a match.
The fuzzy match is necessary because you may have scaling artefacts / anti-aliasing effects.

You'll have to loop over the pixels at some point or another.
Something that is easy to implement yet quite powerful is to calculate the difference between individual color components (RGB) for each pixel, find the average, and see if it crosses a certain threshold. It's certainly not the best method, but for a quick check it should do.

I'd have said roughly what Tom Gullen except I'd just scale down the bigger image to the smaller before comparing (otherwise you're just going to have hard maths if you are comparing a 25x25 with a 30x30 or something).
The other thing I might consider depending on image sizes is to scale them both down to a smaller image. ie if you have one that is 4000x4000 and another that is 3000x3000 then you can scale them both down to 200x200 and compare them at that size.
As others have said you would then need to do a check with a threshold (preferably on colour components) and decide what tolerances work best. I'd suggest this is probably best done by trial and error.

The easiest way is just to scale the biggest image to the smaller images size and compare color difference. Since you don't know if the scaling is cubic or linear (or something else) you have to accept a small difference.
Don't forget to take the absolute value of each pixel difference. ;)

Having absolutely no authority or experience in this area I'm going to make a stab at helping you.
I'd start with the aspect ratio matching by some tolerance, unless you're comparing cropped sections of images, which will makes things a bit harder.
I'd then scan the pixels for regions of similarity, no exactness, again a tolerance level is needed. Then when an area is similar, run along in a straight line comparing one to the other, and find another similarly coloured area. Black & white's gonna be harder.
If you get a hit, you'll have two areas in a line with patches of likeness. With two points you have a reference of length between them and so now you can see what the scaling might be. You could also scale the images first, but this doesn't account for cropped sections where aspects don't match.
Now choose a random point in the source image and get the colour info. Then using the scale factor, find that same random point on the other image and see if the colour checks out. Do it a few times with random points. If many turn up similar it's likely a copy.
You might then want to mark it for further, more CPU intensive, inspection. Either a pixel by pixel comparison or something else.
I know Microsoft (Photosynth) use filters like "outline" (the sort of stuff in Photoshop) to remove the image colours and leave just squrly lines which leave just the 'components' of the picture for matching (they match boundaries and overlap).
For speed, I'd break the problem down into chunks and really think about how humans decide two photos are similar. For non-speed, exhaustively comparing colour will probably get you there.
The process in short:
If you hole punched a sheet of paper randomly 4 times, then put it over two photos, just by seeing the colours coming through you could tell if they were likely a copy and need further inspection.