We Keep Coding

Algorithm to scale down YUV 4:2:2

Trying to write an efficient algorithm to scale down YUV 4:2:2 by a factor of 2 - and which doesn't require a conversion to RGB (which is CPU intensive).
I've seen plenty of code on stack overflow for YUV to RGB conversion - but only an example of scaling for YUV 4:2:0 here which I have started based my code on. However, this produces an image which is effectively 3 columns of the same image with corrupt colours, so something is wrong with the algo when applied to 4:2:2.
Can anybody see what is wrong with this code?
public static byte[] HalveYuv(byte[] data, int imageWidth, int imageHeight)
{
byte[] yuv = new byte[imageWidth / 2 * imageHeight / 2 * 3 / 2];
int i = 0;
for (int y = 0; y < imageHeight; y += 2)
{
for (int x = 0; x < imageWidth; x += 2)
{
yuv[i] = data[y * imageWidth + x];
i++;
}
}
for (int y = 0; y < imageHeight / 2; y += 2)
{
for (int x = 0; x < imageWidth; x += 4)
{
yuv[i] = data[(imageWidth * imageHeight) + (y * imageWidth) + x];
i++;
yuv[i] = data[(imageWidth * imageHeight) + (y * imageWidth) + (x + 1)];
i++;
}
}
return yuv;
}

A fast way to generate a low quality thumbnail would be to discard half of the data in each dimension.
We break the image in 4x2 grid of pixels - each pair of pixels in the grid is represented by 4 bytes. In the down-scaled image, we take the color values for the first 2 pixels in the grid by copying the first 4 bytes, whilst discarding the other 12 bytes worth of data.
This scaling can be generalized to any power of 2 (1/2, 1/4, 1/8, ...) - this method is quick because it doesn't use any interpolation. This will give a lower quality image which appears blocky however - for better results consider some sampling approach.
public static byte[] FastResize(
byte[] data,
int imageWidth,
int imageHeight,
int scaleDownExponent)
{
var scaleDownFactor = (uint)Math.Pow(2, scaleDownExponent);
var outputImageWidth = imageWidth / scaleDownFactor;
var outputImageHeight = imageHeight / scaleDownFactor;
// 2 bytes per pixel.
byte[] yuv = new byte[outputImageWidth * outputImageHeight * 2];
var pos = 0;
// Process every other line.
for (uint pixelY = 0; pixelY < imageHeight; pixelY += scaleDownFactor)
{
// Work in blocks of 2 pixels, we discard the second.
for (uint pixelX = 0; pixelX < imageWidth; pixelX += 2*scaleDownFactor)
{
// Position of pixel bytes.
var start = ((pixelY * imageWidth) + pixelX) * 2;
yuv[pos] = data[start];
yuv[pos + 1] = data[start + 1];
yuv[pos + 2] = data[start + 2];
yuv[pos + 3] = data[start + 3];
pos += 4;
}
}
return yuv;
}

I assume that the original data is in the following order (as it seems so from your example code): First there are the luminance (Y) values of the pixels of the image (size = imageWidth*imageHeight bytes). After that there are the chrominance components UV, s.t., the values for a single pixel are given after each other. This means that the total size of the original image is 3*size.
Now for 4:2:2 subsampling means that every other value of the horizontal chrominance component are discarded. This reduces the data to size size + 0.5*size + 0.5*size = 2*size, i.e., luminance is kept completely and both chrominance components are divided to half. Therefore, the result image should be allocated as:
byte[] yuv = new byte[2*imageWidth*imageHeight];
As the first part of the image is copied in full the first loop becomes:
int i = 0;
for (int y = 0; y < imageHeight; y++)
{
for (int x = 0; x < imageWidth; x++)
{
yuv[i] = data[y * imageWidth + x];
i++;
}
}
Because this just copies the beginning of data this can be simplified to
int size = imageHeight*imageWidth;
int i = 0;
for (; i < size; i++)
{
yuv[i] = data[i];
}
Now to copy the rest we need to skip every other horizontal coordinate
for (int y = 0; y < imageHeight; y++)
{
for (int x = 0; x < imageWidth; x += 2) // +2 skip each other horizontal component
{
yuv[i] = data[size + y*2*imageWidth + 2*x];
i++;
yuv[i] = data[size + y*2*imageWidth + 2*x + 1];
i++;
}
}
The factor two in data-array index is needed because there are 2 bytes for each pixel (both chrominance components), so each "row" has 2*imageWidth bytes of data.

Low pass filter with directsound implementation issue

Currently I am developing a c# low pass filter in real time using directsound API. The problem that I've encountered is that various implementations of low pass filters seem not to work with my echo implementation. At the begining of the filtering I have an array of bytes (read) that is read from the capturing buffer. The values of this array are between 0 and 255.
When I subject the array to low-pass filtering algorithm I obtain values that fall between 0 and 1 and as an effect the play array is silent (zeros only). I attach the relevant code.
I imagine that that this approach is wrong and probably the input data should look different as the low pass algorithm seems to kill it completely. Any ideas? Thanks in advance.
Updated relevant code after KillaKem's comments:
//buffer format
var format = new WaveFormat
{
SamplesPerSecond = 44100,
BitsPerSample = 16,
Channels = 2,
FormatTag = WaveFormatTag.Pcm
};
//reading the buffer
byte[] read = (byte[])_dwCapBuffer.Read(offset, typeof(byte), LockFlag.None, _dwOutputBufferSize);
byte[] play = new byte [read.Length];
byte[] readL= new byte [read.Length/2];
byte[] readR= new byte [read.Length/2];
byte[] playL = new byte[read.Length / 2];
byte[] playR = new byte[read.Length / 2];
float[] readLfloat = new float[read.Length / 4];
float[] readRfloat = new float[read.Length / 4];
float[] playLfloat = new float[read.Length / 4];
float[] playRfloat = new float[read.Length / 4];
//dividing into channels and casting to float
for(int i = 0; i<read.Length; i=i+1)
{
if (i % 4 == 0)
{
readL[(int)i / 2] = read[i];
}
if(i%4==1)
{
readL[(int)i/2]=read[i];
readLfloat[(int)i / 4] = (((short)(read[i-1] << 8 | read[i])) / 32768f);
}
if (i % 4 == 2)
{
readR[(int)i / 2] = read[i];
}
if (i % 4 == 3)
{
readR[(int)i / 2] = read[i];
readRfloat[(int)i / 4] = (((short)(read[i - 1] << 8 | read[i])) / 32768f);
}
}
//filter coefficients
float frequency = 1000f;
float sampleRate = (float)_dwCapBuffer.Format.SamplesPerSecond;
float resonance = 0.5f;
float c = 1.0f / (float)Math.Tan(Math.PI * frequency / sampleRate);
float a0 = 1.0f / (1.0f + resonance * c + c * c);
float a1 = 2f * a0;
float a2 = a0;
float b1 = 2.0f * (1.0f - c * c) * a0;
float b2 = (1.0f - resonance * c + c * c) * a0;
//filtering
for(int i = 0; i < readLfloat.Length; i++)
{
float readCurrSample = readLfloat[i];
float playCurrSample = playLfloat[i];
float filtered=readCurrSample;
float readOneSample;
float readTwoSample;
float playOneSample;
float playTwoSample;
if (i ==0)
{
filtered = ((float)a0 * readCurrSample) + ((float)a1 * savelastRead) + ((float)a2 * saveprelastRead) - ((float)b1 * savelastPlay) - ((float)b2 * saveprelastPlay);
}
else if (i==1)
{
readOneSample = readLfloat[i-1];
playOneSample = playLfloat[i-1];
filtered = ((float)a0 * readCurrSample) + ((float)a1 * readOneSample) + ((float)a2 * savelastRead) - ((float)b1 * playOneSample) - ((float)b2 * savelastPlay);
}
else
{
readOneSample = readLfloat[i - 1];
playOneSample = playLfloat[i - 1];
readTwoSample = readLfloat[i - 2];
playTwoSample = playLfloat[i - 2];
filtered = ((float)a0 * readCurrSample) + ((float)a1 * readOneSample) + ((float)a2 * readTwoSample) - ((float)b1 * playOneSample) - ((float)b2 * playTwoSample);
}
if (i == readL.Length - 4)
{
saveprelastPlay = playCurrSample;
saveprelastRead = readCurrSample;
}
if (i == readL.Length-2)
{
savelastPlay = playCurrSample;
savelastRead = readCurrSample;
}
if (filtered > 1 || filtered < -1)
{
int x = 0;
}
playLfloat[i] = filtered;
}
playRfloat = playLfloat; //ignoring Right channel operations
//Recasting to bytes array
for (int i = 0; i < read.Length; i = i + 1)
{
if (i % 4 == 1)
{
byte[] bytes;
bytes = BitConverter.GetBytes((short)(playLfloat[(int)(i-1)/4] * 32768f));
read[i] = bytes[0];
read[i - 1] = bytes[1];
}
if (i % 4 == 3)
{
byte[] bytes;
bytes = BitConverter.GetBytes((short)(playRfloat[(int)(i - 1) / 4] * 32768f));
read[i] = bytes[0];
read[i - 1] = bytes[1];
}
}

Hamming window generates a line

I have an audio file and read all data from the sound card buffer. Then, I convert byte[] to float[] to use them for hamming window. The waveform of the audio is:
after using hamming window:
is the waveform of audio with hamming window right? Where is my mistake?
by the way i use naudio library to process audio:
WaveChannel32 wave = new WaveChannel32(new WaveFileReader("sesDosyası.wav"));
byte []buffer = new byte[wave.length];
float []data = new float[wave.length / 4];
int read = wave.Read(buffer, 0, wave.length);
for (int i = 0; i < read / 4; i++)
{
data[i] = BitConverter.ToSingle(buffer, i * 4); //converting byte to float
chart1.Series["wave"].Points.Add(data[i]); //first waveform
}
for (int j = 0; j < read/4; j++)
{
data[j] = (float)(0.54 - 0.46 * Math.Cos((2 * Math.PI * data[j]) / (read / 4 - 1)));//hamming
chart2.Series["wave"].Points.Add(data[j]); //second waveform
}

It appears you are applying the window to the whole wave, so read is going to be huge, so the term inside the cos is always going to be very near 0 for a data between [-1,1].
So you are always getting .54 - .46*cos(0) = .54 - .46*1.0 = .08
From wikipedia, only j should be inside the cosine - that gives the window, which you then multiply by data[j]:
window = 0.54 - 0.46 * Math.Cos( (2*Math.PI * j)/(total - 1) ).
hammed_signal = data[j]*window;
Why are you trying to apply a hamming window to the whole wave?

I think your hamming line is wrong:
data[j] = (float)(0.54 - 0.46 * Math.Cos((2 * Math.PI * data[j]) / (read - 1)));
As your loop goes from 0 to read/4, and you are dividing by read, so if you have read 16 samples, your for loop looks only at the first 4, but divides by 15, not 3.
var total = read / 4;
for (int j = 0; j < total; j++)
{
data[j] = (float)(0.54 - 0.46 * Math.Cos((2 * Math.PI * data[j]) / (total- 1)));//hamming
chart2.Series["wave"].Points.Add(data[j]); //second waveform
}

IndexOutOfRangeException while running FFTPitchDetector

I'm facing some problems on implementing FFTPitchDetector. What I actually want to do is to get real-time frequency from guitar input, I not so sure how to use the functions in FftPitchDetector.cs. Any idea?
private void sourceStream_DataAvailable(object sender, NAudio.Wave.WaveInEventArgs e)
{
if (waveWriter == null) return;
byte[] buffer = e.Buffer;
float[] floats = new float[buffer.Length];
float sample32 = 0;
int bytesRecorded = e.BytesRecorded;
waveWriter.Write(buffer, 0, bytesRecorded);
for (int index = 0; index < e.BytesRecorded; index += 2)
{
short sample = (short)((buffer[index + 1] << 8) |
buffer[index + 0]);
sample32 = sample / 32768f;
sampleAggregator.Add(sample32);
}
floats = bytesToFloats(buffer);
FftPitchDetector PitchDetect = new FftPitchDetector(sample32);
PitchDetect.DetectPitch(floats, bytesRecorded);
Console.WriteLine("{0}",sample32);
}
private static float[] bytesToFloats(byte[] bytes)
{
float[] floats = new float[bytes.Length / 2];
for (int i = 0; i < bytes.Length; i += 2)
{
floats[i / 2] = bytes[i] | (bytes[i + 1] << 8);
}
return floats;
}
When I execute the code, there's an error IndexOutOfRangeException was unhandled which points to the line
fftBuffer[n * 2] = buffer[n-inFrames] * window(n, frames);
in fftPitchDetector.cs. What is the problem in my code?
Is there any open source code of C# Guitar Tuner? I wish to outsource it into my project.

The issue is that in either fftBuffer[] or buffer[] you are trying to access an index in the array which does not exist.
So if fftBuffer[] has 4 items and n * 2 totals 6 then you're trying to access fftBuffer[6] which doesn't exist.
So basically you need to check how many items are in each array before trying to access their values.
You will need to place checks in your code, for example:
if ((n * 2) < fftBuffer.length && (n - inFrames) < buffer.length) {
fftBuffer[n * 2] = buffer[n-inFrames] * window(n, frames);
}
You first ensure that the indexes you are about to use for each array is not greater then the number of items the array.

Memory Object Allocation failure using c# and opencl

I am writing an image processing program with the express purpose to alter large images, the one I'm working with is 8165 pixels by 4915 pixels. I was told to implement gpu processing, so after some research I decided to go with OpenCL. I started implementing the OpenCL C# wrapper OpenCLTemplate.
My code takes in a bitmap and uses lockbits to lock its memory location. I then copy the order of each bit into an array, run the array through the openCL kernel, and it inverts each bit in the array. I then run the inverted bits back into the memory location of the image. I split this process into ten chunks so that i can increment a progress bar.
My code works perfectly with smaller images, but when I try to run it with my big image I keep getting a MemObjectAllocationFailure when trying to execute the kernel. I don't know why its doing this and i would appreciate any help in figuring out why or how to fix it.
using OpenCLTemplate;
public static void Invert(Bitmap image, ToolStripProgressBar progressBar)
{
string openCLInvert = #"
__kernel void Filter(__global uchar * Img0,
__global float * ImgF)
{
// Gets information about work-item
int x = get_global_id(0);
int y = get_global_id(1);
// Gets information about work size
int width = get_global_size(0);
int height = get_global_size(1);
int ind = 4 * (x + width * y );
// Inverts image colors
ImgF[ind]= 255.0f - (float)Img0[ind];
ImgF[1 + ind]= 255.0f - (float)Img0[1 + ind];
ImgF[2 + ind]= 255.0f - (float)Img0[2 + ind];
// Leave alpha component equal
ImgF[ind + 3] = (float)Img0[ind + 3];
}";
//Lock the image in memory and get image lock data
var imageData = image.LockBits(new Rectangle(0, 0, image.Width, image.Height), ImageLockMode.ReadWrite, PixelFormat.Format32bppArgb);
CLCalc.InitCL();
for (int i = 0; i < 10; i++)
{
unsafe
{
int adjustedHeight = (((i + 1) * imageData.Height) / 10) - ((i * imageData.Height) / 10);
int count = 0;
byte[] Data = new byte[(4 * imageData.Stride * adjustedHeight)];
var startPointer = (byte*)imageData.Scan0;
for (int y = ((i * imageData.Height) / 10); y < (((i + 1) * imageData.Height) / 10); y++)
{
for (int x = 0; x < imageData.Width; x++)
{
byte* Byte = (byte*)(startPointer + (y * imageData.Stride) + (x * 4));
Data[count] = *Byte;
Data[count + 1] = *(Byte + 1);
Data[count + 2] = *(Byte + 2);
Data[count + 3] = *(Byte + 3);
count += 4;
}
}
CLCalc.Program.Compile(openCLInvert);
CLCalc.Program.Kernel kernel = new CLCalc.Program.Kernel("Filter");
CLCalc.Program.Variable CLData = new CLCalc.Program.Variable(Data);
float[] imgProcessed = new float[Data.Length];
CLCalc.Program.Variable CLFiltered = new CLCalc.Program.Variable(imgProcessed);
CLCalc.Program.Variable[] args = new CLCalc.Program.Variable[] { CLData, CLFiltered };
kernel.Execute(args, new int[] { imageData.Width, adjustedHeight });
CLCalc.Program.Sync();
CLFiltered.ReadFromDeviceTo(imgProcessed);
count = 0;
for (int y = ((i * imageData.Height) / 10); y < (((i + 1) * imageData.Height) / 10); y++)
{
for (int x = 0; x < imageData.Width; x++)
{
byte* Byte = (byte*)(startPointer + (y * imageData.Stride) + (x * 4));
*Byte = (byte)imgProcessed[count];
*(Byte + 1) = (byte)imgProcessed[count + 1];
*(Byte + 2) = (byte)imgProcessed[count + 2];
*(Byte + 3) = (byte)imgProcessed[count + 3];
count += 4;
}
}
}
progressBar.Owner.Invoke((Action)progressBar.PerformStep);
}
//Unlock image
image.UnlockBits(imageData);
}

You may have reached a memory allocation limit of your OpenCL driver/device. Check the values returned by clGetDeviceInfo. There is a limit for the size of one single memory object. The OpenCL driver may allow the total size of all allocated memory objects to exceed the memory size on your device, and will copy them to/from host memory when needed.
To process large images, you may have to split them into smaller pieces, and process them separately.