We Keep Coding

Unity Converting Microphone input into Hertz

I'm working on a Unity app that has some Microphone controls. At one point, I have to convert the Microphone input into Hertz (Hz) values and show them to the user. Now, I did some research and I made the following script for this purpose:
int amountSamples = 1024;
void Start ()
{
_fSample = AudioSettings.outputSampleRate;
}
void Update() {
if (focused && Initialized) {
if (Microphone.IsRecording(selectedDevice) && recording) {
spectrumData = GetSpectrumAnalysis();
if (spectrumCurve.keys.Length <= spectrumData.Length) {
float keyTimeValue = 0;
float currentHighestKeyTime = 0;
//create a curvefield if none exists
spectrumCurve = new AnimationCurve();
for (int t = 0; t < spectrumData.Length; t++) {
spectrumCurve.AddKey(1 / spectrumData.Length + t, spectrumData[t]);
spectrumCurve.MoveKey(1 / spectrumData.Length + t, new Keyframe(1 / spectrumData.Length + t, keyTimeValue = spectrumData[t])); //update keyframe value
if (keyTimeValue > currentHighestKeyTime) {
currentHighestKeyTime = keyTimeValue;
}
}
HighestKeyTimeValue = currentHighestKeyTime;
float freqN = HighestKeyTimeValue;
float f = freqN * (_fSample / 2) / amountSamples;
Debug.Log(f); //hz
}
}
}
audioSource.volume = 1;
}
And the GetSpectrumAnalysis()
public float[] GetSpectrumAnalysis ()
{
float[] dataSpectrum = new float[amountSamples];
audioSource.GetSpectrumData (dataSpectrum, 0, FFTWindow.BlackmanHarris);
for (int i = 0; i <= dataSpectrum.Length - 1; i++)
{
dataSpectrum[i] = Mathf.Abs (dataSpectrum[i] * sensitivity);
}
return dataSpectrum;
}
Now, with this code, the Hz value should be calculated in float f, it does work but the Hz values aren't too accurate, for example, I'm getting 400-500 Hz where I should get around 880 Hz. Similarly I'm getting 130 Hz instead of 220 Hz, etc.. So, I have 2 issues: I'm getting less Hz then I should and the Hz value is jumping too much and too fast so it's not consistent even if the sound playing is constant. Any idea how to improve this code? Where did I made a mistake?
EDIT
Check my answer for the solution.

Ok, nevermind, I found the solution, maybe this will help someone stumbling across this thread, change GetSpectrumAnalysis method to this:
public float test() {
float Threshold = 0.02f;
float[] dataSpectrum = new float[amountSamples];
audioSource.GetSpectrumData(dataSpectrum, 0, FFTWindow.BlackmanHarris); //Rectangular
float maxV = 0;
var maxN = 0;
for (int i = 0; i < amountSamples; i++) {
if (!(dataSpectrum[i] > maxV) || !(dataSpectrum[i] > Threshold)) {
continue;
}
maxV = dataSpectrum[i];
maxN = i; // maxN is the index of max
}
float freqN = maxN; // pass the index to a float variable
if (maxN > 0 && maxN < amountSamples - 1) { // interpolate index using neighbours
var dL = dataSpectrum[maxN - 1] / dataSpectrum[maxN];
var dR = dataSpectrum[maxN + 1] / dataSpectrum[maxN];
freqN += 0.5f * (dR * dR - dL * dL);
}
return freqN * (_fSample / 2) / amountSamples; // convert index to frequency
}
Then just call this in the update method like this:
Text.text = test().ToString("00");
For more info check out this thread: Unity answers

How to play audio samples without writing it into to a file?

I get audio samples (as float array) from another method
and I want to play this float array without writing any files to disk.
For this example I get samples from audio file:
var file = new AudioFileReader(stereoFile);
int startPos = 0;
int endPos = 138890;
var sampleArray = new float[endPos - startPos];
file.Read(sampleArray, startPos, sampleArray.Length);
How can I play sampleArray?

I find next solution and it work for me:
var file = new AudioFileReader(stereoFile);
WaveFormat waveFormat = file.WaveFormat;
int startPos = 2403;
int endPos = 41265;
if (startPos % 2 != 0)
{
startPos += 1;
}
if (endPos % 2 != 0)
{
endPos += 1;
}
if (waveFormat.Channels == 2)
{
startPos *= 2;
endPos *= 2;
}
var allSamples = new float[file.Length / (file.WaveFormat.BitsPerSample / 8)];
file.Read(allSamples, 0, allSamples.Length);
Span<float> samplesSpan = allSamples;
Span<float> trackSpan = samplesSpan.Slice(startPos, endPos - startPos);
^
upper code to take correct samples for tests.
code below - my solution
MemoryStream ms = new MemoryStream();
IgnoreDisposeStream ids = new IgnoreDisposeStream(ms);
using (WaveFileWriter waveFileWriter = new WaveFileWriter(ids, waveFormat))
{
waveFileWriter.WriteSamples(trackSpan.ToArray(), 0, endPos-startPos);
}
ms.Position = 0;
WaveStream waveStream = new WaveFileReader(ms);
WaveOutEvent waveOutEvent = new WaveOutEvent();
waveOutEvent.Init(waveStream);
waveOutEvent.Play();

RawSourceWaveStream is designed for the situation where you have audio in a byte array or MemoryStream and want to play it directly. You can learn more about it and see an example of how to use it here

C# Convolution algorithm is producing a very loud .wav file

I am trying to create a convolution reverb algorithm that takes a sound input signal and convolves it in the frequency domain with an impulse response. I have been trying to debug the code for a week and I cannot seem to find where the error is. The output of the algorithm is a .wav file that for some reason is very loud in the beginning but still sounds echoey at the end. I am using Aforge .net libraryLink to compute the FFT and the fault does not lie within the coding of that algorithm. Here is my code:
using System;
using System.Collections;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using AForge.Math;
using NWaves.Operations.Convolution;
using UnityEngine;
public class ConvolutionReverbTest : MonoBehaviour {
[SerializeField] private AudioClip input;
[SerializeField] private AudioClip impulseResponse;
// ir
private float[] irData;
private Complex[] irc;
// input
private float[] inputData;
private Complex[] inputc;
// conv. reverb values
private int fftSize = 0; // F = N + M - 1 --> Also to the power of 2
// muliplication in freq domain
private Complex[] multiplication;
// output signal
private float[] output;
private void Start() {
GetAudioData();
Convolve();
}
private void Convolve() {
// calculate the fft of input data
FourierTransform.FFT(inputc, FourierTransform.Direction.Forward);
// calculate fft of impulse response
FourierTransform.FFT(irc, FourierTransform.Direction.Forward);
// multiply them in frequency domain
multiplication = new Complex[fftSize];
for (int i = 0; i < fftSize; i++) {
multiplication[i] = inputc[i] * irc[i];
multiplication[i] *= fftSize; // normalize it
}
// calculate the ifft of the multiplication
FourierTransform.FFT(multiplication, FourierTransform.Direction.Backward);
// save output signal as wave file
output = new float[fftSize];
for (int i = 0; i < fftSize; i++) {
output[i] = (float) multiplication[i].Re;
}
SaveOutputSignal(output);
}
private Complex ComplexMultiply(Complex a, Complex b) {
return new Complex(a.Re * b.Re - a.Im * b.Im, a.Re * b.Im + a.Im * b.Re);
}
private void CalculateFftSize() {
fftSize = Mathf.NextPowerOfTwo(inputData.Length + irData.Length - 1);
}
private void GetAudioData() {
// get data of ir signal
irData = new float[impulseResponse.samples];
impulseResponse.GetData(irData, 0);
// get data of input signal
inputData = new float[input.samples];
input.GetData(inputData, 0);
CalculateFftSize();
// change values accordingly of both signals --> zero padding + complex numbers
irData = ZeroPadding(irData);
irc = new Complex[irData.Length];
for (int i = 0; i < irData.Length; i++) {
irc[i].Re = irData[i];
}
inputData = ZeroPadding(inputData);
inputc = new Complex[inputData.Length];
for (int i = 0; i < inputData.Length; i++) {
inputc[i].Re = inputData[i];
}
}
private float[] ZeroPadding(float[] data) {
List<float> newData = new List<float>(data);
int N = data.Length;
for (int i = 0; i < (fftSize - N); i++) {
newData.Add(0);
}
return newData.ToArray();
}
private void SaveOutputSignal(float[] outputSignal) {
string filename = "outputSignal.txt";
// save the output signal to a txt file
File.WriteAllLines(Path.Combine(Application.streamingAssetsPath, filename),
outputSignal.Select(d => d.ToString()));
new WaveFile().SaveAudio(Path.Combine(Application.streamingAssetsPath, "output.wav"), outputSignal.Length, 1, outputSignal, input.frequency);
}
}
Here is also a picture of the output signal in audacity:
Any help is greatly appreciated.

Unity: Live Video Streaming ARCore Camera Texture

I'm trying to use TCP sockets to stream the texture from the ARCore camera on one device to the background image on another device. I've found some great resources that have got me close. This question helped me a lot: Unity: Live Video Streaming
The only difference is that this example uses a WebcamTexture, which I've got working without any lag and smoothly.
However, obtaining the camera texture from ARCore is a bit of a different process. I think I'm doing this correctly but maybe I'm missing something. When I run this with the ARCore camera texture and my minor changes, the server lags significantly.
Below is my code. I made no changes to the client side code so I know it has to do with my server side changes.
SERVER:
public class MySocketServer : MonoBehaviour
{
WebCamTexture webCam;
public RawImage myImage;
public bool enableLog = false;
Texture2D currentTexture;
private TcpListener listner;
private const int port = 8010;
private bool stop = false;
private List<TcpClient> clients = new List<TcpClient>();
//This must be the-same with SEND_COUNT on the client
const int SEND_RECEIVE_COUNT = 15;
private void Start()
{
Application.runInBackground = true;
//Start WebCam coroutine
StartCoroutine(initAndWaitForWebCamTexture());
}
//Converts the data size to byte array and put result to the fullBytes array
void byteLengthToFrameByteArray(int byteLength, byte[] fullBytes)
{
//Clear old data
Array.Clear(fullBytes, 0, fullBytes.Length);
//Convert int to bytes
byte[] bytesToSendCount = BitConverter.GetBytes(byteLength);
//Copy result to fullBytes
bytesToSendCount.CopyTo(fullBytes, 0);
}
//Converts the byte array to the data size and returns the result
int frameByteArrayToByteLength(byte[] frameBytesLength)
{
int byteLength = BitConverter.ToInt32(frameBytesLength, 0);
return byteLength;
}
IEnumerator initAndWaitForWebCamTexture()
{
// Open the Camera on the desired device, in my case IPAD pro
//webCam = new WebCamTexture();
// Get all devices , front and back camera
//webCam.deviceName = WebCamTexture.devices[WebCamTexture.devices.Length - 1].name;
// request the lowest width and heigh possible
//webCam.requestedHeight = 10;
//webCam.requestedWidth = 10;
//myImage.texture = webCam;
//webCam.Play();
//currentTexture = new Texture2D(Screen.width, Screen.height, TextureFormat.RGB24,false);
//currentTexture = new Texture2D(webCam.width, webCam.height);
// Connect to the server
listner = new TcpListener(IPAddress.Any, port);
listner.Start();
while (Screen.width < 100)
{
yield return null;
}
//Start sending coroutine
StartCoroutine(senderCOR());
}
WaitForEndOfFrame endOfFrame = new WaitForEndOfFrame();
IEnumerator senderCOR()
{
bool isConnected = false;
TcpClient client = null;
NetworkStream stream = null;
// Wait for client to connect in another Thread
Loom.RunAsync(() =>
{
while (!stop)
{
// Wait for client connection
client = listner.AcceptTcpClient();
// We are connected
clients.Add(client);
isConnected = true;
stream = client.GetStream();
}
});
//Wait until client has connected
while (!isConnected)
{
yield return null;
}
LOG("Connected!");
bool readyToGetFrame = true;
byte[] frameBytesLength = new byte[SEND_RECEIVE_COUNT];
while (!stop)
{
//Wait for End of frame
yield return endOfFrame;
//currentTexture.ReadPixels(new Rect(0,0,Screen.width,Screen.height), 0, 0);
//currentTexture.Apply();
//NEW CODE TO TRY
using (var image = Frame.CameraImage.AcquireCameraImageBytes()) {
if(!image.IsAvailable)
{
yield return null;
}
_OnImageAvailable(image.Width, image.Height, image.Y, image.Width * image.Height);
}
//currentTexture.SetPixels(webCam.GetPixels());
byte[] pngBytes = currentTexture.EncodeToPNG();
//Fill total byte length to send. Result is stored in frameBytesLength
byteLengthToFrameByteArray(pngBytes.Length, frameBytesLength);
//Set readyToGetFrame false
readyToGetFrame = false;
Loom.RunAsync(() =>
{
//Send total byte count first
stream.Write(frameBytesLength, 0, frameBytesLength.Length);
LOG("Sent Image byte Length: " + frameBytesLength.Length);
//Send the image bytes
stream.Write(pngBytes, 0, pngBytes.Length);
LOG("Sending Image byte array data : " + pngBytes.Length);
//Sent. Set readyToGetFrame true
readyToGetFrame = true;
});
//Wait until we are ready to get new frame(Until we are done sending data)
while (!readyToGetFrame)
{
LOG("Waiting To get new frame");
yield return null;
}
}
}
void LOG(string messsage)
{
if (enableLog)
Debug.Log(messsage);
}
private void Update()
{
myImage.texture = webCam;
}
// stop everything
private void OnApplicationQuit()
{
if (webCam != null && webCam.isPlaying)
{
webCam.Stop();
stop = true;
}
if (listner != null)
{
listner.Stop();
}
foreach (TcpClient c in clients)
c.Close();
}
private void _OnImageAvailable(int width, int height, IntPtr pixelBuffer, int bufferSize){
currentTexture = new Texture2D(width, height, TextureFormat.RGBA32, false, false);
byte[] bufferYUV = new byte[width * height * 3 / 2];
bufferSize = width * height * 3 / 2;
System.Runtime.InteropServices.Marshal.Copy(pixelBuffer, bufferYUV, 0, bufferSize);
Color color = new Color();
for (int y = 0; y < height; y++)
{
for (int x = 0; x < width; x++)
{
float Yvalue = bufferYUV[y * width + x];
float Uvalue = bufferYUV[(y / 2) * (width / 2) + x / 2 + (width * height)];
float Vvalue = bufferYUV[(y / 2) * (width / 2) + x / 2 + (width * height) + (width * height) / 4];
color.r = Yvalue + (float)(1.37705 * (Vvalue - 128.0f));
color.g = Yvalue - (float)(0.698001 * (Vvalue - 128.0f)) - (float)(0.337633 * (Uvalue - 128.0f));
color.b = Yvalue + (float)(1.732446 * (Uvalue - 128.0f));
color.r /= 255.0f;
color.g /= 255.0f;
color.b /= 255.0f;
if (color.r < 0.0f)
color.r = 0.0f;
if (color.g < 0.0f)
color.g = 0.0f;
if (color.b < 0.0f)
color.b = 0.0f;
if (color.r > 1.0f)
color.r = 1.0f;
if (color.g > 1.0f)
color.g = 1.0f;
if (color.b > 1.0f)
color.b = 1.0f;
color.a = 1.0f;
currentTexture.SetPixel(width - 1 - x, y, color);
}
}
currentTexture.Apply();
//this.GetComponent<RawImage>().texture = m_TextureRender;
}
}
CLIENT
public class MySocketsClient : MonoBehaviour
{
public RawImage image;
public bool enableLog = false;
const int port = 8010;
//public string IP = "xxx.xxx.x.xx";
public string IP = "xxx.xxx.x.x";
TcpClient client;
Texture2D tex;
private bool stop = false;
//This must be the-same with SEND_COUNT on the server
const int SEND_RECEIVE_COUNT = 15;
// Use this for initialization
void Start()
{
Application.runInBackground = true;
tex = new Texture2D(0, 0);
client = new TcpClient();
//Connect to server from another Thread
Loom.RunAsync(() =>
{
LOGWARNING("Connecting to server...");
// if on desktop
//client.Connect(IPAddress.Loopback, port);
// if using the IPAD
client.Connect(IPAddress.Parse(IP), port);
LOGWARNING("Connected!");
imageReceiver();
});
}
void imageReceiver()
{
Debug.Log("Here");
//While loop in another Thread is fine so we don't block main Unity Thread
Loom.RunAsync(() =>
{
while (!stop)
{
//Read Image Count
int imageSize = readImageByteSize(SEND_RECEIVE_COUNT);
LOGWARNING("Received Image byte Length: " + imageSize);
//Read Image Bytes and Display it
readFrameByteArray(imageSize);
}
});
}
//Converts the data size to byte array and put result to the fullBytes array
void byteLengthToFrameByteArray(int byteLength, byte[] fullBytes)
{
//Clear old data
Array.Clear(fullBytes, 0, fullBytes.Length);
//Convert int to bytes
byte[] bytesToSendCount = BitConverter.GetBytes(byteLength);
//Copy result to fullBytes
bytesToSendCount.CopyTo(fullBytes, 0);
}
//Converts the byte array to the data size and returns the result
int frameByteArrayToByteLength(byte[] frameBytesLength)
{
int byteLength = BitConverter.ToInt32(frameBytesLength, 0);
return byteLength;
}
/////////////////////////////////////////////////////Read Image SIZE from Server///////////////////////////////////////////////////
private int readImageByteSize(int size)
{
bool disconnected = false;
NetworkStream serverStream = client.GetStream();
byte[] imageBytesCount = new byte[size];
var total = 0;
do
{
var read = serverStream.Read(imageBytesCount, total, size - total);
//Debug.LogFormat("Client recieved {0} bytes", total);
if (read == 0)
{
disconnected = true;
break;
}
total += read;
} while (total != size);
int byteLength;
if (disconnected)
{
byteLength = -1;
}
else
{
byteLength = frameByteArrayToByteLength(imageBytesCount);
}
return byteLength;
}
/////////////////////////////////////////////////////Read Image Data Byte Array from Server///////////////////////////////////////////////////
private void readFrameByteArray(int size)
{
bool disconnected = false;
NetworkStream serverStream = client.GetStream();
byte[] imageBytes = new byte[size];
var total = 0;
do
{
var read = serverStream.Read(imageBytes, total, size - total);
//Debug.LogFormat("Client recieved {0} bytes", total);
if (read == 0)
{
disconnected = true;
break;
}
total += read;
} while (total != size);
bool readyToReadAgain = false;
//Display Image
if (!disconnected)
{
//Display Image on the main Thread
Loom.QueueOnMainThread(() =>
{
displayReceivedImage(imageBytes);
readyToReadAgain = true;
});
}
//Wait until old Image is displayed
while (!readyToReadAgain)
{
System.Threading.Thread.Sleep(1);
}
}
void displayReceivedImage(byte[] receivedImageBytes)
{
tex.LoadImage(receivedImageBytes);
image.texture = tex;
}
// Update is called once per frame
void Update()
{
}
void LOG(string messsage)
{
if (enableLog)
Debug.Log(messsage);
}
void LOGWARNING(string messsage)
{
if (enableLog)
Debug.LogWarning(messsage);
}
void OnApplicationQuit()
{
LOGWARNING("OnApplicationQuit");
stop = true;
if (client != null)
{
client.Close();
}
}
}
Would appreciate any help or thoughts on what might be causing the severe lag. Thanks!

How to adjust the brightness of an Image efficiently

Does anyone know of a more efficient way of adjusting the brightness of an image at runtime in UWP?
I found this question which works fine but runs terribly slow.
However, I can't find any documentation online suggesting there is an alternative method.
Here is my problematic code.
// TODO Make Image Brightness Slider quicker and more intuitive.
private WriteableBitmap ChangeBrightness(WriteableBitmap source, int increment)
{
var dest = new WriteableBitmap(source.PixelWidth, source.PixelHeight);
byte[] color = new byte[4];
using (var srcBuffer = source.PixelBuffer.AsStream())
using (var dstBuffer = dest.PixelBuffer.AsStream())
{
while (srcBuffer.Read(color, 0, 4) > 0)
{
for (int i = 0; i < 3; i++)
{
var value = (float)color[i];
var alpha = color[3] / (float)255;
value /= alpha;
value += increment;
value *= alpha;
if (value > 255)
{
value = 255;
}
color[i] = (byte)value;
}
dstBuffer.Write(color, 0, 4);
}
}
return dest;
}

This might work. I didn't test it:
private async Task<WriteableBitmap> ChangeBrightness(WriteableBitmap source, float increment)
{
var canvasBitmap = CanvasBitmap.CreateFromBytes(CanvasDevice.GetSharedDevice(), source.PixelBuffer,
source.PixelWidth, source.PixelHeight, DirectXPixelFormat.B8G8R8A8UIntNormalized);
var brightnessFx = new BrightnessEffect
{
Source = canvasBitmap,
BlackPoint = new Vector2(0, increment)
};
var crt = new CanvasRenderTarget(CanvasDevice.GetSharedDevice(), source.PixelWidth, source.PixelHeight, 96);
using (var ds = crt.CreateDrawingSession())
{
ds.DrawImage(brightnessFx);
}
crt.GetPixelBytes(source.PixelBuffer);
return source;
}
You have to reference win2d nuget