I am trying to access the raw (float[]) values of the HoloLens 2's embedded microphone in real time. I do not need to record this data or playback this data, purely to sample is the user is speaking at any given slice in time, as recorded by the HL2. I am using the MicrophoneAmplitudeDemo.cs demo here nearly verbatim. I attach this script to a Unity GameObject, and have modified the script only to print the average amplitude every update, purely as a way to debug the output. When the script runs, the returned float values are always 0. I have already double checked the permissions for the microphone in the manifest and the initial popup permission windows are answered "yes". The code, modified from the original MS sample only to print the average amplitude, is below.
TO try to fix the problem, I have already diabled all other functionality in this program (eye tracking, onboard ML inference, etc) to ensure that wasn't the problem. I have also tried another MS sample (MicStreamDemo) with the exact same result. The debugging window throws no errors, but merely print zeros when I print the current values of the mic stream.
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
[RequireComponent(typeof(AudioSource))]
public class AudioCaptureUtility : MonoBehaviour
{
[SerializeField]
[Tooltip("Gain to apply to the microphone input.")]
[Range(0, 10)]
private float inputGain = 1.0f;
[SerializeField]
[Tooltip("Stream Type")]
public WindowsMicrophoneStreamType streamType= WindowsMicrophoneStreamType.HighQualityVoice;
/// <summary>
/// Class providing microphone stream management support on Microsoft Windows based devices.
/// </summary>
private WindowsMicrophoneStream micStream = null;
/// <summary>
/// The average amplitude of the sound captured during the most recent microphone update.
/// </summary>
private float averageAmplitude = 0.0f;
private void Awake()
{
// We do not wish to play the ambient room sound from the audio source.
//gameObject.GetComponent<AudioSource>().volume = 0.0f;
micStream = new WindowsMicrophoneStream();
if (micStream == null)
{
Debug.Log("Failed to create the Windows Microphone Stream object");
}
micStream.Gain = inputGain;
// Initialize the microphone stream.
WindowsMicrophoneStreamErrorCode result = micStream.Initialize(streamType);
if (result != WindowsMicrophoneStreamErrorCode.Success)
{
Debug.Log($"Failed to initialize the microphone stream. {result}");
return;
}
// Start the microphone stream.
// Do not keep the data and do not preview.
result = micStream.StartStream(false, false);
if (result != WindowsMicrophoneStreamErrorCode.Success)
{
Debug.Log($"Failed to start the microphone stream. {result}");
}
}
private void OnDestroy()
{
if (micStream == null) { return; }
// Stop the microphone stream.
WindowsMicrophoneStreamErrorCode result = micStream.StopStream();
if (result != WindowsMicrophoneStreamErrorCode.Success)
{
Debug.Log($"Failed to stop the microphone stream. {result}");
}
// Uninitialize the microphone stream.
micStream.Uninitialize();
micStream = null;
}
// Start is called before the first frame update
void Start()
{
}
// Update is called once per frame
private void Update()
{
if (micStream == null) { return; }
// Update the gain, if changed.
if (micStream.Gain != inputGain)
{
micStream.Gain = inputGain;
}
float[] tempBuffer = new float[5];
OnAudioFilterRead(tempBuffer, 2);
if(averageAmplitude == 0.0f)
{
Debug.Log("Average Amp is Zero");
//Debug.Log(averageAmplitude.ToString("F9"));
}
}
private void OnAudioFilterRead(float[] buffer, int numChannels)
{
if (micStream == null) { return; }
// Read the microphone stream data.
WindowsMicrophoneStreamErrorCode result = micStream.ReadAudioFrame(buffer, numChannels);
if (result != WindowsMicrophoneStreamErrorCode.Success)
{
Debug.Log($"Failed to read the microphone stream data. {result}");
}
float sumOfValues = 0;
// Calculate this frame's average amplitude.
for (int i = 0; i < buffer.Length; i++)
{
if (float.IsNaN(buffer[i]))
{
buffer[i] = 0;
}
buffer[i] = Mathf.Clamp(buffer[i], -1.0f, 1.0f);
sumOfValues += Mathf.Clamp01(Mathf.Abs(buffer[i]));
}
averageAmplitude = sumOfValues / buffer.Length;
}
}
EDIT: The pictures below are screenshots of the errors. I was able to get some raw float data printed, but the data stream ends during initialization each time. I simply print the current value of averageAmplitude each Update(). The InitializeFrameReader message are from a Windows MediaCapture instance. To ensure that this isn't the culprit, I remove this functionality and the issues remain. The float values cease and never return. I have waited as long as 5 minutes to ensure they never did come back.
I run into an issue after a few tests, and I am not sure if it is the same issue you have. When initialization of micStream, it sometimes returns 'Already Running', and OnAudioFilterRead() will return 'NotEnoughData'. Then I made some modifications of micStream initialization to solve this issue. You can refer to it.
WindowsMicrophoneStreamErrorCode result = micStream.Initialize(WindowsMicrophoneStreamType.HighQualityVoice);
if (result != WindowsMicrophoneStreamErrorCode.Success && result!=WindowsMicrophoneStreamErrorCode.AlreadyRunning)
{
Debug.Log($"Failed to initialize the microphone stream. {result}");
return;
}
Also, you can get some log information on Windows Device Portal to troubleshoot the issue. Or you can try remote debugging on Unity.
Related
Let me start by saying that this is my first time opening up a question here. If I'm unclear in my code formatting or have expressed myself poorly, please let me know and I'd be happy to adjust.
I'm doing conceptual design for a tool to be used in Unity (C#) to allow us to "stream" the output of an AudioSource, in real-time, and then have that same audio be played back from a different GameObject. In essence, we would have a parallel signal being stored in a buffer while the original AudioSource functions as one would expect, sending it's playing clip into the mixer just as is normal.
To achieve this I'm trying to use the audio thread and the OnAudioFilterRead() function. To extract the floating point audio data to pipe into OnAudioFilterRead() i'm using AudioSource.GetOutputData, storing that into an array and then supplying the array to the audio filter. I'm also creating an empty AudioClip, setting it's data from the same array and playing that AudioClip on the new GameObject.
Right now I have the audio being played from the new GameObject, but the result is very distorted and unpleasant, which i'm chalking up to one or more of the following;
The audio buffer is being written to/read from in an unsynched manner
Unitys samplerate is causing problems with the clip. Have tried both 44.1khz and 48khz with subpar results, as well as playing around with import settings on clips. Documentation for Unity 2018.2 is very weak and a lot of older methods are now deprecated.
Aliens.
Ideally, I would be able to stream back the audio without audible artefacts. Some latency i can live with (~30-50ms) but not poor audio quality.
Below is the code that is being used. This script is attached to the GameObject that is to receive this audio signal from the original emitter, and it has its own AudioSource for playback and positioning.
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using UnityEngine;
[RequireComponent(typeof(AudioSource))]
public class ECO_receiver : MonoBehaviour {
public AudioSource emitterSend;
private AudioSource audioSource;
public AudioClip streamedClip;
public bool debugSampleData;
private float[] sampleDataArray;
public float sampleSize;
public float sampleFreq;
private int outputSampleRate;
public bool _bufferReady;
void Start () {
audioSource = GetComponent<AudioSource>();
sampleSize = emitterSend.clip.samples;
sampleFreq = emitterSend.clip.frequency;
sampleDataArray = new float[2048];
streamedClip = AudioClip.Create("audiostream", (int)sampleSize, 1, (int)sampleFreq, false);
audioSource.clip = streamedClip;
audioSource.Play();
_bufferReady = true;
}
private void FixedUpdate()
{
if (emitterSend.isPlaying && _bufferReady == true)
{
FillAudioBuffer();
}
else if (!emitterSend.isPlaying)
{
Debug.Log("Emitter is not playing!");
}
if (debugSampleData && sampleDataArray != null && Input.GetKeyDown("p"))
{
for (int i = 0; i < sampleDataArray.Length; i++)
{
Debug.Log(sampleDataArray[i]);
}
}
else if (sampleDataArray == null)
{
Debug.Log("No data in array!");
}
}
void FillAudioBuffer()
{
emitterSend.GetOutputData(sampleDataArray, 0);
streamedClip.SetData(sampleDataArray, 0);
_bufferReady = false;
}
void OnAudioFilterRead(float[] data, int channels)
{
if (!_bufferReady)
{
for (int i = 0; i < data.Length; i++)
{
data[i] = (float)sampleDataArray[i];
}
_bufferReady = true;
}
}
}
Greatly appreciate any wisdom I might be granted! Thank you!
The name 'Microphone' does not exist in the current context. getting this error when opening an unity(version 5.6.0f3) project is visual studio 2017 in window 8.
[RequireComponent (typeof (AudioSource))]
public class SingleMicrophoneCapture : MonoBehaviour
{
//A boolean that flags whether there's a connected microphone
private bool micConnected = false;
//The maximum and minimum available recording frequencies
private int minFreq;
private int maxFreq;
//A handle to the attached AudioSource
public AudioSource goAudioSource;
public AudioClip recordedAudioClip;
[HideInInspector]
public AudioClip myAudioClip;
//public Text fileExist;
bool startRecording = false;
public Sprite[] recordingSprites;
public int count =0;
//int recordedFileCount =0;
public bool isDefaultAudioPlaying = false;
[SerializeField]
public Sprite[] playSprites;
public GameObject forwardButton;
public GameObject backwardButton;
public GameObject playButton;
public GameObject replayButton;
//Use this for initialization
public AudioClip[] allAudioClips;
public string storyName;
float[] samples;
public Dictionary<int,float> recordedClipDict;
void Start()
{
//ReplayButtonClicked ();
//Check if there is at least one microphone connected
recordedAudioClip= null;
if(Microphone.devices.Length <= 0)
{
//Throw a warning message at the console if there isn't
Debug.LogWarning("Microphone not connected!");
}
else //At least one microphone is present
{
//Set 'micConnected' to true
micConnected = true;
//Get the default microphone recording capabilities
Microphone.GetDeviceCaps(null, out minFreq, out maxFreq);
//According to the documentation, if minFreq and maxFreq are zero, the microphone supports any frequency...
if(minFreq == 0 && maxFreq == 0)
{
//...meaning 44100 Hz can be used as the recording sampling rate
maxFreq = 44100;
}
//Get the attached AudioSource component
goAudioSource = this.GetComponent<AudioSource>();
// mainAudioSource = Camera.main.GetComponent<AudioSource> ();
}
}
how to solve this.
You are getting this error because you using a platform that do not support the Microphone API. One of the platforms that do not support the Microphone API is the WebGL. There might be other platforms other than WebGL without Microphone support.
Switch to a platform that supports the Microphone API from the Build Settings.
You can also use Unity's preprocessor directives to guard it and make sure that the Microphone API is not used when using platforms that do not support it or did not implement it.
#if !UNITY_WEBGL
//YOUR Microphone CODE HERE
#endif
If you really need Microphone in WebGL with Unity, make a plugin or use this one(Not free).
We can not see your using statements.
But it seems like your are missing
using UnityEngine.AudioModule;
This scene is common in real time video proccessing. And I need timestamps to synchronize with other devices.
I have tried cv::VideoCapture, but it can not extract the timestamps frome video stream.
So I have two questions here:
Does video stream provided by USB camera indeed contains the timestamp information ?
If it has. What should I do to extract it ? A C# solution is best, while C++ is OK.
Addition:
Using these two properties doesn't work:
secCounter = (long) cap.get(CAP_PROP_POS_MSEC);
frameNumber = (long) cap.get(CAP_PROP_POS_FRAMES);
It always gives the following result:
VIDEOIO ERROR: V4L2: getting property #1 is not supported
msecCounter = 0
frameNumber = -1
OpenCV's VideoCapture class is a very high level interface to retrieve frames from a camera, so it "hides" a lot of the details that are necessary to connect to the camera, retrieve frames from the camera, and decode those frames in to a useful color space like BGR. This is nice because you don't have to worry about the details of grabbing frames, but the downside is that you don't have direct access to other data you might want, like the frame number or frame timestamp. That doesn't mean it's impossible to get the data you want, though!
Here's a sample frame grabbing loop that will get you what you want, loosely based on the example code from here. This is in C++.
#include "opencv2/opencv.hpp"
using namespace cv;
int main(int, char**)
{
VideoCapture cap(0); // open the default camera
if(!cap.isOpened()) // check if we succeeded
return -1;
// TODO: change the width, height, and capture FPS to your desired
// settings.
cap.set(CAP_PROP_FRAME_WIDTH, 1920);
cap.set(CAP_PROP_FRAME_HEIGHT, 1080);
cap.set(CAP_PROP_FPS, 30);
Mat frame;
long msecCounter = 0;
long frameNumber = 0;
for(;;)
{
// Instead of cap >> frame; we'll do something different.
//
// VideoCapture::grab() tells OpenCV to grab a frame from
// the camera, but to not worry about all the color conversion
// and processing to convert that frame into BGR.
//
// This means there's less processing overhead, so the time
// stamp will be more accurate because we are fetching it
// immediately after.
//
// grab() should also wait for the next frame to be available
// based on the capture FPS that is set, so it's okay to loop
// continuously over it.
if(cap.grab())
{
msecCounter = (long) cap.get(CAP_PROP_POS_MSEC);
frameNumber = (long) cap.get(CAP_PROP_POS_FRAMES);
// VideoCapture::retrieve color converts the image and places
// it in the Mat that you provide.
if(cap.retrieve(&frame))
{
// Pass the frame and parameters to your processing
// method.
ProcessFrame(&frame, msecCounter, frameNumber);
}
}
// TODO: Handle your loop termination condition here
}
// the camera will be deinitialized automatically in VideoCapture destructor
return 0;
}
void ProcessFrame(Mat& frame, long msecCounter, long frameNumber)
{
// TODO: Make a copy of frame if you are going to process it
// asynchronously or put it in a buffer or queue and then return
// control from this function. This is because the reference Mat
// being passed in is "owned" by the processing loop, and on each
// iteration it will be destructed, so any references to it will be
// invalid. Hence, if you do any work async, you need to copy frame.
//
// If all your processing happens synchronously in this function,
// you don't need to make a copy first because the loop is waiting
// for this function to return.
// TODO: Your processing logic goes here.
}
If you're using C# and Emgu CV it will look a bit different. I haven't tested this code, but it should work or be very close to the solution.
using System;
using Emgu.CV;
using Emgu.CV.CvEnum;
static class Program
{
[STAThread]
static void Main()
{
VideoCapture cap = new VideoCapture(0);
if(!cap.IsOpened)
{
return;
}
cap.SetCaptureProperty(CapProp.FrameWidth, 1920);
cap.SetCaptureProperty(CapProp.FrameHeight, 1080);
cap.SetCaptureProperty(CapProp.Fps, 30);
Mat frame = new Mat();
long msecCounter = 0;
long frameNumber = 0;
for(;;)
{
if(cap.Grab())
{
msecCounter = (long) cap.GetCaptureProperty(CapProp.PosMsec);
frameNumber = (long) cap.GetCaptureProperty(CapProp.PosFrames);
if(cap.Retrieve(frame))
{
ProcessFrame(frame, msecCounter, frameNumber);
}
}
// TODO: Determine when to quit the processing loop
}
}
private static void ProcessFrame(Mat frame, long msecCounter, long frameNumber)
{
// Again, copy frame here if you're going to queue the frame or
// do any async processing on it.
// TODO: Your processing code goes here.
}
}
Emgu's VideoCapture implementation also allows for asynchronous Grab operations to be done for you, and notifications when a grabbed frame is ready to be used with Retrieve. That looks like this:
using System;
using Emgu.CV;
using Emgu.CV.CvEnum;
static class Program
{
private static Mat s_frame;
private static VideoCapture s_cap;
private static object s_retrieveLock = new object();
[STAThread]
static void Main()
{
s_cap = new VideoCapture(0);
if(!s_cap.IsOpened)
{
return;
}
s_frame = new Mat();
s_cap.SetCaptureProperty(CapProp.FrameWidth, 1920);
s_cap.SetCaptureProperty(CapProp.FrameHeight, 1080);
s_cap.SetCaptureProperty(CapProp.Fps, 30);
s_cap.ImageGrabbed += FrameIsReady;
s_cap.Start();
// TODO: Wait here until you're done with the capture process,
// the same way you'd determine when to exit the for loop in the
// above example.
s_cap.Stop();
s_cap.ImageGrabbed -= FrameIsReady;
}
private static void FrameIsReady(object sender, EventArgs e)
{
// This function is being called from VideoCapture's thread,
// so if you rework this code to run with a UI, be very careful
// about updating Controls here because that needs to be Invoke'd
// back to the UI thread.
// I used a lock here to be extra careful and protect against
// re-entrancy, but this may not be necessary if Emgu's
// VideoCapture thread blocks for completion of this event
// handler.
lock(s_retrieveLock)
{
msecCounter = (long) s_cap.GetCaptureProperty(CapProp.PosMsec);
frameNumber = (long) s_cap.GetCaptureProperty(CapProp.PosFrames);
if(s_cap.Retrieve(s_frame))
{
ProcessFrame(s_frame, msecCounter, frameNumber);
}
}
}
private static void ProcessFrame(Mat frame, long msecCounter, long frameNumber)
{
// Again, copy frame here if you're going to queue the frame or
// do any async processing on it.
// TODO: Your processing code goes here.
}
}
I am writing an application in Unity which will be required to capture an image from a camera every frame (at ~60fps), and send the resultant data to another service running locally.
The issue is, I am aware that capturing the rendered data from the camera can cause massive frame rate drops (as explained in this article) when using the GetPixels() method. The article explains that "GetPixels() blocks for ReadPixels() to complete" and "ReadPixels() blocks while flushing the GPU" which is why the GPU and CPU have to sync up, resulting in a lag.
I have produced a sample project with a script attached which simply outputs frames to a file as a PNG to replicate the functionality of the program I wish to create. I have done my best to implement what is described in the article, namely allowing the GPU to render a frame, then wait a few frames before calling GetPixels() so as not to cause the GPU and CPU to forcefully sync up. However, I really haven't made any progress with it. The project still plays at about 10-15fps.
How can I achieve a realtime capture of 60 frames per second in Unity?
using System;
using System.Collections;
using System.IO;
using UnityEngine;
namespace Assets
{
public class MyClass: MonoBehaviour
{
private const float reportInterval = 0.5f;
private int screenshotCount = 0;
private const float maxElapsedSecond = 20;
private string screenshotsDirectory = "UnityHeadlessRenderingScreenshots";
public Camera camOV;
public RenderTexture currentRT;
private int frameCount = 0;
private Texture2D resultantImage;
public void Start()
{
camOV.forceIntoRenderTexture = true;
if (Directory.Exists(screenshotsDirectory))
{
Directory.Delete(screenshotsDirectory, true);
}
if (!Application.isEditor)
{
Directory.CreateDirectory(screenshotsDirectory);
camOV.targetTexture = currentRT;
}
}
// Update is called once per frame
public void Update()
{
//Taking Screenshots
frameCount += 1;
if (frameCount == 1)
{
TakeScreenShot();
}
else if (frameCount == 3)
{
ReadPixelsOut("SS_"+screenshotCount+".png");
}
if (frameCount >= 3)
{
frameCount = 0;
}
}
public void TakeScreenShot()
{
screenshotCount += 1;
RenderTexture.active = camOV.targetTexture;
camOV.Render();
resultantImage = new Texture2D(camOV.targetTexture.width, camOV.targetTexture.height, TextureFormat.RGB24, false);
resultantImage.ReadPixels(new Rect(0, 0, camOV.targetTexture.width, camOV.targetTexture.height), 0, 0);
resultantImage.Apply();
}
private void ReadPixelsOut(string filename)
{
if (resultantImage != null)
{
resultantImage.GetPixels();
RenderTexture.active = currentRT;
byte[] bytes = resultantImage.EncodeToPNG();
// save on disk
var path = screenshotsDirectory + "/" + filename;
File.WriteAllBytes(path, bytes);
Destroy(resultantImage);
}
}
}
}
The article implies that it is possible, but I haven't managed to get it to work.
Many thanks in advance for your help.
I am not sure if OP still need the answer. But in case someone in the future getting the same problem, Let me share what i found.
https://github.com/unity3d-jp/FrameCapturer
This is a plugin designed for rendering animation video in Unity editor. But it can also work in standalone. In my case, i take some part of it, and make my app stream Motion Jpeg. I did it with 30fps, never tried 60fps
I have the a script called Timer.cs. This script is connected to some GUI Text, which displays the amount of time remaining in the game.
Also attached to this script is an Audio Source with my desired sound selected. When the clock reaches zero, the text changes to say "GAME OVER!" and the character controls lock up; however, the sound does not play.
All other instances of audio.Play() in my scene are working fine, and when I set the Audio Source to "Play On Awake", it plays without a problem. What could be the problem?
Using UnityEngine;
using System.Collections;
public class Timer : MonoBehaviour {
public float timer = 300; // set duration time in seconds in the Inspector
public static int sound = 1;
public static int go = 1;
bool isFinishedLevel = false; // while this is false, timer counts down
void Start(){
PlayerController.speed = 8;
PlayerController.jumpHeight = 12;
}
void Update (){
if (!isFinishedLevel) // has the level been completed
{
timer -= Time.deltaTime; // I need timer which from a particular time goes to zero
}
if (timer > 0)
{
guiText.text = timer.ToString();
}
else
{
guiText.text = "GAME OVER!"; // when it goes to the end-0,game ends (shows time text over...)
audio.Play();
int getspeed = PlayerController.speed;
PlayerController.speed = 0;
int getjumpHeight = PlayerController.jumpHeight;
PlayerController.jumpHeight = 0;
}
if (Input.GetKeyDown("r")) // And then i can restart game: pressing restart.
{
Application.LoadLevel(Application.loadedLevel); // reload the same level
}
}
}
Given that you are calling it as part of your Update routine, I'd have to guess that the problem is you calling it repeatedly. I.e. you're calling it every frame as long as timer <= 0.
You shouldn't call Play() more than once. Or at least not again while it is playing. A simple fix would be something along the lines of
if(!audio.isPlaying)
{
audio.Play();
}
See if that solves your problem, and then you can take it from there.
I had error using audio.Play(); and used following it fixed the error for me
GetComponent<AudioSource>().Play();