We Keep Coding

Mixed Reality WebRTC - Screen capturing with GraphicsCapturePicker

Setup
Hey,
I'm trying to capture my screen and send/communicate the stream via MR-WebRTC. Communication between two PCs or PC with HoloLens worked with webcams for me, so I thought the next step could be streaming my screen. So I took the uwp application that I already had, which worked with my webcam and tried to make things work:
UWP App is based on the example uwp app from MR-WebRTC.
For Capturing I'm using the instruction from MS about screen capturing via GraphicsCapturePicker.
So now I'm stuck in the following situation:
I get a frame from the screen capturing, but its type is Direct3D11CaptureFrame. You can see it below in the code snipped.
MR-WebRTC takes a frame type I420AVideoFrame (also in a code snipped).
How can I "connect" them?
I420AVideoFrame wants a frame in the I420A format (YUV 4:2:0).
Configuring the framePool I can set the DirectXPixelFormat, but it has no YUV420.
I found this post on so, saying that it its possible.
Code Snipped Frame from Direct3D:
_framePool = Direct3D11CaptureFramePool.Create(
_canvasDevice, // D3D device
DirectXPixelFormat.B8G8R8A8UIntNormalized, // Pixel format
3, // Number of frames
_item.Size); // Size of the buffers
_session = _framePool.CreateCaptureSession(_item);
_session.StartCapture();
_framePool.FrameArrived += (s, a) =>
{
using (var frame = _framePool.TryGetNextFrame())
{
// Here I would take the Frame and call the MR-WebRTC method LocalI420AFrameReady
}
};
Code Snippet Frame from WebRTC:
// This is the way with the webcam; so LocalI420 was subscribed to
// the event I420AVideoFrameReady and got the frame from there
_webcamSource = await DeviceVideoTrackSource.CreateAsync();
_webcamSource.I420AVideoFrameReady += LocalI420AFrameReady;
// enqueueing the newly captured video frames into the bridge,
// which will later deliver them when the Media Foundation
// playback pipeline requests them.
private void LocalI420AFrameReady(I420AVideoFrame frame)
{
lock (_localVideoLock)
{
if (!_localVideoPlaying)
{
_localVideoPlaying = true;
// Capture the resolution into local variable useable from the lambda below
uint width = frame.width;
uint height = frame.height;
// Defer UI-related work to the main UI thread
RunOnMainThread(() =>
{
// Bridge the local video track with the local media player UI
int framerate = 30; // assumed, for lack of an actual value
_localVideoSource = CreateI420VideoStreamSource(
width, height, framerate);
var localVideoPlayer = new MediaPlayer();
localVideoPlayer.Source = MediaSource.CreateFromMediaStreamSource(
_localVideoSource);
localVideoPlayerElement.SetMediaPlayer(localVideoPlayer);
localVideoPlayer.Play();
});
}
}
// Enqueue the incoming frame into the video bridge; the media player will
// later dequeue it as soon as it's ready.
_localVideoBridge.HandleIncomingVideoFrame(frame);
}

I found a solution for my problem by creating an issue on the github repo. Answer was provided by KarthikRichie:
You have to use the ExternalVideoTrackSource
You can convert from the Direct3D11CaptureFrame to Argb32VideoFrame
 
// Setting up external video track source
_screenshareSource = ExternalVideoTrackSource.CreateFromArgb32Callback(FrameCallback);
struct WebRTCFrameData
{
public IntPtr Data;
public uint Height;
public uint Width;
public int Stride;
}
public void FrameCallback(in FrameRequest frameRequest)
{
try
{
if (FramePool != null)
{
using (Direct3D11CaptureFrame _currentFrame = FramePool.TryGetNextFrame())
{
if (_currentFrame != null)
{
WebRTCFrameData webRTCFrameData = ProcessBitmap(_currentFrame.Surface).Result;
frameRequest.CompleteRequest(new Argb32VideoFrame()
{
data = webRTCFrameData.Data,
height = webRTCFrameData.Height,
width = webRTCFrameData.Width,
stride = webRTCFrameData.Stride
});
}
}
}
}
catch (Exception ex)
{
}
}
private async Task<WebRTCFrameData> ProcessBitmap(IDirect3DSurface surface)
{
SoftwareBitmap softwareBitmap = await SoftwareBitmap.CreateCopyFromSurfaceAsync(surface, Windows.Graphics.Imaging.BitmapAlphaMode.Straight);
byte[] imageBytes = new byte[4 * softwareBitmap.PixelWidth * softwareBitmap.PixelHeight];
softwareBitmap.CopyToBuffer(imageBytes.AsBuffer());
WebRTCFrameData argb32VideoFrame = new WebRTCFrameData();
argb32VideoFrame.Data = GetByteIntPtr(imageBytes);
argb32VideoFrame.Height = (uint)softwareBitmap.PixelHeight;
argb32VideoFrame.Width = (uint)softwareBitmap.PixelWidth;
var test = softwareBitmap.LockBuffer(BitmapBufferAccessMode.Read);
int count = test.GetPlaneCount();
var pl = test.GetPlaneDescription(count - 1);
argb32VideoFrame.Stride = pl.Stride;
return argb32VideoFrame;
}
private IntPtr GetByteIntPtr(byte[] byteArr)
{
IntPtr intPtr2 = System.Runtime.InteropServices.Marshal.UnsafeAddrOfPinnedArrayElement(byteArr, 0);
return intPtr2;
}

Playback of FFMPEG Audio Using NAudio Has Lots of Noise

I have a video that I'm trying to play back using the FFMPEG framework, and NAudio to playback the audio frames. The video plays back just fine, but when I play back the audio using a BufferedWaveProvider while the underlying audio sounds correct, it has a ton of static/noise. What's important to note in the attached code is when the events fire off from the managed C++ class, a C# class receives the event and then grabs the latest relevant AVFrame data. This works flawlessly for video, but like I said, when I try to play back the audio I have a bunch of noise. For now I've hardcoded the NAudio WaveFormat settings, but they should match the video's audio.
///////////////////////// C# /////////////////////////////
// Gets the size of the data from the audio AVFrame (audioFrame->linesize[0])
int size = _Decoder.GetAVAudioFrameSize();
if (!once)
{
managedArray = new byte[size];
once = true;
}
// Gets the data array from the audio AVFrame
IntPtr data = _Decoder.GetAVAudioFrameData();
Marshal.Copy(data, managedArray, 0, size);
if (_waveProvider == null)
{
_waveProvider = new BufferedWaveProvider(new WaveFormat(44100, 16, 2));
_waveOut.NumberOfBuffers = 2;
_waveOut.DesiredLatency = 300;
_waveOut.Init(_waveProvider);
_waveOut.Play();
}
_waveProvider.AddSamples(managedArray, 0, size);
//////////////////////// Managed C++ /////////////////////////
while (_shouldRead)
{
// if we're not initialized, sleep
if (!_initialized || !_FFMpegObject->AVFormatContextReady())
{
Thread::Sleep(READ_INIT_WAIT_TIME);
}
else if (_sequentialFailedReadCount > MAX_SEQUENTIAL_READ_FAILS)
{
// we've failed a lot and probably lost the stream, try to reconnect.
_sequentialFailedReadCount = 0;
_initialized = false;
StartInitThread();
Thread::Sleep(READ_INIT_WAIT_TIME << 1);
}
else // otherwise, try reading one AV packet
{
if (_FFMpegObject->AVReadFrame())
{
if (_FFMpegObject->GetAVPacketStreamIndex() == _videoStreamIndex)
{
_sequentialFailedReadCount = 0;
// decode the video packet
_frameLock->WaitOne();
frameFinished = _FFMpegObject->AVCodecDecodeVideo2();
_frameLock->ReleaseMutex();
if (frameFinished)
{
_framesDecoded++;
if (!_isPaused) // If paused and AFAP playback, just don't call the callback
{
FrameFinished(this, EventArgs::Empty);
Thread::Sleep((int)(1000 / (GetFrameRate() * _speed)));
}
}
}
else if (_FFMpegObject->GetAVPacketStreamIndex() == _audioStreamIndex)
{
// decode the audio packet.
_frameLock->WaitOne();
audioFrameFinished = _FFMpegObject->AVCodecDecodeAudio4();
_frameLock->ReleaseMutex();
if (audioFrameFinished)
{
if (!_isPaused) // If paused and AFAP playback, just don't call the callback
{
// Fire the event - leaving it up to video to do the sleeps -- not sure if that will work or not
AudioFrameFinished(this, EventArgs::Empty);
}
}
}
_FFMpegObject->AVFreeFramePacket();
}
else // failed to read an AV packet
{
_sequentialFailedReadCount++;
_failedReadCount++;
Thread::Sleep(READ_FAILED_WAIT_TIME);
}
}
}

Playing live audio stream from byte[] packet

I'm trying to play sound provided by an Ethernet microphone.
The device stream live audio via udp packets, that I read in a network receiver thread :
MemoryStream msAudio = new MemoryStream();
private void process_stream(byte[] buffer)
{
msAudio.Write(fragment, 0, fragment.Length);
}
process_stream is called in a task
Then I have another task to play the stream in NAudio (NAudio isn't mandatory) :
while (IsConnected)
{
msAudio.Position = 0;
var waveFormat = new WaveFormat(8000, 16, 1); // Same format
using (WaveStream blockAlignedStream = new BlockAlignReductionStream(
WaveFormatConversionStream.CreatePcmStream(
new RawSourceWaveStream(msAudio , waveFormat))))
{
using (WaveOut waveOut = new WaveOut(WaveCallbackInfo.FunctionCallback()))
{
waveOut.Init(blockAlignedStream);
waveOut.Play();
while (waveOut.PlaybackState == PlaybackState.Playing)
{
System.Threading.Thread.Sleep(100);
}
}
}
}
My problems is :
I hear Toc Toc Toc noise (about 4 times by second)
The audio is sloooowed, voice is deformed like bitrate is too low (but 8khz is correct)
The audio is looped, I think i have to flush my stream, but I don't see were...
Thanks a lot if you can tell me some advice...
P.S :
for helping, original code is working in android using AudioTrack. the code is here
P.S 2 : Here the "image" of the audio noise that I have :

I don't like to respond myself.. but I think that my question is too specific...
So I resolve my problems with :
Using BufferedWaveProvider in place of a memoryStream
Doubling SampleRate (16KHz)
Removing first 4 byte of the buffer
Now My code look like :
public ctor()
{
var waveFormat = new WaveFormat(16000, 16, 1);
buffer = new BufferedWaveProvider(waveFormat)
{
BufferDuration = TimeSpan.FromSeconds(10),
DiscardOnBufferOverflow = true
};
}
internal void OnDataReceived(byte[] currentFrame)
{
if (mPlaying && mAudioTrack != null)
{
buffer.AddSamples(currentFrame, 4, currentFrame.Length);
}
}
internal void ConfigureCodec()
{
mAudioTrack = new WaveOut(WaveCallbackInfo.FunctionCallback());
mAudioTrack.Init(buffer);
if (mPlaying)
{
mAudioTrack.Play();
}
}
No more Thread....

C# Frequency Retrieval

What I need to do is calculate the frequency of the microphone input. I'm using IWaveProvider for this and its implemented Read(). The buffer always has a size of 8820 elements and something seems to be going wrong with the conversion from byte array to float array as well (the FloatBuffer property part).
Here are some of the important bits...
This is where I start my recording:
private void InitializeSoundRecording()
{
WaveIn waveIn = new WaveIn();
waveIn.DeviceNumber = 0;
waveIn.DataAvailable += (s, e) => this.waveIn_DataAvailable(s, e);
waveIn.RecordingStopped += (s, e) => this.waveIn_RecordingStopped(s, e);
waveIn.WaveFormat = new WaveFormat(44100, 1);
waveIn.StartRecording();
}
When the DataAvailable event handler is called, the following is executed:
private void waveIn_DataAvailable(object sender, WaveInEventArgs e)
{
WaveBuffer wb = new WaveBuffer(e.Buffer.Length);
IWaveProvider iWaveProvider = new PitchDetector(new WaveInProvider(sender as WaveIn), new WaveBuffer(e.Buffer));
iWaveProvider.Read(wb, 0, e.Buffer.Length);
PitchDetector pd = iWaveProvider as PitchDetector;
this.ShowPitch(pd.Pitch);
}
And lastly, this is the "actual" important bit:
private const int FLOAT_BUFFER_SIZE = 8820;
private IWaveProvider source;
private WaveBuffer waveBuffer;
private int sampleRate;
private float[] fftBuffer;
private float[] prevBuffer;
public float Pitch { get; private set; }
public WaveFormat WaveFormat { get { return this.source.WaveFormat; } }
internal PitchDetector(IWaveProvider waveProvider, WaveBuffer waveBuffer = null)
{
this.source = waveProvider;
this.sampleRate = waveProvider.WaveFormat.SampleRate;
this.waveBuffer = waveBuffer;
}
/// <summary>
/// UNSAFE METHOD!
/// </summary>
/// <param name="input"></param>
/// <returns></returns>
private unsafe float[] ByteArrayToFloatArray(byte[] input)
{
float[] fb = new float[FLOAT_BUFFER_SIZE];
unsafe
{
fixed (byte* ptrBuffer = input)
{
float* ptrFloatBuffer = (float*)ptrBuffer;
for (int i = 0; i < FLOAT_BUFFER_SIZE; i++)
{
fb[i] = *ptrFloatBuffer;
ptrFloatBuffer++;
}
}
}
return fb;
}
public int Read(byte[] buffer, int offset = 0, int count = 0)
{
if (this.waveBuffer == null || this.waveBuffer.MaxSize < count)
this.waveBuffer = new WaveBuffer(count);
int readBytes = this.source.Read(this.waveBuffer, 0, count);
if (readBytes > 0) readBytes = count;
int frames = readBytes / sizeof(float);
this.Pitch = this.DeterminePitch(this.waveBuffer.FloatBuffer, frames);
return frames * 4;
}
Strangely enough, when it enters the constructor, waveBuffer contains some data (255, 1, 0, etc.), but when I check the "buffer" parameter of Read(), it's entirely 0. Every element.
Out of curiosity also, why does Read() have a buffer parameter, but isn't actually used in the method at all (I got that piece of code from one of your articles)?
Any help to resolve this issue would be greatly appreciated! I've been at this for quite a while already, but can make no sense out of it.
Thanks,
Alain

It is not clear what article you are referring to and I am not familiar with this library. However, the Read method is clearly reading in your 'time-series'/or other data. From this, the buffer parameter you speak of is likely to be the padding length that you want to place on either end of your data set.
This padding is known as 'Zero Padding' and it pads your recorded signal with zeros (places n zeros on either end of the signal, where n is set according to the radix used). This allows one to use a longer FFT, which will produce a longer FFT resulting vector.
A longer FFT result has more frequency bins that are more closely spaced in frequency. But they will be essentially providing the same result as a high quality Sinc interpolation of a shorter non-zero-padded FFT of the original data.
This might result in a smoother looking spectrum when plotted without further interpolation.
For further in formation see
https://dsp.stackexchange.com/questions/741/why-should-i-zero-pad-a-signal-before-taking-the-fourier-transform
I hope this helps.

This is not really an answer to your question but I wrote a safe generic alternative to your array conversion function.
using System;
using System.Runtime.InteropServices;
public static class Extensions
{
public staitc TDestination[] Transform<TSource, TDestination>(
this TSource[] source)
where TSource : struct
where TDestination : struct
{
if (source.Length == 0)
{
return new TDestination[0];
}
var sourceSize = Marshal.SizeOf(typeof(TSource));
var destinationSize = Marshal.SizeOf(typeof(TDestination));
var byteLength = source.Length * sourceSize;
int remainder;
var destinationLength = Math.DivRem(
byteLength,
destinationSize,
out remainder);
if (remainder > 0)
{
destinationLength++;
}
var destination = new TDestination[destinationLength];
Buffer.BlockCopy(source, 0, destination, 0, byteLength);
return destination;
}
}
Which obviously, you could use like
var bytes = new byte[] { 1, 1, 2, 3, 5, 8, 13, 21 };
var floats = bytes.Transform<byte, float>();

Playing WAVE file in C# using DirectX and threading?

at the moment im trying to figure out how i can manage to play a wave file in C# by filling up the secondary buffer with data from the wave file through threading and then play the wave file.
Any help or sample coding i can use?
thanks
sample code being used:
public delegate void PullAudio(short[] buffer, int length);
public class SoundPlayer : IDisposable
{
private Device soundDevice;
private SecondaryBuffer soundBuffer;
private int samplesPerUpdate;
private AutoResetEvent[] fillEvent = new AutoResetEvent[2];
private Thread thread;
private PullAudio pullAudio;
private short channels;
private bool halted;
private bool running;
public SoundPlayer(Control owner, PullAudio pullAudio, short channels)
{
this.channels = channels;
this.pullAudio = pullAudio;
this.soundDevice = new Device();
this.soundDevice.SetCooperativeLevel(owner, CooperativeLevel.Priority);
// Set up our wave format to 44,100Hz, with 16 bit resolution
WaveFormat wf = new WaveFormat();
wf.FormatTag = WaveFormatTag.Pcm;
wf.SamplesPerSecond = 44100;
wf.BitsPerSample = 16;
wf.Channels = channels;
wf.BlockAlign = (short)(wf.Channels * wf.BitsPerSample / 8);
wf.AverageBytesPerSecond = wf.SamplesPerSecond * wf.BlockAlign;
this.samplesPerUpdate = 512;
// Create a buffer with 2 seconds of sample data
BufferDescription bufferDesc = new BufferDescription(wf);
bufferDesc.BufferBytes = this.samplesPerUpdate * wf.BlockAlign * 2;
bufferDesc.ControlPositionNotify = true;
bufferDesc.GlobalFocus = true;
this.soundBuffer = new SecondaryBuffer(bufferDesc, this.soundDevice);
Notify notify = new Notify(this.soundBuffer);
fillEvent[0] = new AutoResetEvent(false);
fillEvent[1] = new AutoResetEvent(false);
// Set up two notification events, one at halfway, and one at the end of the buffer
BufferPositionNotify[] posNotify = new BufferPositionNotify[2];
posNotify[0] = new BufferPositionNotify();
posNotify[0].Offset = bufferDesc.BufferBytes / 2 - 1;
posNotify[0].EventNotifyHandle = fillEvent[0].Handle;
posNotify[1] = new BufferPositionNotify();
posNotify[1].Offset = bufferDesc.BufferBytes - 1;
posNotify[1].EventNotifyHandle = fillEvent[1].Handle;
notify.SetNotificationPositions(posNotify);
this.thread = new Thread(new ThreadStart(SoundPlayback));
this.thread.Priority = ThreadPriority.Highest;
this.Pause();
this.running = true;
this.thread.Start();
}
public void Pause()
{
if (this.halted) return;
this.halted = true;
Monitor.Enter(this.thread);
}
public void Resume()
{
if (!this.halted) return;
this.halted = false;
Monitor.Pulse(this.thread);
Monitor.Exit(this.thread);
}
private void SoundPlayback()
{
lock (this.thread)
{
if (!this.running) return;
// Set up the initial sound buffer to be the full length
int bufferLength = this.samplesPerUpdate * 2 * this.channels;
short[] soundData = new short[bufferLength];
// Prime it with the first x seconds of data
this.pullAudio(soundData, soundData.Length);
this.soundBuffer.Write(0, soundData, LockFlag.None);
// Start it playing
this.soundBuffer.Play(0, BufferPlayFlags.Looping);
int lastWritten = 0;
while (this.running)
{
if (this.halted)
{
Monitor.Pulse(this.thread);
Monitor.Wait(this.thread);
}
// Wait on one of the notification events
WaitHandle.WaitAny(this.fillEvent, 3, true);
// Get the current play position (divide by two because we are using 16 bit samples)
int tmp = this.soundBuffer.PlayPosition / 2;
// Generate new sounds from lastWritten to tmp in the sound buffer
if (tmp == lastWritten)
{
continue;
}
else
{
soundData = new short[(tmp - lastWritten + bufferLength) % bufferLength];
}
this.pullAudio(soundData, soundData.Length);
// Write in the generated data
soundBuffer.Write(lastWritten * 2, soundData, LockFlag.None);
// Save the position we were at
lastWritten = tmp;
}
}
}
public void Dispose()
{
this.running = false;
this.Resume();
if (this.soundBuffer != null)
{
this.soundBuffer.Dispose();
}
if (this.soundDevice != null)
{
this.soundDevice.Dispose();
}
}
}
}
The concept is the same that im using but i can't manage to get a set on wave byte [] data to play

I have not done this.
But the first place i would look is XNA.
I know that the c# managed directx project was ditched in favor of XNA and i have found it to be good for graphics - i prefer using it to directx.

what is the reason that you decided not to just use soundplayer, as per this msdn entry below?
private SoundPlayer Player = new SoundPlayer();
private void loadSoundAsync()
{
// Note: You may need to change the location specified based on
// the location of the sound to be played.
this.Player.SoundLocation = http://www.tailspintoys.com/sounds/stop.wav";
this.Player.LoadAsync();
}
private void Player_LoadCompleted (
object sender,
System.ComponentModel.AsyncCompletedEventArgs e)
{
if (this.Player.IsLoadCompleted)
{
this.Player.PlaySync();
}
}
usually i just load them all up in a thread, or asynch delegate, then play or playsynch them when needed.

You can use the DirectSound support in SlimDX: http://slimdx.org/ :-)

You can use nBASS or better FMOD both are great audio libraries and can work nicely together with .NET.

DirectSound is where you want to go. It's a piece of cake to use, but I'm not sure what formats it can play besides .wav
http://msdn.microsoft.com/en-us/library/windows/desktop/ee416960(v=vs.85).aspx