Sharing kinect stream data between parallel threads - c#

The problem I'm trying to solve is that I'm combining classes from different projects into a single project. The functions in these classes use KinectSensor variable which is a member of their class and recognize different gestures. I'm trying to run the gesture recognition functions in parallel threads but want to know how to pass kinect data to them.
Is there a method to share kinect sensor stream data (Color stream, depth stream etc.) data among threads of a single program running in parallel? Will a event triggered like "skeleton frame ready" be received by all the threads receiving the kinect stream data?
I'm using kinect 360, in c# using sdk 1.8

The KinectSensor object represents a single Kinect, and a single Kinect can only be used by a single program.
If the two thread you are talking about are part of the same program, you can "share the streams" by simply share the access to the same object.
However, the color, depth and skeleton streams are obtained by registering callbacks methods to an event. So what you can do is the following:
share the reference to the same KinectSensor object (preinitialized) to both threads;
in each thread, register a callback to the depth, color and skeleton stream, which update the content of the streams in a variable that is local to the thread (or used only by one thread):
// Reference to the single KinectSensor object
private KinectSensor kinectSensor;
// Local variables with depth, color and skeletal information
private Skeleton[] skeleton_thread1;
private Skeleton[] skeleton_thread2;
private short[] depth_thread1;
private short[] depth_thread2;
private byte[] color_thread1;
private byte[] color_thread2;
// ...
// Register callbacks (you must so this both in thread1 and thread2)
// Assume that here we are refererring to thread1
kinectSensor.ColorFrameReady += new EventHandler<ColorFrameReadyEventArgs>(kinectSensor_ColorFrameReady1);
kinectSensor.DepthFrameReady += new EventHandler<DepthFrameReadyEventArgs>(kinectSensor_DepthFrameReady1);
kinectSensor.SkeletonFrameReady += new EventHandler<SkeletonFrameReadyEventArgs>(kinectSensor_SkeletonFrameReady1);
// ...
private void kinectSensor_SkeletonFrameReady1(object sender, SkeletonFrameReadyEventArgs e)
{
this.skeletonFrame_thread1 =
using (SkeletonFrame skeletonFrame = e.OpenSkeletonFrame())
{
if (skeletonFrame != null)
{
this.skeleton_thread1 = new Skeleton[skeletonFrame.SkeletonArrayLength];
skeletonFrame.CopySkeletonDataTo(this.skeleton_thread1);
// Do stuff
}
else
{
// Do stuff
}
}
}
private void kinectSensor_ColorFrameReady1(object sender, ColorImageFrameReadyEventArgs e)
{
using (ColorImageFrame colorImageFrame = e.OpenColorImageFrame())
{
if (colorImageFrame != null)
{
this.color_thread1 = new byte[colorImageFrame.PixelDataLength];
colorImageFrame.CopyPixelDataTo(this.color_thread1);
// Do Stuff
}
else
{
// Do stuff
}
}
}
private void kinectSensor_DepthFrameReady1(object sender, DepthImageFrameReadyEventArgs e)
{
using (DepthImageFrame depthImageFrame = e.OpenDepthImageFrame())
{
if (depthImageFrame != null)
{
this.depth_thread1 = new short[depthImageFrame.PixelDataLength];
depthImageFrame.CopyPixelDataTo(this.depth_thread1);
// Do Stuff
}
else
{
// Do stuff
}
}
}

Related

Using Vorbis and NAudio to play OGG files in C#

I'm trying to play OGG file stream with NVorbis and NAudio, as described in the documention, I'm trying to run this code when on a Button click, but I get an exception:
System.ArgumentException: 'Could not initialize container!'
I'm targeting .Net Framework 4.5
Here is my code:
private void button1_Click(object sender, EventArgs e)
{
using (var vorbisStream = new NAudio.Vorbis.VorbisWaveReader(#"OGG file path"))
using (var waveOut = new NAudio.Wave.WaveOutEvent())
{
waveOut.Init(vorbisStream);
waveOut.Play();
// wait here until playback stops or should stop
}
}
Two issues here:
The first is that you cannot declare with using statements either the WaveOutReader or WaveOut object, since the latter is playing the sound asynchronously and it needs the WaveOutReader's Stream open
You were targeting .Net Framework 4.5, which is not suitable to handle the current version of both NAudio and NVorbis
I assume you have already successfully installed the NAudio 2.1.0 and NAudio.Vorbis 1.5.0 packages in a new .Net 6+ Project.
You just need to add references to NAudio.Vorbis and NAudio.Wave
Targeting .Net 6+, nullable enabled.
Add two Button Controls to a Form, one named btnPlayAudio and the other btnStopPlayback, initially disabled; subscribe to their Click event using the event handlers shown here.
The WaveOutReader and the WaveOut object that acts as player are declared as instance Fields, initially set to null.
When you press the Play Audio Button, the audio Stream and the Player are initialized, calling the WaveOutInit() method, which also subscribes to the WaveOut object's PlaybackStopped event.
When you call the WaveOut object's Stop() method or the OGG playback terminates, this event is raised. Here, the Stop() method is called in the btnStopPlayback click handler.
When this happens, another method, WaveOutReset(), is called; this method disposes of the WaveOut object and the Stream of the WaveOutReader. The event handler is also removed.
using NAudio.Vorbis;
using NAudio.Wave;
public partial class MainForm : Form {
private VorbisWaveReader? vorbis = null;
private WaveOut? oggPlayer = null;
private void btnPlayAudio_Click(object sender, EventArgs e)
{
btnPlayAudio.Enabled = false;
btnStopPlayback.Enabled = true;
vorbis = new VorbisWaveReader("OGG File path"));
oggPlayer = WaveOutInit(vorbis);
oggPlayer.Play();
}
private void btnStopPlayback_Click(object sender, EventArgs e)
{
oggPlayer?.Stop();
btnStopPlayback.Enabled = false;
}
private void WaveOut_PlaybackStopped(object? sender, StoppedEventArgs e)
{
WaveOutReset(oggPlayer, vorbis);
btnPlayAudio.Enabled = true;
}
private WaveOut WaveOutInit(IWaveProvider reader)
{
var waveOut = new WaveOut();
waveOut.PlaybackStopped += WaveOut_PlaybackStopped;
waveOut.Init(reader);
return waveOut;
}
private void WaveOutReset(WaveOut? player, VorbisWaveReader? reader)
{
if (player != null) {
player.PlaybackStopped -= WaveOut_PlaybackStopped;
player.Dispose();
}
reader?.Dispose();
}
}

Start async task without waiting for it to finish, but check for completion

I've been searching for a correct answer very hard, and all I find is either too complicated or not doing what i'm looking for.
The case is simple:
I want to start a task asynchronously every now and then. The same task always. There's no defined interval of time to start it (suppose it's random).
The task takes no arguments and doesn't return anything.
I don't want to wait for it to finish. It can't interfere with the rest of the program.
I want to check if it finished before firing it again. Don't want the same task running many times simultaneously. Just one each time.
I don't want to use timers or global variables like semaphors or something. Just a plain and clean solution for a very simple problem.
I've tried background workers, but the tasks overlap as I can't find a reliable way to check for completion. I've tried running tasks but they can't be restarted. I've tried with async/await but I don't want to wait for completion.
EDIT:
I'll provide more information. This application is for a facial recognition SW. I have to handle 3 cameras and I'm using EmguCV. Each camera suscribes to an ImageGrabbed Event called "ProcessFrame", so I have ProcessFrame1, ProcessFrame2 and ProcessFrame3. The Events are fired up at nearly the fps of each camera, given so, the frequency is very high. In each Event I take a capture and show it up in an ImageBox (Emgu's pictureBox). Every 5 captures I check if I have at least one capture of each camera, in which case, instead of showing it in the imagebox I perform a facial recognition on each image. This is the task I want to perform in a separate task, to avoid stopping the live video for each camera.
Right now I'm trying with a semaphore, as suggested by some of you, although I had some trouble on setting the chance to perform a DetectFace() in the three Events, so i just left one up.
Here is a snippet:
public Form1()
{
InitializeComponent();
//Instantiate each camera
//Subscribe to ProcessFrame1, ProcessFrame2 and ProcessFrame3
}
private void ProcessFrame1(object sender, EventArgs e)
{
if (captures[0] != null) //captures[0] is the handle for the camera 1
{
Mat snapshot = new Mat();
captures[0].Retrieve(snapshot);
if (snapshot != null)
{
frameCounter1++;
if (frameCounter1 > 5 && taskCompleted)
{
frameCounter1 = 0;
if (images[0] == null)
{
Image<Bgr, Byte> img = snapshot.ToImage<Bgr, Byte>();
images[0] = img.ToBitmap();
}
if (images[0] != null && images[1] != null && images[2] != null)
{
Thread hilo = new Thread(() => DetectFace());
hilo.IsBackground = true;
hilo.Start();
}
return;
}
else
imageBox1.Image = snapshot;
}
}
}
private void ProcessFrame2(object sender, EventArgs e)
{
if (captures[1] != null) //captures[1] is the handle for the camera 2
{
Mat snapshot = new Mat();
captures[1].Retrieve(snapshot);
if (snapshot != null)
{
frameCounter2++;
if (frameCounter2 > 5 && taskCompleted)
{
frameCounter2 = 0;
if (images[1] == null)
{
Image<Bgr, Byte> img = snapshot.ToImage<Bgr, Byte>();
images[1] = img.ToBitmap();
}
//I used to have the checking to fire up another DetectFace here
return;
}
else
imageBox2.Image = snapshot;
}
}
}
private void ProcessFrame3(object sender, EventArgs e) //Same as ProcessFrame2
private void DetectFace()
{
taskCompleted = false;
//Processing of Images
//Clear array of images
taskCompleted = true;
}
Tasks work with states so you can start your task, save a reference in a variable and check the current status of your task whenever you want.
Here is the .NET documentation to read up on the different states of a task:
https://learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.task.status?view=netframework-4.7.2
https://learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.taskstatus?view=netframework-4.7.2
sadly I'm not able to give you a code example at the moment but I hope the idea helps.
I finally achieved the threadsafe solution using BlockingCollection. Thanks to #Damien_The_Unbeliever for pointing out to the right direction.
I think it's been an important lesson when working with images in a multithreading environment. I've learned that images are very vulnerable when sharing them in different threads.
As I posted in the question, here I have to take snapshots from X different webcams, show them in X different imageBoxes and, as fast as I can (without interrupting the fps of the shown video too much), perform an image processing instead of showing the frame in the imageBox. Using a BlockingCollection I don't need to stablish a frequency for processing the frames as I did before (every 5 frames). Now I can show the frame of each camera as long as I already have a frame from that camera added to the BlockingCollection.
Another important detail to note is, in the .NET documentation for BlockingCollection it says that by default it implements a FIFO as it's a ConcurrentQueue in the lower layer, but I think this isn't true, as I have had to define it myself when instantiating:
BlockingCollection<Tuple<int, Image>> tupleCollection = new BlockingCollection<Tuple<int, Image>>(new ConcurrentQueue<Tuple<int, Image>>(), X);
As the Take() method can't target the desired element in the collection, I had to use a tuple to know to which camera the frame belonged, and to take the frames in order I had to define the ConcurrentQueue.
So basically the pseudocode is this:
void Main()
{
//Instantiate cameras
//Subscribe to the ImageGrabbed events of each (producers)
// A simple blocking consumer with no cancellation.
Task.Run(() => DetectFace());
}
producer1(sender, e)
{
//Get snapshot
...
if (!tupleCollection.Any(x => x.Item1 == 1))
{
tupleCollection.Add(new Tuple<int, Image>(1, snapshot));
}
else
imageBox1.Image = snapshot;
}
producer2(sender, e)
{
//Get snapshot
...
if (!tupleCollection.Any(x => x.Item1 == 2))
{
tupleCollection.Add(new Tuple<int, Image>(2, snapshot));
}
else
imageBox2.Image = snapshot;
}
...
producerX(sender, e)
{
//Get snapshot
...
if (!tupleCollection.Any(x => x.Item1 == X))
{
tupleCollection.Add(new Tuple<int, Image>(X, snapshot));
}
else
imageBoxX.Image = snapshot;
}
private void DetectFace()
{
while (true)
{
Tuple<int, Image> data = null;
try
{
data = tupleCollection.Take();
}
catch (InvalidOperationException) { }
if (data != null)
{
//image processing
}
}
}
Most examples I have found use the condition IsCompletedAdded and IsCompleted to stop adding and consuming, but I needed it to be running forever, so the while(true) statement.
I have been running this code 24/7 within the last week, no racing faults by now, and the CPU processor is very limited, so I'm very satisfied with this solution and I think it's the right one.

Maintain a responsive user interface with backgroundWorker and heavy animation

I have a C# application (winforms and wpf, but for this question I'm focusing on the winforms one) where a backgroundWorker is used to work on a data set, and the call to ProgressChanged which then calls the form Refresh method to force a repaint. This then paints a bunch of ellipses based on the current frame of the data set.
A given frame may involve drawing anywhere between zero and several hundred ellipses.
In addition, I have a slider control that allows the user to adjust the playback rate (basically the thread.sleep value within the loop.)
When the user sets the sleep value too low, sometimes the repainting methods get queued up, and the UI becomes unresponsive. (This depends on the number of ellipses in the frame, and the speed of the computer. And the delay is 100% with the repainting on the UI, not with any other processing, which is basically just incrementing a counter and setting a label text.)
I would like to be able to detect the queuing up and automatically adjust the speed slider to accommodate a larger data set and/or slower computer. How can I tell if the UI thread is backed up with multiple calls to Map_Paint?
Current code (paraphrased):
public Map()
{
InitializeComponent();
_worker = new BackgroundWorker();
_worker.DoWork += _worker_DoWork;
_worker.ProgressChanged += _worker_ProgressChanged;
_worker.WorkerReportsProgress = true;
}
private void _worker_DoWork(object sender, DoWorkEventArgs e)
{
_frameCount = _frames.FrameCount();
// For this specific example, _frameCount may be around 30000-40000
for (var i = 0; i < _frameCount; i++)
{
var f = _frames.Frame(i + 1);
_worker.ReportProgress(i, f);
Thread.Sleep(_tickCount);
_suspend.WaitOne(); // Used to Pause the playback
}
}
void _worker_ProgressChanged(object sender, ProgressChangedEventArgs e)
{
// set some variables according to state and progresspercentage snipped here
// ...
// I would like to detect at this point whether the screen repainting is backed up
// and if so, adjust the value of _tickCount to slow down the program.
this.Refresh();
}
private void Map_Paint(object sender, PaintEventArgs e)
{
// Lots of ellipsis drawing stuff here
// Maybe 0-1000 ellipses drawn per cycle.
}
private void tbSpeed_Scroll(object sender, EventArgs e)
{
// This is the Scroll event for the slider.
// Value range is 10-300
// The slider becomes unresponsive when the UI thread backs up.
// I'd like to detect the back up and override the value of _tickCount
_tickCount = tbSpeed.Value;
}
private static object _lock = new object();
private static int _queuedCount = 0;
public Map()
{
InitializeComponent();
_worker = new BackgroundWorker();
_worker.DoWork += _worker_DoWork;
_worker.ProgressChanged += _worker_ProgressChanged;
_worker.WorkerReportsProgress = true;
}
private void _worker_DoWork(object sender, DoWorkEventArgs e)
{
_frameCount = _frames.FrameCount();
// For this specific example, _frameCount may be around 30000-40000
for (var i = 0; i < _frameCount; i++)
{
var f = _frames.Frame(i + 1);
lock(_lock)
{
_queuedCount++;
}
_worker.ReportProgress(i, f);
Thread.Sleep(_tickCount);
_suspend.WaitOne(); // Used to Pause the playback
}
}
void _worker_ProgressChanged(object sender, ProgressChangedEventArgs e)
{
if (_queuedCount > 1)
//now queue is building up
this.Refresh();
lock(_lock)
{
_queuedCount--;
}
}

How to play sounds asynchronuously, but themselves in a queue?

I just want to play 4 sounds after each other (sounds1->sound2->sound3), but without stopping the flow in my code during each play or without waiting for each sound to finish.
I have searched for this about everywhere, but every direction I read, gets stuck in some other problem.
My best bet so far was: using my already used SoundPlayer from System.Media and make my own queue function, but Soundplayer doesn't have a "finished playing" event so I have no idea of knowing when to start the next sound. (Really, Microsoft?)
Other solution and problems:
DirectSound seems complicated to get working in .NET (c#).
Win Playsound doesn't really help because it can't queue either.
You can try to use PlaySync on a thread outside UI, eg: Background thread, as some people have commented.
Here is a sample (untested) using a thread-safe* BlockingCollection for the queue
* which you can use in and outside the thread
You may want to make your own class or methods that rises an event every time the sounds ends. Or you can just loop the queue in the thread since PlaySync will just wait by itself.
using System.Threading;
using System.Collections.Concurrent;
namespace PlaySound
{
public partial class Form1 : Form
{
private Thread soundPlayThread;
private BlockingCollection<string> speakQueue = new BlockingCollection<string>();
private CancellationTokenSource cancelSoundPlay;
private int soundPlayCount = 0;
public Form1()
{
InitializeComponent();
cancelSoundPlay = new CancellationTokenSource();
}
private void btnStartSoundPlay_Click(object sender, EventArgs e)
{
StartSoundPlay();
}
private void btnStopSoundPlay_Click(object sender, EventArgs e)
{
cancelSoundPlay.Cancel();
Console.WriteLine("Sound play cancelled.");
}
private void btnAddToQueue_Click(object sender, EventArgs e)
{
speakQueue.Add("MyFile.wav");
}
private void queueAndPlay(string loc)
{
if (!File.Exists(loc=loc+".wav"))
loc=configPath+"soundnotfound.wav";
speakQueue.Add(loc);
StartSoundPlay();
}
private void StartSoundPlay()
{
//Sound Player Loop Thread
if (this.soundPlayThread == null || !this.soundPlayThread.IsAlive)
{
this.soundPlayThread = new Thread(SoundPlayerLoop);
this.soundPlayThread.Name = "SoundPlayerLoop";
this.soundPlayThread.IsBackground = true;
this.soundPlayThread.Start();
Console.WriteLine("Sound play started");
}
}
//Method that the outside thread will use outside the thread of this class
private void SoundPlayerLoop()
{
var sound = new SoundPlayer();
foreach (String soundToPlay in this.speakQueue.GetConsumingEnumerable(cancelSoundPlay.Token))
{
//http://msdn.microsoft.com/en-us/library/system.media.soundplayer.playsync.aspx
speaker.SoundLocation=soundToPlay;
//Here the outside thread waits for the following play to end before continuing.
sound.PlaySync();
soundPlayCount++;
Console.WriteLine("Sound play end. Count: " + soundPlayCount);
}
}
}
}

Continuous record and play audio in C# visual studio

I have been trying to record and play the audio simultaneously with out using a temporary wav file. And later I would like to create a VOIP chat program.
I have used the Naudio library to capture and play audio in C# and it seems to work quite well.
below is the c # code that i have written:
using System.IO.Ports;
using NAudio.Wave;
using System.IO;
namespace VOIP
{
public partial class Form1 : Form
{
WaveIn wab = new WaveIn();
MemoryStream s;
int k;
public Form1()
{
InitializeComponent();
wab.BufferMilliseconds = 100;
wab.NumberOfBuffers=5;
wab.DataAvailable += new EventHandler<WaveInEventArgs>(wa_DataAvailable);
}
void wa_DataAvailable(object sender, WaveInEventArgs e)
{
Play(e.Buffer);
}
private void Play(byte[] p)
{
WaveOut ou = new WaveOut();
s = new MemoryStream(p);
RawSourceWaveStream r = new RawSourceWaveStream(s, wab.WaveFormat);
ou.Init(r);
ou.Play();
ou.Stop();
ou.Dispose();
s.Dispose();
r.Dispose();
}
private void Form1_Load(object sender, EventArgs e)
{
}
private void button1_Click(object sender, EventArgs e)
{
if (button1.Text == "Stop")
{
wab.StopRecording();
button1.Text = "Record";
}
else if (button1.Text == "Record")
{
wab.StartRecording();
button1.Text = "Stop";
}
}
}
}
The Problem is at the "Play" method .Since the waveout object is created and disposed every time the data is available : i can hear some clicking sound. Is there a way to avoid this way of creating and disposing object and instead just create one object and then initialize with the new data. I also observed that the memory consumed by this program keeps increasing.
Thanks in advance.
sanatan
Use a BufferedWaveProvider, and as audio arrives, decompress it and add it to the BufferedWaveProvider. Then have a single instance of WaveOut that is playing constantly from the BufferedWaveProvider.
The source code for NAudioDemo shows how to do this in the chat example.

Categories