We Keep Coding

How can I draw a waveform efficiently

I am trying to display the waveform of an audio file. I would like the waveform to be drawn progressively as ffmpeg processes the file, as opposed to all at once after it's done. While I have achieved this effect, it's REALLY slow; like painfully slow. Its starts out really fast, but the speed degrades to the point of taking minutes to draw a sample.
I feel there has to be a way to do this more efficiently, as there is a program I use that does it, I just don't know how. The other program can take in >10 hours of audio and progressively display the waveform with no speed degradation. I have set ffmpeg to process the file at 500 samples/sec, but the other program samples at 1000/sec and it still runs faster than what I wrote. The other program's waveform display only takes about 120MB of RAM with a 10 hour file, where mine takes 1.5GB with a 10 minute file.
I'm fairly certain the slowness is caused by all the UI updates and the RAM usage is from all the rectangle objects being created. When I disable drawing the waveform, the async stream completes pretty fast; less than 1 min for a 10 hour file.
This is the only way I could think to accomplish what I want. I would welcome any help to improve what I wrote or any suggestions for an all together different way to accomplish it.
As a side note, this isn't all I want to display. I will eventually want to add a background grid to help judge time, and draggable line annotations to mark specific places in the waveform.
MainWindow.xaml
<ItemsControl x:Name="AudioDisplayItemsControl"
DockPanel.Dock="Top"
Height="100"
ItemsSource="{Binding Samples}">
<ItemsControl.Resources>
<DataTemplate DataType="{x:Type poco:Sample}">
<Rectangle Width="{Binding Width}"
Height="{Binding Height}"
Fill="ForestGreen"/>
</DataTemplate>
</ItemsControl.Resources>
<ItemsControl.ItemsPanel>
<ItemsPanelTemplate>
<Canvas Background="Black"
Width="500"/>
</ItemsPanelTemplate>
</ItemsControl.ItemsPanel>
<ItemsControl.ItemContainerStyle>
<Style TargetType="ContentPresenter">
<Setter Property="Canvas.Top" Value="{Binding Top}"/>
<Setter Property="Canvas.Left" Value="{Binding Left}"/>
</Style>
</ItemsControl.ItemContainerStyle>
</ItemsControl>
MainWindow.xaml.cs
private string _audioFilePath;
public string AudioFilePath
{
get => _audioFilePath;
set
{
if (_audioFilePath != value)
{
_audioFilePath = value;
NotifyPropertyChanged();
}
}
}
private ObservableCollection<IShape> _samples;
public ObservableCollection<IShape> Samples
{
get => _samples;
set
{
if (_samples != value)
{
_samples = value;
NotifyPropertyChanged();
}
}
}
//Eventhandler that starts this whole process
private async void GetGetWaveform_Click(object sender, RoutedEventArgs e)
{
((Button)sender).IsEnabled = false;
await GetWaveformClickAsync();
((Button)sender).IsEnabled = true;
}
private async Task GetWaveformClickAsync()
{
Samples.Clear();
double left = 0;
double width = .01;
double top = 0;
double height = 0;
await foreach (var sample in FFmpeg.GetAudioWaveform(AudioFilePath).ConfigureAwait(false))
{
// Map {-32,768, 32,767} (pcm_16le) to {-50, 50} (height of sample display)
// I don't this this mapping is correct, but that's not important right now
height = ((sample + 32768) * 100 / 65535) - 50;
// "0" pcm values are not drawn in order to save on UI updates,
// but draw position is still advanced
if (height==0)
{
left += width;
continue;
}
// Positive pcm values stretch "height" above the canvas center line
if (height > 0)
top = 50 - height;
// Negative pcm values stretch "height" below the centerline
else
{
top = 50;
height = -height;
}
Samples.Add(new Sample
{
Height = height,
Width = width,
Top = top,
Left = left,
ZIndex = 1
});
left += width;
}
}
Classes used to define a sample
public interface IShape
{
double Top { get; set; }
double Left { get; set; }
}
public abstract class Shape : IShape
{
public double Top { get; set; }
public double Left { get; set; }
public int ZIndex { get; set; }
}
public class Sample : Shape
{
public double Width { get; set; }
public double Height { get; set; }
}
FFMpeg.cs
public static class FFmpeg
{
public static async IAsyncEnumerable<short> GetAudioWaveform(string filename)
{
var args = GetFFmpegArgs(FFmpegTasks.GetWaveform, filename);
await foreach (var sample in RunFFmpegAsyncStream(args))
{
yield return sample;
}
}
/// <summary>
/// Streams raw results of running ffmpeg.exe with given arguments string
/// </summary>
/// <param name="args">CLI argument string used for ffmpeg.exe</param>
private static async IAsyncEnumerable<short> RunFFmpegAsyncStream(string args)
{
using (var process = new Process())
{
process.StartInfo.FileName = #"External\ffmpeg\bin\x64\ffmpeg.exe";
process.StartInfo.Arguments = args;
process.StartInfo.UseShellExecute = false;
process.StartInfo.RedirectStandardError = true;
process.StartInfo.RedirectStandardOutput = true;
process.StartInfo.CreateNoWindow = true;
process.Start();
process.BeginErrorReadLine();
var buffer = new byte[2];
while (true)
{
// Asynchronously read a pcm16_le value from ffmpeg.exe output
var r = await process.StandardOutput.BaseStream.ReadAsync(buffer, 0, 2);
if (r == 0)
break;
yield return BitConverter.ToInt16(buffer);
}
}
}
FFmpegTasks is just an enum.
GetFFmpegArgs uses a switch argument on FFmpegTasks to return the appropriate CLI arguments for ffmpeg.exe.
I tried using the following class instead of the standard ObservableCollection because I was hoping that less UI updates would speed things up, but it actually made drawing the waveform slower.
RangeObservableCollection.cs
public class RangeObservableCollection<T> : ObservableCollection<T>
{
private bool _suppressNotification = false;
protected override void OnCollectionChanged(NotifyCollectionChangedEventArgs e)
{
if (!_suppressNotification)
base.OnCollectionChanged(e);
}
public void AddRange(IEnumerable<T> list)
{
if (list == null)
throw new ArgumentNullException("list");
_suppressNotification = true;
foreach (T item in list)
{
Add(item);
}
_suppressNotification = false;
OnCollectionChanged(new NotifyCollectionChangedEventArgs(NotifyCollectionChangedAction.Reset));
}
}

You could try drawing a Path by hand. I use this to draw histograms of images in an application:
/// <summary>
/// Converts a histogram to a <see cref="PathGeometry"/>.
/// This is used by converters to draw a path.
/// It is easiest to use the default Canvas width and height and then use a ViewBox.
/// </summary>
/// <param name="histogram">The counts of each value.</param>
/// <param name="CanvasWidth">Width of the canvas</param>
/// <param name="CanvasHeight">Height of the canvas.</param>=
/// <returns>A path geometry. This value can be bound to a <see cref="Path"/>'s Data.</returns>
public static PathGeometry HistogramToPathGeometry(int[] histogram, double CanvasWidth = 100.0, double CanvasHeight = 100.0)
{
double xscale = CanvasWidth / histogram.Length;
double histMax = histogram.Max();
double yscale = CanvasHeight / histMax;
List<LineSegment> segments = new List<LineSegment>();
for (int i = 0; i < histogram.Length; i++)
{
double X = i * xscale;
double Y1 = histogram[i] * yscale;
double Y = CanvasHeight - Y1;
if (Y == double.PositiveInfinity) Y = CanvasHeight;
segments.Add(new LineSegment(new Point(X, Y), true));
}
segments.Add(new LineSegment(new Point(CanvasWidth, CanvasHeight), true));
PathGeometry geometry = new PathGeometry();
PathFigure figure = new PathFigure(new Point(0, CanvasHeight), segments, true);
geometry.Figures = new PathFigureCollection
{
figure
};
return geometry;
}
Then have this in your Xaml:
<Canvas Width="100" Height="100">
<Path Data="{Binding ConvertedPathGeometry}" />
</Canvas>
You could change this up so it handles the data as it is coming in. I'm not sure how well it would work with a lot of points, but you could only update the view after a handful of new points have come in. I've dealt with trying to draw many rectangles in a display and have the same issue you are running into.

As promised, here is all the code that is involved with drawing my waveform. I probably added a little more than necessary, but there is it. I was hoping to draw the waveform as it was processed, but gave up on that idea... for now. I hope this helps someone out there, because it took me 2 weeks in all to get this worked out. Comments are welcome.
Just as an FYI, on my pc it takes about 35 seconds to process a ~10hr m4b file and <2ms to display the image.
Also, it assumes a little-endian system. I put no checks in for big-endian systems, as I am not on one. If you are, the Buffer.BlockCopy will need some attention.
Mainwindow.xaml
<Border Background="Black"
Height="100"
Width="720"
BorderThickness="0">
<Image x:Name="WaveformImg" Source="{Binding Waveform, Mode=OneWay}"
Height="100"
Width="{Binding ImageWidth, Mode=OneWayToSource}"
Stretch="Fill" />
</Border>
<Button Content="Get Waveform"
Padding="5 0"
Margin="5 0"
Click="GetGetWaveform_Click" />
Mainwindow.xaml.cs
private async void GetGetWaveform_Click (object sender, RoutedEventArgs e)
{
((Button)sender).IsEnabled = false;
await ProcessAudiofile();
await GetWaveformClickAsync(0, AudioData!.TotalSamples - 1);
((Button)sender).IsEnabled = true;
}
private async Task ProcessAudiofile ()
{
var milliseconds = FFmpeg.GetAudioDurationSeconds(AudioFilePath);
AudioData = new AudioData(await milliseconds, FFmpeg.SampleRate, new Progress<ProgressStatus>(ReportProgress));
await AudioData.ProcessStream(FFmpeg.GetAudioWaveform(AudioFilePath)).ConfigureAwait(false);
}
private const int _IMAGE_WIDTH = 720;
private async Task GetWaveformClickAsync (int min, int max)
{
var sw = new Stopwatch();
sw.Start();
int color_ForestGreen = 0xFF << 24 | 0x22 << 16 | 0x8c << 8 | 0x22 << 0; //
Waveform = new WriteableBitmap(_IMAGE_WIDTH, 100, 96, 96, PixelFormats.Bgra32, null);
int col = 0;
int currSample = 0;
int row;
var sampleTop = 0;
var sampleBottom = 0;
var sampleHeight = 0;
//I thought this would draw the wave form line by line but it blocks the UI and draws the whole waveform at once
foreach (var sample in AudioData.GetSamples(_IMAGE_WIDTH, min, max))
{
sampleBottom = 50 + (int)(sample.min * (double)50 / short.MinValue);
sampleTop = 50 - (int)(sample.max * (double)50 / short.MaxValue);
sampleHeight = sampleBottom - sampleTop;
try
{
Waveform.Lock();
DrawLine(col, sampleTop, sampleHeight, color_ForestGreen);
col++;
}
finally
{
Waveform.Unlock();
}
}
sw.Stop();
Debug.WriteLine($"Image Creation: {sw.Elapsed}");
}
private void DrawLine (int column, int top, int height, int color)
{
unsafe
{
IntPtr pBackBuffer = Waveform.BackBuffer;
for (int i = 0; i < height; i++)
{
pBackBuffer = Waveform.BackBuffer; // Backbuffer start address
pBackBuffer += (top + i) * Waveform.BackBufferStride; // Move to address or desired row
pBackBuffer += column * 4; // Move to address of desired column
*((int*)pBackBuffer) = color;
}
}
try
{
Waveform.AddDirtyRect(new Int32Rect(column, top, 1, height));
}
catch (Exception) { } // I know this isn't a good way to deal with exceptions, but its what i did.
}
AudioData.cs
public class AudioData
{
private List<short> _amp; // pcm_s16le amplitude values
private int _expectedTotalSamples; // Number of samples expected to be returned by FFMpeg
private int _totalSamplesRead; // Current total number of samples read from the file
private IProgress<ProgressStatus>? _progressIndicator; // Communicates progress to the progress bar
private ProgressStatus _progressStatus; // Progress status to be passed to progress bar
/// <summary>
/// Total number of samples obtained from the audio file
/// </summary>
public int TotalSamples
{
get => _amp.Count;
}
/// <summary>
/// Length of audio file in seconds
/// </summary>
public double Duration
{
get => _duration;
private set
{
_duration = value < 0 ? 0 : value;
}
}
private double _duration;
/// <summary>
/// Number of data points per second of audio
/// </summary>
public int SampleRate
{
get => _sampleRate;
private set
{
_sampleRate = value < 0 ? 0 : value;
}
}
private int _sampleRate;
/// <summary>Update the window's size.</summary>
/// <param name = "duration" > How long the audio file is in milliseconds.</param>
/// <param name = "sampleRate" > Number of times per second to sample the audio file</param>
/// <param name = "progressIndicator" >Used to report progress back to a progress bar</param>
public AudioData (double duration, int sampleRate, IProgress<ProgressStatus>? progressIndicator = null)
{
Duration = duration;
SampleRate = sampleRate;
_progressIndicator = progressIndicator;
_expectedTotalSamples = (int)Math.Ceiling(Duration * SampleRate);
_amp = new List<short>();
_progressStatus = new ProgressStatus();
}
/// <summary>
/// Get values from an async pcm_s16le stream from FFMpeg
/// </summary>
/// <param name = "sampleStream" >FFMpeg samples stream</param>
public async Task ProcessStream (IAsyncEnumerable<(int read, short[] samples)> sampleStream)
{
_totalSamplesRead = 0;
_progressStatus = new ProgressStatus
{
Label = "Started",
};
await foreach ((int read, short[] samples) in sampleStream)
{
_totalSamplesRead += read;
_amp.AddRange(samples[..read]); // Only add the number of samples that where read this iteration
UpdateProgress();
}
Duration = _amp.Count() / SampleRate; // update duration to the correct value; incase duration reported by FFMpeg was wrong
UpdateProgress(true);
}
/// <summary>
/// Report progress back to the UI
/// </summary>
/// <param name="finished">Is FFmpeg done processing the file</param>
private void UpdateProgress (bool finished = false)
{
int percent = (int)(100 * (double)_totalSamplesRead / _expectedTotalSamples);
// Calculate progress update interval; once every 1%
if (percent == _progressStatus.Value)
return;
// update progress status bar object
if (finished)
{
_progressStatus.Label = "Done";
_progressStatus.Value = 100;
}
else
{
_progressStatus.Label = $"Running ({_totalSamplesRead} / {_expectedTotalSamples})";
_progressStatus.Value = percent;
}
_progressIndicator?.Report(_progressStatus);
}
/// <summary>
/// Get evenly spaced sample subsets of the entire audio file samples
/// </summary>
/// <param name="count">Number of samples to be returned</param>
/// <returns>An IEnumerable tuple containg the minimum and maximum amplitudes of a range of samples</returns>
public IEnumerable<(short min, short max)> GetSamples (int count)
{
foreach (var sample in GetSamples(count, 0, -_amp.Count - 1))
yield return sample;
}
/// <summary>
/// Get evenly spaced sample subsets of a section of the audio file samples
/// </summary>
/// <param name="count">number of data points to return</param>
/// <param name="min">inclusive starting index</param>
/// <param name="max">inclusive ending index</param>
/// <returns>An IEnumerable tuple containing the minimum and maximum amplitudes of a range of samples</returns>
public IEnumerable<(short min, short max)> GetSamples (int count, int min, int max)
{
// Protect from out of range exception
max = max >= _amp.Count ? _amp.Count - 1 : max;
max = max < 1 ? 1 : max;
min = min >= _amp.Count - 1 ? _amp.Count - 2 : min;
min = min < 0 ? 0 : min;
double sampleSize = (max - min) / (double)count; // Number of samples to inspect for return value
short rMin; // Minimum return value
short rMax; // Maximum return value
int ssOffset;
int ssLength;
for (int n = 0; n < count; n++)
{
// Calculate offset; no account for min
ssOffset = (int)(n * sampleSize);
// Determine how many samples to get, with a minimum of 1
ssLength = sampleSize <= 1 ? 1 : (int)((n + 1) * sampleSize) - ssOffset;
//shift offset to account for min
ssOffset += min;
// Double check that ssLength wont take us out of bounds
ssLength = ssOffset + ssLength >= _amp.Count ? _amp.Count - ssOffset : ssLength;
// Get the minimum and maximum amplitudes in this sample range
rMin = _amp.GetRange(ssOffset, ssLength).Min();
rMax = _amp.GetRange(ssOffset, ssLength).Max();
// In case this sample range has no (-) values, make the lowest one zero. This makes the rendered waveform look better.
rMin = rMin > 0 ? (short)0 : rMin;
rMax = rMax < 0 ? (short)0 : rMax;
yield return (rMin, rMax);
}
}
}
public class ProgressStatus
{
public int Value { get; set; }
public string Label { get; set; }
}
FFMpeg.cs
private const int _BUFER_READ_SIZE = 500; // Number of bytes to read from process.StandardOutput.BaseStream. Must be a multiple of 2
public const int SampleRate = 1000;
public static async IAsyncEnumerable<(int, short[])> GetAudioWaveform (string filename)
{
using (var process = new Process())
{
process.StartInfo = FFMpegStartInfo(FFmpegTasks.GetWaveform, filename);
process.Start();
process.BeginErrorReadLine();
await Task.Delay(1000); // Give process.StandardOutput a chance to build up values in BaseStream
var bBuffer = new byte[_BUFER_READ_SIZE]; // BaseStream.ReadAsync buffer
var sBuffer = new short[_BUFER_READ_SIZE / 2]; // Return value buffer; bBuffer.length
int read = 1; // ReadAsync returns 0 when BaseStream is empty
while (true)
{
read = await process.StandardOutput.BaseStream.ReadAsync(bBuffer, 0, _BUFER_READ_SIZE);
if (read == 0)
break;
Buffer.BlockCopy(bBuffer, 0, sBuffer, 0, read);
yield return (read / 2, sBuffer);
}
}
}
private static ProcessStartInfo FFMpegStartInfo (FFmpegTasks task, string inputFile1, string inputFile2 = "", string outputFile = "", bool overwriteOutput = true)
{
if (string.IsNullOrWhiteSpace(inputFile1))
throw new ArgumentException(nameof(inputFile1), "Path to input file is null or empty");
if (!File.Exists(inputFile1))
throw new FileNotFoundException($"No file found at: {inputFile1}", nameof(inputFile1));
var args = task switch
{
// TODO: Set appropriate sample rate
// TODO: remove -t xx
FFmpegTasks.GetWaveform => $#" -i ""{inputFile1}"" -ac 1 -filter:a aresample={SampleRate} -map 0:a -c:a pcm_s16le -f data -",
FFmpegTasks.DetectSilence => throw new NotImplementedException(),
_ => throw new NotImplementedException(),
};
return new ProcessStartInfo()
{
FileName = #"External\ffmpeg\bin\x64\ffmpeg.exe",
Arguments = args,
UseShellExecute = false,
RedirectStandardError = true,
RedirectStandardOutput = true,
CreateNoWindow = true
};
public enum FFmpegTasks
{
GetWaveform
}
}

Continuously getting data from heart rate monitor

I've been working on a project that reads a person's heart rate from a Polar H7 hrm. I've been successful in connecting the device and getting the heart rate which the program shows as text in the UI. However, there are instances where the program suddenly stops getting input from the device.
I have already checked the connection of the device to my Win 10 laptop and saw that it was stable, there were also no exceptions getting thrown by the program. The text simply stops changing.
Here is the code I've written:
public sealed partial class MainPage : Page
{
private GattDeviceService device;
public MainPage()
{
this.InitializeComponent();
init();
}
async void init()
{
var devices = await DeviceInformation.FindAllAsync(GattDeviceService.GetDeviceSelectorFromUuid(GattServiceUuids.HeartRate));
Status.Text = devices.Count.ToString();
device = await GattDeviceService.FromIdAsync(devices[0].Id);
enableSensor();
}
private async void enableSensor()
{
IReadOnlyList<GattCharacteristic> characteristicList;
characteristicList = device.GetAllCharacteristics();
if (characteristicList != null)
{
GattCharacteristic characteristic = characteristicList[0];
if (characteristic.CharacteristicProperties.HasFlag(GattCharacteristicProperties.Notify))
{
characteristic.ValueChanged += SendNotif;
await characteristic.WriteClientCharacteristicConfigurationDescriptorAsync(GattClientCharacteristicConfigurationDescriptorValue.Notify);
}
}
}
async void SendNotif(GattCharacteristic sender, GattValueChangedEventArgs eventArgs)
{
if (eventArgs.CharacteristicValue.Length != 0) {
byte[] hrData = new byte[eventArgs.CharacteristicValue.Length];
DataReader.FromBuffer(eventArgs.CharacteristicValue).ReadBytes(hrData);
var hrValue = ProcessData(hrData);
await Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () =>
{
Status.Text = hrValue.ToString();
});
}
}
private int ProcessData(byte[] data)
{
// Heart Rate profile defined flag values
const byte heartRateValueFormat = 0x01;
byte currentOffset = 0;
byte flags = data[currentOffset];
bool isHeartRateValueSizeLong = ((flags & heartRateValueFormat) != 0);
currentOffset++;
ushort heartRateMeasurementValue;
if (isHeartRateValueSizeLong)
{
heartRateMeasurementValue = (ushort)((data[currentOffset + 1] << 8) + data[currentOffset]);
currentOffset += 2;
}
else
{
heartRateMeasurementValue = data[currentOffset];
}
return heartRateMeasurementValue;
}
}

Kinect Swipe Conflict between left hand and right hand

I have written a Kinect swipe testing program from scratch that read gbd files produced by Visual gesture builder Kinect Swipe Test Project here.
The Main 4 gestures I want to focus in this questions is Left & right Hands up, Left & right hand Swipe up.
In the github link i posted above it works fine integrated together.
However, when i port the code over to my main project, Left hand Hands
Up and Right hand Swipe up gestures both malfunctioned. Both gesture
are unable to be detected.
I cannot post my Main project here so i will be posting the code snippets of how i implemented on the 2 Main project files, Mainly GestureDetector.cs and GestureResultsView.cs
Here are the respective snippet of the i had issue with files.
GestureDetector.cs :
public class GestureDetector : IDisposable
{
//stores an array of paths that leads to the gbd files
private string[] databaseArray = {//Left Hand
#"Database\LeftHandDownToTop.gbd"
//, #"Database\LeftHandHandsUp.gbd"
//, #"Database\LeftHandLeftToRight.gbd"
//, #"Database\LeftHandRightToLeft.gbd"
//Right Hand
, #"Database\RightHandBottomToUp.gbd"
, #"Database\RightHandHandsUp.gbd"
//, #"Database\RightHandLeftToRight.gbd"
//, #"Database\RightHandRightToLeft.gbd"
};
//stores an array of the names of the gesture found inside the gbd file
private string[] databaseGestureNameArray = { //Swipe Up
"SwipeUp_Left",
"SwipeUp_Right",
//Swipe Right
//"SwipeRight_Left",
//"SwipeRight_Right",
//Swipe Left
// "SwipeLeft_Left",
// "SwipeLeft_Right",
//Hands Up
//"HandsUp_Left",
"HandsUp_Right"
};
for (int i = 0; i < databaseArray.Length; i++)
{
// load the 'Seated' gesture from the gesture database
using (VisualGestureBuilderDatabase database = new VisualGestureBuilderDatabase(databaseArray[i]))
{
// we could load all available gestures in the database with a call to vgbFrameSource.AddGestures(database.AvailableGestures),
// but for this program, we only want to track one discrete gesture from the database, so we'll load it by name
foreach (Gesture gesture in database.AvailableGestures)
{
if (gesture.Name.Equals(databaseGestureNameArray[i]))
{
this.vgbFrameSource.AddGesture(gesture);
}
}
}
}
}
/// <summary>
/// Handles gesture detection results arriving from the sensor for the associated body tracking Id
/// </summary>
/// <param name="sender">object sending the event</param>
/// <param name="e">event arguments</param>
private void Reader_GestureFrameArrived(object sender, VisualGestureBuilderFrameArrivedEventArgs e)
{
VisualGestureBuilderFrameReference frameReference = e.FrameReference;
using (VisualGestureBuilderFrame frame = frameReference.AcquireFrame())
{
if (frame != null)
{
// get the discrete gesture results which arrived with the latest frame
IReadOnlyDictionary<Gesture, DiscreteGestureResult> discreteResults = frame.DiscreteGestureResults;
bool foundGesture = false;
if (discreteResults != null)
{
// we only have one gesture in this source object, but you can get multiple gestures
foreach (Gesture gesture in this.vgbFrameSource.Gestures)
{
for (int i = 0; i < databaseGestureNameArray.Length; i++)
{
if (gesture.Name.Equals(databaseGestureNameArray[i]) && gesture.GestureType == GestureType.Discrete)
{
DiscreteGestureResult result = null;
discreteResults.TryGetValue(gesture, out result);
if (result != null && !foundGesture)
{
// update the GestureResultView object with new gesture result values & update the label
foundGesture = this.GestureResultView.UpdateGestureResult(true, result.Detected, result.Confidence, databaseGestureNameArray[i]);
}
}
}
}
}
}
}
}
GestureResultView.cs :
public bool UpdateGestureResult(bool isBodyTrackingIdValid, bool isGestureDetected, float detectionConfidence, string gestureName)
{
this.IsTracked = isBodyTrackingIdValid;
this.Confidence = 0.0f;
bool gestureFound = false;
if (!this.IsTracked)
{
this.ImageSource = this.notTrackedImage;
this.Detected = false;
this.BodyColor = Brushes.Gray;
}
else
{
this.Detected = isGestureDetected;
this.BodyColor = this.trackedColors[this.BodyIndex];
if (this.Detected)
{
this.Confidence = detectionConfidence;
if (this.Confidence > 0.4)
{
this.ImageSource = this.seatedImage;
//https://stackoverflow.com/questions/15425495/change-wpf-mainwindow-label-from-another-class-and-separate-thread
//to change label in other class wpf
//http://the--semicolon.blogspot.co.id/p/change-wpf-window-label-content-from.html
string state = App.Current.Properties["state"].ToString();
switch (gestureName)
{
case "SwipeUp_Right":
SwipeUp(state);
break;
case "SwipeUp_Left":
SwipeUp(state);
break;
case "SwipeDown_Right":
break;
case "SwipeDown_Left":
break;
case "SwipeLeft_Right":
break;
case "SwipeLeft_Left":
break;
case "HandsUp_Right":
if (state.Equals("GoBackHome"))
{
}
else
{
Thread.Sleep(350);
MainWindow.handsUp();
}
break;
case "HandsUp_Left":
if (state.Equals("GoBackHome"))
{
}
else
{
Thread.Sleep(350);
MainWindow.handsUp();
}
break;
}
//"HandsUp_Right"
// , "SwipeRight_Right"
// , "SwipeUp_Right"
// , "SwipeLeft_Right"
// , "HandsUp_Left"
// , "SwipeRight_Left"
gestureFound = true;
}
}
else
{
this.ImageSource = this.notSeatedImage;
}
}
return gestureFound;
}
}
//Routing for gesture start
/// <summary>
/// Take in the current screen, filtered swipe up gesture
/// </summary>
/// <param name="state"></param>
private void SwipeUp(string state)
{
if (state.Equals("Home"))
{
int milliseconds = 350;
Thread.Sleep(milliseconds);
//await Task.Delay(500);
Home.swipeUp();
}
else if (state.Equals("ItemChoice"))
{
int milliseconds = 350;
Thread.Sleep(milliseconds);
//await Task.Delay(500);
ItemChoice.swipeUp();
}
else if (state.Equals("ItemParentChoice"))
{
int milliseconds = 350;
Thread.Sleep(milliseconds);
//await Task.Delay(500);
ItemParentChoice.swipeUp();
}
else if (state.Equals("LoanSummary"))
{
int milliseconds = 350;
Thread.Sleep(milliseconds);
//await Task.Delay(500);
LoanSummary.swipeUp();
}
else if (state.Equals("Timeslot"))
{
int milliseconds = 350;
Thread.Sleep(milliseconds);
//await Task.Delay(500);
Timeslot.swipeUp();
}
else if (state.Equals("ViewLoans"))
{
int milliseconds = 350;
Thread.Sleep(milliseconds);
//await Task.Delay(500);
ViewLoans.swipeUp();
}
else if (state.Equals("WhatToDo"))
{
int milliseconds = 350;
Thread.Sleep(milliseconds);
//await Task.Delay(500);
//WhatToDo.swipeUp();
}
}
/// <summary>
/// Take in the current screen, filtered swipe right gesture
/// </summary>
/// <param name="state"></param>
private void swipeRight(string state)
{
if (state.Equals("Home"))
{
//int milliseconds = 500;
//Thread.Sleep(milliseconds);
//Home.swipeUp();
}
else if (state.Equals("ItemChoice"))
{
int milliseconds = 500;
Thread.Sleep(milliseconds);
ItemChoice.swipeRight();
}
else if (state.Equals("ItemParentChoice"))
{
int milliseconds = 500;
Thread.Sleep(milliseconds);
ItemParentChoice.swipeRight();
}
else if (state.Equals("LoanSummary"))
{
//int milliseconds = 500;
//Thread.Sleep(milliseconds);
//LoanSummary.swipeUp();
}
else if (state.Equals("Timeslot"))
{
int milliseconds = 500;
Thread.Sleep(milliseconds);
Timeslot.swipeRight();
}
else if (state.Equals("ViewLoans"))
{
//int milliseconds = 500;
//Thread.Sleep(milliseconds);
//ViewLoans.swipeUp();
}
else if (state.Equals("WhatToDo"))
{
//int milliseconds = 500;
//Thread.Sleep(milliseconds);
//Home.swipeUp();
}
}
/// <summary>
/// Take in the current screen, filtered swipe right gesture
/// </summary>
/// <param name="state"></param>
private void swipeLeft(string state)
{
if (state.Equals("Home"))
{
//int milliseconds = 500;
//Thread.Sleep(milliseconds);
//Home.swipeUp();
}
else if (state.Equals("ItemChoice"))
{
int milliseconds = 500;
Thread.Sleep(milliseconds);
ItemChoice.swipeLeft();
}
else if (state.Equals("ItemParentChoice"))
{
int milliseconds = 500;
Thread.Sleep(milliseconds);
ItemParentChoice.swipeLeft();
}
else if (state.Equals("LoanSummary"))
{
//int milliseconds = 500;
//Thread.Sleep(milliseconds);
//LoanSummary.swipeUp();
}
else if (state.Equals("Timeslot"))
{
int milliseconds = 500;
Thread.Sleep(milliseconds);
Timeslot.swipeLeft();
}
else if (state.Equals("ViewLoans"))
{
//int milliseconds = 500;
//Thread.Sleep(milliseconds);
//ViewLoans.swipeUp();
}
else if (state.Equals("WhatToDo"))
{
//int milliseconds = 500;
//Thread.Sleep(milliseconds);
//Home.swipeUp();
}
}
//routing for gesture end
}

C# code acts wrong unless debugging

So I have been working on my C# project which required to do some serial communication. Everything was working fine until I have one really... weird issue.
So right now I am recieving a code in serial which is stored in myString. Then below where I get the issue which is that if I am not debugging the code then a whole if statement just... get ignored. However when I use a breakpoint it does everything right. For referance everything here works except the part of (*l)
(the rest of the code was removed for needless parts. I can attach more if you guys think it can help.)
public delegate void AddDataDelegate(String myString);
public AddDataDelegate myDelegate;
//
//
this.myDelegate = new AddDataDelegate(AddDataMethod);
//
//
private void Receiver(object sender, SerialDataReceivedEventArgs e)
{
Byte[] str = System.Text.Encoding.ASCII.GetBytes(comPort.ReadExisting());
this.Invoke(this.myDelegate, new Object[]
{
System.Text.Encoding.Default.GetString(str)
});
}
//
private void button2_Click(object sender, EventArgs e)
{
if (connectionStatus.Text == "Connected")
{
comPort.Close();
}
else
{
try
{
comPort.PortName = port;
comPort.BaudRate = baudRate;
comPort.DataBits = dataBits;
comPort.StopBits = (StopBits)stopBits;
comPort.Parity = parity;
comPort.DataReceived += new SerialDataReceivedEventHandler(Receiver);
comPort.Open();
connectionButton.Text = "Disconnect";
connectionStatus.Text = "Connected";
}
catch
{
comPort.Close();
}
}
}
public void AddDataMethod(String myString)
{
try
{
string colorSensorValue;
string distanceSensorValue;
string proximitySwitch;
string limitSwitch;
if (myString.Contains("*c*"))
{
string[] colors;
int red;
int blue;
int green;
int max;
colorSensorValue = myString.Substring(myString.IndexOf("*c*") + 3);
if (colorSensorValue.Contains("*e")) colorSensorValue = colorSensorValue.Substring(0, colorSensorValue.IndexOf("*e*"));
colors = colorSensorValue.Split(',');
red = Convert.ToInt16(colors[0]);
green = Convert.ToInt16(colors[1]);
blue = Convert.ToInt16(colors[2]);
max = Math.Max(red, Math.Max(green, blue));
red = red * 255 / max;
blue = blue * 255 / max;
green = green * 255 / max;
color.BackColor = Color.FromArgb(red,blue,green);
colorSensorTextBox.Text = (colorSensorValue);
}
if (myString.Contains("*d"))
{
distanceSensorValue = myString.Substring(myString.IndexOf("*d*") + 3);
if (distanceSensorValue.Contains("*e")) distanceSensorValue = distanceSensorValue.Substring(0, distanceSensorValue.IndexOf("*e*"));
distanceSensorTextBox.Text = (distanceSensorValue);
}
if (myString.Contains("*l"))
{
limitSwitch = myString.Substring(myString.IndexOf("*l*") + 3);
if (limitSwitch.Contains("*e")) limitSwitch = limitSwitch.Substring(0, limitSwitch.IndexOf("*e*"));
limitRead.Text = limitSwitch;
}
if (myString.Contains("*p"))
{
proximitySwitch = myString.Substring(myString.IndexOf("*p*") + 3);
if (proximitySwitch.Contains("*e")) proximitySwitch = proximitySwitch.Substring(0, proximitySwitch.IndexOf("*e*"));
proximityRead.Text = proximitySwitch;
}
comPort.BaseStream.Flush();
}
catch
{
}
}
Example of myString:
*c*96,84,75*e**d*25.5*e**p*0*e**l*0*e*
so it will be read as:
colors: 96 , 84 , 75 (happens right!)
distance: 25.5 (happens right!)
prox : 0 (happens right!)
limit : 0 (doesnt happen..)
note that order of both sent and received data doesn't change which one (limit) that doesn't work unless I breakpoint

According the SerialPort: class information in MSDN:
The DataReceived event is raised on a secondary thread when data is
received from the SerialPort object. Because this event is raised on a
secondary thread, and not the main thread, attempting to modify some
elements in the main thread, such as UI elements, could raise a
threading exception. If it is necessary to modify elements in the main
Form or Control, post change requests back using Invoke, which will do
the work on the proper thread.
You should use use this form:
this.Invoke(this.AddDataMethod, new Object[] { The_Received_String });
}

Playing sinus through XAudio2

I'm making an audio player using XAudio2. We are streaming data in packets of 640 bytes, at a sample rate of 8000Hz and sample depth of 16 bytes. We are using SlimDX to access XAudio2.
But when playing sound, we are noticing that the sound quality is bad. This, for example, is a 3KHz sine curve, captured with Audacity.
I have condensed the audio player to the bare basics, but the audio quality is still bad. Is this a bug in XAudio2, SlimDX, or my code, or is this simply an artifact that occurs when one go from 8KHz to 44.1KHz? The last one seems unreasonable, as we also generate PCM wav files which are played perfectly by Windows Media Player.
The following is the basic implementation, which generates the broken Sine.
public partial class MainWindow : Window
{
private XAudio2 device = new XAudio2();
private WaveFormatExtensible format = new WaveFormatExtensible();
private SourceVoice sourceVoice = null;
private MasteringVoice masteringVoice = null;
private Guid KSDATAFORMAT_SUBTYPE_PCM = new Guid("00000001-0000-0010-8000-00aa00389b71");
private AutoResetEvent BufferReady = new AutoResetEvent(false);
private PlayBufferPool PlayBuffers = new PlayBufferPool();
public MainWindow()
{
InitializeComponent();
Closing += OnClosing;
format.Channels = 1;
format.BitsPerSample = 16;
format.FormatTag = WaveFormatTag.Extensible;
format.BlockAlignment = (short)(format.Channels * (format.BitsPerSample / 8));
format.SamplesPerSecond = 8000;
format.AverageBytesPerSecond = format.SamplesPerSecond * format.BlockAlignment;
format.SubFormat = KSDATAFORMAT_SUBTYPE_PCM;
}
private void OnClosing(object sender, CancelEventArgs cancelEventArgs)
{
sourceVoice.Stop();
sourceVoice.Dispose();
masteringVoice.Dispose();
PlayBuffers.Dispose();
}
private void button_Click(object sender, RoutedEventArgs e)
{
masteringVoice = new MasteringVoice(device);
PlayBuffer buffer = PlayBuffers.NextBuffer();
GenerateSine(buffer.Buffer);
buffer.AudioBuffer.AudioBytes = 640;
sourceVoice = new SourceVoice(device, format, VoiceFlags.None, 8);
sourceVoice.BufferStart += new EventHandler<ContextEventArgs>(sourceVoice_BufferStart);
sourceVoice.BufferEnd += new EventHandler<ContextEventArgs>(sourceVoice_BufferEnd);
sourceVoice.SubmitSourceBuffer(buffer.AudioBuffer);
sourceVoice.Start();
}
private void sourceVoice_BufferEnd(object sender, ContextEventArgs e)
{
BufferReady.Set();
}
private void sourceVoice_BufferStart(object sender, ContextEventArgs e)
{
BufferReady.WaitOne(1000);
PlayBuffer nextBuffer = PlayBuffers.NextBuffer();
nextBuffer.DataStream.Position = 0;
nextBuffer.AudioBuffer.AudioBytes = 640;
GenerateSine(nextBuffer.Buffer);
Result r = sourceVoice.SubmitSourceBuffer(nextBuffer.AudioBuffer);
}
private void GenerateSine(byte[] buffer)
{
double sampleRate = 8000.0;
double amplitude = 0.25 * short.MaxValue;
double frequency = 3000.0;
for (int n = 0; n < buffer.Length / 2; n++)
{
short[] s = { (short)(amplitude * Math.Sin((2 * Math.PI * n * frequency) / sampleRate)) };
Buffer.BlockCopy(s, 0, buffer, n * 2, 2);
}
}
}
public class PlayBuffer : IDisposable
{
#region Private variables
private IntPtr BufferPtr;
private GCHandle BufferHandle;
#endregion
#region Constructors
public PlayBuffer()
{
Index = 0;
Buffer = new byte[640 * 4]; // 640 = 30ms
BufferHandle = GCHandle.Alloc(this.Buffer, GCHandleType.Pinned);
BufferPtr = new IntPtr(BufferHandle.AddrOfPinnedObject().ToInt32());
DataStream = new DataStream(BufferPtr, 640 * 4, true, false);
AudioBuffer = new AudioBuffer();
AudioBuffer.AudioData = DataStream;
}
public PlayBuffer(int index)
: this()
{
Index = index;
}
#endregion
#region Destructor
~PlayBuffer()
{
Dispose();
}
#endregion
#region Properties
protected int Index { get; private set; }
public byte[] Buffer { get; private set; }
public DataStream DataStream { get; private set; }
public AudioBuffer AudioBuffer { get; private set; }
#endregion
#region Public functions
public void Dispose()
{
if (AudioBuffer != null)
{
AudioBuffer.Dispose();
AudioBuffer = null;
}
if (DataStream != null)
{
DataStream.Dispose();
DataStream = null;
}
}
#endregion
}
public class PlayBufferPool : IDisposable
{
#region Private variables
private int _currentIndex = -1;
private PlayBuffer[] _buffers = new PlayBuffer[2];
#endregion
#region Constructors
public PlayBufferPool()
{
for (int i = 0; i < 2; i++)
Buffers[i] = new PlayBuffer(i);
}
#endregion
#region Desctructor
~PlayBufferPool()
{
Dispose();
}
#endregion
#region Properties
protected int CurrentIndex
{
get { return _currentIndex; }
set { _currentIndex = value; }
}
protected PlayBuffer[] Buffers
{
get { return _buffers; }
set { _buffers = value; }
}
#endregion
#region Public functions
public void Dispose()
{
for (int i = 0; i < Buffers.Length; i++)
{
if (Buffers[i] == null)
continue;
Buffers[i].Dispose();
Buffers[i] = null;
}
}
public PlayBuffer NextBuffer()
{
CurrentIndex = (CurrentIndex + 1) % Buffers.Length;
return Buffers[CurrentIndex];
}
#endregion
}
Some extra details:
This is used to replay recorded voice with various compression such as ALAW, µLAW or TrueSpeech. The data is sent in small packets, decoded and sent to this player. This is the reason for why we're using so low sampling rate, and so small buffers.
There are no problems with our data, however, as generating a WAV file with the data results in perfect replay by WMP or VLC.
edit: We have now "solved" this by rewriting the player in NAudio.
I'd still be interested in any input as to what is happening here. Is it our approach in the PlayBuffers, or is it simply a bug/limitation in DirectX, or the wrappers? I tried using SharpDX instead of SlimDX, but that did not change the result anything.

It looks as if the upsampling is done without a proper anti-aliasing (reconstruction) filter. The cutoff frequency is far too high (above the original Nyquist frequency) and therefore a lot of the aliases are being preserved, resulting in output resembling piecewise-linear interpolation between the samples taken at 8000 Hz.
Although all your different options are doing an upconversion from 8kHz to 44.1kHz, the way in which they do that is important, and the fact that one library does it well is no proof that the upconversion is not the source of error in the other.

It's been a while since I worked with sound and frequencies, but here is what I remember: You have a sample rate of 8000Hz and want a sine frequency of 3000Hz. So for 1 second you have 8000 samples and in that second you want your sine to oscillate 3000 times. That is below the Nyquist-frequency (half your sample rate) but barely (see Nyquist–Shannon sampling theorem). So I would not expect a good quality here.
In fact: step through the GenerateSine-method and you'll see that s[0] will contain the values 0, 5792, -8191, 5792, 0, -5792, 8191, -5792, 0, 5792...
None the less this doesn't explain the odd sine you recorded back and I'm not sure how much samples the human ear need to hear a "good" sine wave.