We Keep Coding

Using Audio Graph, learn environment noise and filter in a UWP App

This is an extended question from here Using UWP monitor live audio and detect gun-fire/clap sound
Thanks to Dernis I finally got the code working to monitor live audio and trigger events when decibel count is above a certain range.
This works perfectly when we run it in office/closed/silent area.
But when I take the app to open road, there will be traffic sound, wind sound, people talk sound and other noises and BLOW events are not identified correctly.
I would like to implement something like Lean Environment button. Before app starts monitoring, the user clicks on "Lean Environment" that recognize the sensitivity levels and set filtering to my live audio and then I start monitoring blows.
If it doesn't add too much load, I would like to record the audio to a file.
Any help on where to start would be appreciated.
OnNavigatedTo
protected override async void OnNavigatedTo(NavigationEventArgs e)
{
//other logic
await CreateInputDeviceNodeAsync(_deviceId);
}
CreateInputDeviceNodeAsync
public async Task<bool> CreateInputDeviceNodeAsync(string deviceId)
{
Console.WriteLine("Creating AudioGraphs");
// Create an AudioGraph with default settings
AudioGraphSettings graphSettings = new AudioGraphSettings(AudioRenderCategory.Media)
{
EncodingProperties = new AudioEncodingProperties
{
Subtype = "Float",
SampleRate = 48000,
ChannelCount = 2,
BitsPerSample = 32,
Bitrate = 3072000
}
};
CreateAudioGraphResult audioGraphResult = await AudioGraph.CreateAsync(graphSettings);
if (audioGraphResult.Status != AudioGraphCreationStatus.Success)
{
_rootPage.NotifyUser("Cannot create graph", NotifyType.ErrorMessage);
return false;
}
_audioGraph = audioGraphResult.Graph;
AudioGraphSettings audioGraphSettings =
new AudioGraphSettings(AudioRenderCategory.GameChat)
{
EncodingProperties = AudioEncodingProperties.CreatePcm(48000, 2, 32),
DesiredSamplesPerQuantum = 990,
QuantumSizeSelectionMode = QuantumSizeSelectionMode.ClosestToDesired
};
_frameOutputNode = _audioGraph.CreateFrameOutputNode(_audioGraph.EncodingProperties);
_quantum = 0;
_audioGraph.QuantumStarted += Graph_QuantumStarted;
LoudNoise += BlowDetected;
DeviceInformation selectedDevice = null;
if (!string.IsNullOrWhiteSpace(_deviceId))
selectedDevice = await DeviceInformation.CreateFromIdAsync(_deviceId);
if (selectedDevice == null)
{
string device = Windows.Media.Devices.MediaDevice.GetDefaultAudioCaptureId(
Windows.Media.Devices.AudioDeviceRole.Default);
if (!string.IsNullOrWhiteSpace(device))
selectedDevice = await DeviceInformation.CreateFromIdAsync(device);
else
{
_rootPage.NotifyUser($"Could not select Audio Device {device}", NotifyType.ErrorMessage);
return false;
}
}
CreateAudioDeviceInputNodeResult result =
await _audioGraph.CreateDeviceInputNodeAsync(MediaCategory.Media, audioGraphSettings.EncodingProperties,
selectedDevice);
if (result.Status != AudioDeviceNodeCreationStatus.Success)
{
_rootPage.NotifyUser("Cannot create device output node", NotifyType.ErrorMessage);
return false;
}
_selectedMicrophone = selectedDevice.Name;
_deviceInputNode = result.DeviceInputNode;
_deviceInputNode.AddOutgoingConnection(_frameOutputNode);
_frameOutputNode.Start();
_audioGraph.Start();
return true;
}
Graph_QuantumStarted
private void Graph_QuantumStarted(AudioGraph sender, object args)
{
if (++_quantum % 2 != 0) return;
AudioFrame frame = _frameOutputNode.GetFrame();
float[] dataInFloats;
using (AudioBuffer buffer = frame.LockBuffer(AudioBufferAccessMode.Write))
using (IMemoryBufferReference reference = buffer.CreateReference())
unsafe
{
// Get the buffer from the AudioFrame
// ReSharper disable once SuspiciousTypeConversion.Global
((IMemoryBufferByteAccess) reference).GetBuffer(out byte* dataInBytes,
out var capacityInBytes);
var dataInFloat = (float*) dataInBytes;
dataInFloats = new float[capacityInBytes / sizeof(float)];
for (var i = 0; i < capacityInBytes / sizeof(float); i++)
{
dataInFloats[i] = dataInFloat[i];
}
}
double decibels = dataInFloats.Aggregate<float, double>(0f, (current, sample) => current + Math.Abs(sample));
decibels = 20 * Math.Log10(decibels / dataInFloats.Length);
_decibelList.Add(decibels);
if (double.IsInfinity(decibels) || decibels < _threshold) return;//-45
if (_watch != null && _watch.Elapsed <= TimeSpan.FromSeconds(1)) return;
LoudNoise?.Invoke(this, decibels);
_watch = Stopwatch.StartNew();
}

This is just statistics. You'll want to collect probably at least 50 frames (1 second) of data before actually having it function (maybe let the user decide by holding and releasing a button). Then you'll probably want to determine where the decibel level is usually around. I can think of 3 ways to do that.
private void Graph_QuantumStarted(AudioGraph sender, object args)
{
...
double decibels = dataInFloats.Aggregate<float, double>(0f, (current, sample) => current + Math.Abs(sample)); // I dislike the fact that the decibels variable is initially inaccurate, but it's your codebase.
decibels = 20 * Math.Log10(decibels / dataInFloats.Length);
if (scanning) // class variable (bool), you can set it from the UI thread like this
{
_decibelList.Add(decibels); // I assume you made this a class variable
}
else if (decibels == Double.NaN)
{
// Code by case below
}
else if (decibels > _sensitivity) //_sensitivity is a class variable(double), initialized to Double.NaN
{
LoudNoise?.Invoke(this, true); // Calling events is a wee bit expensive, you probably want to handle the sensitivity before Invoking it, I'm also going to do it like that to make this demo simpler
}
}
If you can control make sure there's no spike loud enough you want it to go off you can just take the max value of all those frames and say if it's over the sensitivity is maxDecibels + Math.Abs(maxDecibels* 0.2) (the decibels could be negative, hence Abs).
double maxDecibels = _decibelList.OrderByDescending(x => x)[0];
_sensitivity = maxDecibels + Math.Abs(maxDecibels* 0.2);
If you can't control when there's a spike, then you could collect those frames, sort, and have it take item [24] (of your 100 item list) and say that's the sensitivity.
sensitivity = _decibelList.OrderByDescending(x => x)[24]; // If you do a variable time you can just take Count/4 - 1 as the index
(I think it's the best but I really don't know statistics) Walk the list of frame's decibels and track the average difference in value and the what index changed it most. Afterwards, find the max value from after that index and say 75% of the change to there is the sensitivty. (Don't use a LinkedList on this)
int greatestChange, changeIndex = 0;
double p = Double.NaN; // Previous
for (int i = 0; i < _decibelList.Count(); i++)
{
if (p != Double.Nan)
{
change = Math.Abs(_decibelList[i] - p);
if (Math.Abs(change > greatestChange)
{
greatestChange = change;
changeIndex = i;
}
}
p = _decibelList[i];
}
int i = changeIndex;
p = Double.NaN; // reused
double c= Double.NaN; // Current
do
{
p = c != Double.NaN ? c : _decibelList[i];
c = _decibelList[++i];
} while (c < p);
_sensitivity = ((3 * c) + _decibelList[changeIndex]) / 4;
Note: You can (kind of) remove the need to sort by having a LinkedList and inserting in the appropiate place

One longer call is causing all parallel loops to pause or blocked

I am calling a VB 6.0 dll in Parallel.ForEach and expecting all calls to be started simultaneously or at least 2 of them based on my PC's cores or threads availability in thread pool
VB6 dll
Public Function DoJunk(ByVal counter As Long, ByVal data As String) As Integer
Dim i As Long
Dim j As Long
Dim s As String
Dim fno As Integer
fno = FreeFile
Open "E:\JunkVB6Dll\" & data & ".txt" For Output Access Write As #fno
Print #fno, "Starting loop with counter = " & counter
For i = 0 To counter
Print #fno, "counting " & i
Next
Close #fno
DoJunk = 1
End Function
counter is being passed from the caller to control execution time of the call and file is being written to make it an IO based process.
C# caller
private void ReportProgress(int value)
{
progressBar.Value = value;
//progressBar.Value++;
}
private void button1_Click(object sender, EventArgs e)
{
progressBar.Value = 0;
counter = 0;
Stopwatch watch = new Stopwatch();
watch.Start();
//var range = Enumerable.Range(0, 100);
var range = Enumerable.Range(0, 20);
bool finished = false;
Task.Factory.StartNew(() =>
{
Parallel.ForEach(range, i =>
{
#region COM CALL
JunkProject.JunkClass junk = new JunkProject.JunkClass();
try
{
Random rnd = new Random();
int dice = rnd.Next(10, 40);
int val = 0;
if (i == 2)
val = junk.DoJunk(9000000, i.ToString());
else
val = junk.DoJunk(dice * 10000, i.ToString());
System.Diagnostics.Debug.Print(junk.GetHashCode().ToString());
if (val == 1)
{
Interlocked.Increment(ref counter);
progressBar.Invoke((Action)delegate { ReportProgress(counter); });
}
junk = null;
}
catch (Exception excep)
{
i = i;
}
finally { junk = null; }
#endregion
});
}).ContinueWith(t =>
{
watch.Stop();
MessageBox.Show(watch.ElapsedMilliseconds.ToString());
});
}
This line is making a specific call longer than the others.
val = junk.DoJunk(9000000, i.ToString());
Here this second process is causing all calls inside the Parallel.ForEach to stop i.e. no other file is created unless this 2nd call gets completed.
Is it an expected behavior or i am doing something wrong?

As #John Wu suggested that you can create AppDomain to allow COM to run on different App Domain, I believe you could run your parallel like this.
Parallel.ForEach(range, i =>
{
AppDomain otherDomain = AppDomain.CreateDomain(i.ToString());
otherDomain.DoCallBack(delegate
{
//Your COM call
});
});
EDIT
Right.. I am not sure how can you set serializable on VB6.0 class. You can try the other way (Marshaling objects by reference). Noted: I haven't actually tested this, but I would like to know if that will work.
Parallel.ForEach(range, i =>
{
AppDomain otherDomain = AppDomain.CreateDomain(i.ToString());
var comCall = (ComCall) otherDomain.CreateInstanceFromAndUnwrap(Assembly.GetExecutingAssembly().Location, typeof(ComCall).ToString());
comCall.Run();
AppDomain.Unload(otherDomain);
});
and the class
public class ComCall : MarshalByRefObject
{
public void Run()
{
//Your COM Call
}
}
Here is also additional reference regarding the topic.
https://www.codeproject.com/Articles/14791/NET-Remoting-with-an-easy-example

Accessing RichTextBox in Multi-threaded application causes OutOfMemoryException

I am probably just doing it very wrong. I am currently working with MSMQ and Webservices. I wanted to learn how MSMQ worked so I found a school example of a Loan Broker.
To make a long story short, I need to be able to stress test my system, so I want to be able to make, say, 100 messages and send them through my messaging system. I want to do that from a Windows Form application, but here lies the problem. I have a form that looks like this:
On the left you see a custom control and on the right, my "console" window that tells me what's going on. When I press the Send button, it should use the data given in the fields above it, to send messages. But when I press the Send Button, the program freezes for a while and then hits the OutOfMemoryException. This is the Send method:
private void Send(List<SimpleRequest.LoanRequest> list)
{
int quantity = int.Parse(numericQuantity.Value.ToString());
int delay = int.Parse(numericDelay.Value.ToString());
if (list.Count == 1)
{
for (int threadnumber = 0; threadnumber < quantity; threadnumber++)
{
Task.Factory.StartNew(() => RequestLoanQuote(threadnumber, list[0]));
if (delay > 0)
{
Thread.Sleep(delay);
}
}
}
else
{
for (int threadnumber = 0; threadnumber < quantity; threadnumber++)
{
Task.Factory.StartNew(() => RequestLoanQuote(threadnumber, list[threadnumber]));
if (delay > 0)
{
Thread.Sleep(delay);
}
}
}
}
Here is the RequestLoanQuote method that the Send method is calling:
private void RequestLoanQuote(object state, SimpleRequest.LoanRequest loanRequest)
{
try
{
if (console.InvokeRequired)
{
SetText("Sending: " + loanRequest.SSN + "\n");
}
StringBuilder sb = new StringBuilder();
var threadnumber = (int)state;
using (var client = new LoanBrokerWS.LoanBrokerWSClient())
{
Utility_Tool.LoanBrokerWS.LoanQuote response = client.GetLoanQuote(loanRequest.SSN, loanRequest.LoanAmount, loanRequest.LoanDuration);
sb.Append(response.SSNk__BackingField + " returned: ");
sb.Append(response.interestRatek__BackingField + " | ");
sb.Append(response.BankNamek__BackingField + "\n");
SetText(sb.ToString());
}
}
catch (Exception e)
{
SetText(e.Message + "\n");
}
}
And finally, the SetText method:
private void SetText(String msg)
{
if (this.console.InvokeRequired)
{
SetTextCallback d = new SetTextCallback(SetText);
this.Invoke(d, new object[] { msg });
}
else
{
this.console.Text += msg;
}
}
So the Send method calls the RequestLoanQuote method which calls the SetText method. I cannot figure out where I went wrong but it's probably a deadlock.

Try using BeginInvoke and AppendText, like so:
public static void SetText(this RichTextBox textBox, string msg)
{
Action append = () => textBox.AppendText(msg);
if (textBox.InvokeRequired)
textBox.BeginInvoke(append);
else
append();
}

Threaded Code Is Causing the UI To Hang

I wrote a simple class to generate arrays as I needed some hard-coded random arrays for my own debugging purposes, however despite calling the timely operations in a separate worker thread my UI still hangs! Here's the code...
private static Random randGenerator = new Random();
private void generateButton_Click(object sender, EventArgs e)
{
string dataType = "System.";
if (typeComboBox.Text != "Byte")
dataType += signedCheckBox.Checked ? "" : "U";
else if (typeComboBox.Text == "Byte")
dataType += signedCheckBox.Checked ? "S" : "";
dataType += typeComboBox.Text;
generateButton.Enabled = false;
new Thread(() =>
{
Process(Type.GetType(dataType), (int)sizeNumericUpDown.Value, hexCheckBox.Checked);
}).Start();
}
private void Process(Type type, int size, bool hex)
{
StringBuilder sBuilder = new StringBuilder();
sBuilder.Append(string.Format(#"{0}[] values = new {0}[{1}] {{", type.Name, size));
for (int i = 0; i < size; i++)
{
int random = randGenerator.Next(0, GetIntegralMaxValue(type));
sBuilder.Append((hex ? "0x" + random.ToString("x2") : random.ToString()) + (i < size - 1 ? ", " : " };"));
}
outputTextBox.BeginInvoke((MethodInvoker)delegate
{
outputTextBox.Text = sBuilder.ToString();
});
}
private int GetIntegralMaxValue(Type type)
{
var field = type.GetField("MaxValue").GetValue(null);
return Convert.ToInt32(field);
}
I thought that maybe the issue lied with trying to access objects created on the main thread so instead I passed them to the method. I also tried declaring my randGenerator object within the class but still no luck. Could anybody identify the issue?

The TextBox is not designed to hold a large set of data - it becomes very slow when amount of data increases. I would say that populating of the StringBuilder takes about 5% of time while assigning this data to TextBox (which executes in UI thread) takes remaining 95% of time. You can easily check this. Just run this code in debug mode and click "Pause" button during hanging. It should break on "outputTextBox.Text = sBuilder.ToString();" line.

C# Async Ping: How to avoid an out of memory exception?

Question: I want to search the subnet for all computers in it.
So I send a ping to all IP addresses in the subnet.
The problem is it works fine if I only scan 192.168.0.".
But if I scan 192.168..*", then I get an "Out of memory" exception.
Why ? Do I have to limit the threads, or is the problem the memory consumed by new ping which doesn't get destructed once finished, or do I need to call gc.collect() ?
static void Main(string[] args)
{
string strFromIP = "192.168.0.1";
string strToIP = "192.168.255.255";
Oyster.Math.IntX omiFromIP = 0;
Oyster.Math.IntX omiToIP = 0;
IsValidIP(strFromIP, ref omiFromIP);
IsValidIP(strToIP, ref omiToIP);
for (Oyster.Math.IntX omiThisIP = omiFromIP; omiThisIP <= omiToIP; ++omiThisIP)
{
Console.WriteLine(IPn2IPv4(omiThisIP));
System.Net.IPAddress sniIPaddress = System.Net.IPAddress.Parse(IPn2IPv4(omiThisIP));
SendPingAsync(sniIPaddress);
}
Console.WriteLine(" --- Press any key to continue --- ");
Console.ReadKey();
} // Main
// http://pberblog.com/post/2009/07/21/Multithreaded-ping-sweeping-in-VBnet.aspx
// http://www.cyberciti.biz/faq/how-can-ipv6-address-used-with-webbrowser/#comments
// http://www.kloth.net/services/iplocate.php
// http://bytes.com/topic/php/answers/829679-convert-ipv4-ipv6
// http://stackoverflow.com/questions/1434342/ping-class-sendasync-help
public static void SendPingAsync(System.Net.IPAddress sniIPaddress)
{
int iTimeout = 5000;
System.Net.NetworkInformation.Ping myPing = new System.Net.NetworkInformation.Ping();
System.Net.NetworkInformation.PingOptions parmPing = new System.Net.NetworkInformation.PingOptions();
System.Threading.AutoResetEvent waiter = new System.Threading.AutoResetEvent(false);
myPing.PingCompleted += new System.Net.NetworkInformation.PingCompletedEventHandler(AsyncPingCompleted);
string data = "ABC";
byte[] dataBuffer = Encoding.ASCII.GetBytes(data);
parmPing.DontFragment = true;
parmPing.Ttl = 32;
myPing.SendAsync(sniIPaddress, iTimeout, dataBuffer, parmPing, waiter);
//waiter.WaitOne();
}
private static void AsyncPingCompleted(Object sender, System.Net.NetworkInformation.PingCompletedEventArgs e)
{
System.Net.NetworkInformation.PingReply reply = e.Reply;
((System.Threading.AutoResetEvent)e.UserState).Set();
if (reply.Status == System.Net.NetworkInformation.IPStatus.Success)
{
Console.WriteLine("Address: {0}", reply.Address.ToString());
Console.WriteLine("Roundtrip time: {0}", reply.RoundtripTime);
}
}

According to this thread, System.Net.NetworkInformation.Ping seems to allocate one thread per async request, and "ping-sweeping a class-B network creates 100's of threads and eventually results in an out-of-memory error."
The workaround that person used was to write their own implementation using raw sockets. You don't have to do that in F#, of course, but there are a number of advantages in doing so.

First: Only start like 1000 pings the first time (in the loop in Main)
Second: Move the following parameters to Program class (member variables)
Oyster.Math.IntX omiFromIP = 0;
Oyster.Math.IntX omiToIP = 0;
Oyster.Math.IntX omiCurrentIp = 0;
object syncLock = new object();
Third: In AsyncPingCompleted do something like this in the bottom:
public void AsyncPingCompleted (bla bla bla)
{
//[..other code..]
lock (syncLock)
{
if (omiToIP < omiCurrentIp)
{
++omiCurrentIp;
System.Net.IPAddress sniIPaddress = System.Net.IPAddress.Parse(IPn2IPv4(omiCurrentIp));
SendPingAsync(sniIPaddress);
}
}
}
Update with complete code example
public class Example
{
// Number of pings that can be pending at the same time
private const int InitalRequests = 10000;
// variables from your Main method
private Oyster.Math.IntX _omiFromIP = 0;
private Oyster.Math.IntX _omiToIP = 0;
private Oyster.Math.IntX _omiCurrentIp = 0;
// synchronoize so that two threads
// cannot ping the same IP.
private object _syncLock = new object();
static void Main(string[] args)
{
string strFromIP = "192.168.0.1";
string strToIP = "192.168.255.255";
IsValidIP(strFromIP, ref _omiFromIP);
IsValidIP(strToIP, ref _omiToIP);
for (_omiCurrentIp = _omiFromIP; _omiCurrentIp <= _omiFromIP + InitalRequests; ++_omiCurrentIp)
{
Console.WriteLine(IPn2IPv4(_omiCurrentIp));
System.Net.IPAddress sniIPaddress = System.Net.IPAddress.Parse(IPn2IPv4(_omiCurrentIp));
SendPingAsync(sniIPaddress);
}
Console.WriteLine(" --- Press any key to continue --- ");
Console.ReadKey();
} // Main
// http://pberblog.com/post/2009/07/21/Multithreaded-ping-sweeping-in-VBnet.aspx
// http://www.cyberciti.biz/faq/how-can-ipv6-address-used-with-webbrowser/#comments
// http://www.kloth.net/services/iplocate.php
// http://bytes.com/topic/php/answers/829679-convert-ipv4-ipv6
// http://stackoverflow.com/questions/1434342/ping-class-sendasync-help
public void SendPingAsync(System.Net.IPAddress sniIPaddress)
{
int iTimeout = 5000;
System.Net.NetworkInformation.Ping myPing = new System.Net.NetworkInformation.Ping();
System.Net.NetworkInformation.PingOptions parmPing = new System.Net.NetworkInformation.PingOptions();
System.Threading.AutoResetEvent waiter = new System.Threading.AutoResetEvent(false);
myPing.PingCompleted += new System.Net.NetworkInformation.PingCompletedEventHandler(AsyncPingCompleted);
string data = "ABC";
byte[] dataBuffer = Encoding.ASCII.GetBytes(data);
parmPing.DontFragment = true;
parmPing.Ttl = 32;
myPing.SendAsync(sniIPaddress, iTimeout, dataBuffer, parmPing, waiter);
//waiter.WaitOne();
}
private void AsyncPingCompleted(Object sender, System.Net.NetworkInformation.PingCompletedEventArgs e)
{
System.Net.NetworkInformation.PingReply reply = e.Reply;
((System.Threading.AutoResetEvent)e.UserState).Set();
if (reply.Status == System.Net.NetworkInformation.IPStatus.Success)
{
Console.WriteLine("Address: {0}", reply.Address.ToString());
Console.WriteLine("Roundtrip time: {0}", reply.RoundtripTime);
}
// Keep starting those async pings until all ips have been invoked.
lock (_syncLock)
{
if (_omiToIP < _omiCurrentIp)
{
++_omiCurrentIp;
System.Net.IPAddress sniIPaddress = System.Net.IPAddress.Parse(IPn2IPv4(_omiCurrentIp));
SendPingAsync(sniIPaddress);
}
}
}
}

I guess the problem is that you are spawning roughly 63K ping requests near-simultaneously. Without further memory profiling it is hard to say which parts consume the memory. You are working with network resources, which probably are limited. Throttling the number of active pings will ease the use of local resources, and also network traffic.
Again I would look into the Task Parallel Library, the Parallel.For construct combined with the Task<T> should make it easy for you.
Note: for .Net 3.5 users, there is hope.

I did something similar to this. The way I solved the problem on my project was to cast the ping instance to IDisposable:
(myPing as IDisposable).Dispose()
So get a list of say 254 ping instances running asynchronously (X.X.X.1/254) and keep track of when all of them have reported in. When they have, iterate through your list of ping instances, run the above code on each instance, and then dump the list.
Works like a charm.

pseudo-code
do
if pings_running > 100 then
sleep 100ms.
else
start ping
endif
loop while morepings

Finally... No ping requried at all...
http://www.codeproject.com/KB/cs/c__ip_scanner.aspx
All I needed to do is to make it thread-safe for debugging.
Changing Add to:
void Add( string m )
{
Invoke(new MethodInvoker(
delegate
{
add.Items.Add(m);
}));
//add.Items.Add( m );
}
Private Sub Add(m As String)
Invoke(New MethodInvoker(Function() Do
add.Items.Add(m)
End Function))
'add.Items.Add(m);'
End Sub