We Keep Coding

Algorithms and techniques for string search across multiple GiB of text files

I have to create a utility that searches through 40 to 60 GiB of text files as quick as possible.
Each file has around 50 MB of data that consists of log lines (about 630.000 lines per file).
A NOSQL document database is unfortunately no option...
As of now I am using a Aho-Corsaick algorithm for the search which I stole from Tomas Petricek off of his blog. It works very well.
I process the files in Tasks. Each file is loaded into memory by simply calling File.ReadAllLines(path). The lines are then fed into the Aho-Corsaick one by one, thus each file causes around 600.000 calls to the algorithm (I need the line number in my results).
This takes a lot of time and requires a lot of memory and CPU.
I have very little expertise in this field as I usually work in image processing.
Can you guys recommend algorithms and approaches which could speed up the processing?
Below is more detailed view to the Task creation and file loading which is pretty standard. For more information on the Aho-Corsaick, please visit the linked blog page above.
private KeyValuePair<string, StringSearchResult[]> FindInternal(
IStringSearchAlgorithm algo,
string file)
{
List<StringSearchResult> result = new List<StringSearchResult>();
string[] lines = File.ReadAllLines(file);
for (int i = 0; i < lines.Length; i++)
{
var results = algo.FindAll(lines[i]);
for (int j = 0; j < results.Length; j++)
{
results[j].Row = i;
}
}
foreach (string line in lines)
{
result.AddRange(algo.FindAll(line));
}
return new KeyValuePair<string, StringSearchResult[]>(
file, result.ToArray());
}
public Dictionary<string, StringSearchResult[]> Find(
params string[] search)
{
IStringSearchAlgorithm algo = new StringSearch();
algo.Keywords = search;
Task<KeyValuePair<string, StringSearchResult[]>>[] findTasks
= new Task<KeyValuePair<string, StringSearchResult[]>>[_files.Count];
Parallel.For(0, _files.Count, i => {
findTasks[i] = Task.Factory.StartNew(
() => FindInternal(algo, _files[i])
);
});
Task.WaitAll(findTasks);
return findTasks.Select(t => t.Result)
.ToDictionary(x => x.Key, x => x.Value);
}

EDIT
See section Initial Answer for the original Answer.
I further optimized my code by doing the following:
Added paging to prevent memory overflow / crash due to large amount of result data.
I offload the search results into local files as soon as they exceed a certain buffer size (64kb in my case).
Offloading the results required me to convert my SearchData struct to binary and back.
Splicing the array of files which are processed and running them in Tasks greatly increased performance (from 35 sec to 9 sec when processing about 25 GiB of search data)
Splicing / scaling the file array
The code below gives a scaled/normalized value for T_min and T_max.
This value can then be used to determine the size of each array holding n-amount of file paths.
private int ScalePartition(int T_min, int T_max)
{
// Scale m to range.
int m = T_max / 2;
int t_min = 4;
int t_max = Math.Max(T_max / 16, T_min);
m = ((T_min - m) / (T_max - T_min)) * (t_max - t_min) + t_max;
return m;
}
This code shows the implementation of the scaling and splicing.
// Get size of file array portion.
int scale = ScalePartition(1, _files.Count);
// Iterator.
int n = 0;
// List containing tasks.
List<Task<SearchData[]>> searchTasks = new List<Task<SearchData[]>>();
// Loop through files.
while (n < _files.Count) {
// Local instance of n.
// You will get an AggregateException if you use n
// as n changes during runtime.
int num = n;
// The amount of items to take.
// This needs to be calculated as there might be an
// odd number of elements in the file array.
int cnt = n + scale > _files.Count ? _files.Count - n : scale;
// Run the Find(int, int, Regex[]) method and add as task.
searchTasks.Add(Task.Run(() => Find(num, cnt, regexes)));
// Increment iterator by the amount of files stored in scale.
n += scale;
}
Initial Answer
I had the best results so far after switching to MemoryMappedFile and moving from the Aho-Corsaick back to Regex (a demand has been made that pattern matching is a must have).
There are still parts that can be optimized or changed and I'm sure this is not the fastest or best solution but for it's alright.
Here is the code which returns the results in 30 seconds for 25 GiB worth of data:
// GNU coreutil wc defined buffer size.
// Had best performance with this buffer size.
//
// Definition in wc.c:
// -------------------
// /* Size of atomic reads. */
// #define BUFFER_SIZE (16 * 1024)
//
private const int BUFFER_SIZE = 16 * 1024;
private KeyValuePair<string, SearchData[]> FindInternal(Regex[] rgx, string file)
{
// Buffer for data segmentation.
byte[] buffer = new byte[BUFFER_SIZE];
// Get size of file.
FileInfo fInfo = new FileInfo(file);
long fSize = fInfo.Length;
fInfo = null;
// List of results.
List<SearchData> results = new List<SearchData>();
// Create MemoryMappedFile.
string name = "mmf_" + Path.GetFileNameWithoutExtension(file);
using (var mmf = MemoryMappedFile.CreateFromFile(
file, FileMode.Open, name))
{
// Create read-only in-memory access to file data.
using (var accessor = mmf.CreateViewStream(
0, fSize,
MemoryMappedFileAccess.Read))
{
// Store current position.
int pos = (int)accessor.Position;
// Check if file size is less then the
// default buffer size.
int cnt = (int)(fSize - BUFFER_SIZE > 0
? BUFFER_SIZE
: fSize - BUFFER_SIZE);
// Iterate through file until end of file is reached.
while (accessor.Position < fSize)
{
// Write data to buffer.
accessor.Read(buffer, 0, cnt);
// Update position.
pos = (int)accessor.Position;
// Update next buffer size.
cnt = (int)(fSize - pos >= BUFFER_SIZE
? BUFFER_SIZE
: fSize - pos);
// Convert buffer data to string for Regex search.
string s = Encoding.UTF8.GetString(buffer);
// Run regex against extracted data.
foreach (Regex r in rgx) {
// Get matches.
MatchCollection matches = r.Matches(s);
// Create SearchData struct to reduce memory
// impact and only keep relevant data.
foreach (Match m in matches) {
SearchData sd = new SearchData();
// The actual matched string.
sd.Match = m.Value;
// The index in the file.
sd.Index = m.Index + pos;
// Index to find beginning of line.
int nFirst = m.Index;
// Index to find end of line.
int nLast = m.Index;
// Go back in line until the end of the
// preceeding line has been found.
while (s[nFirst] != '\n' && nFirst > 0) {
nFirst--;
}
// Append length of \r\n (new line).
// Change this to 1 if you work on Unix system.
nFirst+=2;
// Go forth in line until the end of the
// current line has been found.
while (s[nLast] != '\n' && nLast < s.Length-1) {
nLast++;
}
// Remove length of \r\n (new line).
// Change this to 1 if you work on Unix system.
nLast-=2;
// Store whole line in SearchData struct.
sd.Line = s.Substring(nFirst, nLast - nFirst);
// Add result.
results.Add(sd);
}
}
}
}
}
return new KeyValuePair<string, SearchData[]>(file, results.ToArray());
}
public List<KeyValuePair<string, SearchData[]>> Find(params string[] search)
{
var results = new List<KeyValuePair<string, SearchData[]>>();
// Prepare regex objects.
Regex[] regexes = new Regex[search.Length];
for (int i=0; i<regexes.Length; i++) {
regexes[i] = new Regex(search[i], RegexOptions.Compiled);
}
// Get all search results.
// Creating the Regex once and passing it
// to the sub-routine is best as the regex
// engine adds a lot of overhead.
foreach (var file in _files) {
var data = FindInternal(regexes, file);
results.Add(data);
}
return results;
}
I had a stupid idea yesterday were I though that it might work out converting the file data to a bitmap and looking for the input within pixels as pixel checking is quite fast.
Just for the giggles... here is the non-optimized test code for that stupid idea:
public struct SearchData
{
public string Line;
public string Search;
public int Row;
public SearchData(string l, string s, int r) {
Line = l;
Search = s;
Row = r;
}
}
internal static class FileToImage
{
public static unsafe SearchData[] FindText(string search, Bitmap bmp)
{
byte[] buffer = Encoding.ASCII.GetBytes(search);
BitmapData data = bmp.LockBits(
new Rectangle(0, 0, bmp.Width, bmp.Height),
ImageLockMode.ReadOnly, bmp.PixelFormat);
List<SearchData> results = new List<SearchData>();
int bpp = Bitmap.GetPixelFormatSize(bmp.PixelFormat) / 8;
byte* ptFirst = (byte*)data.Scan0;
byte firstHit = buffer[0];
bool isFound = false;
for (int y=0; y<data.Height; y++) {
byte* ptStride = ptFirst + (y * data.Stride);
for (int x=0; x<data.Stride; x++) {
if (firstHit == ptStride[x]) {
byte[] temp = new byte[buffer.Length];
if (buffer.Length < data.Stride-x) {
int ret = 0;
for (int n=0, xx=x; n<buffer.Length; n++, xx++) {
if (ptStride[xx] != buffer[n]) {
break;
}
ret++;
}
if (ret == buffer.Length) {
int lineLength = 0;
for (int n = 0; n<data.Stride; n+=bpp) {
if (ptStride[n+2] == 255 &&
ptStride[n+1] == 255 &&
ptStride[n+0] == 255)
{
lineLength=n;
}
}
SearchData sd = new SearchData();
byte[] lineBytes = new byte[lineLength];
Marshal.Copy((IntPtr)ptStride, lineBytes, 0, lineLength);
sd.Search = search;
sd.Line = Encoding.ASCII.GetString(lineBytes);
sd.Row = y;
results.Add(sd);
}
}
}
}
}
return results.ToArray();
bmp.UnlockBits(data);
return null;
}
private static unsafe Bitmap GetBitmapInternal(string[] lines, int startIndex, Bitmap bmp)
{
int bpp = Bitmap.GetPixelFormatSize(bmp.PixelFormat) / 8;
BitmapData data = bmp.LockBits(
new Rectangle(0, 0, bmp.Width, bmp.Height),
ImageLockMode.ReadWrite,
bmp.PixelFormat);
int index = startIndex;
byte* ptFirst = (byte*)data.Scan0;
int maxHeight = bmp.Height;
if (lines.Length - startIndex < maxHeight) {
maxHeight = lines.Length - startIndex -1;
}
for (int y = 0; y < maxHeight; y++) {
byte* ptStride = ptFirst + (y * data.Stride);
index++;
int max = lines[index].Length;
max += (max % bpp);
lines[index] += new string('\0', max % bpp);
max = lines[index].Length;
for (int x=0; x+2<max; x+=bpp) {
ptStride[x+0] = (byte)lines[index][x+0];
ptStride[x+1] = (byte)lines[index][x+1];
ptStride[x+2] = (byte)lines[index][x+2];
}
ptStride[max+2] = 255;
ptStride[max+1] = 255;
ptStride[max+0] = 255;
for (int x = max + bpp; x < data.Stride; x += bpp) {
ptStride[x+2] = 0;
ptStride[x+1] = 0;
ptStride[x+0] = 0;
}
}
bmp.UnlockBits(data);
return bmp;
}
public static unsafe Bitmap[] GetBitmap(string filePath)
{
int bpp = Bitmap.GetPixelFormatSize(PixelFormat.Format24bppRgb) / 8;
var lines = System.IO.File.ReadAllLines(filePath);
int y = 0x800; //lines.Length / 0x800;
int x = lines.Max(l => l.Length) / bpp;
int cnt = (int)Math.Ceiling((float)lines.Length / (float)y);
Bitmap[] results = new Bitmap[cnt];
for (int i = 0; i < results.Length; i++) {
results[i] = new Bitmap(x, y, PixelFormat.Format24bppRgb);
results[i] = GetBitmapInternal(lines, i * 0x800, results[i]);
}
return results;
}
}

You can split the file into partitions and regex search each partition in parallel then join the results. There are some sharp edges in the details like handling values that span two partitions. Gigantor is a c# library I have created that does this very thing. Feel free to try it or have a look at the source code.

System Overflow Exception in System.Windows.Forms.DataVisualization.dll

I've got problem that makes me crazy - I think that's so easy that I can't even think about what is causing my problems.
I'm sending data (fft of generated acustic wave - frequencies and magnitude) generated by my uC to PC by serialport.
Everything seems to work fine till I try to chart my data with MS Chart Controls - without charting there are no errors when recieving data.
I'm always able to chart data once or twice, after that I'm getting error like this:
An unhandled exception of type 'System.OverflowException' occurred in
System.Windows.Forms.DataVisualization.dll
Additional information: Value was either too large or too small for a
Decimal.
I checked if vaules of data are over limits of "float32" aka Single - I even applied statement where I make values lower when they are too high - it's useless. The part of my with recievedata event and charting is like this:
private byte[] bArray = new byte[20 * numberOfSamples + 4]; // frequencies (512 samples) and magnitudes (512 values) - each is a single precision float so 4 bytes + 4 - "!!!!" at the beginning
private float[] fArray = new float[2 * numberOfSamples + 1]; // float data array
private void SerialP_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
SerialPort sp = (SerialPort)sender;
int previouse = counterOfRecBytes;
counterOfRecBytes += sp.BytesToRead;
//if (counterOfRecBytes >= 4104)
sp.Read(bArray, previouse, (counterOfRecBytes - previouse));
if (counterOfRecBytes >= 8 * numberOfSamples + 4)
{
for (uint i = 0; i < 2 * numberOfSamples + 1; i++)
{
fArray[i] = Math.Abs(ByteToFloat(bArray, i));
if (fArray[i] < 0) fArray[i] = 0;
if (fArray[i] > 3.4 * Math.Pow(10, 38) || fArray[i] < -3.4 * Math.Pow(10, 38)) fArray[i] = 0;
}
if (InvokeRequired)
{
Invoke(new MethodInvoker(ChartData));
}
else
{
ChartData();
}
// Set counterOfRecBytes to recieve new data
counterOfRecBytes = 0;
}
}
///// <summary>
///// Changes my recieved bArray to single precision floats
///// </summary>
private float ByteToFloat(byte[] input, UInt32 i)
{
byte[] array = new[] { input[4 * i], input[4 * i + 1], input[4 * i + 2], input[4 * i + 3] };
return BitConverter.ToSingle(array, 0);
}
/// <summary>
/// Setting chart data
/// </summary>
private void ChartData()
{
chart1.Series["Widmo"].Points.Clear();
for(int i = 1; i < numberOfSamples + 1; i++)
{
chart1.Series["Widmo"].Points.AddXY(fArray[i], fArray[i + numberOfSamples]);
}
}

Implementing a neural network in C#

I'm following the tutorial at this link: http://www.c-sharpcorner.com/UploadFile/rmcochran/AI_OOP_NeuralNet06192006090112AM/AI_OOP_NeuralNet.aspx
I'm new to neural networking and I'm trying to edit the example in the above tutorial to match my problem. I'm using multiple regression to find coefficients for 3 different sets of data and I then calculate the rsquared value for each set of data. I'm trying to create a neural network that will change the coefficient value to get the rsquared value as close to 100 as possible.
This is how I establish the coefficient and find the rsquared value for that coefficient. All 3 coefficients use these same methods:
Calculations calc = new Calculations();
Vector<double> lowRiskCoefficient = MultipleRegression.QR( Matrix<double>.Build.DenseOfColumnArrays(lowRiskShortRatingList.ToArray(), lowRiskMediumRatingList.ToArray(), lowRiskLongRatingList.ToArray()), Vector<double>.Build.Dense(lowRiskWeekReturnList.ToArray()));
decimal lowRiskShortCoefficient = Convert.ToDecimal(lowRiskCoefficient[0]);
decimal lowRiskMediumCoefficient = Convert.ToDecimal(lowRiskCoefficient[1]);
decimal lowRiskLongCoefficient = Convert.ToDecimal(lowRiskCoefficient[2]);
List<decimal> lowRiskWeekReturnDecimalList = new List<decimal>(lowRiskWeekReturnList.Count);
lowRiskWeekReturnList.ForEach(i => lowRiskWeekReturnDecimalList.Add(Convert.ToDecimal(i)));
List<decimal> lowRiskPredictedReturnList = new List<decimal>(lowRiskWeekReturnList.Count);
List<decimal> lowRiskResidualValueList = new List<decimal>(lowRiskWeekReturnList.Count);
for (int i = 0; i < lowRiskWeekReturnList.Count; i++)
{
decimal lowRiskPredictedValue = (Convert.ToDecimal(lowRiskShortRatingList.ElementAtOrDefault(i)) * lowRiskShortCoefficient) + (Convert.ToDecimal(lowRiskMediumRatingList.ElementAtOrDefault(i)) * lowRiskMediumCoefficient) +
(Convert.ToDecimal(lowRiskLongRatingList.ElementAtOrDefault(i)) * lowRiskLongCoefficient);
lowRiskPredictedReturnList.Add(lowRiskPredictedValue);
lowRiskResidualValueList.Add(calc.calculateResidual(lowRiskWeekReturnDecimalList.ElementAtOrDefault(i), lowRiskPredictedValue));
}
decimal lowRiskTotalSumofSquares = calc.calculateTotalSumofSquares(lowRiskWeekReturnDecimalList, lowRiskWeekReturnDecimalList.Average());
decimal lowRiskTotalSumofRegression = calc.calculateTotalSumofRegression(lowRiskPredictedReturnList, lowRiskWeekReturnDecimalList.Average());
decimal lowRiskTotalSumofErrors = calc.calculateTotalSumofErrors(lowRiskResidualValueList);
decimal lowRiskRSquared = lowRiskTotalSumofRegression / lowRiskTotalSumofSquares;
This is the example that performs the training and I'm currently stuck on how to change this example to match what I'm trying to do.
private void button1_Click(object sender, EventArgs e)
{
net = new NeuralNet();
double high, mid, low;
high = .9;
low = .1;
mid = .5;
// initialize with
// 2 perception neurons
// 2 hidden layer neurons
// 1 output neuron
net.Initialize(1, 2, 2, 1);
double[][] input = new double[4][];
input[0] = new double[] {high, high};
input[1] = new double[] {low, high};
input[2] = new double[] {high, low};
input[3] = new double[] {low, low};
double[][] output = new double[4][];
output[0] = new double[] { low };
output[1] = new double[] { high };
output[2] = new double[] { high };
output[3] = new double[] { low };
double ll, lh, hl, hh;
int count;
count = 0;
do
{
count++;
for (int i = 0; i < 100; i++)
net.Train(input, output);
net.ApplyLearning();
net.PerceptionLayer[0].Output = low;
net.PerceptionLayer[1].Output = low;
net.Pulse();
ll = net.OutputLayer[0].Output;
net.PerceptionLayer[0].Output = high;
net.PerceptionLayer[1].Output = low;
net.Pulse();
hl = net.OutputLayer[0].Output;
net.PerceptionLayer[0].Output = low;
net.PerceptionLayer[1].Output = high;
net.Pulse();
lh = net.OutputLayer[0].Output;
net.PerceptionLayer[0].Output = high;
net.PerceptionLayer[1].Output = high;
net.Pulse();
hh = net.OutputLayer[0].Output;
}
while (hh > mid || lh < mid || hl < mid || ll > mid);
MessageBox.Show((count*100).ToString() + " iterations required for training");
}
How do I use this information to create a neural network to find the coefficient that will in turn have a rsquared value as close to 100 as possible?

Instead of building one, you can use Neuroph framework built in the .NET by using the Neuroph.NET from here https://github.com/starhash/Neuroph.NET/releases/tag/v1.0-beta
It is a light conversion of the original Neuroph they did for the JAVA platform.
Hope this helps you.

How can i implement the Goertzel Algorithm with this?

I got the code here:
https://naudio.codeplex.com/discussions/270762.
The goertzel algorithm goes like this:
public double goertzel(List<double> sngData, long N, float frequency, int samplerate)
{
double skn, skn1, skn2;
skn = skn1 = skn2 = 0;
samplerate = this.sampleRate;
frequency = this.freq;
double c = 2 * pi * frequency / samplerate;
double cosan = Math.Cos(c);
for (int i = 0; i < N; i++)
{
skn2 = skn1;
skn1 = skn;
skn = 2 * cosan * skn1 - skn2 + sngData[i];
}
return skn - Math.Exp(-c) * skn1;
}
I want to transform the audio data (from wave file reader in the link above) by using that algorithm. How can i do that? Thanks

If you're doing DTMF detection, try "phoneToneDecoder" COM. it detects DTMF tones from your sound card. (i think it's proprietary)

Detecting audio silence in WAV files using C#

I'm tasked with building a .NET client app to detect silence in a WAV files.
Is this possible with the built-in Windows APIs? Or alternately, any good libraries out there to help with this?

Audio analysis is a difficult thing requiring a lot of complex math (think Fourier Transforms). The question you have to ask is "what is silence". If the audio that you are trying to edit is captured from an analog source, the chances are that there isn't any silence... they will only be areas of soft noise (line hum, ambient background noise, etc).
All that said, an algorithm that should work would be to determine a minimum volume (amplitude) threshold and duration (say, <10dbA for more than 2 seconds) and then simply do a volume analysis of the waveform looking for areas that meet this criteria (with perhaps some filters for millisecond spikes). I've never written this in C#, but this CodeProject article looks interesting; it describes C# code to draw a waveform... that is the same kind of code which could be used to do other amplitude analysis.

If you want to efficiently calculate the average power over a sliding window: square each sample, then add it to a running total. Subtract the squared value from N samples previous. Then move to the next step. This is the simplest form of a CIC Filter. Parseval's Theorem tells us that this power calculation is applicable to both time and frequency domains.
Also you may want to add Hysteresis to the system to avoid switching on&off rapidly when power level is dancing about the threshold level.

http://www.codeproject.com/Articles/19590/WAVE-File-Processor-in-C
This has all the code necessary to strip silence, and mix wave files.
Enjoy.

I'm using NAudio, and I wanted to detect the silence in audio files so I can either report or truncate.
After a lot of research, I came up with this basic implementation. So, I wrote an extension method for the AudioFileReader class which returns the silence duration at the start/end of the file, or starting from a specific position.
Here:
static class AudioFileReaderExt
{
public enum SilenceLocation { Start, End }
private static bool IsSilence(float amplitude, sbyte threshold)
{
double dB = 20 * Math.Log10(Math.Abs(amplitude));
return dB < threshold;
}
public static TimeSpan GetSilenceDuration(this AudioFileReader reader,
SilenceLocation location,
sbyte silenceThreshold = -40)
{
int counter = 0;
bool volumeFound = false;
bool eof = false;
long oldPosition = reader.Position;
var buffer = new float[reader.WaveFormat.SampleRate * 4];
while (!volumeFound && !eof)
{
int samplesRead = reader.Read(buffer, 0, buffer.Length);
if (samplesRead == 0)
eof = true;
for (int n = 0; n < samplesRead; n++)
{
if (IsSilence(buffer[n], silenceThreshold))
{
counter++;
}
else
{
if (location == SilenceLocation.Start)
{
volumeFound = true;
break;
}
else if (location == SilenceLocation.End)
{
counter = 0;
}
}
}
}
// reset position
reader.Position = oldPosition;
double silenceSamples = (double)counter / reader.WaveFormat.Channels;
double silenceDuration = (silenceSamples / reader.WaveFormat.SampleRate) * 1000;
return TimeSpan.FromMilliseconds(silenceDuration);
}
}
This will accept almost any audio file format not just WAV.
Usage:
using (AudioFileReader reader = new AudioFileReader(filePath))
{
TimeSpan duration = reader.GetSilenceDuration(AudioFileReaderExt.SilenceLocation.Start);
Console.WriteLine(duration.TotalMilliseconds);
}
References:
How audio dB levels are calculated.
Floating-point samples range.
More about amplitude.

Here a nice variant to detect threshold alternatings:
static class AudioFileReaderExt
{
private static bool IsSilence(float amplitude, sbyte threshold)
{
double dB = 20 * Math.Log10(Math.Abs(amplitude));
return dB < threshold;
}
private static bool IsBeep(float amplitude, sbyte threshold)
{
double dB = 20 * Math.Log10(Math.Abs(amplitude));
return dB > threshold;
}
public static double GetBeepDuration(this AudioFileReader reader,
double StartPosition, sbyte silenceThreshold = -40)
{
int counter = 0;
bool eof = false;
int initial = (int)(StartPosition * reader.WaveFormat.Channels * reader.WaveFormat.SampleRate / 1000);
if (initial > reader.Length) return -1;
reader.Position = initial;
var buffer = new float[reader.WaveFormat.SampleRate * 4];
while (!eof)
{
int samplesRead = reader.Read(buffer, 0, buffer.Length);
if (samplesRead == 0)
eof = true;
for (int n = initial; n < samplesRead; n++)
{
if (IsBeep(buffer[n], silenceThreshold))
{
counter++;
}
else
{
eof=true; break;
}
}
}
double silenceSamples = (double)counter / reader.WaveFormat.Channels;
double silenceDuration = (silenceSamples / reader.WaveFormat.SampleRate) * 1000;
return TimeSpan.FromMilliseconds(silenceDuration).TotalMilliseconds;
}
public static double GetSilenceDuration(this AudioFileReader reader,
double StartPosition, sbyte silenceThreshold = -40)
{
int counter = 0;
bool eof = false;
int initial = (int)(StartPosition * reader.WaveFormat.Channels * reader.WaveFormat.SampleRate / 1000);
if (initial > reader.Length) return -1;
reader.Position = initial;
var buffer = new float[reader.WaveFormat.SampleRate * 4];
while (!eof)
{
int samplesRead = reader.Read(buffer, 0, buffer.Length);
if (samplesRead == 0)
eof=true;
for (int n = initial; n < samplesRead; n++)
{
if (IsSilence(buffer[n], silenceThreshold))
{
counter++;
}
else
{
eof=true; break;
}
}
}
double silenceSamples = (double)counter / reader.WaveFormat.Channels;
double silenceDuration = (silenceSamples / reader.WaveFormat.SampleRate) * 1000;
return TimeSpan.FromMilliseconds(silenceDuration).TotalMilliseconds;
}
}
Main usage:
using (AudioFileReader reader = new AudioFileReader("test.wav"))
{
double duratioff = 1;
double duration = 1;
double position = 1;
while (duratioff >-1 && duration >-1)
{
duration = reader.GetBeepDuration(position);
Console.WriteLine(duration);
position = position + duration;
duratioff = reader.GetSilenceDuration(position);
Console.WriteLine(-duratioff);
position = position + duratioff;
}
}

I don't think you'll find any built-in APIs for detection of silence. But you can always use good ol' math/discreete signal processing to find out loudness.
Here's a small example: http://msdn.microsoft.com/en-us/magazine/cc163341.aspx

Use Sox. It can remove leading and trailing silences, but you'll have to call it as an exe from your app.

See code below from Detecting audio silence in WAV files using C#
private static void SkipSilent(string fileName, short silentLevel)
{
WaveReader wr = new WaveReader(File.OpenRead(fileName));
IntPtr format = wr.ReadFormat();
WaveWriter ww = new WaveWriter(File.Create(fileName + ".wav"),
AudioCompressionManager.FormatBytes(format));
int i = 0;
while (true)
{
byte[] data = wr.ReadData(i, 1);
if (data.Length == 0)
{
break;
}
if (!AudioCompressionManager.CheckSilent(format, data, silentLevel))
{
ww.WriteData(data);
}
}
ww.Close();
wr.Close();
}