We Keep Coding

Importing and removing duplicates from a massive amount of text files using C# and Redis

This is a bit of a doozy and it's been a while since I worked with C#, so bear with me:
I'm running a jruby script to iterate through 900 files (5 Mb - 1500 Mb in size) to figure out how many dupes STILL exist within these (already uniq'd) files. I had little luck with awk.
My latest idea was to insert them into a local MongoDB instance like so:
db.collection('hashes').update({ :_id => hash}, { $inc: { count: 1} }, { upsert: true)
... so that later I could just query it like db.collection.where({ count: { $gt: 1 } }) to get all the dupes.
This is working great except it's been over 24 hours and at the time of writing I'm at 72,532,927 Mongo entries and growing.
I think Ruby's .each_line is bottlnecking the IO hardcore:
So what I'm thinking now is compiling a C# program which fires up a thread PER EACH FILE and inserts the line (md5 hash) into a Redis list.
From there, I could have another compiled C# program simply pop the values off and ignore the save if the count is 1.
So the questions are:
Will using a compiled file reader and multithreading the file reads significantly improve performance?
Is using Redis even necessary? With a tremendous amount of AWS memory, could I not just use the threads to fill some sort of a list atomically and proceed from there?
Thanks in advance.

Updated
New solution. Old solution. The main idea is to calculate dummy hashes(just sum of all chars in string) of each line and store it in Dictionary<ulong, List<LinePosition>> _hash2LinePositions. It's possible to have multiple hashes in the same stream and it solves by List in Dictionary Value. When the hashes are the same, we read and compare the strings from the streams. LinePosition is using for storing info about line - position in stream and its length. I don't have such huge files as you, but my tests shows that it works. Here is the full code:
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
public class Solution
{
struct LinePosition
{
public long Start;
public long Length;
public LinePosition(long start, long count)
{
Start = start;
Length = count;
}
public override string ToString()
{
return string.Format("Start: {0}, Length: {1}", Start, Length);
}
}
class TextFileHasher : IDisposable
{
readonly Dictionary<ulong, List<LinePosition>> _hash2LinePositions;
readonly Stream _stream;
bool _isDisposed;
public HashSet<ulong> Hashes { get; private set; }
public string Name { get; private set; }
public TextFileHasher(string name, Stream stream)
{
Name = name;
_stream = stream;
_hash2LinePositions = new Dictionary<ulong, List<LinePosition>>();
Hashes = new HashSet<ulong>();
}
public override string ToString()
{
return Name;
}
public void CalculateFileHash()
{
int readByte = -1;
ulong dummyLineHash = 0;
// Line start position in file
long startPosition = 0;
while ((readByte = _stream.ReadByte()) != -1) {
// Read until new line
if (readByte == '\r' || readByte == '\n') {
// If there was data
if (dummyLineHash != 0) {
// Add line hash and line position to the dict
AddToDictAndHash(dummyLineHash, startPosition, _stream.Position - 1 - startPosition);
// Reset line hash
dummyLineHash = 0;
}
}
else {
// Was it new line ?
if (dummyLineHash == 0)
startPosition = _stream.Position - 1;
// Calculate dummy hash
dummyLineHash += (uint)readByte;
}
}
if (dummyLineHash != 0) {
// Add line hash and line position to the dict
AddToDictAndHash(dummyLineHash, startPosition, _stream.Position - startPosition);
// Reset line hash
dummyLineHash = 0;
}
}
public List<LinePosition> GetLinePositions(ulong hash)
{
return _hash2LinePositions[hash];
}
public List<string> GetDuplicates()
{
List<string> duplicates = new List<string>();
foreach (var key in _hash2LinePositions.Keys) {
List<LinePosition> linesPos = _hash2LinePositions[key];
if (linesPos.Count > 1) {
duplicates.AddRange(FindExactDuplicates(linesPos));
}
}
return duplicates;
}
public void Dispose()
{
if (_isDisposed)
return;
_stream.Dispose();
_isDisposed = true;
}
private void AddToDictAndHash(ulong hash, long start, long count)
{
List<LinePosition> linesPosition;
if (!_hash2LinePositions.TryGetValue(hash, out linesPosition)) {
linesPosition = new List<LinePosition>() { new LinePosition(start, count) };
_hash2LinePositions.Add(hash, linesPosition);
}
else {
linesPosition.Add(new LinePosition(start, count));
}
Hashes.Add(hash);
}
public byte[] GetLineAsByteArray(LinePosition prevPos)
{
long len = prevPos.Length;
byte[] lineBytes = new byte[len];
_stream.Seek(prevPos.Start, SeekOrigin.Begin);
_stream.Read(lineBytes, 0, (int)len);
return lineBytes;
}
private List<string> FindExactDuplicates(List<LinePosition> linesPos)
{
List<string> duplicates = new List<string>();
linesPos.Sort((x, y) => x.Length.CompareTo(y.Length));
LinePosition prevPos = linesPos[0];
for (int i = 1; i < linesPos.Count; i++) {
if (prevPos.Length == linesPos[i].Length) {
var prevLineArray = GetLineAsByteArray(prevPos);
var thisLineArray = GetLineAsByteArray(linesPos[i]);
if (prevLineArray.SequenceEqual(thisLineArray)) {
var line = System.Text.Encoding.Default.GetString(prevLineArray);
duplicates.Add(line);
}
#if false
string prevLine = System.Text.Encoding.Default.GetString(prevLineArray);
string thisLine = System.Text.Encoding.Default.GetString(thisLineArray);
Console.WriteLine("PrevLine: {0}\r\nThisLine: {1}", prevLine, thisLine);
StringBuilder sb = new StringBuilder();
sb.Append(prevPos);
sb.Append(" is '");
sb.Append(prevLine);
sb.Append("'. ");
sb.AppendLine();
sb.Append(linesPos[i]);
sb.Append(" is '");
sb.Append(thisLine);
sb.AppendLine("'. ");
sb.Append("Equals => ");
sb.Append(prevLine.CompareTo(thisLine) == 0);
Console.WriteLine(sb.ToString());
#endif
}
else {
prevPos = linesPos[i];
}
}
return duplicates;
}
}
public static void Main(String[] args)
{
List<TextFileHasher> textFileHashers = new List<TextFileHasher>();
string text1 = "abc\r\ncba\r\nabc";
TextFileHasher tfh1 = new TextFileHasher("Text1", new MemoryStream(System.Text.Encoding.Default.GetBytes(text1)));
tfh1.CalculateFileHash();
textFileHashers.Add(tfh1);
string text2 = "def\r\ncba\r\nwet";
TextFileHasher tfh2 = new TextFileHasher("Text2", new MemoryStream(System.Text.Encoding.Default.GetBytes(text2)));
tfh2.CalculateFileHash();
textFileHashers.Add(tfh2);
string text3 = "def\r\nbla\r\nwat";
TextFileHasher tfh3 = new TextFileHasher("Text3", new MemoryStream(System.Text.Encoding.Default.GetBytes(text3)));
tfh3.CalculateFileHash();
textFileHashers.Add(tfh3);
List<string> totalDuplicates = new List<string>();
Dictionary<ulong, Dictionary<TextFileHasher, List<LinePosition>>> totalHashes = new Dictionary<ulong, Dictionary<TextFileHasher, List<LinePosition>>>();
textFileHashers.ForEach(tfh => {
foreach(var dummyHash in tfh.Hashes) {
Dictionary<TextFileHasher, List<LinePosition>> tfh2LinePositions = null;
if (!totalHashes.TryGetValue(dummyHash, out tfh2LinePositions))
totalHashes[dummyHash] = new Dictionary<TextFileHasher, List<LinePosition>>() { { tfh, tfh.GetLinePositions(dummyHash) } };
else {
List<LinePosition> linePositions = null;
if (!tfh2LinePositions.TryGetValue(tfh, out linePositions))
tfh2LinePositions[tfh] = tfh.GetLinePositions(dummyHash);
else
linePositions.AddRange(tfh.GetLinePositions(dummyHash));
}
}
});
HashSet<TextFileHasher> alreadyGotDuplicates = new HashSet<TextFileHasher>();
foreach(var hash in totalHashes.Keys) {
var tfh2LinePositions = totalHashes[hash];
var tfh = tfh2LinePositions.Keys.FirstOrDefault();
// Get duplicates in the TextFileHasher itself
if (tfh != null && !alreadyGotDuplicates.Contains(tfh)) {
totalDuplicates.AddRange(tfh.GetDuplicates());
alreadyGotDuplicates.Add(tfh);
}
if (tfh2LinePositions.Count <= 1) {
continue;
}
// Algo to get duplicates in more than 1 TextFileHashers
var tfhs = tfh2LinePositions.Keys.ToArray();
for (int i = 0; i < tfhs.Length; i++) {
var tfh1Positions = tfhs[i].GetLinePositions(hash);
for (int j = i + 1; j < tfhs.Length; j++) {
var tfh2Positions = tfhs[j].GetLinePositions(hash);
for (int k = 0; k < tfh1Positions.Count; k++) {
var tfh1Pos = tfh1Positions[k];
var tfh1ByteArray = tfhs[i].GetLineAsByteArray(tfh1Pos);
for (int m = 0; m < tfh2Positions.Count; m++) {
var tfh2Pos = tfh2Positions[m];
if (tfh1Pos.Length != tfh2Pos.Length)
continue;
var tfh2ByteArray = tfhs[j].GetLineAsByteArray(tfh2Pos);
if (tfh1ByteArray.SequenceEqual(tfh2ByteArray)) {
var line = System.Text.Encoding.Default.GetString(tfh1ByteArray);
totalDuplicates.Add(line);
}
}
}
}
}
}
Console.WriteLine();
if (totalDuplicates.Count > 0) {
Console.WriteLine("Total number of duplicates: {0}", totalDuplicates.Count);
Console.WriteLine("#######################");
totalDuplicates.ForEach(x => Console.WriteLine("{0}", x));
Console.WriteLine("#######################");
}
// Free resources
foreach (var tfh in textFileHashers)
tfh.Dispose();
}
}

If you have tons of ram... You guys are overthinking it...
var fileLines = File.ReadAllLines(#"c:\file.csv").Distinct();

Suppressing Frequencies From FFT

What I am trying to do is to retrieve the frequencies from some song and suppress all the frequencies that do not appear in the human vocal range or in general any range. Here is my suppress function.
public void SupressAndWrite(Func<FrequencyUnit, bool> func)
{
this.WaveManipulated = true;
while (this.mainWave.WAVFile.NumSamplesRemaining > 0)
{
FrequencyUnit[] freqUnits = this.mainWave.NextFrequencyUnits();
Complex[] compUnits = (from item
in freqUnits
select (func(item)
? new Complex(item.Frequency, 0) :Complex.Zero))
.ToArray();
FourierTransform.FFT(compUnits, FourierTransform.Direction.Backward);
short[] shorts = (from item
in compUnits
select (short)item.Real).ToArray();
foreach (short item in shorts)
{
this.ManipulatedFile.AddSample16bit(item);
}
}
this.ManipulatedFile.Close();
}
Here is my class for my wave.
public sealed class ComplexWave
{
public readonly WAVFile WAVFile;
public readonly Int32 SampleSize;
private FourierTransform.Direction fourierDirection { get; set; }
private long position;
/// <param name="file"></param>
/// <param name="sampleSize in BLOCKS"></param>
public ComplexWave(WAVFile file, int sampleSize)
{
file.NullReferenceExceptionCheck();
this.WAVFile = file;
this.SampleSize = sampleSize;
if (this.SampleSize % 8 != 0)
{
if (this.SampleSize % 16 != 0)
{
throw new ArgumentException("Sample Size");
}
}
if (!MathTools.IsPowerOf2(sampleSize))
{
throw new ArgumentException("Sample Size");
}
this.fourierDirection = FourierTransform.Direction.Forward;
}
public Complex[] NextSampleFourierTransform()
{
short[] newInput = this.GetNextSample();
Complex[] data = newInput.CopyToComplex();
if (newInput.Any((x) => x != 0))
{
Debug.Write("done");
}
FourierTransform.FFT(data, this.fourierDirection);
return data;
}
public FrequencyUnit[] NextFrequencyUnits()
{
Complex[] cm = this.NextSampleFourierTransform();
FrequencyUnit[] freqUn = new FrequencyUnit[(cm.Length / 2)];
int max = (cm.Length / 2);
for (int i = 0; i < max; i++)
{
freqUn[i] = new FrequencyUnit(cm[i], this.WAVFile.SampleRateHz, i, cm.Length);
}
Array.Sort(freqUn);
return freqUn;
}
private short[] GetNextSample()
{
short[] retval = new short[this.SampleSize];
for (int i = 0; i < this.SampleSize; i++)
{
if (this.WAVFile.NumSamplesRemaining > 0)
{
retval[i] = this.WAVFile.GetNextSampleAs16Bit();
this.position++;
}
}
return retval;
}
}
Both FFT forward and FFT backwards work correctly. Could you please tell me what my error is.

Unfortunately, human voice, even when singing, isn't in 'frequency range'. It usually has one main frequency and multitude of harmonics that follow it, depending on the phoneme.
Use this https://play.google.com/store/apps/details?id=radonsoft.net.spectralview&hl=en or some similar app to see what I mean - and then re-define your strategy. Also google 'karaoke' effect.
NEXT:
It's not obvious from your example, but you should scan whole file in windows (google 'fft windowing') to process it whole.

How to list camera available video resolution

if I have more than one camera attached to my PC ... I want to know the best available resolutions for a specific camera ...
for example some cameras are HD or FullHD (1,280×720 pixels (720p) or 1,920×1,080 pixels (1080i/1080p)) or the most common are web cameras....
I want to know at least the best video mode that the camera work properly...(the mode that the camera made to work with)
my work is on WPF using C# (I am using Directshow)
thanks in advance

This is a code that I wrote, its working perfectly for me
public static List<Point> GetAllAvailableResolution(DsDevice vidDev)
{
try
{
int hr;
int max = 0;
int bitCount = 0;
IBaseFilter sourceFilter = null;
var m_FilterGraph2 = new FilterGraph() as IFilterGraph2;
hr = m_FilterGraph2.AddSourceFilterForMoniker(vidDev.Mon, null, vidDev.Name, out sourceFilter);
var pRaw2 = DsFindPin.ByCategory(sourceFilter, PinCategory.Capture, 0);
var AvailableResolutions = new List<Point>();
VideoInfoHeader v = new VideoInfoHeader();
IEnumMediaTypes mediaTypeEnum;
hr = pRaw2.EnumMediaTypes(out mediaTypeEnum);
AMMediaType[] mediaTypes = new AMMediaType[1];
IntPtr fetched = IntPtr.Zero;
hr = mediaTypeEnum.Next(1, mediaTypes, fetched);
while (fetched != null && mediaTypes[0] != null)
{
Marshal.PtrToStructure(mediaTypes[0].formatPtr, v);
if (v.BmiHeader.Size != 0 && v.BmiHeader.BitCount != 0)
{
if (v.BmiHeader.BitCount > bitCount)
{
AvailableResolutions.Clear();
max = 0;
bitCount = v.BmiHeader.BitCount;
}
AvailableResolutions.Add(new Point(v.BmiHeader.Width, v.BmiHeader.Height));
if (v.BmiHeader.Width > max || v.BmiHeader.Height > max)
max = (Math.Max(v.BmiHeader.Width, v.BmiHeader.Height));
}
hr = mediaTypeEnum.Next(1, mediaTypes, fetched);
}
return AvailableResolutions;
}
catch (Exception ex)
{
Log(ex);
return new List<Point>();
}
}
(E.g. this can be added to VideoCaptureElement in WPF-MediaKit)

i use this to get max frame size, just change to suit your needs ;)
private Point GetMaxFrameSize(IPin pStill)
{
VideoInfoHeader v;
IAMStreamConfig videoStreamConfig = pStill as IAMStreamConfig;
int iCount = 0, iSize = 0;
videoStreamConfig.GetNumberOfCapabilities(out iCount, out iSize);
IntPtr TaskMemPointer = Marshal.AllocCoTaskMem(iSize);
int iMaxHeight = 0;
int iMaxWidth = 0;
for (int iFormat = 0; iFormat < iCount; iFormat++)
{
AMMediaType pmtConfig = null;
IntPtr ptr = IntPtr.Zero;
videoStreamConfig.GetStreamCaps(iFormat, out pmtConfig, TaskMemPointer);
v = (VideoInfoHeader)Marshal.PtrToStructure(pmtConfig.formatPtr, typeof(VideoInfoHeader));
if (v.BmiHeader.Width > iMaxWidth)
{
iMaxWidth = v.BmiHeader.Width;
iMaxHeight = v.BmiHeader.Height;
}
DsUtils.FreeAMMediaType(pmtConfig);
}
Marshal.FreeCoTaskMem(TaskMemPointer);
return new Point(iMaxWidth, iMaxHeight);
}
/// <summary>
/// Free the nested structures and release any
/// COM objects within an AMMediaType struct.
/// </summary>
public static void FreeAMMediaType(AMMediaType mediaType)
{
if (mediaType != null)
{
if (mediaType.formatSize != 0)
{
Marshal.FreeCoTaskMem(mediaType.formatPtr);
mediaType.formatSize = 0;
mediaType.formatPtr = IntPtr.Zero;
}
if (mediaType.unkPtr != IntPtr.Zero)
{
Marshal.Release(mediaType.unkPtr);
mediaType.unkPtr = IntPtr.Zero;
}
}
}

According to this webpage:
http://www.e-consystems.com/blog/camera/?p=651, you should use this call for getting the capabilities of this device:
g_DShowCaptureGraph.GetNumberOfCapabilities(nStream, &iCount, &iSize);
g_DShowCaptureGraph.GetStreamCaps(nStream,iFormat, &pmtConfig, (BYTE*)&scc);
They are C++, however.

FileReader class in C#

I am looking for fast class for to work with text files and comfortable reading different object (methods like NextInt32, NextDouble, NextLine, etc). Can you advice me something?
Edit: BinaryReader is bad class in my case. Format of my data is not binary. I have file like
1 2 3
FirstToken NextToken
1.23 2,34
And I want read this file with code like:
int a = FileReader.NextInt32();
int b = FileReader.NextInt32();
int c = FileReader.NextInt32();
int d = FileReader.NextString();
int e = FileReader.NextString();
int f = FileReader.NextDouble();
int g = FileReader.NextDouble();
Edit2: I am looking for analog Scanner from Java

I believe this extension method for TextReader would do the trick:
public static class TextReaderTokenizer
{
// Adjust as needed. -1 is EOF.
private static int[] whitespace = { -1, ' ', '\r' , '\n', '\t' };
public static T ReadToken<T>(this TextReader reader)
{
StringBuilder sb = new StringBuilder();
while (Array.IndexOf(whitespace, reader.Peek()) < 0)
{
sb.Append((char)reader.Read());
}
return (T)Convert.ChangeType(sb.ToString(), typeof(T));
}
}
It can be used thus:
TextReader reader = File.OpenText("foo.txt");
int n = reader.ReadToken<int>();
string s = reader.ReadToken<string>();
[EDIT] As requested in question comments, here's an instance wrapper version of the above that is parametrized with delimiters and CultureInfo:
public class TextTokenizer
{
private TextReader reader;
private Predicate<char> isDelim;
private CultureInfo cultureInfo;
public TextTokenizer(TextReader reader, Predicate<char> isDelim, CultureInfo cultureInfo)
{
this.reader = reader;
this.isDelim = isDelim;
this.cultureInfo = cultureInfo;
}
public TextTokenizer(TextReader reader, char[] delims, CultureInfo cultureInfo)
{
this.reader = reader;
this.isDelim = c => Array.IndexOf(delims, c) >= 0;
this.cultureInfo = cultureInfo;
}
public TextReader BaseReader
{
get { return reader; }
}
public T ReadToken<T>()
{
StringBuilder sb = new StringBuilder();
while (true)
{
int c = reader.Peek();
if (c < 0 || isDelim((char)c))
{
break;
}
sb.Append((char)reader.Read());
}
return (T)Convert.ChangeType(sb.ToString(), typeof(T));
}
}
Sample usage:
TextReader reader = File.OpenText("foo.txt");
TextTokenizer tokenizer = new TextTokenizer(
reader,
new[] { ' ', '\r', '\n', '\t' },
CultureInfo.InvariantCulture);
int n = tokenizer.ReadToken<int>();
string s = tokenizer.ReadToken<string>();

I'm going to add this as a separate answer because it's quite distinct from the answer I already gave. Here's how you could start creating your own Scanner class:
class Scanner : System.IO.StringReader
{
string currentWord;
public Scanner(string source) : base(source)
{
readNextWord();
}
private void ReadNextWord()
{
System.Text.StringBuilder sb = new StringBuilder();
char nextChar;
int next;
do
{
next = this.Read();
if (next < 0)
break;
nextChar = (char)next;
if (char.IsWhiteSpace(nextChar))
break;
sb.Append(nextChar);
} while (true);
while((this.Peek() >= 0) && (char.IsWhiteSpace((char)this.Peek())))
this.Read();
if (sb.Length > 0)
currentWord = sb.ToString();
else
currentWord = null;
}
public bool HasNextInt()
{
if (currentWord == null)
return false;
int dummy;
return int.TryParse(currentWord, out dummy);
}
public int NextInt()
{
try
{
return int.Parse(currentWord);
}
finally
{
readNextWord();
}
}
public bool HasNextDouble()
{
if (currentWord == null)
return false;
double dummy;
return double.TryParse(currentWord, out dummy);
}
public double NextDouble()
{
try
{
return double.Parse(currentWord);
}
finally
{
readNextWord();
}
}
public bool HasNext()
{
return currentWord != null;
}
}

You should define exactly what your file format is meant to look like. How would you represent a string with a space in it? What determines where the line terminators go?
In general you can use TextReader and its ReadLine method, followed by double.TryParse, int.TryParse etc - but you'll need to pin the format down more first.

Have you checked out the BinaryReader class? Yes it's a text file but there is nothing stopping you from treating it as binary data and hence using BinaryReader. It has all of the methods that you are looking for with the exception of ReadLine. However it wouldn't be too difficult to implement that method on top of BinaryReader.

If you do need text files (ie UTF-8 or ASCII encoding) then the binary writer will not work.
You can use the TextReader, but unlike the BinaryReader and the TextWriter it does not support any types other than Line and char. You will have to define what separators are allowed and parse the Line base data yourself.

The System.IO.BinaryReader class is what you need.
Example of implementation of a ReadLine method:
public static class Extensions
{
public static String ReadLine(this BinaryReader binaryReader)
{
var bytes = new List<Byte>();
byte temp;
while ((temp = (byte)binaryReader.Read()) < 10)
bytes.Add(temp);
return Encoding.Default.GetString(bytes.ToArray());
}
}
Example for using this class:
using System;
using System.IO;
using System.Security.Permissions;
class Test
{
static void Main()
{
// Load application settings.
AppSettings appSettings = new AppSettings();
Console.WriteLine("App settings:\nAspect Ratio: {0}, " +
"Lookup directory: {1},\nAuto save time: {2} minutes, " +
"Show status bar: {3}\n",
new Object[4]{appSettings.AspectRatio.ToString(),
appSettings.LookupDir, appSettings.AutoSaveTime.ToString(),
appSettings.ShowStatusBar.ToString()});
// Change the settings.
appSettings.AspectRatio = 1.250F;
appSettings.LookupDir = #"C:\Temp";
appSettings.AutoSaveTime = 10;
appSettings.ShowStatusBar = true;
// Save the new settings.
appSettings.Close();
}
}
// Store and retrieve application settings.
class AppSettings
{
const string fileName = "AppSettings####.dat";
float aspectRatio;
string lookupDir;
int autoSaveTime;
bool showStatusBar;
public float AspectRatio
{
get{ return aspectRatio; }
set{ aspectRatio = value; }
}
public string LookupDir
{
get{ return lookupDir; }
set{ lookupDir = value; }
}
public int AutoSaveTime
{
get{ return autoSaveTime; }
set{ autoSaveTime = value; }
}
public bool ShowStatusBar
{
get{ return showStatusBar; }
set{ showStatusBar = value; }
}
public AppSettings()
{
// Create default application settings.
aspectRatio = 1.3333F;
lookupDir = #"C:\AppDirectory";
autoSaveTime = 30;
showStatusBar = false;
if(File.Exists(fileName))
{
BinaryReader binReader =
new BinaryReader(File.Open(fileName, FileMode.Open));
try
{
// If the file is not empty,
// read the application settings.
// First read 4 bytes into a buffer to
// determine if the file is empty.
byte[] testArray = new byte[3];
int count = binReader.Read(testArray, 0, 3);
if (count != 0)
{
// Reset the position in the stream to zero.
binReader.BaseStream.Seek(0, SeekOrigin.Begin);
aspectRatio = binReader.ReadSingle();
lookupDir = binReader.ReadString();
autoSaveTime = binReader.ReadInt32();
showStatusBar = binReader.ReadBoolean();
}
}
// If the end of the stream is reached before reading
// the four data values, ignore the error and use the
// default settings for the remaining values.
catch(EndOfStreamException e)
{
Console.WriteLine("{0} caught and ignored. " +
"Using default values.", e.GetType().Name);
}
finally
{
binReader.Close();
}
}
}
// Create a file and store the application settings.
public void Close()
{
using(BinaryWriter binWriter =
new BinaryWriter(File.Open(fileName, FileMode.Create)))
{
binWriter.Write(aspectRatio);
binWriter.Write(lookupDir);
binWriter.Write(autoSaveTime);
binWriter.Write(showStatusBar);
}
}
}

You can probably use the System.IO.File Class to read the file and System.Convert to parse the strings you read from the file.
string line = String.Empty;
while( (line = file.ReadLine()).IsNullOrEmpty() == false )
{
TYPE value = Convert.ToTYPE( line );
}
Where TYPE is whatever type you're dealing with at that particular line / file.
If there are multiple values on one line you could do a split and read the individual values e.g.
string[] parts = line.Split(' ');
if( parts.Length > 1 )
{
foreach( string item in parts )
{
TYPE value = Convert.ToTYPE( item );
}
}
else
{
// Use the code from before
}

C# : Redirect console application output : How to flush the output?

I am spawning external console application and use async output redirect.
as shown in this SO post
My problem is it seems that the spawned process needs to produce certain amount of output before I get the OutputDataReceived event notification.
I want to receive the OutputDataReceived event as soon as possible.
I have a bare-bones redirecting application, and here are some observations:
1. When I call a simple 'while(true) print("X");' console application (C#) I receive output event immediately.
2. When I call a 3d party app I am trying to wrap from the command line I see the line-by-line output.
3. When I call that 3d party app from my bare-bone wrapper (see 1) - the output comes in chunks (about one page size).
What happens inside that app?
FYI: The app in question is a "USBee DX Data Exctarctor (Async bus) v1.0".

I did some more research and have a fix to microsofts Process class. But as my last answer was deleted without a reason, I have had to create a new one.
So take this example...
Create a windows app and stick a rich textbox on the main form, then add this to the form load...
Process p = new Process()
{
StartInfo = new ProcessStartInfo()
{
FileName = "cmd.exe",
CreateNoWindow = true,
UseShellExecute = false,
ErrorDialog = false,
RedirectStandardInput = true,
RedirectStandardOutput = true,
RedirectStandardError = true,
},
EnableRaisingEvents = true,
SynchronizingObject = this
};
p.OutputDataReceived += (s, ea) => this.richTextBox1.AppendText(ea.Data);
p.Start();
p.BeginOutputReadLine();
This will output something like this...
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.
The OutputDataReceived event is not fired for the last line. After some ILSpying it appears that this is deliberate because the last line does not end with a crlf, it assumes there is more comming and appends it to the start of the next event.
To correct this, I have written a wrapper for the Process class and taken some of the required internal classes out with it so that it all works neatly. Here is the FixedProcess class...
using System;
using System.Collections;
using System.IO;
using System.Text;
using System.Threading;
namespace System.Diagnostics
{
internal delegate void UserCallBack(string data);
public delegate void DataReceivedEventHandler(object sender, DataReceivedEventArgs e);
public class FixedProcess : Process
{
internal AsyncStreamReader output;
internal AsyncStreamReader error;
public event DataReceivedEventHandler OutputDataReceived;
public event DataReceivedEventHandler ErrorDataReceived;
public new void BeginOutputReadLine()
{
Stream baseStream = StandardOutput.BaseStream;
this.output = new AsyncStreamReader(this, baseStream, new UserCallBack(this.FixedOutputReadNotifyUser), StandardOutput.CurrentEncoding);
this.output.BeginReadLine();
}
public void BeginErrorReadLine()
{
Stream baseStream = StandardError.BaseStream;
this.error = new AsyncStreamReader(this, baseStream, new UserCallBack(this.FixedErrorReadNotifyUser), StandardError.CurrentEncoding);
this.error.BeginReadLine();
}
internal void FixedOutputReadNotifyUser(string data)
{
DataReceivedEventHandler outputDataReceived = this.OutputDataReceived;
if (outputDataReceived != null)
{
DataReceivedEventArgs dataReceivedEventArgs = new DataReceivedEventArgs(data);
if (this.SynchronizingObject != null && this.SynchronizingObject.InvokeRequired)
{
this.SynchronizingObject.Invoke(outputDataReceived, new object[]
{
this,
dataReceivedEventArgs
});
return;
}
outputDataReceived(this, dataReceivedEventArgs);
}
}
internal void FixedErrorReadNotifyUser(string data)
{
DataReceivedEventHandler errorDataReceived = this.ErrorDataReceived;
if (errorDataReceived != null)
{
DataReceivedEventArgs dataReceivedEventArgs = new DataReceivedEventArgs(data);
if (this.SynchronizingObject != null && this.SynchronizingObject.InvokeRequired)
{
this.SynchronizingObject.Invoke(errorDataReceived, new object[]
{
this,
dataReceivedEventArgs
});
return;
}
errorDataReceived(this, dataReceivedEventArgs);
}
}
}
internal class AsyncStreamReader : IDisposable
{
internal const int DefaultBufferSize = 1024;
private const int MinBufferSize = 128;
private Stream stream;
private Encoding encoding;
private Decoder decoder;
private byte[] byteBuffer;
private char[] charBuffer;
private int _maxCharsPerBuffer;
private Process process;
private UserCallBack userCallBack;
private bool cancelOperation;
private ManualResetEvent eofEvent;
private Queue messageQueue;
private StringBuilder sb;
private bool bLastCarriageReturn;
public virtual Encoding CurrentEncoding
{
get
{
return this.encoding;
}
}
public virtual Stream BaseStream
{
get
{
return this.stream;
}
}
internal AsyncStreamReader(Process process, Stream stream, UserCallBack callback, Encoding encoding)
: this(process, stream, callback, encoding, 1024)
{
}
internal AsyncStreamReader(Process process, Stream stream, UserCallBack callback, Encoding encoding, int bufferSize)
{
this.Init(process, stream, callback, encoding, bufferSize);
this.messageQueue = new Queue();
}
private void Init(Process process, Stream stream, UserCallBack callback, Encoding encoding, int bufferSize)
{
this.process = process;
this.stream = stream;
this.encoding = encoding;
this.userCallBack = callback;
this.decoder = encoding.GetDecoder();
if (bufferSize < 128)
{
bufferSize = 128;
}
this.byteBuffer = new byte[bufferSize];
this._maxCharsPerBuffer = encoding.GetMaxCharCount(bufferSize);
this.charBuffer = new char[this._maxCharsPerBuffer];
this.cancelOperation = false;
this.eofEvent = new ManualResetEvent(false);
this.sb = null;
this.bLastCarriageReturn = false;
}
public virtual void Close()
{
this.Dispose(true);
}
void IDisposable.Dispose()
{
this.Dispose(true);
GC.SuppressFinalize(this);
}
protected virtual void Dispose(bool disposing)
{
if (disposing && this.stream != null)
{
this.stream.Close();
}
if (this.stream != null)
{
this.stream = null;
this.encoding = null;
this.decoder = null;
this.byteBuffer = null;
this.charBuffer = null;
}
if (this.eofEvent != null)
{
this.eofEvent.Close();
this.eofEvent = null;
}
}
internal void BeginReadLine()
{
if (this.cancelOperation)
{
this.cancelOperation = false;
}
if (this.sb == null)
{
this.sb = new StringBuilder(1024);
this.stream.BeginRead(this.byteBuffer, 0, this.byteBuffer.Length, new AsyncCallback(this.ReadBuffer), null);
return;
}
this.FlushMessageQueue();
}
internal void CancelOperation()
{
this.cancelOperation = true;
}
private void ReadBuffer(IAsyncResult ar)
{
int num;
try
{
num = this.stream.EndRead(ar);
}
catch (IOException)
{
num = 0;
}
catch (OperationCanceledException)
{
num = 0;
}
if (num == 0)
{
lock (this.messageQueue)
{
if (this.sb.Length != 0)
{
this.messageQueue.Enqueue(this.sb.ToString());
this.sb.Length = 0;
}
this.messageQueue.Enqueue(null);
}
try
{
this.FlushMessageQueue();
return;
}
finally
{
this.eofEvent.Set();
}
}
int chars = this.decoder.GetChars(this.byteBuffer, 0, num, this.charBuffer, 0);
this.sb.Append(this.charBuffer, 0, chars);
this.GetLinesFromStringBuilder();
this.stream.BeginRead(this.byteBuffer, 0, this.byteBuffer.Length, new AsyncCallback(this.ReadBuffer), null);
}
private void GetLinesFromStringBuilder()
{
int i = 0;
int num = 0;
int length = this.sb.Length;
if (this.bLastCarriageReturn && length > 0 && this.sb[0] == '\n')
{
i = 1;
num = 1;
this.bLastCarriageReturn = false;
}
while (i < length)
{
char c = this.sb[i];
if (c == '\r' || c == '\n')
{
if (c == '\r' && i + 1 < length && this.sb[i + 1] == '\n')
{
i++;
}
string obj = this.sb.ToString(num, i + 1 - num);
num = i + 1;
lock (this.messageQueue)
{
this.messageQueue.Enqueue(obj);
}
}
i++;
}
// Flush Fix: Send Whatever is left in the buffer
string endOfBuffer = this.sb.ToString(num, length - num);
lock (this.messageQueue)
{
this.messageQueue.Enqueue(endOfBuffer);
num = length;
}
// End Flush Fix
if (this.sb[length - 1] == '\r')
{
this.bLastCarriageReturn = true;
}
if (num < length)
{
this.sb.Remove(0, num);
}
else
{
this.sb.Length = 0;
}
this.FlushMessageQueue();
}
private void FlushMessageQueue()
{
while (this.messageQueue.Count > 0)
{
lock (this.messageQueue)
{
if (this.messageQueue.Count > 0)
{
string data = (string)this.messageQueue.Dequeue();
if (!this.cancelOperation)
{
this.userCallBack(data);
}
}
continue;
}
break;
}
}
internal void WaitUtilEOF()
{
if (this.eofEvent != null)
{
this.eofEvent.WaitOne();
this.eofEvent.Close();
this.eofEvent = null;
}
}
}
public class DataReceivedEventArgs : EventArgs
{
internal string _data;
/// <summary>Gets the line of characters that was written to a redirected <see cref="T:System.Diagnostics.Process" /> output stream.</summary>
/// <returns>The line that was written by an associated <see cref="T:System.Diagnostics.Process" /> to its redirected <see cref="P:System.Diagnostics.Process.StandardOutput" /> or <see cref="P:System.Diagnostics.Process.StandardError" /> stream.</returns>
/// <filterpriority>2</filterpriority>
public string Data
{
get
{
return this._data;
}
}
internal DataReceivedEventArgs(string data)
{
this._data = data;
}
}
}
Stick that in your project and then change ...
Process p = new Process()
{
....
to
FixedProcess p = new FixedProcess()
{
....
Now your application should display something like this...
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.
C:\Projects\FixedProcess\bin\Debug>
without needing to make any other changes to your existing code. It is also still async and wrapped up nicely. The one caveat is that now you will get multiple events for large output with potential breaks in-between, so you will need to handle this scenario yourself. Other than that, it should be all good.

It seems as the problem was that the dummy app was written in c# which flushes the output automatically one every println while the 3rd party app was written in c/c++ and therefore only wrote when the stdoutbuffer was full. The only solution which ive found is to make sure the c/c++ app flushes after every print or to set its buffer to 0.

Check out this answer.
How to send input to the console as if the user is typing?
The idea is that you will receive the output received events when any is thrown after the process is started.