We Keep Coding

Importing and removing duplicates from a massive amount of text files using C# and Redis

This is a bit of a doozy and it's been a while since I worked with C#, so bear with me:
I'm running a jruby script to iterate through 900 files (5 Mb - 1500 Mb in size) to figure out how many dupes STILL exist within these (already uniq'd) files. I had little luck with awk.
My latest idea was to insert them into a local MongoDB instance like so:
db.collection('hashes').update({ :_id => hash}, { $inc: { count: 1} }, { upsert: true)
... so that later I could just query it like db.collection.where({ count: { $gt: 1 } }) to get all the dupes.
This is working great except it's been over 24 hours and at the time of writing I'm at 72,532,927 Mongo entries and growing.
I think Ruby's .each_line is bottlnecking the IO hardcore:
So what I'm thinking now is compiling a C# program which fires up a thread PER EACH FILE and inserts the line (md5 hash) into a Redis list.
From there, I could have another compiled C# program simply pop the values off and ignore the save if the count is 1.
So the questions are:
Will using a compiled file reader and multithreading the file reads significantly improve performance?
Is using Redis even necessary? With a tremendous amount of AWS memory, could I not just use the threads to fill some sort of a list atomically and proceed from there?
Thanks in advance.

Updated
New solution. Old solution. The main idea is to calculate dummy hashes(just sum of all chars in string) of each line and store it in Dictionary<ulong, List<LinePosition>> _hash2LinePositions. It's possible to have multiple hashes in the same stream and it solves by List in Dictionary Value. When the hashes are the same, we read and compare the strings from the streams. LinePosition is using for storing info about line - position in stream and its length. I don't have such huge files as you, but my tests shows that it works. Here is the full code:
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
public class Solution
{
struct LinePosition
{
public long Start;
public long Length;
public LinePosition(long start, long count)
{
Start = start;
Length = count;
}
public override string ToString()
{
return string.Format("Start: {0}, Length: {1}", Start, Length);
}
}
class TextFileHasher : IDisposable
{
readonly Dictionary<ulong, List<LinePosition>> _hash2LinePositions;
readonly Stream _stream;
bool _isDisposed;
public HashSet<ulong> Hashes { get; private set; }
public string Name { get; private set; }
public TextFileHasher(string name, Stream stream)
{
Name = name;
_stream = stream;
_hash2LinePositions = new Dictionary<ulong, List<LinePosition>>();
Hashes = new HashSet<ulong>();
}
public override string ToString()
{
return Name;
}
public void CalculateFileHash()
{
int readByte = -1;
ulong dummyLineHash = 0;
// Line start position in file
long startPosition = 0;
while ((readByte = _stream.ReadByte()) != -1) {
// Read until new line
if (readByte == '\r' || readByte == '\n') {
// If there was data
if (dummyLineHash != 0) {
// Add line hash and line position to the dict
AddToDictAndHash(dummyLineHash, startPosition, _stream.Position - 1 - startPosition);
// Reset line hash
dummyLineHash = 0;
}
}
else {
// Was it new line ?
if (dummyLineHash == 0)
startPosition = _stream.Position - 1;
// Calculate dummy hash
dummyLineHash += (uint)readByte;
}
}
if (dummyLineHash != 0) {
// Add line hash and line position to the dict
AddToDictAndHash(dummyLineHash, startPosition, _stream.Position - startPosition);
// Reset line hash
dummyLineHash = 0;
}
}
public List<LinePosition> GetLinePositions(ulong hash)
{
return _hash2LinePositions[hash];
}
public List<string> GetDuplicates()
{
List<string> duplicates = new List<string>();
foreach (var key in _hash2LinePositions.Keys) {
List<LinePosition> linesPos = _hash2LinePositions[key];
if (linesPos.Count > 1) {
duplicates.AddRange(FindExactDuplicates(linesPos));
}
}
return duplicates;
}
public void Dispose()
{
if (_isDisposed)
return;
_stream.Dispose();
_isDisposed = true;
}
private void AddToDictAndHash(ulong hash, long start, long count)
{
List<LinePosition> linesPosition;
if (!_hash2LinePositions.TryGetValue(hash, out linesPosition)) {
linesPosition = new List<LinePosition>() { new LinePosition(start, count) };
_hash2LinePositions.Add(hash, linesPosition);
}
else {
linesPosition.Add(new LinePosition(start, count));
}
Hashes.Add(hash);
}
public byte[] GetLineAsByteArray(LinePosition prevPos)
{
long len = prevPos.Length;
byte[] lineBytes = new byte[len];
_stream.Seek(prevPos.Start, SeekOrigin.Begin);
_stream.Read(lineBytes, 0, (int)len);
return lineBytes;
}
private List<string> FindExactDuplicates(List<LinePosition> linesPos)
{
List<string> duplicates = new List<string>();
linesPos.Sort((x, y) => x.Length.CompareTo(y.Length));
LinePosition prevPos = linesPos[0];
for (int i = 1; i < linesPos.Count; i++) {
if (prevPos.Length == linesPos[i].Length) {
var prevLineArray = GetLineAsByteArray(prevPos);
var thisLineArray = GetLineAsByteArray(linesPos[i]);
if (prevLineArray.SequenceEqual(thisLineArray)) {
var line = System.Text.Encoding.Default.GetString(prevLineArray);
duplicates.Add(line);
}
#if false
string prevLine = System.Text.Encoding.Default.GetString(prevLineArray);
string thisLine = System.Text.Encoding.Default.GetString(thisLineArray);
Console.WriteLine("PrevLine: {0}\r\nThisLine: {1}", prevLine, thisLine);
StringBuilder sb = new StringBuilder();
sb.Append(prevPos);
sb.Append(" is '");
sb.Append(prevLine);
sb.Append("'. ");
sb.AppendLine();
sb.Append(linesPos[i]);
sb.Append(" is '");
sb.Append(thisLine);
sb.AppendLine("'. ");
sb.Append("Equals => ");
sb.Append(prevLine.CompareTo(thisLine) == 0);
Console.WriteLine(sb.ToString());
#endif
}
else {
prevPos = linesPos[i];
}
}
return duplicates;
}
}
public static void Main(String[] args)
{
List<TextFileHasher> textFileHashers = new List<TextFileHasher>();
string text1 = "abc\r\ncba\r\nabc";
TextFileHasher tfh1 = new TextFileHasher("Text1", new MemoryStream(System.Text.Encoding.Default.GetBytes(text1)));
tfh1.CalculateFileHash();
textFileHashers.Add(tfh1);
string text2 = "def\r\ncba\r\nwet";
TextFileHasher tfh2 = new TextFileHasher("Text2", new MemoryStream(System.Text.Encoding.Default.GetBytes(text2)));
tfh2.CalculateFileHash();
textFileHashers.Add(tfh2);
string text3 = "def\r\nbla\r\nwat";
TextFileHasher tfh3 = new TextFileHasher("Text3", new MemoryStream(System.Text.Encoding.Default.GetBytes(text3)));
tfh3.CalculateFileHash();
textFileHashers.Add(tfh3);
List<string> totalDuplicates = new List<string>();
Dictionary<ulong, Dictionary<TextFileHasher, List<LinePosition>>> totalHashes = new Dictionary<ulong, Dictionary<TextFileHasher, List<LinePosition>>>();
textFileHashers.ForEach(tfh => {
foreach(var dummyHash in tfh.Hashes) {
Dictionary<TextFileHasher, List<LinePosition>> tfh2LinePositions = null;
if (!totalHashes.TryGetValue(dummyHash, out tfh2LinePositions))
totalHashes[dummyHash] = new Dictionary<TextFileHasher, List<LinePosition>>() { { tfh, tfh.GetLinePositions(dummyHash) } };
else {
List<LinePosition> linePositions = null;
if (!tfh2LinePositions.TryGetValue(tfh, out linePositions))
tfh2LinePositions[tfh] = tfh.GetLinePositions(dummyHash);
else
linePositions.AddRange(tfh.GetLinePositions(dummyHash));
}
}
});
HashSet<TextFileHasher> alreadyGotDuplicates = new HashSet<TextFileHasher>();
foreach(var hash in totalHashes.Keys) {
var tfh2LinePositions = totalHashes[hash];
var tfh = tfh2LinePositions.Keys.FirstOrDefault();
// Get duplicates in the TextFileHasher itself
if (tfh != null && !alreadyGotDuplicates.Contains(tfh)) {
totalDuplicates.AddRange(tfh.GetDuplicates());
alreadyGotDuplicates.Add(tfh);
}
if (tfh2LinePositions.Count <= 1) {
continue;
}
// Algo to get duplicates in more than 1 TextFileHashers
var tfhs = tfh2LinePositions.Keys.ToArray();
for (int i = 0; i < tfhs.Length; i++) {
var tfh1Positions = tfhs[i].GetLinePositions(hash);
for (int j = i + 1; j < tfhs.Length; j++) {
var tfh2Positions = tfhs[j].GetLinePositions(hash);
for (int k = 0; k < tfh1Positions.Count; k++) {
var tfh1Pos = tfh1Positions[k];
var tfh1ByteArray = tfhs[i].GetLineAsByteArray(tfh1Pos);
for (int m = 0; m < tfh2Positions.Count; m++) {
var tfh2Pos = tfh2Positions[m];
if (tfh1Pos.Length != tfh2Pos.Length)
continue;
var tfh2ByteArray = tfhs[j].GetLineAsByteArray(tfh2Pos);
if (tfh1ByteArray.SequenceEqual(tfh2ByteArray)) {
var line = System.Text.Encoding.Default.GetString(tfh1ByteArray);
totalDuplicates.Add(line);
}
}
}
}
}
}
Console.WriteLine();
if (totalDuplicates.Count > 0) {
Console.WriteLine("Total number of duplicates: {0}", totalDuplicates.Count);
Console.WriteLine("#######################");
totalDuplicates.ForEach(x => Console.WriteLine("{0}", x));
Console.WriteLine("#######################");
}
// Free resources
foreach (var tfh in textFileHashers)
tfh.Dispose();
}
}

If you have tons of ram... You guys are overthinking it...
var fileLines = File.ReadAllLines(#"c:\file.csv").Distinct();

Having trouble getting past NullReferenceException [duplicate]

This question already has answers here:
What is a NullReferenceException, and how do I fix it?
(27 answers)
Closed 8 years ago.
I'm a newbie in programming, especially in c#. I have written some code but I keep getting an error when running it and I can't move on until I get that fixed.
The error in question is a NullReferenceException. It also tells me "Object reference not set to an instance of an object".
It seems like a pretty clear error message indicating that an object hasn't been instantiated yet. However I thought I had done that. I hope someone can explain to me what I'm doing wrong. Here's my code.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.IO;
namespace EvenHelemaalOvernieuw
{
class Globals
{
public static int size = 50;
public static int factor = 3;
public static int puzzleNumber = 1;
public static Square[,] allSquares = new Square[Globals.factor * Globals.factor, Globals.factor * Globals.factor];
public static String path = #"" + factor.ToString() + "\\" + puzzleNumber.ToString() + ".txt";
public static int[,][,] values = new int[factor, factor][,];
public Globals() { }
public void setSize(int s)
{
size = s;
if (size > 100)
{
size = 100;
}
if (size < 20)
{
size = 20;
}
}
public void setFactor(int f)
{
factor = f;
if (factor > 5)
{
factor = 5;
}
if (factor < 2)
{
factor = 2;
}
}
public Square getSquare(int x, int y)
{
return allSquares[x, y];
}
public static void readPuzzle()
{
List<int> conversion = new List<int>();
int count = 0;
using (StreamReader codeString = new StreamReader(path))
{
String line = codeString.ReadToEnd();
Array characters = line.ToCharArray();
foreach (char a in characters)
{
if (a.ToString() != ",")
{
conversion.Add(Convert.ToInt32(a));
}
}
for (int panelX = 0; panelX < factor; panelX++)
{
for (int panelY = 0; panelY < factor; panelY++)
{
for (int squareX = 0; squareX < factor; squareX++)
{
for (int squareY = 0; squareY < factor; squareY++)
{
values[panelX, panelY][squareX, squareY] = conversion[count];
count++;
}
}
}
}
}
}
}
}
The line that is indicated by the error message is near the bottom and reads values[panelX, panelY][squareX, squareY] = conversion[count];.

The problem is the following line
public static int[,][,] values = new int[factor, factor][,];
This is an array of arrays but this code only creates the outer array. The inner array is uninitialized and will be null. Hence when the following code runs it will throw a NullReferenceException trying to access the inner array
values[panelX, panelY][squareX, squareY] = conversion[count];
To fix this just initialize the array elements right before the 3rd nested loop
values[panelX, panelY] = new int[factor, factor];
for (int squareX = 0; squareX < factor; squareX++)

C# - For vs Foreach - Huge performance difference

i was making some optimizations to an algorithm that finds the smallest number that is bigger than X, in a given array, but then a i stumbled on a strange difference. On the code bellow, the "ForeachUpper" ends in 625ms, and the "ForUpper" ends in, i believe, a few hours (insanely slower). Why so?
class Teste
{
public double Valor { get; set; }
public Teste(double d)
{
Valor = d;
}
public override string ToString()
{
return "Teste: " + Valor;
}
}
private static IEnumerable<Teste> GetTeste(double total)
{
for (int i = 1; i <= total; i++)
{
yield return new Teste(i);
}
}
static void Main(string[] args)
{
int total = 1000 * 1000*30 ;
double test = total/2+.7;
var ieTeste = GetTeste(total).ToList();
Console.WriteLine("------------");
ForeachUpper(ieTeste.Select(d=>d.Valor), test);
Console.WriteLine("------------");
ForUpper(ieTeste.Select(d => d.Valor), test);
Console.Read();
}
private static void ForUpper(IEnumerable<double> bigList, double find)
{
var start1 = DateTime.Now;
double uppper = 0;
for (int i = 0; i < bigList.Count(); i++)
{
var toMatch = bigList.ElementAt(i);
if (toMatch >= find)
{
uppper = toMatch;
break;
}
}
var end1 = (DateTime.Now - start1).TotalMilliseconds;
Console.WriteLine(end1 + " = " + uppper);
}
private static void ForeachUpper(IEnumerable<double> bigList, double find)
{
var start1 = DateTime.Now;
double upper = 0;
foreach (var toMatch in bigList)
{
if (toMatch >= find)
{
upper = toMatch;
break;
}
}
var end1 = (DateTime.Now - start1).TotalMilliseconds;
Console.WriteLine(end1 + " = " + upper);
}
Thanks

IEnumerable<T> is not indexable.
The Count() and ElementAt() extension methods that you call in every iteration of your for loop are O(n); they need to loop through the collection to find the count or the nth element.
Moral: Know thy collection types.

The reason for this difference is that your for loop will execute bigList.Count() at every iteration. This is really costly in your case, because it will execute the Select and iterate the complete result set.
Furthermore, you are using ElementAt which again executes the select and iterates it up to the index you provided.

attempting to create string of random values

I'm trying to generate a string of a random length which consists out of random chars.
To do so I have this code:
class Program
{
static void Main(string[] args)
{
for (int i = 0; i < 1000; i++)
{
MyString test = new MyString();
test.Random();
Console.WriteLine(test.Value);
}
Console.ReadLine();
}
}
public class MyString
{
private string value = string.Empty;
private Random r = new Random();
public string Value
{
get { return this.value; }
set { this.value = value; }
}
public void Random()
{
int length = (r.Next() % (100)) + 1;
for(int i = 0; i < length; i++)
{
value = value + RandomChar();
}
}
public char RandomChar()
{
// 32 to 126
int c = (r.Next() % 95) + 32;
char ch = (char)c;
return ch;
}
}
Now, lets look at a part of the output:
As you can see, the output is far from random, it contains a lot of repeating strings. How is this possible, and how do I solve it?

It looks like you are creating a new instance of the Random class every time your MyString constructor is called. The Random class probably seeds itself based on the current time (to some resolution). Random number generators seeded with the same value will generate the same pseudo-random sequence.
The solution is to construct one instance of Random and use that everywhere.

http://msdn.microsoft.com/en-us/library/system.io.path.getrandomfilename.aspx
string randomName = Path.GetRandomFileName();
randomName = randomName.Replace(".", string.Empty);
// take substring...

getting the best record from a file

I have a file with the following text inside
mimi,m,70
tata,f,60
bobo,m,100
soso,f,30
I did the reading from file thing and many many other methods and functions, but how I can get the best male name and his grade according to the grade.
here is the code I wrote. Hope it's not so long
using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
namespace practice_Ex
{
class Program
{
public static int[] ReadFile(string FileName, out string[] Name, out char[] Gender)
{
Name = new string[1];
int[] Mark = new int[1];
Gender = new char[1];
if (File.Exists(FileName))
{
FileStream Input = new FileStream(FileName, FileMode.Open, FileAccess.Read);
StreamReader SR = new StreamReader(Input);
string[] Current;
int Counter = 0;
string Str = SR.ReadLine();
while (Str != null)
{
Current = Str.Split(',');
Name[Counter] = Current[0];
Mark[Counter] = int.Parse(Current[2]);
Gender[Counter] = char.Parse(Current[1].ToUpper());
Counter++;
Array.Resize(ref Name, Counter + 1);
Array.Resize(ref Mark, Counter + 1);
Array.Resize(ref Gender, Counter + 1);
Str = SR.ReadLine();
}
}
return Mark;
}
public static int MostFreq(int[] M, out int Frequency)
{
int Counter = 0;
int Frequent = 0;
Frequency = 0;
for (int i = 0; i < M.Length; i++)
{
Counter = 0;
for (int j = 0; j < M.Length; j++)
if (M[i] == M[j])
Counter++;
if (Counter > Frequency)
{
Frequency = Counter;
Frequent = M[i];
}
}
return Frequent;
}
public static int Avg(int[] M)
{
int total = 0;
for (int i = 0; i < M.Length; i++)
total += M[i];
return total / M.Length;
}
public static int AvgCond(char[] G, int[] M, char S)
{
int total = 0;
int counter = 0;
for (int i = 0; i < G.Length; i++)
if (G[i] == S)
{
total += M[i];
counter++;
}
return total / counter;
}
public static int BelowAvg(int[] M, out int AboveAvg)
{
int Bcounter = 0;
AboveAvg = 0;
for (int i = 0; i < M.Length; i++)
{
if (M[i] < Avg(M))
Bcounter++;
else
AboveAvg++;
}
return Bcounter;
}
public static int CheckNames(string[] Name, char C)
{
C = char.Parse(C.ToString().ToLower());
int counter = 0;
string Str;
for (int i = 0; i < Name.Length - 1; i++)
{
Str = Name[i].ToLower();
if (Str[0] == C || Str[Str.Length - 1] == C)
counter++;
}
return counter;
}
public static void WriteFile(string FileName, string[] Output)
{
FileStream FS = new FileStream(FileName, FileMode.OpenOrCreate, FileAccess.Write);
StreamWriter SW = new StreamWriter(FS);
for (int i = 0; i < Output.Length; i++)
SW.WriteLine(Output[i]);
}
static void Main(string[] args)
{
int[] Mark;
char[] Gender;
string[] Name;
string[] Output = new string[8];
int Frequent, Frequency, AvgAll, MaleAvg, FemaleAvg, BelowAverage, AboveAverage, NamesCheck;
Mark = ReadFile("c:\\IUST1.txt", out Name, out Gender);
Frequent = MostFreq(Mark, out Frequency);
AvgAll = Avg(Mark);
MaleAvg = AvgCond(Gender, Mark, 'M');
FemaleAvg = AvgCond(Gender, Mark, 'F');
BelowAverage = BelowAvg(Mark, out AboveAverage);
NamesCheck = CheckNames(Name, 'T');
Output [0]= "Frequent Mark = " + Frequent.ToString();
Output [1]= "Frequency = " + Frequency.ToString();
Output [2]= "Average Of All = " + AvgAll.ToString();
Output [3]= "Average Of Males = " + MaleAvg.ToString();
Output [4]= "Average Of Females = " + FemaleAvg.ToString();
Output [5]= "Below Average = " + BelowAverage.ToString();
Output [6]= "Above Average = " + AboveAverage.ToString();
Output [7]= "Names With \"T\" = " + NamesCheck.ToString();
WriteFile("c:\\Output.txt", Output);
}
}
}

Well, I like LINQ (update: excluded via comments) for querying, especially if I can do it without buffering the data (so I can process a huge file efficiently). For example below (update: removed LINQ); note the use of iterator blocks (yield return) makes this fully "lazy" - only one record is held in memory at a time.
This also shows separation of concerns: one method deals with reading a file line by line; one method deals with parsing a line into a typed data record; one (or more) method(s) work with those data record(s).
using System;
using System.Collections.Generic;
using System.IO;
enum Gender { Male, Female, Unknown }
class Record
{
public string Name { get; set; }
public Gender Gender { get; set; }
public int Score { get; set; }
}
static class Program
{
static IEnumerable<string> ReadLines(string path)
{
using (StreamReader reader = File.OpenText(path))
{
string line;
while ((line = reader.ReadLine()) != null)
{
yield return line;
}
}
}
static IEnumerable<Record> Parse(string path)
{
foreach (string line in ReadLines(path))
{
string[] segments = line.Split(',');
Gender gender;
switch(segments[1]) {
case "m": gender = Gender.Male; break;
case "f": gender = Gender.Female; break;
default: gender = Gender.Unknown; break;
}
yield return new Record
{
Name = segments[0],
Gender = gender,
Score = int.Parse(segments[2])
};
}
}
static void Main()
{
Record best = null;
foreach (Record record in Parse("data.txt"))
{
if (record.Gender != Gender.Male) continue;
if (best == null || record.Score > best.Score)
{
best = record;
}
}
Console.WriteLine("{0}: {1}", best.Name, best.Score);
}
}
The advantage of writing things as iterators is that you can easily use either streaming or buffering - for example, you can do:
List<Record> data = new List<Record>(Parse("data.txt"));
and then manipulate data all day long (assuming it isn't too large) - useful for multiple aggregates, mutating data, etc.

This question asks how to find a maximal element by a certain criterion. Combine that with Marc's LINQ part and you're away.

In the real world, of course, these would be records in a database, and you would use one line of SQL to select the best record, ie:
SELECT Name, Score FROM Grades WHERE Score = MAX(Score)
(This returns more than one record where there's more than one best record, of course.) This is an example of the power of using the right tool for the job.

I think the fastest and least-code way would be to transform the txt to xml and then use Linq2Xml to select from it. Here's a link.
Edit: That might be more work than you'd like to do. Another option is to create a class called AcademicRecord that has properties for the persons name gender etc. Then when you read the file, add to a List for each line in the file. Then use a Sort predicate to sort the list; the highest record would then be the first one in the list. Here's a link.

Your assignment might have different requirements, but if you only want to get "best male name and grade" from a file you described, a compact way is:
public String FindRecord()
{
String[] lines = File.ReadAllLines("MyFile.csv");
Array.Sort(lines, CompareByBestMaleName);
return lines[0];
}
int SortByBestMaleName(String a, String b)
{
String[] ap = a.Split();
String[] bp = b.Split();
// Always rank male higher
if (ap[1] == "m" && bp[1] == "f") { return 1; }
if (ap[1] == "f" && bp[1] == "m") { return -1; }
// Compare by score
return int.Parse(ap[2]).CompareTo(int.Parse(bp[2]));
}
Note that this is neither fast nor robust.