We Keep Coding

C# - Split CSV File by Removing Bad Rows

I have a csv file with 2 million rows and file size of 2 GB. But due to a couple of free text form columns, these contain redundant CRLF and cause the file to not load in the SQL Server table. I get an error that the last column does not end with ".
I have the following code, but it gives an OutOfMemoryException when reading from fileName. The line is:
var lines = File.ReadAllLines(fileName);
How can I fix it? Ideally, I would like to split the file into two good and bad rows. Or delete rows that do not end with "CRLF.
int goodRow = 0;
int badRow = 0;
String badRowFileName = fileName.Substring(0, fileName.Length - 4) + "BadRow.csv";
String goodRowFileName = fileName.Substring(0, fileName.Length - 4) + "GoodRow.csv";
var charGood = "\"\"";
String lineOut = string.Empty;
String str = string.Empty;
var lines = File.ReadAllLines(fileName);
StringBuilder sbGood = new StringBuilder();
StringBuilder sbBad = new StringBuilder();
foreach (string line in lines)
{
if (line.Contains(charGood))
{
goodRow++;
sbGood.AppendLine(line);
}
else
{
badRow++;
sbBad.AppendLine(line);
}
}
if (badRow > 0)
{
File.WriteAllText(badRowFileName, sbBad.ToString());
}
if (goodRow > 0)
{
File.WriteAllText(goodRowFileName, sbGood.ToString());
}
sbGood.Clear();
sbBad.Clear();
msg = msg + "Good Rows - " + goodRow.ToString() + " Bad Rows - " + badRow.ToString() + " Done.";

You can translate that code like this to be much more efficient:
int goodRow = 0, badRow = 0;
String badRowFileName = fileName.Substring(0, fileName.Length - 4) + "BadRow.csv";
String goodRowFileName = fileName.Substring(0, fileName.Length - 4) + "GoodRow.csv";
var charGood = "\"\"";
using (var lines = File.ReadLines(fileName))
using (var swGood = new StreamWriter(goodRowFileName))
using (var swBad = new StreamWriter(badRowFileName))
{
foreach (string line in lines)
{
if (line.Contains(charGood))
{
goodRow++;
swGood.WriteLine(line);
}
else
{
badRow++;
swBad.WriteLine(line);
}
}
}
msg += $"Good Rows: {goodRow,9} Bad Rows: {badRow,9} Done.";
But I'd also look at using a real csv parser for this. There are plenty on NuGet. That might even let you clean up the data on the fly.

I would not suggest reading the entire file into memory, then processing the file, then writing all modified contents out to the new file.
Instead using file streams:
using (var rdr = new StreamReader(fileName))
using (var wrtrGood = new StreamWriter(goodRowFileName))
using (var wrtrBad = new StreamWriter(badRowFileName))
{
string line = null;
while ((line = rdr.ReadLine()) != null)
{
if (line.Contains(charGood))
{
goodRow++;
wrtr.WriteLine(line);
}
else
{
badRow++;
wrtrBad.WriteLine(line);
}
}
}

c# store textbox(file content) lines into array

Hey I am currently stuck. I have the following code that takes a line from a file, based on specific string and returns it, then it can be edited, and applied back to a textbox that contains the file content into the line that the string was initially taken from.
private void citationChange()
{
List<string> matchedList = new List<string>();
string[] linesArr = File.ReadAllLines(fileName);
//find matches
for (int a = 0; a < linesArr.Length; a++)
{
string s = linesArr[a];
if (s.Contains(citation))
{
matchedList.Add(linesArr[a]); //matched
lineBeingEdited = a;
break; //breaks the loop when a match is found
}
}
//output
foreach (string s in matchedList)
{
string citationLine = s;
string[] lineData = citationLine.Split(',');
editModuleComboBox.Text = lineData[1];
selectedModuleLabel.Text = lineData[2];
moduleTitleTextBox.Text = lineData[3];
creditsTextBox.Text = lineData[4];
semesterTextBox.Text = lineData[5];
examWeightingTextBox.Text = lineData[6];
examMarkTextBox.Text = lineData[7];
testWeightingTextBox.Text = lineData[8];
testMarkTextBox.Text = lineData[9];
courseworkWeightingTextBox.Text = lineData[10];
courseworkMarkTexbox.Text = lineData[11];
}
}
How can I recreate/change this code but for it to read a textbox instead of a file?
Thanks

Change this:
string[] linesArr = File.ReadAllLines(fileName);
to:
string[] linesArr = theTextBox.Text.Split(new string[] { Environment.NewLine }, StringSplitOptions.None);

When reading your file File.ReadAllLines basically splits the entire text by \r\n. So you could do this with your text from the textbox:
exchange this line:
string[] linesArr = File.ReadAllLines(fileName);
to this:
string[] linesArr = YourTextBox.Text.Split(new string[] { Environment.NewLine }, StringSplitOptions.None);
or this:
string[] linesArr = YourTextBox.Text.Split(new char[] {'\r', '\n'});

Reading, Manipulating, and Writing text files C#

I need to convert any flat delimited file into a pipe delimited format. I wrote this console app as a POC but the second file it tries to write will include the all of the text from the first file. Any suggestions?
string sourceDir = #"c:\temp\";
string targetDir = #"c:\dest\";
List<string> listLines = new List<string>();
string[] files = Directory.GetFiles(sourceDir);
foreach(string file in files)
{
using (StreamReader sr = new StreamReader(sourceDir + Path.GetFileName(file)))
{
do
{
listLines.Add(sr.ReadLine());
} while (!sr.EndOfStream);
for (int i = 0; i < listLines.Count; i++)
{
listLines[i] = listLines[i].Replace(',', '|');
listLines[i] = listLines[i].Replace('\t', '|');
}
}
using (StreamWriter sw = new StreamWriter(targetDir + Path.GetFileName(file)))
{
foreach (string line in listLines)
{
sw.WriteLine(line);
}
}
}

You need to either move the instantiation of listLines into the foreach, or re-initialize the list at the end of the loop.
string sourceDir = #"c:\temp\";
string targetDir = #"c:\dest\";
string[] files = Directory.GetFiles(sourceDir);
foreach(string file in files)
{
List<string> listLines = new List<string>();
using (StreamReader sr = new StreamReader(sourceDir + Path.GetFileName(file)))
{
do
{
listLines.Add(sr.ReadLine());
} while (!sr.EndOfStream);
for (int i = 0; i < listLines.Count; i++)
{
listLines[i] = listLines[i].Replace(',', '|');
listLines[i] = listLines[i].Replace('\t', '|');
}
}
using (StreamWriter sw = new StreamWriter(targetDir + Path.GetFileName(file)))
{
foreach (string line in listLines)
{
sw.WriteLine(line);
}
}
}

You're adding lines to listLines and never clearing the list after the foreach iteration.
by #Jonathan Carroll
Beside that, you can improve your code to this:
string sourceDir = #"c:\temp\";
string targetDir = #"c:\dest\";
List<string> listLines = new List<string>();
string[] files = Directory.GetFiles(sourceDir);
foreach (string file in files)
{
using (StreamReader sr = new StreamReader(sourceDir + Path.GetFileName(file)))
using (StreamWriter sw = new StreamWriter(targetDir + Path.GetFileName(file)))
{
do
{
var line = sr.ReadLine();
line = line.Replace(',', '|').Replace('\t', '|');
sw.WriteLine(line);
} while (!sr.EndOfStream);
}
}

I'd argue that the other answers are considerably clearer, but just thought I'd throw in a short alternative solution using LINQ and Regex:
foreach (var file in Directory.GetFiles(sourceDir).Select(x => Path.GetFileName(x)))
File.WriteAllText(targetDir + file, new Regex("[,\t]").Replace(File.ReadAllText(sourceDir + file), "|"));
The LINQ select query is used to transform the full paths into file names - this collection of file names is then iterated over.
Regex is used to match all ',' and '\t' characters that are read from the source file and replace them with the '|' character. This resulting string is then written to the target file.

How to create a generic text file parser for any find of text file?

Want to create a generic text file parser in c# for any find of text file.Actually i have 4 application all 4 getting input data from txt file format but text files are not homogeneous in nature.i have tried fixedwithdelemition.
private static DataTable FixedWidthDiliminatedTxtRead()
{
string[] fields;
StringBuilder sb = new StringBuilder();
List<StringBuilder> lst = new List<StringBuilder>();
DataTable dtable = new DataTable();
ArrayList aList;
using (TextFieldParser tfp = new TextFieldParser(testOCC))
{
tfp.TextFieldType = FieldType.FixedWidth;
tfp.SetFieldWidths(new int[12] { 2,25,8,12,13,5,6,3,10,11,10,24 });
for (int col = 1; col < 13; ++col)
dtable.Columns.Add("COL" + col);
while (!tfp.EndOfData)
{
fields = tfp.ReadFields();
aList = new ArrayList();
for (int i = 0; i < fields.Length; ++i)
aList.Add(fields[i] as string);
if (dtable.Columns.Count == aList.Count)
dtable.Rows.Add(aList.ToArray());
}
}
return dtable;
}
but i feel its very rigid one and really varies application to application making it configgurable .any better way ..
tfp.SetFieldWidths(new int[12] { 2,25,8,12,13,5,6,3,10,11,10,24 });
File nature :
Its a report kind of file .
position of columns are very similar
row data of file id different .
I get this as a reference
http://www.codeproject.com/Articles/11698/A-Portable-and-Efficient-Generic-Parser-for-Flat-F
any other thoughts ?

If the only thing different is the field widths, you could just try sending the field widths in as a parameter:
private static DataTable FixedWidthDiliminatedTxtRead(int[] fieldWidthArray)
{
string[] fields;
StringBuilder sb = new StringBuilder();
List<StringBuilder> lst = new List<StringBuilder>();
DataTable dtable = new DataTable();
ArrayList aList;
using (TextFieldParser tfp = new TextFieldParser(testOCC))
{
tfp.TextFieldType = FieldType.FixedWidth;
tfp.SetFieldWidths(fieldWidthArray);
for (int col = 1; col < 13; ++col)
dtable.Columns.Add("COL" + col);
while (!tfp.EndOfData)
{
fields = tfp.ReadFields();
aList = new ArrayList();
for (int i = 0; i < fields.Length; ++i)
aList.Add(fields[i] as string);
if (dtable.Columns.Count == aList.Count)
dtable.Rows.Add(aList.ToArray());
}
}
return dtable;
}
If you will have more logic to grab the data, you might want to consider defining an interface or abstract class for a GenericTextParser and create concrete implementations for each other file.

Hey I made one of these last week.
I did not write it with the intentions of other people using it so I appologize in advance if its not documented well but I cleaned it up for you. ALSO I grabbed several segments of code from stack overflow so I am not the original author of several pieces of this.
The places you need to edit are the path and pathout and the seperators of text.
char[] delimiters = new char[]
So it searches for part of a word and then grabs the whole word. I used a c# console application for this.
Here you go:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.IO;
namespace UniqueListofStringFinder
{
class Program
{
static void Main(string[] args)
{
string path = #"c:\Your Path\in.txt";
string pathOut = #"c:\Your Path\out.txt";
string data = "!";
Console.WriteLine("Current Path In is set to: " + path);
Console.WriteLine("Current Path Out is set to: " + pathOut);
Console.WriteLine(Environment.NewLine + Environment.NewLine + "Input String to Search For:");
Console.Read();
string input = Console.ReadLine();
// Delete the file if it exists.
if (!File.Exists(path))
{
// Create the file.
using (FileStream fs = File.Create(path))
{
Byte[] info =
new UTF8Encoding(true).GetBytes("This is some text in the file.");
// Add some information to the file.
fs.Write(info, 0, info.Length);
}
}
System.IO.StreamReader file = new System.IO.StreamReader(path);
List<string> Spec = new List<string>();
using (StreamReader sr = File.OpenText(path))
{
while (!file.EndOfStream)
{
string s = file.ReadLine();
if (s.Contains(input))
{
char[] delimiters = new char[] { '\r', '\n', '\t', ')', '(', ',', '=', '"', '\'', '<', '>', '$', ' ', '#', '[', ']' };
string[] parts = s.Split(delimiters,
StringSplitOptions.RemoveEmptyEntries);
foreach (string word in parts)
{
if (word.Contains(input))
{
if( word.IndexOf(input) == 0)
{
Spec.Add(word);
}
}
}
}
}
Spec.Sort();
// Open the stream and read it back.
//while ((s = sr.ReadLine()) != null)
//{
// Console.WriteLine(s);
//}
}
Console.WriteLine();
StringBuilder builder = new StringBuilder();
foreach (string s in Spec) // Loop through all strings
{
builder.Append(s).Append(Environment.NewLine); // Append string to StringBuilder
}
string result = builder.ToString(); // Get string from StringBuilder
Program a = new Program();
data = a.uniqueness(result);
int i = a.writeFile(data,pathOut);
}
public string uniqueness(string rawData )
{
if (rawData == "")
{
return "Empty Data Set";
}
List<string> dataVar = new List<string>();
List<string> holdData = new List<string>();
bool testBool = false;
using (StringReader reader = new StringReader(rawData))
{
string line;
while ((line = reader.ReadLine()) != null)
{
foreach (string s in holdData)
{
if (line == s)
{
testBool = true;
}
}
if (testBool == false)
{
holdData.Add(line);
}
testBool = false;
// Do something with the line
}
}
int i = 0;
string dataOut = "";
foreach (string s in holdData)
{
dataOut += s + "\r\n";
i++;
}
// Write the string to a file.
return dataOut;
}
public int writeFile(string dataOut, string pathOut)
{
try
{
System.IO.StreamWriter file = new System.IO.StreamWriter(pathOut);
file.WriteLine(dataOut);
file.Close();
}
catch (Exception ex)
{
dataOut += ex.ToString();
return 1;
}
return 0;
}
}
}

private static DataTable FixedWidthTxtRead(string filename, int[] fieldWidths)
{
string[] fields;
DataTable dtable = new DataTable();
ArrayList aList;
using (TextFieldParser tfp = new TextFieldParser(filename))
{
tfp.TextFieldType = FieldType.FixedWidth;
tfp.SetFieldWidths(fieldWidths);
for (int col = 1; col <= fieldWidths.length; ++col)
dtable.Columns.Add("COL" + col);
while (!tfp.EndOfData)
{
fields = tfp.ReadFields();
aList = new ArrayList();
for (int i = 0; i < fields.Length; ++i)
aList.Add(fields[i] as string);
if (dtable.Columns.Count == aList.Count) dtable.Rows.Add(aList.ToArray());
}
}
return dtable;
}

Here's what I did:
I built a factory for the type of processor needed (based on file type/format), which abstracted the file reader.
I then built a collection object that contained a set of triggers for each field I was interested in (also contained the property name for which this field is destined). This settings collection is loaded in via an XML configuration file, so all I need to change are the settings, and the base parsing process can react to how the settings are configured. Finally I built a reflection wrapper wherein once a field is parsed, the corresponding property on the model object is set.
As the file flowed through, the triggers for each setting evaluated each lines value. When it found what it was set to find (via pattern matching, or column length values) it fired and event that bubbled up and set a property on the model object. I can show some pseudo code if you're interested. It needs some work for efficiency's sake, but I like the concept.

How to make seperate lines with c# (Textfile)

I'm making this application for fun but i have a problem
I want this string/file to be read in seperate lines.
this the file(not the whole file):
1ChampSelectPack.Ahri.mp31ChampSelectPack.Akali.mp31ChampSelectPack.Alistar.mp31ChampSelectPack.Amumu.mp31ChampSelectPack.Anivia.mp31ChampSelectPack.Annie.mp31ChampSelectPack.Ashe.mp31ChampSelectPack.Blitzcrank.mp31ChampSelectPack.Brand.mp31ChampSelectPack.Caitlyn.mp3
and this is what i got so far:
List<SoundPath> paths = new List<SoundPath>();
StreamReader reader = File.OpenText("C:/Users/Esat/Documents/Visual Studio 2010/Projects/WikiLoL/WikiLoL/lolSoundBoard/1ChampSelectPack/files.txt");
while (!reader.EndOfStream)
{
SoundPath path = new SoundPath();
path.Path = reader.ReadLine();
paths.Add(path);
}
reader.Close();
return paths;

Not sure if that is what you want:
"YourString".Split(new string[] {"mp3"}, StringSplitOptions.None)
You would have to append the "mp3" on each line afterwards.

You can do it using splitting on .mp3 and adding .mp3 in each element of resultant array.
string text = File.ReadAllText("C:/Users/Esat/Documents/Visual Studio 2010/Projects/WikiLoL/WikiLoL/lolSoundBoard/1ChampSelectPack/files.txt");
string[] lines = text.Split(new string[] { ".mp3" }, StringSplitOptions.RemoveEmptyEntries);
for (int i = 0; i < lines.Length; i++)
lines[i] = lines[i] + ".mp3";