We Keep Coding

Reading and writing very large text files in C#

I have a very large file, almost 2GB in size. I am trying to write a process to read the file in and write it out without the first row. I pretty much have been only able to read and write one line at a time which takes forever. I can open it, remove the first row and save it faster in TextPad, though that is still very slow.
I use this code to get the number of records in the file:
private long getNumRows(string strFileName)
{
long lngNumRows = 0;
string strMsg;
try
{
lngNumRows = 0;
using (var strReader = File.OpenText(#strFileName))
{
while (strReader.ReadLine() != null)
{
lngNumRows++;
}
strReader.Close();
strReader.Dispose();
}
}
catch (Exception excExcept)
{
strMsg = "The File could not be read: ";
strMsg += excExcept.Message;
System.Windows.MessageBox.Show(strMsg);
//Console.WriteLine("Thee was an error reading the file: ");
//Console.WriteLine(excExcept.Message);
//Console.ReadLine();
}
return lngNumRows;
}
This only takes seconds to run. When I add the following code it takes forever to run. Am I doing something wrong? Why does the write add so much time? Any ideas on how I can make this faster?
private void ProcessTextFiles(string strFileName)
{
string strDataLine;
string strFullOutputFileName;
string strSubFileName;
int intPos;
long lngTotalRows = 0;
long lngCurrNumRows = 0;
long lngModNumber = 0;
double dblProgress = 0;
double dblProgressPct = 0;
string strPrgFileName = "";
string strOutName = "";
string strMsg;
long lngFileNumRows;
try
{
using (StreamReader srStreamRdr = new StreamReader(strFileName))
{
while ((strDataLine = srStreamRdr.ReadLine()) != null)
{
lngCurrNumRows++;
if (lngCurrNumRows > 1)
{
WriteDataRow(strDataLine, strFullOutputFileName);
}
}
srStreamRdr.Dispose();
}
}
catch (Exception excExcept)
{
strMsg = "The File could not be read: ";
strMsg += excExcept.Message;
System.Windows.MessageBox.Show(strMsg);
//Console.WriteLine("The File could not be read:");
//Console.WriteLine(excExcept.Message);
}
}
public void WriteDataRow(string strDataRow, string strFullFileName)
{
//using (StreamWriter file = new StreamWriter(#strFullFileName, true, Encoding.GetEncoding("iso-8859-1")))
using (StreamWriter file = new StreamWriter(#strFullFileName, true, System.Text.Encoding.UTF8))
{
file.WriteLine(strDataRow);
file.Close();
}
}

Not sure how much this will improve the performance, but surely, opening and closing the output file for every line that you want to write is not a good idea.
Instead open both files just one time and then write the line directly
using (StreamWriter file = new StreamWriter(#strFullFileName, true, System.Text.Encoding.UTF8))
using (StreamReader srStreamRdr = new StreamReader(strFileName))
{
while ((strDataLine = srStreamRdr.ReadLine()) != null)
{
lngCurrNumRows++;
if (lngCurrNumRows > 1)
file.WriteLine(strDataRow);
}
}
You could also remove the check on lngCurrNumRow simply making an empty read before entering the while loop
strDataLine = srStreamRdr.ReadLine();
if(strDataLine != null)
{
while ((strDataLine = srStreamRdr.ReadLine()) != null)
{
file.WriteLine(strDataRow);
}
}

Depending on the memory of your machine. You could try the following (my big file was "D:\savegrp.log" I had a 2gb file knocking about) This used about 6gb memory when I tried it
int counter = File.ReadAllLines(#"D:\savegrp.log").Length;
Console.WriteLine(counter);
It does depends on the memory available..
File.WriteAllLines(#"D:\savegrp2.log",File.ReadAllLines(#"D:\savegrp.log").Skip(1));
Console.WriteLine("file saved");

How to overwrite text in same text file

I am in a fight with overwriting of a text file with some of changes using a console application. Here I am reading the file line by line. Can any one help me.
StreamReader sr = new StreamReader(#"C:\abc.txt");
string line;
line = sr.ReadLine();
while (line != null)
{
if (line.StartsWith("<"))
{
if (line.IndexOf('{') == 29)
{
string s = line;
int start = s.IndexOf("{");
int end = s.IndexOf("}");
string result = s.Substring(start+1, end - start - 1);
Guid g= Guid.NewGuid();
line = line.Replace(result, g.ToString());
File.WriteAllLines(#"C:\abc.txt", line );
}
}
Console.WriteLine(line);
line = sr.ReadLine();
}
//close the file
sr.Close();
Console.ReadLine();
Here I am getting the error file is already open by another process.
Please help me, anyone. Main task is to overwrite the same texfile with modifications

You need a single stream,
open it for both reading and writing.
FileStream fileStream = new FileStream(
#"c:\words.txt", FileMode.OpenOrCreate,
FileAccess.ReadWrite, FileShare.None);
now you can use fileStream.Read() and fileStream.Write() methods
please see this link for extended discussion
How to both read and write a file in C#

The problem is that you're trying to write to a file that is used by the StreamReader. You have to close it or - better - use the using-statement which disposes/closes it even on error.
using(StreamReader sr = new StreamReader(#"C:\abc.txt"))
{
// ...
}
File.WriteAllLines(...);
File.WriteAllLines also writes all lines to the file not only the currrent line, so it's pointless to do it in the loop.
Can i suggest you a different method to read the lines of a text-file? You can use File.ReadAllLines which reads all lines into a string[] or File.ReadLines which works similar to a StreamReader by reading all lines lazily.
Here's a version doing the same but using a ( more readable?) LINQ query:
var lines = File.ReadLines(#"C:\abc.txt")
.Where(l => l.StartsWith("<") && l.IndexOf('{') == 29)
.Select(l =>
{
int start = l.IndexOf("{");
int end = l.IndexOf("}", start);
string result = l.Substring(start + 1, end - start - 1);
Guid g = Guid.NewGuid();
return l.Replace(result, g.ToString());
}).ToList();
File.WriteAllLines(#"C:\abc.txt", lines);

Problem is that you have opened the file and reading from same file at the same time you are writing in that file. But what you should do is,
Read the changes from the file
Close the file
Write the contents back to file
So your code should be like
List<string> myAppendedList = new List<string>();
using (StreamReader sr = new StreamReader(#"C:\abc.txt"))
{
string line;
line = sr.ReadLine();
while (line != null)
{
if (line.StartsWith("<"))
{
if (line.IndexOf('{') == 29)
{
string s = line;
int start = s.IndexOf("{");
int end = s.IndexOf("}");
string result = s.Substring(start + 1, end - start - 1);
Guid g = Guid.NewGuid();
line = line.Replace(result, g.ToString());
myAppendedList.Add(line);
}
}
Console.WriteLine(line);
line = sr.ReadLine();
}
}
if(myAppendedList.Count > 0 )
File.WriteAllLines(#"C:\abc.txt", myAppendedList);

How to prevent adding an empty line to my file?

Im creating a text file and the last line is ""
private void lastRunDate()
{
String lastLine = readLastDate();
String[] date = lastLine.Split('/');
DateTime dt = new DateTime(Int32.Parse(date[2]), Int32.Parse(date[0]), Int32.Parse(date[1]));
DateTime currentDT = DateTime.Now;
argValue = 1;
if ((dt.Month == currentDT.Month) && (argValue == 0))
{
MessageBox.Show("This application has already been run this month");
this.Close();
}
}
private void AddRecordToFile()
{
DateTime now = DateTime.Now;
prepareToEmail();
string path = filepath;
bool dirtyData = true;
// This text is added only once to the file.
if (!File.Exists(path))
{
// Create a file to write to.
using (StreamWriter sw = File.CreateText(path))
{
sw.Write(now.ToShortDateString());
}
dirtyData = false;
}
if (dirtyData)
{
// This text is always added, making the file longer over time
// if it is not deleted.
using (StreamWriter sw = File.AppendText(path))
{
sw.Write(now.ToShortDateString());
}
}
}
private String readLastDate()
{
using (StreamReader sr = new StreamReader(filepath))
{
// Initialize to null so we are not stuck in loop forever in case there is nothing in the file to read
String line = null;
do
{
line = sr.ReadLine();
// Is this the end of the file?
if (line == null)
{
// Yes, so bail out of loop
return "01/01/1900"; // I had to put something
}
// Is the line empty?
if (line == String.Empty)
{
// Yes, so skip it
continue;
}
// Here you process the non-empty line
return line;
} while (true);
}
}
is what I am using to create the file (or append it)
now is a DateTime object
I used your (Karl) code to create a method called "readLastDate()"
I get the 1st date instead.

I'm probably being way to pragmatic and simple, but skip all the stream stuff and use File class directly like this...
string newLine = "";
if (!isFirstLine)
newLine = Environment.NewLine;
File.AppendAllText(
filePath,
string.Format("{0}{1}", newLine, DateTime.Now.ToString()));

You could use a sw.Write and PRE-pend a linefeed. Unfortunately that will give you an empty line at the start of the file.

Have you tried using the command .Trimend ('\n')?
http://msdn.microsoft.com/en-us/library/system.string.trimend.aspx

Do this:
sw.Write(now.ToShortDateString());
Here is the MSDN documentation for StreamWriter.WriteLine.
Here is the MSDN documentation for StreamWriter.Write.
UPDATE:
Keep using the WriteLine, but change the way you read your values in from the file:
using (StreamReader sr = new StreamReader(path))
{
// Initialize to null so we are not stuck in loop forever in case there is nothing in the file to read
String line = null;
do
{
line = sr.ReadLine();
// Is this the end of the file?
if (line == null)
{
// Yes, so bail out of loop
return;
}
// Is the line empty?
if (line == String.Empty)
{
// Yes, so skip it
continue;
}
// Here you process the non-empty line
} while (true);
}

Adding a record should be a simple matter of calling File.AppendAllText, as pointed out in another answer. Although I would recommend:
File.AppendAllText(filePath, DateTime.Now.ToString() + Environment.NewLine);
To read the last date from the file is also very easy:
string lastGoodLine = "01/01/1900";
using (StringReader sr = new StringReader(filePath))
{
while (!sr.EndOfStream)
{
string line = sr.ReadLine();
if (!string.IsNullOrEmpty(line))
lastGoodLine = line;
}
}
return lastGoodLine;

Reading a Line from text file and return back

I am developing a C# application in which I need to read a line from a text file and return back to first of line.
As file size may be too large I can't copy it into an array .
I tried this code
StreamReader str1 = new StreamReader(#"c:\file1.txt");
StreamReader str2 = new StreamReader(#"c:\file2.txt");
int a, b;
long pos1, pos2;
while (!str1.EndOfStream && !str2.EndOfStream)
{
pos1 = str1.BaseStream.Position;
pos2 = str2.BaseStream.Position;
a = Int32.Parse(str1.ReadLine());
b = Int32.Parse(str2.ReadLine());
if (a <= b)
{
Console.WriteLine("File1 ---> " + a.ToString());
str2.BaseStream.Seek(pos2, SeekOrigin.Begin);
}
else
{
Console.WriteLine("File2 ---> " + b.ToString());
str1.BaseStream.Seek(pos1, SeekOrigin.Begin);
}
}
When I debuged the program I found out str1.BaseStream.Position and str2.BaseStream.Position are same in every loop , so nothing will change.
Is there any better way ?
Thanks

You can use ReadLines for large file, it is deferred execution and does not load the whole file into memory, so you can manipulate lines in IEnumerable type:
var lines = File.ReadLines("path");
If you are in old .NET version, below is how to build ReadLines by yourself:
public IEnumerable<string> ReadLine(string path)
{
using (var streamReader = new StreamReader(path))
{
string line;
while((line = streamReader.ReadLine()) != null)
{
yield return line;
}
}
}

Another way Which I prefer to use.
Create a Function like this:
string ReadLine( Stream sr,bool goToNext)
{
if (sr.Position >= sr.Length)
return string.Empty;
char readKey;
StringBuilder strb = new StringBuilder();
long position = sr.Position;
do
{
readKey = (char)sr.ReadByte();
strb.Append(readKey);
}
while (readKey != (char)ConsoleKey.Enter && sr.Position<sr.Length);
if(!goToNext)
sr.Position = position;
return strb.ToString();
}
Then , Create a stream from file for It's argument
Stream stream = File.Open("C:\\1.txt", FileMode.Open);

Delete specific line from a text file?

I need to delete an exact line from a text file but I cannot for the life of me workout how to go about doing this.
Any suggestions or examples would be greatly appreciated?
Related Questions
Efficient way to delete a line from a text file (C#)

If the line you want to delete is based on the content of the line:
string line = null;
string line_to_delete = "the line i want to delete";
using (StreamReader reader = new StreamReader("C:\\input")) {
using (StreamWriter writer = new StreamWriter("C:\\output")) {
while ((line = reader.ReadLine()) != null) {
if (String.Compare(line, line_to_delete) == 0)
continue;
writer.WriteLine(line);
}
}
}
Or if it is based on line number:
string line = null;
int line_number = 0;
int line_to_delete = 12;
using (StreamReader reader = new StreamReader("C:\\input")) {
using (StreamWriter writer = new StreamWriter("C:\\output")) {
while ((line = reader.ReadLine()) != null) {
line_number++;
if (line_number == line_to_delete)
continue;
writer.WriteLine(line);
}
}
}

The best way to do this is to open the file in text mode, read each line with ReadLine(), and then write it to a new file with WriteLine(), skipping the one line you want to delete.
There is no generic delete-a-line-from-file function, as far as I know.

One way to do it if the file is not very big is to load all the lines into an array:
string[] lines = File.ReadAllLines("filename.txt");
string[] newLines = RemoveUnnecessaryLine(lines);
File.WriteAllLines("filename.txt", newLines);

Hope this simple and short code will help.
List linesList = File.ReadAllLines("myFile.txt").ToList();
linesList.RemoveAt(0);
File.WriteAllLines("myFile.txt"), linesList.ToArray());
OR use this
public void DeleteLinesFromFile(string strLineToDelete)
{
string strFilePath = "Provide the path of the text file";
string strSearchText = strLineToDelete;
string strOldText;
string n = "";
StreamReader sr = File.OpenText(strFilePath);
while ((strOldText = sr.ReadLine()) != null)
{
if (!strOldText.Contains(strSearchText))
{
n += strOldText + Environment.NewLine;
}
}
sr.Close();
File.WriteAllText(strFilePath, n);
}

You can actually use C# generics for this to make it real easy:
var file = new List<string>(System.IO.File.ReadAllLines("C:\\path"));
file.RemoveAt(12);
File.WriteAllLines("C:\\path", file.ToArray());

This can be done in three steps:
// 1. Read the content of the file
string[] readText = File.ReadAllLines(path);
// 2. Empty the file
File.WriteAllText(path, String.Empty);
// 3. Fill up again, but without the deleted line
using (StreamWriter writer = new StreamWriter(path))
{
foreach (string s in readText)
{
if (!s.Equals(lineToBeRemoved))
{
writer.WriteLine(s);
}
}
}

Read and remember each line
Identify the one you want to get rid
of
Forget that one
Write the rest back over the top of
the file

I cared about the file's original end line characters ("\n" or "\r\n") and wanted to maintain them in the output file (not overwrite them with what ever the current environment's char(s) are like the other answers appear to do). So I wrote my own method to read a line without removing the end line chars then used it in my DeleteLines method (I wanted the option to delete multiple lines, hence the use of a collection of line numbers to delete).
DeleteLines was implemented as a FileInfo extension and ReadLineKeepNewLineChars a StreamReader extension (but obviously you don't have to keep it that way).
public static class FileInfoExtensions
{
public static FileInfo DeleteLines(this FileInfo source, ICollection<int> lineNumbers, string targetFilePath)
{
var lineCount = 1;
using (var streamReader = new StreamReader(source.FullName))
{
using (var streamWriter = new StreamWriter(targetFilePath))
{
string line;
while ((line = streamReader.ReadLineKeepNewLineChars()) != null)
{
if (!lineNumbers.Contains(lineCount))
{
streamWriter.Write(line);
}
lineCount++;
}
}
}
return new FileInfo(targetFilePath);
}
}
public static class StreamReaderExtensions
{
private const char EndOfFile = '\uffff';
/// <summary>
/// Reads a line, similar to ReadLine method, but keeps any
/// new line characters (e.g. "\r\n" or "\n").
/// </summary>
public static string ReadLineKeepNewLineChars(this StreamReader source)
{
if (source == null)
throw new ArgumentNullException(nameof(source));
char ch = (char)source.Read();
if (ch == EndOfFile)
return null;
var sb = new StringBuilder();
while (ch != EndOfFile)
{
sb.Append(ch);
if (ch == '\n')
break;
ch = (char)source.Read();
}
return sb.ToString();
}
}

Are you on a Unix operating system?
You can do this with the "sed" stream editor. Read the man page for "sed"

What?
Use file open, seek position then stream erase line using null.
Gotch it? Simple,stream,no array that eat memory,fast.
This work on vb.. Example search line culture=id where culture are namevalue and id are value and we want to change it to culture=en
Fileopen(1, "text.ini")
dim line as string
dim currentpos as long
while true
line = lineinput(1)
dim namevalue() as string = split(line, "=")
if namevalue(0) = "line name value that i want to edit" then
currentpos = seek(1)
fileclose()
dim fs as filestream("test.ini", filemode.open)
dim sw as streamwriter(fs)
fs.seek(currentpos, seekorigin.begin)
sw.write(null)
sw.write(namevalue + "=" + newvalue)
sw.close()
fs.close()
exit while
end if
msgbox("org ternate jua bisa, no line found")
end while
that's all..use #d