We Keep Coding

Streamreader reading the same line multiple times C#

My old method (other than being wrong in general) takes too long to get multiple lines from a file and then store the parameters into a dictionary.
Essentially it's open file, grab every second line one at a time, modify the line then store the data (line pos and the first element of the line (minus) ">") close the file and then repeat.
for (int i = 0; i < linecount - 1; i += 2)
{
string currentline = File.ReadLines
(datafile).Skip(i).Take(1).First();
string[] splitline = currentline.Split(' ');
string filenumber = splitline[0].Trim('>');
} for (int i = 0; i < linecount - 1; i += 2)

You need to read next line inside while loop, otherwise loop body will always analyse first line (that's why there are Dictionary error) and never exist:
while (line != null)
{
// your current code here
line = sr.ReadLine();
}

The issue is that you only ever read the first line of the file. To solve this you need to ensure you call sr.ReadLine() on every iteration through the loop. This would look like:
using (StreamReader sr = File.OpenText(datafile))
{
string line = sr.ReadLine();
while (line != null)
{
count = count ++;
if (count % 2 == 0)
{
string[] splitline = line.Split(' ');
string datanumber = splitline[0].Trim('>');
index.Add(datanumber, count);
}
line = sr.ReadLine();
}
}
This means on each iteration, the value of line will be a new value (from the next line of the file).

C# Console Word Wrap

I have a string with newline characters and I want to wrap the words. I want to keep the newline characters so that when I display the text it looks like separate paragraphs. Anyone have a good function to do this? Current function and code below.(not my own function). The WordWrap function seems to be stripping out \n characters.
static void Main(string[] args){
StreamReader streamReader = new StreamReader("E:/Adventure Story/Intro.txt");
string intro = "";
string line;
while ((line = streamReader.ReadLine()) != null)
{
intro += line;
if(line == "")
{
intro += "\n\n";
}
}
WordWrap(intro);
public static void WordWrap(string paragraph)
{
paragraph = new Regex(#" {2,}").Replace(paragraph.Trim(), #" ");
var left = Console.CursorLeft; var top = Console.CursorTop; var lines = new List<string>();
for (var i = 0; paragraph.Length > 0; i++)
{
lines.Add(paragraph.Substring(0, Math.Min(Console.WindowWidth, paragraph.Length)));
var length = lines[i].LastIndexOf(" ", StringComparison.Ordinal);
if (length > 0) lines[i] = lines[i].Remove(length);
paragraph = paragraph.Substring(Math.Min(lines[i].Length + 1, paragraph.Length));
Console.SetCursorPosition(left, top + i); Console.WriteLine(lines[i]);
}
}

Here is a word wrap function that works by using regular expressions to find the places that it's ok to break and places where it must break. Then it returns pieces of the original text based on the "break zones". It even allows for breaks at hyphens (and other characters) without removing the hyphens (since the regex uses a zero-width positive lookbehind assertion).
IEnumerable<string> WordWrap(string text, int width)
{
const string forcedBreakZonePattern = #"\n";
const string normalBreakZonePattern = #"\s+|(?<=[-,.;])|$";
var forcedZones = Regex.Matches(text, forcedBreakZonePattern).Cast<Match>().ToList();
var normalZones = Regex.Matches(text, normalBreakZonePattern).Cast<Match>().ToList();
int start = 0;
while (start < text.Length)
{
var zone =
forcedZones.Find(z => z.Index >= start && z.Index <= start + width) ??
normalZones.FindLast(z => z.Index >= start && z.Index <= start + width);
if (zone == null)
{
yield return text.Substring(start, width);
start += width;
}
else
{
yield return text.Substring(start, zone.Index - start);
start = zone.Index + zone.Length;
}
}
}

If you want another newline to make text look-like paragraphs, just use Replace method of your String object.
var str =
"Line 1\n" +
"Line 2\n" +
"Line 3\n";
Console.WriteLine("Before:\n" + str);
str = str.Replace("\n", "\n\n");
Console.WriteLine("After:\n" + str);

Recently I've been working on creating some abstractions that imitate window-like features in a performance- and memory-sensitive console context.
To this end I had to implement word-wrapping functionality without any unnecessary string allocations.
The following is what I managed to simplify it into. This method:
preserves new-lines in the input string,
allows you to specify what characters it should break on (space, hyphen, etc.),
returns the start indices and lengths of the lines via Microsoft.Extensions.Primitives.StringSegment struct instances (but it's very simple to replace this struct with your own, or append directly to a StringBuilder).
public static IEnumerable<StringSegment> WordWrap(string input, int maxLineLength, char[] breakableCharacters)
{
int lastBreakIndex = 0;
while (true)
{
var nextForcedLineBreak = lastBreakIndex + maxLineLength;
// If the remainder is shorter than the allowed line-length, return the remainder. Short-circuits instantly for strings shorter than line-length.
if (nextForcedLineBreak >= input.Length)
{
yield return new StringSegment(input, lastBreakIndex, input.Length - lastBreakIndex);
yield break;
}
// If there are native new lines before the next forced break position, use the last native new line as the starting position of our next line.
int nativeNewlineIndex = input.LastIndexOf(Environment.NewLine, nextForcedLineBreak, maxLineLength);
if (nativeNewlineIndex > -1)
{
nextForcedLineBreak = nativeNewlineIndex + Environment.NewLine.Length + maxLineLength;
}
// Find the last breakable point preceding the next forced break position (and include the breakable character, which might be a hypen).
var nextBreakIndex = input.LastIndexOfAny(breakableCharacters, nextForcedLineBreak, maxLineLength) + 1;
// If there is no breakable point, which means a word is longer than line length, force-break it.
if (nextBreakIndex == 0)
{
nextBreakIndex = nextForcedLineBreak;
}
yield return new StringSegment(input, lastBreakIndex, nextBreakIndex - lastBreakIndex);
lastBreakIndex = nextBreakIndex;
}
}

c# how to store values from a file in an array [duplicate]

This question already has answers here:
C# parsing a text file and storing the values in an array
(3 answers)
Closed 5 years ago.
I am trying to store values in an array from reading from a file. I have the reading from a file part but I can't get it to store in an array because it gives me an error "Value cannot be null" because after the loop the value of my variable becomes null and the array cannot be null. Here's what I have. And I realize that the for loop probably isn't in the correct spot so any help with where to put it would be great.
Program p = new Program();
int MAX = 50;
int[] grades = new int[MAX];
string environment = System.Environment.GetFolderPath(System.Environment.SpecialFolder.Personal) + "\\";
string path = environment + "grades.txt";
StreamReader myFile = new StreamReader(path);
string input;
int count = 0;
do
{
input = myFile.ReadLine();
if (input != null)
{
WriteLine(input);
count++;
}
} while (input != null);
for (int i = 0; i < count; i++)
{
grades[i] = int.Parse(input);
}

You start the for loop just after exiting from the while loop. And the condition to exit from the while loop is true when input is null. Of course this is not well accepted by Int.Parse.
Instead you can use a single loop, taking in consideration that you don't want to loop more than 50 times otherwise you exceed the array dimensions
int count = 0;
while((input = myFile.ReadLine()) != null && count < 50)
{
WriteLine(input);
grades[count] = int.Parse(input);
count++;
}
However you can have a more flexible way to handle your input if you use a List<int> instead of an array of integers. In this way you don't have to check for the number of lines present in your file
List<int> grades = new List<int>();
while((input = myFile.ReadLine()) != null)
grades.Add(int.Parse(input));

if we want to get really condensed
var grades = File.ReadAllLines(path).Select(l=>Int.Parse(l)).ToArray();

Utilize the Path.Combine() to help you in concatenating paths.
string environment = System.Environment.GetFolderPath(System.Environment.SpecialFolder.Personal);
String fullPath = Path.Combine(environment, "grades.txt");
int[] grades = File.ReadAllLines(fullPath).Select(p => int.Parse(p)).ToArray<int>();
Console.WriteLine(grades);
Refer to https://www.dotnetperls.com/file-readalllines on how to use File.ReadAllLines() its very handy.
I'm using LINQ here, which sometimes simplifies things. Even though it looks a bit intimidating now. We read all lines, the result of that is then parsed by selecting each one and converting it to an integer then outputting an array of integers and saving that to grades.

Program p = new Program();
int MAX = 50;
int[] grades = new int[MAX];
string environment = System.Environment.GetFolderPath(System.Environment.SpecialFolder.Personal) + "\\";
string path = environment + "grades.txt";
using (StreamReader myFile = new StreamReader(path))
{
string input;
int count = 0;
while((!myFile.EndOfStream) && (count < MAX))
{
input = myFile.ReadLine();
if (!String.IsNullOrWhiteSpace(input))
{
WriteLine(input);
grades[count] = int.Parse(input);
count++;
}
}
}
You should definitely use the "using" pattern around your stream object. Got rid of the for-loop for you while maintaining mostly your code and style. Your issue was that you weren't using the input value before moving on to the next line. You only ever had the last value in your original code.

String array throws OutOfMemoryException for large multi-line entries

In a Windows Forms C# app, I have a textbox where users paste log data, and it sorts it. I need to check each line individualy so I split the input by the new line, but if there are a lot of lines, greater than 100,000 or so, it throws a OutOfMemoryException.
My code looks like this:
StringSplitOptions splitOptions = new StringSplitOptions();
if(removeEmptyLines_CB.Checked)
splitOptions = StringSplitOptions.RemoveEmptyEntries;
else
splitOptions = StringSplitOptions.None;
List<string> outputLines = new List<string>();
foreach(string line in input_TB.Text.Split(new string[] { "\r\n", "\n" }, splitOptions))
{
if(line.Contains(inputCompare_TB.Text))
outputLines.Add(line);
}
output_TB.Text = string.Join(Environment.NewLine, outputLines);
The problem comes from when I split the textbox text by line, here input_TB.Text.Split(new string[] { "\r\n", "\n" }
Is there a better way to do this? I've thought about taking the first X amount of text, truncating at a new line and repeat until everything has been read, but this seems tedious. Or is there a way to allocate more memory for it?
Thanks,
Garrett
Update
Thanks to Attila, I came up with this and it seems to work. Thanks
StringReader reader = new StringReader(input_TB.Text);
string line;
while((line = reader.ReadLine()) != null)
{
if(line.Contains(inputCompare_TB.Text))
outputLines.Add(line);
}
output_TB.Text = string.Join(Environment.NewLine, outputLines);

The better way to do this would be to extract and process one line at a time, and use a StringBuilder to create the result:
StringBuilder outputTxt = new StringBuilder();
string txt = input_TB.Text;
int txtIndex = 0;
while (txtIndex < txt.Length) {
int startLineIndex = txtIndex;
GetMore:
while (txtIndex < txt.Length && txt[txtIndex] != '\r' && txt[txtIndex] != '\n')) {
txtIndex++;
}
if (txtIndex < txt.Length && txt[txtIndex] == '\r' && (txtIndex == txt.Length-1 || txt[txtIndex+1] != '\n') {
txtIndex++;
goto GetMore;
}
string line = txt.Substring(startLineIndex, txtIndex-startLineIndex);
if (line.Contains(inputCompare_TB.Text)) {
if (outputTxt.Length > 0)
outputTxt.Append(Environment.NewLine);
outputTxt.Append(line);
}
txtIndex++;
}
output_TB.Text = outputTxt.ToString();
Pre-emptive comment: someone will object to the goto - but it is what's needed here, the alternatives are much more complex (reg exp for example), or fake the goto with another loop and continue or break
Using a StringReader to split the lines is a much cleaner solution, but it does not handle both \r\n and \n as a new line:
StringReader reader = new StringReader(input_TB.Text);
StringBuilder outputTxt = new StringBuilder();
string compareTxt = inputCompare_TB.Text;
string line;
while((line = reader.ReadLine()) != null) {
if (line.Contains(compareTxt)) {
if (outputTxt.Length > 0)
outputTxt.Append(Environment.NewLine);
outputTxt.Append(line);
}
}
output_TB.Text = outputTxt.ToString();

Split will have to duplicate the memory need of the original text, plus overhead of string objects for each line. If this causes memory issues, a reliable way of processing the input is to parse one line at a time.

I guess the only way to do this on large text files is to open the file manually and use a StreamReader. Here is an example how to do this.

You can avoid creating strings for all lines and the array by creating the string for each line one at a time:
var eol = new[] { '\r', '\n' };
var pos = 0;
while (pos < input.Length)
{
var i = input.IndexOfAny(eol, pos);
if (i < 0)
{
i = input.Length;
}
if (i != pos)
{
var line = input.Substring(pos, i - pos);
// process line
}
pos = i + 1;
}

On other hand, In this article say that the point is that "split" method is implemented poorly. Read it, and make your conclusions.
Like Attila said, you have to parse line by line.

How to read last "n" lines of log file [duplicate]

This question already has answers here:
Get last 10 lines of very large text file > 10GB
(21 answers)
Closed 1 year ago.
need a snippet of code which would read out last "n lines" of a log file. I came up with the following code from the net.I am kinda new to C sharp. Since the log file might be
quite large, I want to avoid overhead of reading the entire file.Can someone suggest any performance enhancement. I do not really want to read each character and change position.
var reader = new StreamReader(filePath, Encoding.ASCII);
reader.BaseStream.Seek(0, SeekOrigin.End);
var count = 0;
while (count <= tailCount)
{
if (reader.BaseStream.Position <= 0) break;
reader.BaseStream.Position--;
int c = reader.Read();
if (reader.BaseStream.Position <= 0) break;
reader.BaseStream.Position--;
if (c == '\n')
{
++count;
}
}
var str = reader.ReadToEnd();

Your code will perform very poorly, since you aren't allowing any caching to happen.
In addition, it will not work at all for Unicode.
I wrote the following implementation:
///<summary>Returns the end of a text reader.</summary>
///<param name="reader">The reader to read from.</param>
///<param name="lineCount">The number of lines to return.</param>
///<returns>The last lneCount lines from the reader.</returns>
public static string[] Tail(this TextReader reader, int lineCount) {
var buffer = new List<string>(lineCount);
string line;
for (int i = 0; i < lineCount; i++) {
line = reader.ReadLine();
if (line == null) return buffer.ToArray();
buffer.Add(line);
}
int lastLine = lineCount - 1; //The index of the last line read from the buffer. Everything > this index was read earlier than everything <= this indes
while (null != (line = reader.ReadLine())) {
lastLine++;
if (lastLine == lineCount) lastLine = 0;
buffer[lastLine] = line;
}
if (lastLine == lineCount - 1) return buffer.ToArray();
var retVal = new string[lineCount];
buffer.CopyTo(lastLine + 1, retVal, 0, lineCount - lastLine - 1);
buffer.CopyTo(0, retVal, lineCount - lastLine - 1, lastLine + 1);
return retVal;
}

Had trouble with your code. This is my version. Since its' a log file, something might be writing to it, so it's best making sure you're not locking it.
You go to the end. Start reading backwards until you reach n lines. Then read everything from there on.
int n = 5; //or any arbitrary number
int count = 0;
string content;
byte[] buffer = new byte[1];
using (FileStream fs = new FileStream("text.txt", FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
// read to the end.
fs.Seek(0, SeekOrigin.End);
// read backwards 'n' lines
while (count < n)
{
fs.Seek(-1, SeekOrigin.Current);
fs.Read(buffer, 0, 1);
if (buffer[0] == '\n')
{
count++;
}
fs.Seek(-1, SeekOrigin.Current); // fs.Read(...) advances the position, so we need to go back again
}
fs.Seek(1, SeekOrigin.Current); // go past the last '\n'
// read the last n lines
using (StreamReader sr = new StreamReader(fs))
{
content = sr.ReadToEnd();
}
}

A friend of mine uses this method (BackwardReader can be found here):
public static IList<string> GetLogTail(string logname, string numrows)
{
int lineCnt = 1;
List<string> lines = new List<string>();
int maxLines;
if (!int.TryParse(numrows, out maxLines))
{
maxLines = 100;
}
string logFile = HttpContext.Current.Server.MapPath("~/" + logname);
BackwardReader br = new BackwardReader(logFile);
while (!br.SOF)
{
string line = br.Readline();
lines.Add(line + System.Environment.NewLine);
if (lineCnt == maxLines) break;
lineCnt++;
}
lines.Reverse();
return lines;
}

Does your log have lines of similar length? If yes, then you can calculate average length of the line, then do the following:
seek to end_of_file - lines_needed*avg_line_length (previous_point)
read everything up to the end
if you grabbed enough lines, that's fine. If no, seek to previous_point - lines_needed*avg_line_length
read everything up to previous_point
goto 3
memory-mapped file is also a good method -- map the tail of file, calculate lines, map the previous block, calculate lines etc. until you get the number of lines needed

Here is my answer:-
private string StatisticsFile = #"c:\yourfilename.txt";
// Read last lines of a file....
public IList<string> ReadLastLines(int nFromLine, int nNoLines, out bool bMore)
{
// Initialise more
bMore = false;
try
{
char[] buffer = null;
//lock (strMessages) Lock something if you need to....
{
if (File.Exists(StatisticsFile))
{
// Open file
using (StreamReader sr = new StreamReader(StatisticsFile))
{
long FileLength = sr.BaseStream.Length;
int c, linescount = 0;
long pos = FileLength - 1;
long PreviousReturn = FileLength;
// Process file
while (pos >= 0 && linescount < nFromLine + nNoLines) // Until found correct place
{
// Read a character from the end
c = BufferedGetCharBackwards(sr, pos);
if (c == Convert.ToInt32('\n'))
{
// Found return character
if (++linescount == nFromLine)
// Found last place
PreviousReturn = pos + 1; // Read to here
}
// Previous char
pos--;
}
pos++;
// Create buffer
buffer = new char[PreviousReturn - pos];
sr.DiscardBufferedData();
// Read all our chars
sr.BaseStream.Seek(pos, SeekOrigin.Begin);
sr.Read(buffer, (int)0, (int)(PreviousReturn - pos));
sr.Close();
// Store if more lines available
if (pos > 0)
// Is there more?
bMore = true;
}
if (buffer != null)
{
// Get data
string strResult = new string(buffer);
strResult = strResult.Replace("\r", "");
// Store in List
List<string> strSort = new List<string>(strResult.Split('\n'));
// Reverse order
strSort.Reverse();
return strSort;
}
}
}
}
catch (Exception ex)
{
System.Diagnostics.Debug.WriteLine("ReadLastLines Exception:" + ex.ToString());
}
// Lets return a list with no entries
return new List<string>();
}
const int CACHE_BUFFER_SIZE = 1024;
private long ncachestartbuffer = -1;
private char[] cachebuffer = null;
// Cache the file....
private int BufferedGetCharBackwards(StreamReader sr, long iPosFromBegin)
{
// Check for error
if (iPosFromBegin < 0 || iPosFromBegin >= sr.BaseStream.Length)
return -1;
// See if we have the character already
if (ncachestartbuffer >= 0 && ncachestartbuffer <= iPosFromBegin && ncachestartbuffer + cachebuffer.Length > iPosFromBegin)
{
return cachebuffer[iPosFromBegin - ncachestartbuffer];
}
// Load into cache
ncachestartbuffer = (int)Math.Max(0, iPosFromBegin - CACHE_BUFFER_SIZE + 1);
int nLength = (int)Math.Min(CACHE_BUFFER_SIZE, sr.BaseStream.Length - ncachestartbuffer);
cachebuffer = new char[nLength];
sr.DiscardBufferedData();
sr.BaseStream.Seek(ncachestartbuffer, SeekOrigin.Begin);
sr.Read(cachebuffer, (int)0, (int)nLength);
return BufferedGetCharBackwards(sr, iPosFromBegin);
}
Note:-
Call ReadLastLines with nLineFrom starting at 0 for the last line and nNoLines as the number of lines to read back from.
It reverses the list so the 1st one is the last line in the file.
bMore returns true if there are more lines to read.
It caches the data in 1024 char chunks - so it is fast, you may want to increase this size for very large files.
Enjoy!

This is in no way optimal but for quick and dirty checks with small log files I've been using something like this:
List<string> mostRecentLines = File.ReadLines(filePath)
// .Where(....)
// .Distinct()
.Reverse()
.Take(10)
.ToList()

Something that you can now do very easily in C# 4.0 (and with just a tiny bit of effort in earlier versions) is use memory mapped files for this type of operation. Its ideal for large files because you can map just a portion of the file, then access it as virtual memory.
There is a good example here.

As #EugeneMayevski stated above, if you just need an approximate number of lines returned, each line has roughly the same line length and you're more concerned with performance especially for large files, this is a better implementation:
internal static StringBuilder ReadApproxLastNLines(string filePath, int approxLinesToRead, int approxLengthPerLine)
{
//If each line is more or less of the same length and you don't really care if you get back exactly the last n
using (FileStream fs = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
var totalCharsToRead = approxLengthPerLine * approxLinesToRead;
var buffer = new byte[1];
//read approx chars to read backwards from end
fs.Seek(totalCharsToRead > fs.Length ? -fs.Length : -totalCharsToRead, SeekOrigin.End);
while (buffer[0] != '\n' && fs.Position > 0) //find new line char
{
fs.Read(buffer, 0, 1);
}
var returnStringBuilder = new StringBuilder();
using (StreamReader sr = new StreamReader(fs))
{
returnStringBuilder.Append(sr.ReadToEnd());
}
return returnStringBuilder;
}
}

Most log files have a DateTime stamp. Although can be improved, the code below works well if you want the log messages from the last N days.
/// <summary>
/// Returns list of entries from the last N days.
/// </summary>
/// <param name="N"></param>
/// <param name="cSEP">field separator, default is TAB</param>
/// <param name="indexOfDateColumn">default is 0; change if it is not the first item in each line</param>
/// <param name="bFileHasHeaderRow"> if true, it will not include the header row</param>
/// <returns></returns>
public List<string> ReadMessagesFromLastNDays(int N, char cSEP ='\t', int indexOfDateColumn = 0, bool bFileHasHeaderRow = true)
{
List<string> listRet = new List<string>();
//--- replace msFileName with the name (incl. path if appropriate)
string[] lines = File.ReadAllLines(msFileName);
if (lines.Length > 0)
{
DateTime dtm = DateTime.Now.AddDays(-N);
string sCheckDate = GetTimeStamp(dtm);
//--- process lines in reverse
int iMin = bFileHasHeaderRow ? 1 : 0;
for (int i = lines.Length - 1; i >= iMin; i--) //skip the header in line 0, if any
{
if (lines[i].Length > 0) //skip empty lines
{
string[] s = lines[i].Split(cSEP);
//--- s[indexOfDateColumn] contains the DateTime stamp in the log file
if (string.Compare(s[indexOfDateColumn], sCheckDate) >= 0)
{
//--- insert at top of list or they'd be in reverse chronological order
listRet.Insert(0, s[1]);
}
else
{
break; //out of loop
}
}
}
}
return listRet;
}
/// <summary>
/// Returns DateTime Stamp as formatted in the log file
/// </summary>
/// <param name="dtm">DateTime value</param>
/// <returns></returns>
private string GetTimeStamp(DateTime dtm)
{
// adjust format string to match what you use
return dtm.ToString("u");
}