We Keep Coding

How to search string in large text file?

I want to get the line containing a certain word that cannot be repeated like profile ID without make loop to read each of line separately, Because if the word I am looking for is in the last line of the text file, this will take a lot of time to get it, and if the search process is for more than one word and extract the line that contains it, I think it will take a lot of time.
Example for line text file
name,id,image,age,place,link
string word = "13215646";
string output = string.Empty;
using (var fileStream = File.OpenRead(FileName))
using (var streamReader = new StreamReader(fileStream, Encoding.UTF8))
{
String line;
while ((line = streamReader.ReadLine()) != null)
{
string[] strList = line.Split(',');
if (word == strList[1]) // check if word = id
{
output = line;
break;
}
}
}

You can use this to search the file:
var output = File.ReadLines(FileName).
Where(line => line.Split(',')[1] == word).
FirstOrDefault();
But it won't solve this:
if the word I am looking for is in the last line of the text file, this will take a lot of time to get it, and if the search process is for more than one word and extract the line that contains it, I think it will take a lot of time.
There's not a practical way to avoid this for a basic file.
The only ways around actually reading through the file is either maintaining an index, which requires absolute control over everything that might write into the file, or if you can guarantee the file is already sorted by the columns that matter, in which case you can do something like a binary search.
But neither is likely for a random csv file. This is one of the reasons people use databases.
However, we also need to stop and check whether this is really a problem for you. I'd expect the code above to handle files up to a couple hundred MB in around 1 to 2 seconds on modern hardware, even if you need to look through the whole file.

You can optimise the code. Here are few ideas:
var ids = new ["13215646", "113"];
foreach(var line in File.ReadLines(FileName))
{
var id = line.Split(',', count: 3)[1]; // Optimization 1: Use: `count: 3`
if(ids.Contains(id) // Optimization 2: Search for multiple ids
{
//Do what you need with the line
}
}

How do I separate each line of a .csv file into a string list>

I am new to c# and am attempting to read in a .csv file and put each line of text in to a separate list item so I can sort it later.
the .csv file is organised like so:
1;"final60";"United Kingdom";"2013-12-06 15:48:16";
2;"donnyr8";"Netherlands";"2013-12-06 15:54:32";
etc
This is my first attempt that doesn't work.It shows no errors in Visual studios 2010 but when I run the console program it displays the following Exception instead of the list.
Exception of type 'System.OutOFMemoryException' was thrown. Which is bizarre because the .csv file only contains a small list.
try
{
// load csv file
using (StreamReader file = new StreamReader("file.csv"))
{
string line = file.ReadLine();
List<string> fileList = new List<string>();
// Do something with the lines from the file until the end of
// the file is reached.
while (line != null)
{
fileList.Add(line);
}
foreach (string fileListLine in fileList)
{
Console.WriteLine(fileListLine);
}
}
}
catch (Exception e)
{
// Let the user know what went wrong.
Console.WriteLine("The file could not be read:");
Console.WriteLine(e.Message);
}
So am I approaching this the correct way?

If the file you are loading isn't really big then you can use File.ReadAllLines:
List<string> list = File.ReadAllLines("file.csv").ToList();
As Servy pointed out in comment it would be better to use File.ReadLines method.
File.ReadLines - MSDN
The ReadLines and ReadAllLines methods differ as follows: When you use
ReadLines, you can start enumerating the collection of strings before
the whole collection is returned; when you use ReadAllLines, you must
wait for the whole array of strings be returned before you can access
the array. Therefore, when you are working with very large files,
ReadLines can be more efficient.
If you need a List<string> then you can do:
List<string> list = File.ReadLines("file.csv").ToList();

You are not updating the line variable so the line will be always different from null infinite loop which cause OutOfMemoryException
try
{
// load csv file
using (StreamReader file = new StreamReader("file.csv"))
{
string line = file.ReadLine();
List<string> fileList = new List<string>();
// Do something with the lines from the file until the end of
// the file is reached.
while (line != null)
{
fileList.Add(line);
line = file.ReadLine();
}
foreach (string fileListLine in fileList)
{
Console.WriteLine(fileListLine);
}
}
}
but the correct approaches will be
List<string> list = File.ReadLines("file.csv").ToList();
which is better than File.ReadAllLines for the following reason
From MSDN:
When you use ReadLines, you can start enumerating the collection of strings before the whole collection is returned;

You should use File.ReadAllLines() and then parse the strings in the array.
For extremely large files this might not be feasible and you'll have to stream the single lines in and process them one by one.
But this is something you can only decide AFTER you have seen this quick approach failing miserably. Until then, stick to the quick and dirty.

c# write an arraylist to a text file

I need to write my array list into a text file and so far have come up with this code.
Now im confused as to how to write the 'line' to my text file using the textwriter?
One procedure loads the list out of a txt file below.
public void LoadArrayList()
{
TextReader tr;
tr = File.OpenText("C:\\Users\\Mirro\\Documents\\Visual Studio 2010\\Projects\\Assessment2\\Assessment2\\act\\actors.txt");
string line = tr.ReadToEnd();
Console.WriteLine(line);
if (line != null)
{
ActorArrayList.Add(line);
}
else
tr.Close();
}
Then i have one that will populate the combo box in my form.
public void PopulateActors()
{
cboActor.Items.Clear();
foreach (string line in ActorArrayList)
{
cboActor.Items.AddRange(File.ReadAllLines("C:\\Users\\Mirro\\Documents\\Visual Studio 2010\\Projects\\Assessment2\\Assessment2\\act\\actors.txt"));
}
}
and this procedure i need it to write my whole array "ActoryArrayList" into the text file.
public void WriteArrayList()
{
}
Im sorry for being confusing originally.

Try with following code
// Example #1: Write an array of strings to a file.
// Create a string array that consists of three lines.
string[] lines = { "First line", "Second line", "Third line" };
// WriteAllLines creates a file, writes a collection of strings to the file,
// and then closes the file.
System.IO.File.WriteAllLines(#"C:\Users\Mirro\Documents\Visual Studio 2010\Projects\Assessment2\Assessment2\act\actors.txt", lines);
OUTPUT :
// First line
// Second line
// Third line

The best way is #Leez's way, but You also may use TextWriter and foreach operator to make this:
//your array
string[] yourArray = { "fisrt", "second", "third" };
string text = "C:\\Users\\Mirro\\Documents\\Visual Studio 2010\\Projects\\Assessment2\\Assessment2\\act\\actors.txt";
using (TextWriter writer = File.CreateText(text))
{
foreach (string i in yourArray)
{
writer.WriteLine(i);
}
}

System.IO.File.WriteAllText("FILE_PATH", line);
BTW, where is the ArrayList in your code? Also, consider using System.IO.File.ReadAllText("FILE_PATH") for everyday file reading.
If you were to actually write an ArrayList to a disk file, that would require you to first serialize the contents of the ArrayList to maybe XML or binary etc. Then you can use the above methods to write that serialized representation to a file. Also note that serializing collections involves a concept called deep and shallow copying. This question may help you better understand the concept.

File.WriteAllLines(filePath, ActorArrayList.ToArray());

WriteAllLines outputs two end of line characters (carriage return and line feed - \r\n). If you don't want two end of line characters at the end of each line (\r\n), you can output only one character (\n) by using StreamWriter.
using (StreamWriter sw = new StreamWriter(#"C:\mypath\file.txt"))
{
foreach (string s in linesArray)
sw.Write(s + "\n");
}

What's the best way to get all the content in between two tagged lines of a file so that you can deserialize it?

I've been noticing that the following segment of code does not scale well for large files (I think that appending to the paneContent string is slow):
string paneContent = String.Empty;
bool lineFound = false;
foreach (string line in File.ReadAllLines(path))
{
if (line.Contains(tag))
{
lineFound = !lineFound;
}
else
{
if (lineFound)
{
paneContent += line;
}
}
}
using (TextReader reader = new StringReader(paneContent))
{
data = (PaneData)(serializer.Deserialize(reader));
}
What's the best way to speed this all up? I have a file that looks like this (so I wanna get all the content in between the two different tags and then deserialize all that content):
A line with some tag
A line with content I want to get into a single stream or string
A line with content I want to get into a single stream or string
A line with content I want to get into a single stream or string
A line with content I want to get into a single stream or string
A line with content I want to get into a single stream or string
A line with some tag
Note: These tags are not XML tags.

You could use a StringBuilder as opposed to a string, that is what the StringBuilder is for. Some example code is below:
var paneContent = new StringBuilder();
bool lineFound = false;
foreach (string line in File.ReadLines(path))
{
if (line.Contains(tag))
{
lineFound = !lineFound;
}
else
{
if (lineFound)
{
paneContent.Append(line);
}
}
}
using (TextReader reader = new StringReader(paneContent.ToString()))
{
data = (PaneData)(serializer.Deserialize(reader));
}
As mentioned in this answer, a StringBuilder is preferred to a string when you are concatenating in a loop, which is the case here.

Here is an example of how to use groups with regexes and retrieve their contents afterwards.
What you want is a regex that will match your tags, label this as a group then retrieve the data of the group as in the example

Use a StringBuilder to build your data string (paneContent). It's much faster because concatenating strings results in new memory allocations. StringBuilder pre-allocates memory (if you expect large data strings, you can customize the initial allocation).
It's a good idea to read your input file line-by-line so you can avoid loading the whole file into memory if you expect files with many lines of text.

How to read a file into a string with CR/LF preserved?

If I asked the question "how to read a file into a string" the answer would be obvious. However -- here is the catch with CR/LF preserved.
The problem is, File.ReadAllText strips those characters. StreamReader.ReadToEnd just converted LF into CR for me which led to long investigation where I have bug in pretty obvious code ;-)
So, in short, if I have file containing foo\n\r\nbar I would like to get foo\n\r\nbar (i.e. exactly the same content), not foo bar, foobar, or foo\n\n\nbar. Is there some ready to use way in .Net space?
The outcome should be always single string, containing entire file.

Are you sure that those methods are the culprits that are stripping out your characters?
I tried to write up a quick test; StreamReader.ReadToEnd preserves all newline characters.
string str = "foo\n\r\nbar";
using (Stream ms = new MemoryStream(Encoding.ASCII.GetBytes(str)))
using (StreamReader sr = new StreamReader(ms, Encoding.UTF8))
{
string str2 = sr.ReadToEnd();
Console.WriteLine(string.Join(",", str2.Select(c => ((int)c))));
}
// Output: 102,111,111,10,13,10,98,97,114
// f o o \n \r \n b a r
An identical result is achieved when writing to and reading from a temporary file:
string str = "foo\n\r\nbar";
string temp = Path.GetTempFileName();
File.WriteAllText(temp, str);
string str2 = File.ReadAllText(temp);
Console.WriteLine(string.Join(",", str2.Select(c => ((int)c))));
It appears that your newlines are getting lost elsewhere.

This piece of code will preserve LR and CR
string r = File.ReadAllText(#".\TestData\TR120119.TRX", Encoding.ASCII);

The outcome should be always single string, containing entire file.
It takes two hops. First one is File.ReadAllBytes() to get all the bytes in the file. Which doesn't try to translate anything, you get the raw data in the file so the weirdo line-endings are preserved as-is.
But that's bytes, you asked for a string. So second hop is to apply Encoding.GetString() to convert the bytes to a string. The one thing you have to do is pick the right Encoding class, the one that matches the encoding used by the program that wrote the file. Given that the file is pretty messed up if it contains \n\r\n sequences, and you didn't document anything else about the file, your best bet is to use Encoding.Default. Tweak as necessary.

You can read the contents of a file using File.ReadAllLines, which will return an array of the lines. Then use String.Join to merge the lines together using a separator.
string[] lines = File.ReadAllLines(#"C:\Users\User\file.txt");
string allLines = String.Join("\r\n", lines);
Note that this will lose the precision of the actual line terminator characters. For example, if the lines end in only \n or \r, the resulting string allLines will have replaced them with \r\n line terminators.
There are of course other ways of acheiving this without losing the true EOL terminator, however ReadAllLines is handy in that it can detect many types of text encoding by itself, and it also takes up very few lines of code.

ReadAllText doesn't return carriage returns.
This method opens a file, reads each line of the file, and then adds each line as an element of a string. It then closes the file. A line is defined as a sequence of characters followed by a carriage return ('\r'), a line feed ('\n'), or a carriage return immediately followed by a line feed. The resulting string does not contain the terminating carriage return and/or line feed.
From MSDN - https://msdn.microsoft.com/en-us/library/ms143368(v=vs.110).aspx

This is similar to the accepted answer, but wanted to be more to the point. sr.ReadToEnd() will read the bytes like is desired:
string myFilePath = #"C:\temp\somefile.txt";
string myEvents = String.Empty;
FileStream fs = new FileStream(myFilePath, FileMode.Open);
StreamReader sr = new StreamReader(fs);
myEvents = sr.ReadToEnd();
sr.Close();
fs.Close();
You could even also do those in cascaded using statements. But I wanted to describe how the way you write to that file in the first place will determine how to read the content from the myEvents string, and might really be where the problem lies. I wrote to my file like this:
using System.Reflection;
using System.IO;
private static void RecordEvents(string someEvent)
{
string folderLoc = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
if (!folderLoc.EndsWith(#"\")) folderLoc += #"\";
folderLoc = folderLoc.Replace(#"\\", #"\"); // replace double-slashes with single slashes
string myFilePath = folderLoc + "myEventFile.txt";
if (!File.Exists(myFilePath))
File.Create(myFilePath).Close(); // must .Close() since will conflict with opening FileStream, below
FileStream fs = new FileStream(myFilePath, FileMode.Append);
StreamWriter sr = new StreamWriter(fs);
sr.Write(someEvent + Environment.NewLine);
sr.Close();
fs.Close();
}
Then I could use the code farther above to get the string of the contents. Because I was going further and looking for the individual strings, I put this code after THAT code, up there:
if (myEvents != String.Empty) // we have something
{
// (char)2660 is ♠ -- I could have chosen any delimiter I did not
// expect to find in my text
myEvents = myEvents.Replace(Environment.NewLine, ((char)2660).ToString());
string[] eventArray = myEvents.Split((char)2660);
foreach (string s in eventArray)
{
if (!String.IsNullOrEmpty(s))
// do whatever with the individual strings from your file
}
}
And this worked fine. So I know that myEvents had to have the Environment.NewLine characters preserved because I was able to replace it with (char)2660 and do a .Split() on that string using that character to divide it into the individual segments.