We Keep Coding

How to search string in large text file?

I want to get the line containing a certain word that cannot be repeated like profile ID without make loop to read each of line separately, Because if the word I am looking for is in the last line of the text file, this will take a lot of time to get it, and if the search process is for more than one word and extract the line that contains it, I think it will take a lot of time.
Example for line text file
name,id,image,age,place,link
string word = "13215646";
string output = string.Empty;
using (var fileStream = File.OpenRead(FileName))
using (var streamReader = new StreamReader(fileStream, Encoding.UTF8))
{
String line;
while ((line = streamReader.ReadLine()) != null)
{
string[] strList = line.Split(',');
if (word == strList[1]) // check if word = id
{
output = line;
break;
}
}
}

You can use this to search the file:
var output = File.ReadLines(FileName).
Where(line => line.Split(',')[1] == word).
FirstOrDefault();
But it won't solve this:
if the word I am looking for is in the last line of the text file, this will take a lot of time to get it, and if the search process is for more than one word and extract the line that contains it, I think it will take a lot of time.
There's not a practical way to avoid this for a basic file.
The only ways around actually reading through the file is either maintaining an index, which requires absolute control over everything that might write into the file, or if you can guarantee the file is already sorted by the columns that matter, in which case you can do something like a binary search.
But neither is likely for a random csv file. This is one of the reasons people use databases.
However, we also need to stop and check whether this is really a problem for you. I'd expect the code above to handle files up to a couple hundred MB in around 1 to 2 seconds on modern hardware, even if you need to look through the whole file.

You can optimise the code. Here are few ideas:
var ids = new ["13215646", "113"];
foreach(var line in File.ReadLines(FileName))
{
var id = line.Split(',', count: 3)[1]; // Optimization 1: Use: `count: 3`
if(ids.Contains(id) // Optimization 2: Search for multiple ids
{
//Do what you need with the line
}
}

C# so I need to split out a string, I think

so I have this application that I have inherited from someone that is long gone. The gist of the application is that it reads in a .cvs file that has about 5800 lines in it, copies it over to another .cvs, which it creates new each time, after striping out a few things , #, ', &. Well everything works great, or it has until about a month ago. so I started checking into it, and what I have found so far is that there are about 131 items missing from the spreadsheet. Now I read someplace that the maximun amount of data a string can hold is over 1,000,000,000 chars, and my spreadsheet is way under that, around 800,000 chars, but the only thing I can think is doing it is the string object.
So anyway, here is the code in question, this piece appears
to both read in from the existing field, and output to the new file:
StreamReader s = new StreamReader(File);
//Read the rest of the data in the file.
string AllData = s.ReadToEnd();
//Split off each row at the Carriage Return/Line Feed
//Default line ending in most windows exports.
//You may have to edit this to match your particular file.
//This will work for Excel, Access, etc. default exports.
string[] rows = AllData.Split("\r\n".ToCharArray(), System.StringSplitOptions.RemoveEmptyEntries);
//Now add each row to the DataSet
foreach (string r in rows)
{
//Split the row at the delimiter.
string[] items = r.Split(delimiter.ToCharArray());
//Add the item
result.Rows.Add(items);
}
If anyone can help me I would really appreciate it. I either need to figure out how to split the data better, or I need to figure out why it is cutting out the last 131 lines from the existing excel file to the new excel file.

One easier way to do this, since you're using "\r\n" for lines, would be to just use the built-in line reading method: File.ReadLines(path)
foreach(var line in File.ReadLines(path))
{
var items = line.Split(',');
result.Rows.Add(items);
}

You may want to check out the TextFieldParser class, which is part of the Microsoft.VisualBasic.FileIO namespace (yes, you can use this with C# code)
Something along the lines of:
using(var reader = new TextFieldParser("c:\\path\\to\\file"))
{
//configure for a delimited file
reader.TextFieldType = FieldType.Delimited;
//configure the delimiter character (comma)
reader.Delimiters = new[] { "," };
while(!reader.EndOfData)
{
string[] row = reader.ReadFields();
//do stuff
}
}
This class can help with some of the issues of splitting a line into its fields, when the field may contain the delimiter.

How to read CSV file group by group

I want to read a .csv file which has some repeated lines. I want to read between this repeated line. For example a is the repeated line and before the second a, I have lines such as 1,2,3.
I want to read write these values between a values. Can you help me?

It was a bit unclear. But perhaps this will do:
var lines = new List<string>(File.ReadLines("input.csv"));
foreach (string line in lines)
{
if (line.StartsWith("a")) continue;
// insert code to modify the other lines
}
// ... and later
File.WriteAllLines("output.csv", lines);

read only given last x lines in txt file [duplicate]

This question already has answers here:
Get last 10 lines of very large text file > 10GB
(21 answers)
Closed 9 years ago.
Currently I'm reading file content using File.ReadAllText(), but now I need to read last x lines in my txt file. How can I do that?
content of myfile.txt
line1content
line2content
line3content
line4content
string contentOfLastTwoLines = ...

What about this
List <string> text = File.ReadLines("file.txt").Reverse().Take(2).ToList()

Use Queue<string> to store last X lines and replace the first one with currently read:
int x = 4; // number of lines you want to get
var buffor = new Queue<string>(x);
var file = new StreamReader("Input.txt");
while (!file.EndOfStream)
{
string line = file.ReadLine();
if (buffor.Count >= x)
buffor.Dequeue();
buffor.Enqueue(line);
}
string[] lastLines = buffor.ToArray();
string contentOfLastLines = String.Join(Environment.NewLine, lastLines);

You can use ReadLines to avoid reading the entire file into memory, like this:
const int neededLines = 5;
var lines = new List<String>();
foreach (var s in File.ReadLines("c:\\myfile.txt")) {
lines.Add(s);
if (lines.Count > neededLines) {
lines.RemoveAt(0);
}
}
Once the for loop is finished, the lines list contains up to the last neededLines of text from the file. Of course if the file does not contain as many lines as required, fewer lines will be placed in the lines list.

Read the lines into an array, then extract the last two:
string[] lines = File.ReadAllLines();
string last2 = lines[lines.Count-2] + Environment.NewLine + lines[lines.Count-1];
Assuming your file is reasonably small, it's easier to just read the whole thing and throw away what you don't need.

Since reading a file is done linearly, usually line-by-line. Simply read line-by-line and remember last two lines (you can use queue or something if you want... or just two string variables). When you get to EOF, you'll have your last two lines.

You want to read the file backwards using ReverseLineReader:
How to read a text file reversely with iterator in C#
Then run .Take(2) on it.
var lines = new ReverseLineReader(filename);
var last = lines.Take(2);
OR
Use a System.IO.StreamReader.
string line1, line2;
using(StreamReader reader = new StreamReader("myFile.txt")) {
line1 = reader.ReadLine();
line2 = reader.ReadLine();
}

txt file read/overwrite/append. Is this feasible? (Visual C#)

I'm writing a program for some data entry I have to periodically do. I have begun testing a few things that the program will have to do but i'm not sure about this part.
What i need this part to do is:
read a .txt file of data
take the first 12 characters from each line
take the first 12 characters from each line of the data that has been entered in a multi-line text box
compare the two lists line by line
if one of the 12 character blocks from the multi-line text box match one of the blocks in the .txt file then overwrite that entire line (only 17 characters in total)
if one of the 12 character blocks from the multi-line text box DO NOT match any of the blocks in the.txt file then append that entire line to the file
thats all it has to do.
i'll do an example:
TXT FILE:
G01:78:08:32 JG05
G08:80:93:10 JG02
G28:58:29:28 JG04
MULTI-LINE TEXT BOX:
G01:78:08:32 JG06
G28:58:29:28 JG03
G32:10:18:14 JG01
G32:18:50:78 JG07
RESULTING TXT FILE:
G01:78:08:32 JG06
G08:80:93:10 JG02
G28:58:29:28 JG03
G32:10:18:14 JG01
G32:18:50:78 JG07
as you can see lines 1 and 3 were overwriten, line 2 was left alone as it did not match any blocks in the text box, lines 4 and 5 were appended to the file.
thats all i want it to do.
How do i go about this?
Thanks in advance
Edit
The code i'm using is this:
private void WriteToFile()
{
// Read all lines from file into memory
System.IO.StreamReader objReader = new System.IO.StreamReader("Jumpgate List.JG");
List<String> fileTextList = new List<String>();
do
{
fileTextList.Add(objReader.ReadLine());
}
while (objReader.Peek() != -1);
objReader.Close();
// Read all lines from the Input textbox into memory
System.IO.StringReader objReaderi = new System.IO.StringReader(txtInput.Text);
List<String> inputTextList = new List<String>();
do
{
inputTextList.Add(objReaderi.ReadLine());
}
while (objReaderi.Peek() != -1);
objReaderi.Close();
for(int i=0;i<fileTextList.Count;i++)
{
for(int j=0;j<inputTextList.Count;j++)
//compare the first 12 characters of each string
if (String.Compare(fileTextList[i], 0, inputTextList[j], 0, 12) == 0) // strings are equal
{
//replace the fileTextList string with the inputTextList string
fileTextList[i] = inputTextList[j];
// now that you have matched you inputTextList line you remember not to append it at the end
inputTextList[j] = String.Empty; // or nothing
}
}
for(int i=0;i<inputTextList.Count;i++)
{
if (!string.IsNullOrEmpty(inputTextList[i])) fileTextList.Add(inputTextList[i]);
}
System.IO.StreamWriter objWriter = new System.IO.StreamWriter("Jumpgate List.JG");
// Overwrite the Jumpgate List.JG file using the updated fileTextList
objWriter.Write(fileTextList);
objWriter.Close();
}
However, when i open the txt file all i get is: System.Collections.Generic.List`1[System.String]

I'm not going to write the whole code for doing this but it would be something like this:
Disclaimer: I have not used a code editor to try the code, just wrote it here, hopefully you'll get the idea and fill in the missing pieces :)
1) get all the lines in the file in a list. Something like this
StreamReader rd = new StreamReader("sadasd");
List<String> llist = new List<String>();
do
{
llist.Add(rd.ReadLine());
} while (rd.Peek() != -1);
2) get all the lines in your multiline text box (the procedure should be similar to the one above): multiTextList
3) now that you can compare the content of the 2 lists iterating through them
for(int i=0;i<fileTextList.Count;i++)
{
for(int j=0;j<multiTextList.Count;j++)
//compare the first 12 characters of each string
if String.Compare(fileTextList[i], 0, multiTextList[j], 0, 12) == 0 // strings are equal
{
//replace the initial line with whatever you want
fileTextList[i] = //whatever
// now that you have matched you multiTextList line you remember not to append it at the end
multiTextList[j] = String.empty // or nothing
}
}
4) at the end you will have in fileTextList the initial rows, modified where necessary
In multiTextList you will have only the lines that were not matched so we add them to the initial file rows
for(int i=0;i<multiTextList.Count;i++)
{
if !string.isnullorempty(multitextlist[i]) fileTextList.add(multitextlist[i])
}
5) now in fileTextList you have all the rows you require so you can print them one by one in a file and you have your result
StringBuilder lSb = new StringBuilder();
for (int i = 0; i < fileTextList.Count; i++)
{
lSb.AppendLine(fileTextList[i]);
}
File.WriteAllText(#"C:/test2.txt",lSb.ToString());
In C:/test2.txt you should have the results.
Hope this helps!

// this variable maps the timestamps to complete lines
var dict = new Dictionary<string, string>();
// create the map of stamp => line for the original text file
string fileLine = file.ReadLine();
string fileStamp = fileLine.Substring(0, 12);
dict[fileStamp] = fileLine;
// now update the map with results from the text input. This will overwrite text
// strings that already exist in the file
foreach (string inputLine in textInputString.Split('\n'))
{
string inputStamp = inputLine.Substring(0, 12);
dict[inputStamp] = inputLine;
}
// write out the new file with the updated lines
foreach (string line in dict.Values)
{
outputFile.WriteLine(line);
}

if the file is large, loading the entire file into a dictionary to update a handful of lines from a textfield is probably excessive.
In pseudocode I would probably:
Create a list of booleans or other structure to track if a line was matched.
open the file in read/write mode.
For each line in file
{
for each unmatched line in text field
{
If stamps match
Update file
record that it was matched
}
}
for each unmatched line in text field
{
append to file
}
If the lines are fixed width, you can probably optimize by only reading the stamp rather than the whole line. If they match your file pointer is in the right spot to start writing, if not you move to the next line.