Read last 30,000 lines of a file [duplicate] - c#
This question already has answers here:
How to read last "n" lines of log file [duplicate]
(9 answers)
Closed 9 years ago.
If has a csv file whose data will increase by time to time. Now what i need to do is to read the last 30,000 lines.
Code :
string[] lines = File.ReadAllLines(Filename).Where(r => r.ToString() != "").ToArray();
int count = lines.Count();
int loopCount = count > 30000 ? count - 30000 : 0;
for (int i = loopCount; i < lines.Count(); i++)
{
string[] columns = lines[i].Split(',');
orderList.Add(columns[2]);
}
It is working fine but the problem is
File.ReadAllLines(Filename)
Read a complete file which causes performance lack. I want something like it only reads the last 30,000 lines which iteration through the complete file.
PS : i am using .Net 3.5 . Files.ReadLines() not exists in .Net 3.5
You can Use File.ReadLines() Method instead of using File.ReadAllLines()
From MSDN:File.ReadLines()
The ReadLines and ReadAllLines methods differ as follows:
When you use ReadLines, you can start enumerating the collection of strings before
the whole collection is returned; when you use ReadAllLines, you must
wait for the whole array of strings be returned before you can access
the array.
Therefore, when you are working with very large files,
ReadLines can be more efficient.
Solution 1 :
string[] lines = File.ReadAllLines(FileName).Where(r => r.ToString() != "").ToArray();
int count = lines.Count();
List<String> orderList = new List<String>();
int loopCount = count > 30000 ? 30000 : 0;
for (int i = count-1; i > loopCount; i--)
{
string[] columns = lines[i].Split(',');
orderList.Add(columns[2]);
}
Solution 2: if you are using .NET Framework 3.5 as you said in comments below , you can not use File.ReadLines() method as it is avaialble since .NET 4.0 .
You can use StreamReader as below:
List<string> lines = new List<string>();
List<String> orderList = new List<String>();
String line;
int count=0;
using (StreamReader reader = new StreamReader("c:\\Bethlehem-Deployment.txt"))
{
while ((line = reader.ReadLine()) != null)
{
lines.Add(line);
count++;
}
}
int loopCount = (count > 30000) ? 30000 : 0;
for (int i = count-1; i > loopCount; i--)
{
string[] columns = lines[i].Split(',');
orderList.Add(columns[0]);
}
You can use File.ReadLines by you can start enumerating the collection of strings before the whole collection is returned.
After that you can use the linq to make things lot more easier. Reverse will reverse the order of collection and Take will take the n number of items. Now put again Reverse to get the last n lines in original format.
var lines = File.ReadLines(Filename).Reverse().Take(30000).Reverse();
If you are using the .NET 3.5 or earlier you can create your own method which works same as File.ReadLines like this. Here is the code for the method originally written by #Jon
public IEnumerable<string> ReadLines(string file)
{
using (TextReader reader = File.OpenText(file))
{
string line;
while ((line = reader.ReadLine()) != null)
{
yield return line;
}
}
}
Now you can use linq over this function as well like the above statement.
var lines = ReadLines(Filename).Reverse().Take(30000).Reverse();
The problem is that you do not know where to start reading the file to get the last 30,000 lines. Unless you want to maintain a separate index of line offsets you can either read the file from the start counting lines only retaining the last 30,000 lines or you can start from the end counting lines backwards. The last approach can be efficient if the file is very large and you only want a few lines. However, 30,000 does not seem like "a few lines" so here is an approach that reads the file from the start and uses a queue to keep the last 30,000 lines:
var filename = #" ... ";
var linesToRead = 30000;
var queue = new Queue<String>();
using (var streamReader = File.OpenText(fileName)) {
while (!streamReader.EndOfStream) {
queue.Enqueue(streamReader.ReadLine());
if (queue.Count > linesToRead)
queue.Dequeue();
}
}
Now you can access the lines that are stored in queue. This class implements IEnumerable<String> allowing you to use foreach to iterate the lines. However, if you want random access you will have to use the ToArray method to convert the queue into an array which adds some overhead to the computation.
This solution is efficient in terms memory because at most 30,000 lines has to be kept in memory and the garbage collector can free any extra lines when required. Using File.ReadAllLines will pull all the lines into memory at once possibly increasing the memory required by the process.
Or I have a diffrent ideo for this.
Try splitting the csv to categories like A-D , E-G ....
and acces what first character you need .
Or you can split data with count of entites. Every file will contain 15.000 entites for example. And a text file which will contain tiny data about entits and location Like :
Txt File:
entitesID | inWhich.Csv
....
Related
C# :- How to sort a large csv file with 10 columns. Based on 5th column (Period Column). Without memory
How to sort a large csv file with 10 columns? The sorting should be based on data type for example, string, Date, integer etc Assuming Based on 5th column (Period Column) we need to sort. As it is large CSV file, Without loading the same in memory we have to do. I tried using logparser, but beyond certain size it throws error saying "log parser tool has stopped working" So please suggest any algorithm which i can implement in c#. Or if there is any other component or code which can help me. Thanks in advance
Do know that running a program without memory is hard, specially if you have an algorithm that by its nature requires memory allocation.
I've looked at the External sort method mentioned by Jim Menschel and this is my implementation.
I didn't implement sorting on the fifth field but left some hints in the code so you can add that yourself.
This code reads a file, line by line and creates, in a temporary directory for each line a new file. Then we open two of those files and create a new target file. After reading a line from the two open files, we can compare them (or their fields). Based on their comparison we write the smallest one to the target file and read the next line from the file we used.
Although this doesn't keep much strings in memory it is hard on the diskdrive. I checked the NTFS limits and 50,000,000 files is within the specs.
Here are the main methods of the class:
main entry point
This take the file to be sorted
public void Sort(string file)
{
Directory.CreateDirectory(sortdir);
Split(file);
var sortedFile = SortAndCombine();
// if you feel confident you can override the original file
File.Move(sortedFile, file + ".sorted");
Directory.Delete(sortdir);
}
Split file
Split the file in a new file for each line
Yes, that will be a lot of files but it guarantees the least amount of memory used. It is easy to optimize though, read a couple of lines, sort those and write to a file.
void Split(string file)
{
using (var sr = new StreamReader(file, Encoding.UTF8))
{
var line = sr.ReadLine();
while (!String.IsNullOrEmpty(line))
{
// whatever you do, make sure this file your writed
// is ordered, just writing a single line is the easiest
using (var sw = new StreamWriter(CreateUniqueFilename()))
{
sw.WriteLine(line);
}
line = sr.ReadLine();
}
}
}
Combine the files
Iterate over all files and take one and the next one, merge those files
string SortAndCombine()
{
long processed; // keep track of how much we processed
do
{
// iterate the folder
var files = Directory.EnumerateFiles(sortdir).GetEnumerator();
bool hasnext = files.MoveNext();
processed = 0;
while (hasnext)
{
processed++;
// have one
string fileOne = files.Current;
hasnext = files.MoveNext();
if (hasnext)
{
// we have number two
string fileTwo = files.Current;
// do the work
MergeSort(fileOne, fileTwo);
hasnext = files.MoveNext();
}
}
} while (processed > 1);
var lastfile = Directory.EnumerateFiles(sortdir).GetEnumerator();
lastfile.MoveNext();
return lastfile.Current; // by magic is the name of the last file
}
Merge and Sort
Open two files and create one target file. Read a line from both of these and write sthe mallest of the two to the target file.
Keep doing that until both lines are null
void MergeSort(string fileOne, string fileTwo)
{
string result = CreateUniqueFilename();
using(var srOne = new StreamReader(fileOne, Encoding.UTF8))
{
using(var srTwo = new StreamReader(fileTwo, Encoding.UTF8))
{
// I left the actual field parsing as an excersise for the reader
string lineOne, lineTwo; // fieldOne, fieldTwo;
using(var target = new StreamWriter(result))
{
lineOne = srOne.ReadLine();
lineTwo = srTwo.ReadLine();
// naive field parsing
// fieldOne = lineOne.Split(';')[4];
// fieldTwo = lineTwo.Split(';')[4];
while(
!String.IsNullOrEmpty(lineOne) ||
!String.IsNullOrEmpty(lineTwo))
{
// use your parsed fieldValues here
if (lineOne != null && (lineOne.CompareTo(lineTwo) < 0 || lineTwo==null))
{
target.WriteLine(lineOne);
lineOne = srOne.ReadLine();
// fieldOne = lineOne.Split(';')[4];
}
else
{
if (lineTwo!=null)
{
target.WriteLine(lineTwo);
lineTwo = srTwo.ReadLine();
// fieldTwo = lineTwo.Split(';')[4];
}
}
}
}
}
}
// all is perocessed, remove the input files.
File.Delete(fileOne);
File.Delete(fileTwo);
}
Helper variable and method
There is one shared member for the temporary directory and a method for generating temporary unique filenames.
private string sortdir = Path.Combine(Path.GetTempPath(), Guid.NewGuid().ToString("N"));
string CreateUniqueFilename()
{
return Path.Combine(sortdir, Guid.NewGuid().ToString("N"));
}
Memory analysis
I've created a small file with 5000 lines in it with the following code:
using(var sw= new StreamWriter("c:\\temp\\test1.txt"))
{
for(int line=0; line<5000; line++)
{
sw.WriteLine(Guid.NewGuid().ToString());
}
}
I then ran the sorting code with the memory profiler. This is what the summary looked like on my box with Windows 10, 4GB RAM and a spinning disk:
The object lifetime shows as expected a lot of String, char[] and byte[] allocations, but none of those have survived a Gen 0 collection, which means they are all short lived and I don't expect this to be a problem if the number of lines to sort increases.
This is the simplest solution that works for me. From here easy alterations and improvements are possible, either leading to even less memory consumption, reduce allocations or a higher speed. Make sure to measure, select the area where you can make the biggest impact and compare successive results. That should give you the optimum between memory usage and performance.
Instead of reading CSV completely you can simply index it:
Read unsorted CSV line by line and remember 5th element (column) value and something to identify this line later: line number or offset of this line from beginning of the file and size.
You will have some kind of List<Tuple<string, ...>>. Sort that
var sortedList = unsortedList.OrderBy(item => item.Item1);
Now you can create sorted CSV by enumerating sorted list, reading line from source file and appending it to new CSV:
using (var sortedCSV = File.AppendText(newCSVFileName))
foreach(var item in sortedList)
{
... // read line from unsorted csv using item.Item2, etc.
sortedCSV.WriteLine(...);
}
Select rows from CSV based on a list of IDs
I have a task of extracting a few hundred thousand rows from CSV files where the row contains a specified ID. So I have about 300,000 IDs stored in a string List and need to extract any row in the CSV that contains any of these IDs.
At the minute I am using a Linq statement to see if each row contains any of the IDs in the List:
using (StreamReader sr = new StreamReader(csvFile))
{
string inLine = sr.ReadLine();
if(searchStrings.Any(sr.ReadLine().Contains))
{
stremWriter.Write(inLine);
}
}
This kind of works ok but it is Very slow since there are 300,000 values in the searchStrings List and a few million rows in the CSVs that I need to search.
Does anyone know how to make this search more efficient to speed it up?
Or an alternative method for extracting the required rows?
Thanks
I've faced a similarish problem before, I had to iterate through a several hundred thousand line .csv and parse each row.
I went with a threaded approach where I tried to do the reading and parsing simultaneously in batches.
Here's roughly how I did it;
using System.Collections.Concurrent; using System.Threading;
private static ConcurrentBag<String> items = new ConcurrentBag<String>();
private static List<String> searchStrings;
static void Main(string[] args)
{
using (StreamReader sr = new StreamReader(csvFile))
{
const int buffer_size = 10000;
string[] buffer = new string[buffer_size];
int count = 0;
String line = null;
while ((line = sr.ReadLine()) != null)
{
buffer[count] = line;
count++;
if (count == buffer_size)
{
new Thread(() =>
{
find(buffer);
}).Start();
buffer = new String[buffer_size];
count = 0;
}
}
if (count > 0)
{
find(buffer);
}
//some kind of sync here, can be done with a bool - make sure all the threads have finished executing
foreach (var str in searchStrings)
streamWriter.write(str);
}
}
private static void find(string[] buffer)
{
//do your search algorithm on the array of strings
//add to the concurrentbag if they match
}
I just quickly threw this code together from what I remember doing before so it might not be entirely correct. Doing it like this certainly speeds things up though (with very large files at least).
The idea is to always be reading from the hdd as string parsing can be pretty expensive, and thus batching the work on multiple cores can make it significantly faster.
With this, I was able to parse (splitting each line into about 50 items and parsing the key/value pairs and building objects in memory from them - by far the most time consuming part) around 250k lines in just over 7s.
Just throwing this out there, it's not specifically relevant to any of the tags on your question but the *nix "grep -f" functionality would work here. Essentially, you'd have a file with the list of strings you want to match (e.g., StringsToFind.txt) and you'd have your csv input file (e.g., input.csv) and the following command would output the matching lines to output.csv grep -f StringsToFind.txt input.csv > output.csv See grep man page for more details.
Merging CSV lines in huge file
I have a CSV that looks like this 783582893T,2014-01-01 00:00,0,124,29.1,40.0,0.0,40,40,5,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,Y 783582893T,2014-01-01 00:15,1,124,29.1,40.0,0.0,40,40,5,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,Y 783582893T,2014-01-01 00:30,2,124,29.1,40.0,0.0,40,40,5,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,Y 783582855T,2014-01-01 00:00,0,128,35.1,40.0,0.0,40,40,5,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,Y 783582855T,2014-01-01 00:15,1,128,35.1,40.0,0.0,40,40,5,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,Y 783582855T,2014-01-01 00:30,2,128,35.1,40.0,0.0,40,40,5,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,Y ... 783582893T,2014-01-02 00:00,0,124,29.1,40.0,0.0,40,40,5,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,Y 783582893T,2014-01-02 00:15,1,124,29.1,40.0,0.0,40,40,5,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,Y 783582893T,2014-01-02 00:30,2,124,29.1,40.0,0.0,40,40,5,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,Y although there are 5 billion records. If you notice the first column and part of the 2nd column (the day), three of the records are all 'grouped' together and are just a breakdown of 15 minute intervals for the first 30 minutes of that day. I want the output to look like 783582893T,2014-01-01 00:00,0,124,29.1,40.0,0.0,40,40,5,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,Y,40.0,0.0,40,40,5,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,Y,40.0,0.0,40,40,5,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,Y 783582855T,2014-01-01 00:00,0,128,35.1,40.0,0.0,40,40,5,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,Y,40.0,0.0,40,40,5,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,Y,40.0,0.0,40,40,5,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,Y ... 783582893T,2014-01-02 00:00,0,124,29.1,40.0,0.0,40,40,5,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,Y,40.0,0.0,40,40,5,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,Y,40.0,0.0,40,40,5,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,Y Where the first 4 columns of the repeating rows are ommitted and the rest of the columns are combined with the first record of it's kind. Basically I am converting the day from being each line is 15 minutes, to each line is 1 day. Since I will be processing 5 billion records, I think the best thing is to use regular expressions (and EmEditor) or some tool that is made for this (multithreading, optimized), rather than a custom programmed solution. Althought I am open to ideas in nodeJS or C# that are relatively simple and super quick. How can this be done?
If there's always a set number of records records and they're in order, it'd be fairly easy to just read a few lines at a time and parse and output them. Trying to do regex on billions of records would take forever. Using StreamReader and StreamWriter should make it possible to read and write these large files since they read and write one line at a time.
using (StreamReader sr = new StreamReader("inputFile.txt"))
using (StreamWriter sw = new StreamWriter("outputFile.txt"))
{
string line1;
int counter = 0;
var lineCountToGroup = 3; //change to 96
while ((line1 = sr.ReadLine()) != null)
{
var lines = new List<string>();
lines.Add(line1);
for(int i = 0; i < lineCountToGroup - 1; i++) //less 1 because we already added line1
lines.Add(sr.ReadLine());
var groupedLine = lines.SomeLinqIfNecessary();//whatever your grouping logic is
sw.WriteLine(groupedLine);
}
}
Disclaimer- untested code with no error handling and assuming that there are indeed the correct number of lines repeated, etc. You'd obviously need to do some tweaks for your exact scenario.
You could do something like this (untested code without any error handling - but should give you the general gist of it):
using (var sin = new SteamReader("yourfile.csv")
using (var sout = new SteamWriter("outfile.csv")
{
var line = sin.ReadLine(); // note: should add error handling for empty files
var cells = line.Split(","); // note: you should probably check the length too!
var key = cells[0]; // use this to match other rows
StringBuilder output = new StringBuilder(line); // this is the output line we build
while ((line = sin.ReadLine()) != null) // if we have more lines
{
cells = line.Split(","); // split so we can get the first column
while(cells[0] == key) // if the first column matches the current key
{
output.Append(String.Join(",",cells.Skip(4))); // add this row to our output line
}
// once the key changes
sout.WriteLine(output.ToString()); // write out the line we've built up
output.Clear();
output.Append(line); // update the new line to build
key = cells[0]; // and update the key
}
// once all lines have been processed
sout.WriteLine(output.ToString()); // We'll have just the last line to write out
}
The idea is to loop through each line in turn and keep track of the current value of the first column. When that value changes, you write out the output line you've been building up and update the key. This way you don't have to worry about exactly how many matches you have or if you might be missing a few points.
One note, it might be more efficient to use a StringBuilder for output rather than a String if you are going to concatentate 96 rows.
Define the ProcessOutputLine to store merged lines.
Call ProcessLine after each ReadLine and at end of file.
string curKey ="" ;
string keyLength = ... ; // set totalength of 4 first columns
string outputLine = "" ;
private void ProcessInputLine(string line)
{
string newKey=line.substring(0,keyLength) ;
if (newKey==curKey) outputline+=line.substring(keyLength) ;
else
{
if (outputline!="") ProcessOutPutLine(outputLine)
curkey = newKey ;
outputLine=Line ;
}
EDIT : this solution is very similar to that of Matt Burland, the only noticable difference is that I don't use the Split function.
reading string each number c#
suppose this is my txt file:
line1
line2
line3
line4
line5
im reading content of this file with:
string line;
List<string> stdList = new List<string>();
StreamReader file = new StreamReader(myfile);
while ((line = file.ReadLine()) != null)
{
stdList.Add(line);
}
finally
{//need help here
}
Now i want to read data in stdList, but read only value every 2 line(in this case i've to read "line2" and "line4").
can anyone put me in the right way?
Even shorter than Yuck's approach and it doesn't need to read the whole file into memory in one go :) var list = File.ReadLines(filename) .Where((ignored, index) => index % 2 == 1) .ToList(); Admittedly it does require .NET 4. The key part is the overload of Where which provides the index as well as the value for the predicate to act on. We don't really care about the value (which is why I've named the parameter ignored) - we just want odd indexes. Obviously we care about the value when we build the list, but that's fine - it's only ignored for the predicate.
You can simplify your file read logic into one line, and just loop through every other line this way:
var lines = File.ReadAllLines(myFile);
for (var i = 1; i < lines.Length; i += 2) {
// do something
}
EDIT: Starting at i = 1 which is line2 in your example.
Add a conditional block and a tracking mechanism inside of a loop. (The body of the loop is as follows:)
int linesProcessed = 0;
if( linesProcessed % 2 == 1 ){
// Read the line.
stdList.Add(line);
}
else{
// Don't read the line (Do nothing.)
}
linesProcessed++;
The line linesProcessed % 2 == 1 says: take the number of lines we have processed already, and find the mod 2 of this number. (The remainder when you divide that integer by 2.) That will check to see if the number of lines processed is even or odd.
If you have processed no lines, it will be skipped (such as line 1, your first line.) If you have processed one line or any odd number of lines already, go ahead and process this current line (such as line 2.)
If modular math gives you any trouble, see the question: https://stackoverflow.com/a/90247/758446
try this:
string line;
List<string> stdList = new List<string>();
StreamReader file = new StreamReader(myfile);
while ((line = file.ReadLine()) != null)
{
stdList.Add(line);
var trash = file.ReadLine(); //this advances to the next line, and doesn't do anything with the result
}
finally
{
}
Create text files of every combination of specific lines within a base text file
Ok, so hopefully I can explain this in enough detail for somebody to be able to help me.. I am writing a program in C# that is supposed to take a text file and replace specific text, which happen to be names of files, and print a new text file for every single combination of the given filenames. The specific places to change the text of filenames have their own set of possible filenames, listed as an array described below. The program should run regardless of how many filenames are available for each location as well as how many total locations for the filenames. If you really wanted to make it awesome, it can be slightly optimized knowing that no filenames should be duplicated throughout any single text file.
text is an array of lines that make up the base of the total file.
lineNum holds an array of the line locations of the filename entries.
previousFiles is an array of previously used filenames, starting with what is already in the file.
files is a jagged 2-dimensional array of possible filenames where files[1] would be an array of all the possible filenames for the 2nd location
Here is an example of how it would work with 3 separate filename locations, the first one given 3 possible filenames, the second given 8 possible filenames, and the third given 3 possible filenames.
Oh and assume buildNewFile works.
int iterator = 0;
for (int a = 0; a < 3; a++)
{
for (int b = 0; b < 8; b++)
{
for (int c = 0; c < 3; c++)
{
iterator++;
text[lineNums[0]] = text[lineNums[0]].Replace(previousFiles[0], files[0][a]);
text[lineNums[1]] = text[lineNums[1]].Replace(previousFiles[0], files[0][a]);
text[lineNums[2]] = text[lineNums[2]].Replace(previousFiles[1], files[1][b]);
text[lineNums[3]] = text[lineNums[3]].Replace(previousFiles[1], files[1][b]);
text[lineNums[4]] = text[lineNums[4]].Replace(previousFiles[2], files[2][c]);
text[lineNums[5]] = text[lineNums[5]].Replace(previousFiles[2], files[2][c]);
previousFiles = new string[] { files[0][a], files[1][b], files[2][c] };
buildNewFile(text, Info.baseFolder + "networks\\" + Info.dsnFilename + iterator + ".dsn");
}
}
}
If you guys can help me, thank you so much, I just can't figure out how to do it recursively or anything. If you have any questions I'll answer them and edit up here to reflect that.
It took me a little while to figure out what you really wanted to do. This problem can be solved without recursion, the trick is to look at the data you have and get it into a more usable format.
Your "files" array is the one that is the most inconvenient. The trick is to transform the data into usable permutations. To do that, I suggest taking advantage of yield and using a method that returns IEnumerable. The code for it is here:
public IEnumerable<string[]> GenerateFileNameStream(string[][] files)
{
int[] current_indices = new int[files.Length];
current_indices.Initialize();
List<string> file_names = new List<string>();
while (current_indices[0] < files[0].Length)
{
file_names.Clear();
for (var index_index = 0; index_index < current_indices.Length; index_index++)
{
file_names.Add(files[index_index][current_indices[index_index]]);
}
yield return file_names.ToArray();
// increment the indices, trickle down as needed
for (var check_index = 0; check_index < current_indices.Length; check_index++)
{
current_indices[check_index]++;
// if the index hasn't rolled over, we're done here
if (current_indices[check_index] < files[check_index].Length) break;
// if the last location rolls over, then we are totally done
if (check_index == current_indices.Length - 1) yield break;
// reset this index, increment the next one in the next iteration
current_indices[check_index] = 0;
}
}
}
Basically, it keeps track of the current index for each row of the files 2D array and returns the file name at each current index. Then it increments the first index. If the first index rolls over, then it resets to 0 and increments the next index instead. This way we can iterate through every permutation of the file names.
Now, looking at the relationship between lineNum and files, I assume that each location in the file is copied to two lines. The rest of the code is here:
public void MakeItWork(string[][] files, int[] lineNum, string[] text, string[] previousFiles)
{
var iterator = 0;
var filenames = GenerateFileNameStream(files);
// work a copy of the text, assume the "previousFiles" are in this text
var text_copy = new string[text.Length];
foreach (var filenameset in filenames)
{
iterator++;
Array.Copy(text, text_copy, text.Length);
for (var line_index = 0; line_index < lineNum.Length; line_index++)
{
var line_number = lineNum[line_index];
text[line_number] = text[line_number].Replace(previousFiles[line_index], filenameset[line_index / 2]);
}
buildNewFile(text_copy, Info.baseFolder + "networks\\" + Info.dsnFilename + iterator + ".dsn");
}
}
This code just takes the results from the enumerator and generates the files for you. The assumption based on your sample code is that each filename location is used twice per file (since the lineNum array was twice as long as the files location count.
I haven't fully tested all the code, but the crux of the algorithm is there. The key is to transform your data into a more usable form, then process it. The other suggestion I have when asking a question here is to describe the problem more as a "problem" and not in the terms of your current solution. If you detailed the goal you are trying to achieve instead of showing code, you can get more insights into the problem.