I have a C# console application where an external text file is read. Each line of the file has values separated by spaces, such as:
1 -88 30.1
2 -89 30.1
So line one should be split into '1', '-88', and '30.1'.
What I need to do is to populate an array (or any other better object) so that it duplicate each line; the array should have 3 elements per row. I must be having a brain-lock to not figure it out today. Here's my code:
string line;
int[] intArray;
intArray = new int[3];
int i = 0;
//Read Input file
using (StreamReader file = new StreamReader("Score_4.dat"))
{
while ((line = file.ReadLine()) != null && line.Length > 10)
{
line.Trim();
string[] parts;
parts = line.Split(' ');
intArray[0][i] = parts[0];//error: cannot apply indexing
i++;
}
}
Down the road in my code, I intend to make some API calls to a server by constructing a Json object while looping through the array (or alternate object).
Any idea?
Thanks
If you only need the data to be transferred to JSON then you don't need to process the values of the data, just reformat it to JSON arrays.
As you don't know the number of lines in the input file, it is easier to use a List<>, whose capacity expands automatically, to hold the data rather than an array, whose size you would need to know in advance.
I took your sample data and repeated it a few times into a text file and used this program:
static void Main(string[] args)
{
string src = #"C:\temp\Score_4.dat";
List<string> dataFromFile = new List<string>();
using (var sr = new StreamReader(src))
{
while (!sr.EndOfStream)
{
string thisLine = sr.ReadLine();
string[] parts = thisLine.Split(" ".ToCharArray(), StringSplitOptions.RemoveEmptyEntries);
if (parts.Length == 3)
{
string jsonArray = "[" + string.Join(",", parts) + "]";
dataFromFile.Add(jsonArray);
}
else
{
/* the line did not have three entries */
/* Maybe keep a count of the lines processed to give an error message to the user */
}
}
}
/* Do something with the data... */
int totalEntries = dataFromFile.Count();
int maxBatchSize = 50;
int nBatches = (int)Math.Ceiling((double)totalEntries / maxBatchSize);
for(int i=0;i<nBatches;i+=1)
{
string thisBatchJsonArray = "{\"myData\":[" + string.Join(",", dataFromFile.Skip(i * maxBatchSize).Take(maxBatchSize)) + "]}";
Console.WriteLine(thisBatchJsonArray);
}
Console.ReadLine();
}
to get this output:
{"myData":[[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1]]}
{"myData":[[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1],[1,-88,30.1],[2,-89,30.1]]}
It should be easy to adjust the format as required.
I would create a custom Item class and then populate a list, for easy access and sorting, with self contained items. something like:
public Class MyItem
{
public int first { get; set; }
public int second { get; set; }
public float third { get; set; }
public MyItem(int one, int two, float three)
{
this.first = one;
this.second = two;
this.third = three;
}
}
then you could do:
List<MyItem> mylist = new List<MyItem>();
and then in your loop:
using (StreamReader file = new StreamReader("Score_4.dat"))
{
while ((line = file.ReadLine()) != null && line.Length > 10)
{
line.Trim();
string[] parts;
parts = line.Split(' ');
MyItem item = new Item(Int32.Parse(parts[0]),Int32.Parse(parts[1]),Float.Parse(parts[2]));
mylist.Add(item);
i++;
}
}
As there are numbers like 30.1 so int is not suitable for this, and also it must not be a double[] but double[][]:
string[] lines = File.ReadAllLines("file.txt");
double[][] array = lines.Select(x => s.Split(' ').Select(a => double.Parse(a)).ToArray()).ToArray();
Issue is that int array is single dimensional.
My suggestion is that you can put a class with 3 properties and populate a list of class there. It's better to have class with same property names that you require to build JSON. So that you can easily serialize this class to JSON using some nugets like Newtonsoft and make api calls easily.
Your int array is a single dimensional array yet you're trying to index it like a multidemensional array. It should be something like this:
intArray[i] = parts[0]
(However you'll need to handle converting to int for parts that are fractional)
Alternatively, if you want to use a multidimensional array, you have to declare one.
int[][] intArray = new int[*whatever your expected number of records are*][3]
Arrays have a static size. Since you're reading from a file and may not know how many records there are until your file finishes reading, I recommend using something like a List of Tuples or a Dictionary depending on your needs.
A dictionary will allow you to have quick lookup of your records without iterating over them by using a key value pair, so if you wanted your records to match up with their line numbers, you could do something like this:
Dictionary<int, int[]> test = new Dictionary<int, int[]>();
int lineCount = 1;
while ((line = file.ReadLine()) != null && line.Length > 10)
{
int[] intArray = new int[3];
line.Trim();
string[] parts = line.Split(' ');
for (int i = 0; i < 3; i++)
{
intArray[i] = int.Parse(parts[i]);
}
test[lineCount] = intArray;
lineCount++;
}
This will let you access your values by line count like so:
test[3] = *third line of file*
I am reading and counting the number of lines in a file, then initializing an array with the same number of spaces as the number of lines. The file is then read again and each line is assigned to that position of the array. For example, the first line would be stored in the index position of 0. I have the following code:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.IO;
namespace testProg
{
class program
{
static void main(){
Console.WriteLine("enter your filename for reading!");
fileName = Console.ReadLine();
using (StreamReader rs = new StreamReader(fileName))
{
string line2;
while ((line2 = rs.ReadLine()) != null)
{
arraysize = arraysize+1;//this goes through and gets the number of lines
}
}
Console.WriteLine(arraysize);
string[] unenc = new string[arraysize]; //this creates the array dynamically
int i = -1;//starts at position -1 then +1 so starts at 0
using (StreamReader fr = new StreamReader(fileName))
{
string linefinal;
while ((linefinal = fr.ReadLine()) != null)
{
Console.WriteLine(linefinal);//this will write the current line
unenc[i + 1] = linefinal;// this should store the string above in the current position
Console.WriteLine(unenc[i]);//this should output the same line it does not the index is just empty ? but it should be stored yet it is not
}
}
}
}
}
The problem is that you're not saving the incremented value of i anywhere.
You could modify you code like so:
while ((linefinal = fr.ReadLine()) != null)
{
Console.WriteLine(linefinal);
unenc[i + 1] = linefinal;
Console.WriteLine(unenc[i]);
i++;
}
So your query in comments is
would the array not update the value of i in the section where unenc[i+1]?
What i + 1 does is return the value of "i plus 1".
If you want to increment the value, you have two options:
Increment after returning value:
var oldValue = i++;
var newValue = i;
Increment before returning value:
var oldValue = i;
var newValue = ++i;
Code optimisation
You're having to read your file twice in order to get the number of lines so you can find out how big your array is. .NET provides a lovely class which will help you in this instance: List<>.
List<int> is a wrapper for int[] which manages the length of its internal array on the fly, which means while with an array you have to do this:
var array = int[3];
array[0] = 1;
array[1] = 2;
array[3] = 3:
with a List<int> you can simply do the following
var list = new List<int>();
list.Add(1);
list.Add(2);
list.Add(3);
With an array you'd have to copy to a larger array if you wanted to add another element, with list, this is all done for you internally.
Of course you can still access a list by var item = list[3];.
So utilising this, you can do away with the first read of the file and just keep adding to a list.
Instead of an array, consider using a List object. You can use the Add() method to continuously add items as you read them. Once complete, you can simply call the ToArray() method on the List object to get the array you need. You'll have all of the index values matching each line.
The line
unenc[i + 1] = linefinal;
isn't quite right. I belive you mean
unenc[i++] = linefinal;
Your line does not change the value of i as it goes through the loop.
Simple way is to use ArrayList. Please see the below code snippet
ArrayList lines = new ArrayList();
using (StreamReader rs = new StreamReader(#"C:\Users\vimal\Desktop\test.txt"))
{
string line = null;
while ((line = rs.ReadLine()) != null)
{
lines.Add(line);
}
}
Suppose I have a text file with data like below, I want to read first row and store the elements in one array. Read the second row and store in second array and so on. I will be making some manipulations on the array later on. Can you help me to do this in C#?
Input text file:
5,7,3,6,9,8,3,5,7
5,6,8,3,4,5
6,4,3,2,65,8,6,3,3,5,7,4
4,5,6,78,9,4,2,5,6
The Code I am trying out is:
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using System.IO;
namespace ReadFile
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}
static void Main(string[] args)
{
TextReader tr = new StreamReader("Data.txt");
// write a line of text to the file
string word = tr.ReadLine();
//now split this line into words
string[] val = word.Split(new Char[] { ',' });
}
}
}
If I use the above technique, I can get the first line in array val. Is there a way to loop it for all the rows?
File.ReadAllLines will help you read all lines from file as string array.
String.Split will help you split line in pieces.
Int.Parse will help you transform string to int (double has similar method).
Ok, since you are a beginner my suggestions on what to focalize your attention.
First of all, are your consumer always needing all the lines or can it stop at some point? If so, use a yield strategy to return the results one by one. You can use a StreamReader to read line one by one in any case, and use double.TryParse(...) or int.TryParse() after splitting the string by the separator. Keep an eye to the fact that separator can change, so use it with some kind of configurability, and in the case of double, ensure your code works even in machines with a different decimal point configured. If you are sure your csv alwais uses '.' as a decimal point separator, specify
double.TryParse("",System.Globalization.CultureInfo.InvariantCulture);
Something roughly like this would work:
StreamReader reader = new (File.OpenRead(#"YourFile.txt"));
List<string> LstIntegers = new List<string>();
while (!reader.EndOfStream)
{
var line = reader.ReadLine();
//values is actually a string array.
var values = line.Split(',');
//add the entire array fetched into string List
LstIntegers.AddRange(values);
}
// important to close the reader. You can also use using statement for reader. It
// will close the reader automatically when reading finishes.
reader.Close();
// You can then further manipulate it like below, you can also use int.Parse or int.TryParse:
foreach (var v in LstIntegers)
{
// use int.TryParse if there is a chance that a non-int is read.
int someNum = Convert.ToInt32(v);
}
Try this
using System;
using System.Collections.Generic;
using System.Linq;
using System.Xml;
using System.IO;
using System.Xml.Linq;
using System.Diagnostics;
namespace ConsoleApplication5
{
class Program
{
static void Main(string[] args)
{
List<int[]> arrays = new List<int[]>();
int counter = 0;
string line;
// Read the file
System.IO.StreamReader file =
new System.IO.StreamReader("c:\\temp\\test.txt");
while ((line = file.ReadLine()) != null)
{
// split the line into a string array on , separator
string[] splitLine = line.ToString().Split(',');
// if our split isnt null and has a positive length
if (splitLine != null && splitLine.Length > 0)
{
// create a lineArray the same size as our new string[]
int[] lineArray = new int[splitLine.Length];
int posCounter = 0;
foreach (string splitValue in splitLine)
{
// loop through each value in the split, try and convert
// it into an int and push it into the array
try
{
lineArray[posCounter] = Int32.Parse(splitValue);
}
catch { }
posCounter++;
}
// if our lineArray has a positive length then at it to our
// list of arrays for processing later.
if (lineArray.Length > 0)
{
arrays.Add(lineArray);
}
}
counter++;
}
file.Close();
// go through the List<int[]> and print to screen
foreach (int[] row in arrays)
{
foreach (int rowCol in row)
{
Console.Write(rowCol + ",");
}
Console.WriteLine();
}
// Suspend the screen.
Console.ReadLine();
}
}
}
See this lines of code:
List<List<int>> numbers = new List<List<int>>();
foreach (string line in File.ReadAllLines(""))
{
var list = new List<int>();
foreach (string s in line.Split(new[]{',', ' '},
StringSplitOptions.RemoveEmptyEntries))
{
int i;
if(int.TryParse(s, out i))
{
list.Add(i);
}
}
numbers.Add(list);
}
var specialNumber = numbers[3][4]; // gives line 3 number 4
var specialLine = numbers[2].ToArray(); // gives an array of numbers of line 2
description:
I used this useful Generic:
List: Represents a strongly typed list of objects that can be accessed by index
and these useful Classes:
File.ReadAllLines: Opens a text file, reads all lines of the file into a string array
String.Split : Returns a string array that contains the substrings in this instance that are delimited by elements of a specified string or Unicode character array.
Int.TryParse: Converts the string representation of a number to its 32-bit signed integer equivalent.
I have presented the following code in two sections, just to show you in steps.
SECTION 1
This code is just to show that you just needed to loop through all lines and store string numbers from each line in a List.
SECTION 1
static void Main(string[] args)
{
List<string[]> allLines = new List<string[]>();
TextReader tr = new StreamReader("Data.txt");
string word = tr.ReadLine();
// write a line of text to the file
while ( word != null ) {
//now split this line into words
string[] vals = word.Split(new Char[] { ',' });
//Add this line into allLines
allLines.Add(vals);
//Now read the next line
word = tr.ReadLine();
}
}
SECTION 2
This section will get you the results as int.
static void Main(string[] args)
{
//A list of arrays of integers
//A single array will have numbers from a single line
List<int[]> allNumbers = new List<int[]>();
TextReader tr = new StreamReader("Data.txt");
string word = tr.ReadLine();
// write a line of text to the file
while ( word != null ) {
//now split this line into words
string[] vals = word.Split(new Char[] { ',' });
int[] intVals = new int[vals.Length];
for ( int i = 0; i < vals.Length; i++) {
Int32.TryParse(vals[i], out intVals[i]);
}
//Add this array of integers into allNumbers
allNumbers.Add(intVals);
//Now read the next line
word = tr.ReadLine();
}
}
NOTE: I have not compiled or tested the code above.
I figured it out. Thanks all for your time!
private void ReadFile()
{
var lines = File.ReadLines("Data.csv");
var numbers = new List<List<double>>();
var separators = new[] { ',', ' ' };
/*System.Threading.Tasks.*/
Parallel.ForEach(lines, line =>
{
var list = new List<double>();
foreach (var s in line.Split(separators, StringSplitOptions.RemoveEmptyEntries))
{
double i;
if (double.TryParse(s, out i))
{
list.Add(i);
}
}
lock (numbers)
{
numbers.Add(list);
}
});
I have millions of lines generated from data updated every second which look like this:
104500 4783
104501 8930
104502 21794
104503 21927
104505 5746
104506 9968
104509 5867
104510 46353
104511 7767
104512 4903
The column on the left represents time (hhmmss format), and the column on the right is data which is updated second-by-second. As you can see however, it isn't actually second-by-second, and there are some missing times (10:45:04, 10:45:07, 10:45:08 are missing in this example). My goal is to add in the missing seconds, and to use the data from the previous second for that missing second, like this:
104500 4783
104501 8930
104502 21794
104503 21927
104504 21927 --
104505 5746
104506 9968
104507 9968 --
104508 9968 --
104509 5867
104510 46353
104511 7767
104512 4903
I don't want the "--" in the result, I just put those there to mark the added lines. So far I've tried to accomplish this using StreamReader and StreamWriter, but it doesn't seem like they're going to get me what I want. I'm a newbie programmer and a newbie to C#, so if you could just point me in the right direction, that would be great. I'm really just wondering if this is even possible to do in C#...I've spent a lot of time on MSDN and here on SO looking for a solution to this, but so far haven't found any.
Edit: The lines are in a text file, and I want to store the newly created data in a new text file.
There are a few things you need to put together.
Read a file line-by-line: See here: Reading a Text File One Line at a Time
Writing a file line-by-line : StreamWriter.WriteLine
Keep track of the last read line. (Just use a variable in your while loop where you read the lines)
Check whether there is a gap. Maybe by parsing the first column (string.Split) using TimeSpan.Parse. If there is a gap then write the last read line, incrementing the timespan.
ok, here is the whole shooting match, tested and working against your test data:
public void InjectMissingData()
{
DataLine lastDataLine = null;
using (var writer = new StreamWriter(File.Create("c:\\temp\\out.txt")))
{
using (var reader = new StreamReader("c:\\temp\\in.txt"))
{
while (!reader.EndOfStream)
{
var dataLine = DataLine.Parse(reader.ReadLine());
while (lastDataLine != null && dataLine.Occurence - lastDataLine.Occurence > TimeSpan.FromSeconds(1))
{
lastDataLine = new DataLine(lastDataLine.Occurence + TimeSpan.FromSeconds(1), lastDataLine.Data);
writer.WriteLine(lastDataLine.Line);
}
writer.WriteLine(dataLine.Line);
lastDataLine = dataLine;
}
}
}
}
public class DataLine
{
public static DataLine Parse(string line)
{
var timeString = string.Format("{0}:{1}:{2}", line.Substring(0, 2), line.Substring(2, 2),
line.Substring(4, 2));
return new DataLine(TimeSpan.Parse(timeString), long.Parse(line.Substring(7, line.Length - 7).Trim()));
}
public DataLine(TimeSpan occurence, long data)
{
Occurence = occurence;
Data = data;
}
public TimeSpan Occurence { get; private set; }
public long Data { get; private set; }
public string Line
{
get { return string.Format("{0}{1}{2} {3}",
Occurence.Hours.ToString().PadLeft(2, Char.Parse("0")),
Occurence.Minutes.ToString().PadLeft(2, Char.Parse("0")),
Occurence.Seconds.ToString().PadLeft(2, Char.Parse("0")),
Data); }
}
}
In adition to all answers, considering that you are talking about a huge files, consider use of MemoryMappedFiles, can read here to see how to use them from C#.
This is not performance improvement, but memory improvement definetely is.
So far as inserting new entries between certain ones goes, I would advise reading in the text file into separated lines, and then storing them in a List. That way, you can use the Insert(...) method to insert your new lines. From there, you can write the lines back into the file.
When reading the lines, you can use either of the static helper methods in the System.IO.File class: ReadAllText and ReadAllLines.
Note: I've added links to the MSDN Documentation for each of the methods and classes I've mentioned, since you said you are new to C# and programming in general.
String prevTime;
String prevData;
while(String line = myStreamReader.ReadLine())
{
String[] parts = line.Split(new Char[] { ' ' });
String time = parts[0];
String data = parts[1];
Int32 iPrevTime = Int32.Parse(prevTime);
Int32 iCurrentTime = Int32.Parse(time);
// May need to loop here if you're missing more than one second
if(iCurrentTime > iPrevTime + 1)
AddData((iPrevTime + 1).ToString(), prevData);
AddData(time, data);
prevTime = time;
prevData = data;
}
Here is some pseudo-code to get you started. I think you will want this type of algorithm.
Here's some rough code for you. I'm not properly disposing everything, it's just to get you started.
DateTime lastTime;
string lastValue = null;
StreamReader reader = File.OpenText("path");
StreamWriter writer = new StreamWriter(File.OpenWrite("newPath"));
while (!reader.EndOfStream)
{
string[] lineData = reader.ReadLine().Split(' ');
DateTime currentTime = DateTime.Parse(lineData[0]);
string value = lineData[1];
if (lastValue != null)
{
while (lastTime < currentTime.AddSeconds(-1))
{
lastTime = lastTime.AddSeconds(1);
writer.WriteLine("{0} {1}", lastTime, lastValue);
}
}
writer.WriteLine("{0} {1}", currentTime, value);
lastTime = currentTime;
lastValue = value;
}
This assumes the times are never more than a second apart. If that assumption is wrong, it's easy enough to modify the below so it writes the lastValue in a loop for each second missing.
Update I missed in your example that it can in fact miss multiple seconds. I changed the example below to address that.
using (StreamReader reader = OpenYourInputFile())
using (StreamWriter writer = OpenYourOutputFile())
{
TimeSpan? lastTime;
TimeSpan currentTime, maxDiff = TimeSpan.FromSeconds(1);
string lastValue, currentline, currentValue, format = "{0:hhmmss} {1}";
while( (currentLine = reader.ReadLine()) != null)
{
string[] s = currentLine.Split(' ');
currentTime = DateTime.ParseExact("hhmmss", s[0] CultureInfo.InvariantCulture).TimeOfDay;
currentValue = s[1];
if (lastTime.HasValue && currentTime - lastTime.Value > maxDiff)
{
for(int x = 1; x <= (currentTime - lastTime).Seconds; x++) writer.WriteLine(string.Format(format, DateTime.Today.Add(lastTime).AddSeconds(x), lastValue);
}
writer.WriteLine(string.Format(format, DateTime.Today.Add(currentTime), currentValue);
lastTime = currentTime;
lastValue = currentValue;
}
}
string line;//The line that is read.
string previousLine = "0 0";
int prevTime = 0;
//These "using"'s are so that the resources they use will be freed when the block ( i.e. {} ) is finished.
using (System.IO.StreamReader originalFile = new System.IO.StreamReader("c:\\users\\Me\\t.txt"))
using (System.IO.StreamWriter newFile = new System.IO.StreamWriter("c:\\users\\Me\\t2.txt"))
{
while ((line = originalFile.ReadLine()) != null)
{
//"Split" changes the words in "line" (- that are separated by a space) to an array.
//"Parse" takes the first in that array (by using "[0]") and changes it into an integer.
int time = int.Parse(line.Split(' ')[0]);
while (prevTime != 0 && time > ++prevTime) newFile.WriteLine(prevTime.ToString() + " " + previousLine.Split(' ')[1]);
previousLine = line;
prevTime = time;
newFile.WriteLine(line);
}
}