We Keep Coding

C#: Opening and processing very large GeoJSON file

I'm trying to process a 25GB GeoJSON file using GeoJSON.net
The accepted answer here works on a small test file but is causing Memory Exception errors with a large 25GB file
There isn't a huge amount of info about how to process the FeatureCollection so I'm just looping through
Can anyone advise what I'm doing wrong?
CODE
try
{
JsonSerializer serializer = new JsonSerializer();
using (FileStream s = File.Open(jsonFile, FileMode.Open))
using (StreamReader sr = new StreamReader(s, Encoding.UTF8))
using (JsonReader reader = new JsonTextReader(sr))
{
while (reader.Read())
{
// deserialize only when there's "{" character in the stream
if (reader.TokenType == JsonToken.StartObject)
{
FeatureCollection FC = serializer.Deserialize<FeatureCollection>(reader);
// Errors Here
foreach (var Feature in FC.Features)
{
if (Feature.Properties.ContainsKey("place"))
{
foreach (var p in Feature.Properties)
{
var Key = p.Key;
var Value = p.Value;
Console.WriteLine("Tags K: {0} Value: {1} ", Key, Value);
}
}
}
}
}
}
}
catch (Exception e)
{
Console.WriteLine("Err: " + e.Message);
}
The second answer on the same page isn't doing anything
BTW JsonReaderExtensions is in a separate file copied from that page
Regex regex = new Regex(#"^\[\d+\]$");
using (FileStream s = File.Open(jsonFile, FileMode.Open))
using (StreamReader sr = new StreamReader(s, Encoding.UTF8))
using (JsonReader reader = new JsonTextReader(sr))
{
IEnumerable<FeatureCollection> objects = reader.SelectTokensWithRegex<FeatureCollection>(regex);
foreach (var Feature in objects)
{
Console.WriteLine("Hello");
// Doesn't get here
}
Update:
I think the problem is with GeoJSON.net not Newtonsoft.Json as I've used the same method above to open bigger json files using
dynamic jsonFeatures = serializer.Deserialize<ExpandoObject>(reader);
Following the comment by #Martin Costello
I've come up with opening the file using a standard StreamReader line by line then convert the filtered lines back into valid GeoJSON
I'm sure there must be a better way to do this?
string Start = #"{
""type"": ""FeatureCollection"",
""name"": ""points"",
""crs"": { ""type"": ""name"", ""properties"": { ""name"": ""urn:ogc:def:crs:OGC:1.3:CRS84"" } },
""features"": [";
String End = #"]
}
";
try
{
string line;
StreamReader INfile = new StreamReader(jsonFile);
while ((line = INfile.ReadLine()) != null)
{
// for debugging
CurrentLine = line;
// Filter only the line with "place"
if (Regex.Match(line, #"\bplace\b", RegexOptions.IgnoreCase).Success)
{
// rebuild to valid GeoJSON
string json = Start + line + End;
FeatureCollection FC = JsonConvert.DeserializeObject<FeatureCollection>(json);
foreach (var Feature in FC.Features)
{
foreach (var p in Feature.Properties)
{
var Key = p.Key;
var Value = p.Value;
switch (Key.ToLower())
{
// Do Stuff
}
}
}
}
}
}
catch (Exception e)
{
Console.OutputEncoding = System.Text.Encoding.UTF8;
Console.WriteLine("Err: " + e.Message + "\n" + CurrentLine);
}

C# write multiple console lines to text file [duplicate]

I want to open a text file, append a single line to it, then close it.

You can use File.AppendAllText for that:
File.AppendAllText(#"c:\path\file.txt", "text content" + Environment.NewLine);

using (StreamWriter w = File.AppendText("myFile.txt"))
{
w.WriteLine("hello");
}

Choice one! But the first is very simple. The last maybe util for file manipulation:
//Method 1 (I like this)
File.AppendAllLines(
"FileAppendAllLines.txt",
new string[] { "line1", "line2", "line3" });
//Method 2
File.AppendAllText(
"FileAppendAllText.txt",
"line1" + Environment.NewLine +
"line2" + Environment.NewLine +
"line3" + Environment.NewLine);
//Method 3
using (StreamWriter stream = File.AppendText("FileAppendText.txt"))
{
stream.WriteLine("line1");
stream.WriteLine("line2");
stream.WriteLine("line3");
}
//Method 4
using (StreamWriter stream = new StreamWriter("StreamWriter.txt", true))
{
stream.WriteLine("line1");
stream.WriteLine("line2");
stream.WriteLine("line3");
}
//Method 5
using (StreamWriter stream = new FileInfo("FileInfo.txt").AppendText())
{
stream.WriteLine("line1");
stream.WriteLine("line2");
stream.WriteLine("line3");
}

Or you could use File.AppendAllLines(string, IEnumerable<string>)
File.AppendAllLines(#"C:\Path\file.txt", new[] { "my text content" });

Might want to check out the TextWriter class.
//Open File
TextWriter tw = new StreamWriter("file.txt");
//Write to file
tw.WriteLine("test info");
//Close File
tw.Close();

The technically best way is probably this here:
private static async Task AppendLineToFileAsync([NotNull] string path, string line)
{
if (string.IsNullOrWhiteSpace(path))
throw new ArgumentOutOfRangeException(nameof(path), path, "Was null or whitepsace.");
if (!File.Exists(path))
throw new FileNotFoundException("File not found.", nameof(path));
using (var file = File.Open(path, FileMode.Append, FileAccess.Write))
using (var writer = new StreamWriter(file))
{
await writer.WriteLineAsync(line);
await writer.FlushAsync();
}
}

File.AppendText will do it:
using (StreamWriter w = File.AppendText("textFile.txt"))
{
w.WriteLine ("-------HURRAY----------");
w.Flush();
}

//display sample reg form in notepad.txt
using (StreamWriter stream = new FileInfo("D:\\tt.txt").AppendText())//ur file location//.AppendText())
{
stream.WriteLine("Name :" + textBox1.Text);//display textbox data in notepad
stream.WriteLine("DOB : " + dateTimePicker1.Text);//display datepicker data in notepad
stream.WriteLine("DEP:" + comboBox1.SelectedItem.ToString());
stream.WriteLine("EXM :" + listBox1.SelectedItem.ToString());
}

We can use
public StreamWriter(string path, bool append);
while opening the file
string path="C:\\MyFolder\\Notes.txt"
StreamWriter writer = new StreamWriter(path, true);
First parameter is a string to hold a full file path
Second parameter is Append Mode, that in this case is made true
Writing to the file can be done with:
writer.Write(string)
or
writer.WriteLine(string)
Sample Code
private void WriteAndAppend()
{
string Path = Application.StartupPath + "\\notes.txt";
FileInfo fi = new FileInfo(Path);
StreamWriter SW;
StreamReader SR;
if (fi.Exists)
{
SR = new StreamReader(Path);
string Line = "";
while (!SR.EndOfStream) // Till the last line
{
Line = SR.ReadLine();
}
SR.Close();
int x = 0;
if (Line.Trim().Length <= 0)
{
x = 0;
}
else
{
x = Convert.ToInt32(Line.Substring(0, Line.IndexOf('.')));
}
x++;
SW = new StreamWriter(Path, true);
SW.WriteLine("-----"+string.Format("{0:dd-MMM-yyyy hh:mm:ss tt}", DateTime.Now));
SW.WriteLine(x.ToString() + "." + textBox1.Text);
}
else
{
SW = new StreamWriter(Path);
SW.WriteLine("-----" + string.Format("{0:dd-MMM-yyyy hh:mm:ss tt}", DateTime.Now));
SW.WriteLine("1." + textBox1.Text);
}
SW.Flush();
SW.Close();
}

Convert .XYZ to .csv using c#

Hi i am using this method to replace " " to "," but is failing when i try to use it on data that have 32 millions lines. Is anyone knows how to modify it to make it running?
List<String> lines = new List<String>();
//loop through each line of file and replace " " sight to ","
using (StreamReader sr = new StreamReader(inputfile))
{
int id = 1;
int i = File.ReadAllLines(inputfile).Count();
while (sr.Peek() >= 0)
{
//Out of memory issuee
string fileLine = sr.ReadLine();
//do something with line
string ttt = fileLine.Replace(" ", ", ");
//Debug.WriteLine(ttt);
lines.Add(ttt);
//lines.Add(id++, 'ID');
}
using (StreamWriter writer = new StreamWriter(outputfile, false))
{
foreach (String line in lines)
{
writer.WriteLine(line+","+id);
id++;
}
}
}
//change extension to .csv
FileInfo f = new FileInfo(outputfile);
f.MoveTo(Path.ChangeExtension(outputfile, ".csv"));
I general i am trying to convert big .XYZ file to .csv format and add incremental field at the end. I am using c# for first time in my life to be honest :) Can you help me?

See my comment above - you could modify your reading / writing as follows :
using (StreamReader sr = new StreamReader(inputfile))
{
using (StreamWriter writer = new StreamWriter(outputfile, false))
{
int id = 1;
while (sr.Peek() >= 0)
{
string fileLine = sr.ReadLine();
//do something with line
string ttt = fileLine.Replace(" ", ", ");
writer.WriteLine(ttt + "," + id);
id++;
}
}
}

Modify/replace Text in tab-base text file in C#

This question is a continuation of a past discussion HERE. So now I managed to read every line in my text file and also read the exact string in a certain column. My issue now is that I wish to modify a text value in say Column (4) of the tab-base file with another string value.
For example, the original file is like this:
ID1 25 800 Low
ID2 25 700 Low
ID3 25 600 Low
I want to Change to:
ID1 25 800 High
ID2 25 700 High
ID3 25 600 High
… here is my full code. I appreciate your help.
string route = #"C:\MyFile.txt";
FileStream fileStream2 = new FileStream(route, FileMode.Open);
var m_readFile2 = new StreamReader(fileStream2);
var m_writeFile2 = new StreamWriter(fileStream2);
string[] colArr1 = new string[100];
string[] colArr2 = new string[100];
string[] colArr3 = new string[100];
string[] colArr4 = new string[100];
int arrcount = 1;
while ((line = m_readFile2.ReadLine()) != null)
{
string col1 = "";
string col2 = "";
string col3 = "";
string col4 = "";
col1 = line.Split('\t')[0];
col2 = line.Split('\t')[1];
col3 = line.Split('\t')[2];
col4 = line.Split('\t')[3];
colArr1[arrcount] = col1;
colArr2[arrcount] = col2;
colArr3[arrcount] = col3;
colArr4[arrcount] = col4;
m_writeFile2.WriteLine("Serv" + arrcount + "\t" + "25" + "\t" + "400" + "\t" + "High");
arrcount = arrcount + 1;

KISS
string text = File.ReadAllText(route);
text = text.Replace("Low", "High");
File.WriteAllText(route, text);

I would suggest you split the lines into an array and put a new line back together:
string source = #"D:\MyFile.txt";
string destination = #"D:\MyFile2.txt";
int columnToChange = 3;
string newValueForColumn = "High";
using (FileStream sourceStream = new FileStream(source, FileMode.Open))
{
using (FileStream destinationStream = new FileStream(destination, FileMode.CreateNew))
{
using (StreamReader sourceReader = new StreamReader(sourceStream))
{
using (StreamWriter destinationWriter = new StreamWriter(destinationStream))
{
string oldLine = string.Empty;
while ((oldLine = sourceReader.ReadLine()) != null)
{
string[] values = oldLine.Split('\t');
StringBuilder newLine = new StringBuilder();
if (values.Length > columnToChange)
{
values[columnToChange] = newValueForColumn;
for (int i = 0; i < values.Length; i++)
{
newLine.Append(values[i]);
if (i + 1 < values.Length)
{
newLine.Append('\t');
}
}
}
else
{
newLine.Append(oldLine);
}
destinationWriter.WriteLine(newLine.ToString());
}
}
}
}
}
// File.Delete(source);
File.Move(source, source + ".bak");
File.Move(destination, source);
}

//Works fine Thomas Voß - I've just added a line to ensure that Column Header is
//also not changed
string source = #"D:\MyFile.txt";
string destination = #"D:\MyFile2.txt";
int columnToChange = 3;
string newValueForColumn = "High";
using (FileStream sourceStream = new FileStream(source, FileMode.Open))
{
using (FileStream destinationStream = new FileStream(destination, FileMode.CreateNew))
{
using (StreamReader sourceReader = new StreamReader(sourceStream))
{
using (StreamWriter destinationWriter = new StreamWriter(destinationStream))
{
string oldLine = string.Empty;
while ((oldLine = sourceReader.ReadLine()) != null)
{
string[] values = oldLine.Split('\t');
StringBuilder newLine = new StringBuilder();
if (values.Length > columnToChange)
{
if (values[columnToChange] != "ColumnHeaderName")
{
values[columnToChange] = newValueForColumn;
}
for (int i = 0; i < values.Length; i++)
{
newLine.Append(values[i]);
if (i + 1 < values.Length)
{
newLine.Append('\t');
}
}
}
else
{
newLine.Append(oldLine);
}
destinationWriter.WriteLine(newLine.ToString());
}
}
}
}
}
// File.Delete(source);
File.Move(source, source + ".bak");
File.Move(destination, source);
}

Reading in files that contain specific characters in C#

I have a text file named C:/test.txt:
1 2 3 4
5 6
I want to read every number in this file using StreamReader.
How can I do that?

Do you really need to use a StreamReader to do this?
IEnumerable<int> numbers =
Regex.Split(File.ReadAllText(#"c:\test.txt"), #"\D+").Select(int.Parse);
(Obviously if it's impractical to read the entire file in one hit then you'll need to stream it, but if you're able to use File.ReadAllText then that's the way to do it, in my opinion.)
For completeness, here's a streaming version:
public IEnumerable<int> GetNumbers(string fileName)
{
using (StreamReader sr = File.OpenText(fileName))
{
string line;
while ((line = sr.ReadLine()) != null)
{
foreach (string item in Regex.Split(line, #"\D+"))
{
yield return int.Parse(item);
}
}
}
}

using (StreamReader reader = new StreamReader(stream))
{
string contents = reader.ReadToEnd();
Regex r = new Regex("[0-9]");
Match m = r.Match(contents );
while (m.Success)
{
int number = Convert.ToInt32(match.Value);
// do something with the number
m = m.NextMatch();
}
}

Something like so might do the trick, if what you want is to read integers from a file and store them in a list.
try
{
StreamReader sr = new StreamReader("C:/test.txt"))
List<int> theIntegers = new List<int>();
while (sr.Peek() >= 0)
theIntegers.Add(sr.Read());
sr.Close();
}
catch (Exception e)
{
//Do something clever to deal with the exception here
}

Solution for big files:
class Program
{
const int ReadBufferSize = 4096;
static void Main(string[] args)
{
var result = new List<int>();
using (var reader = new StreamReader(#"c:\test.txt"))
{
var readBuffer = new char[ReadBufferSize];
var buffer = new StringBuilder();
while ((reader.Read(readBuffer, 0, readBuffer.Length)) > 0)
{
foreach (char c in readBuffer)
{
if (!char.IsDigit(c))
{
// we found non digit character
int newInt;
if (int.TryParse(buffer.ToString(), out newInt))
{
result.Add(newInt);
}
buffer.Remove(0, buffer.Length);
}
else
{
buffer.Append(c);
}
}
}
// check buffer
if (buffer.Length > 0)
{
int newInt;
if (int.TryParse(buffer.ToString(), out newInt))
{
result.Add(newInt);
}
}
}
result.ForEach(Console.WriteLine);
Console.ReadKey();
}
}

I might be wrong but with StreamReader you cannot set delimeter.
But you can use String.Split() to set delimeter (it is space in your case?) and extract all numbers into separate array.

Something like this ought to work:
using (var sr = new StreamReader("C:/test.txt"))
{
var s = sr.ReadToEnd();
var numbers = (from x in s.Split('\n')
from y in x.Split(' ')
select int.Parse(y));
}

Something like this:
using System;
using System.IO;
class Test
{
public static void Main()
{
string path = #"C:\Test.txt";
try
{
if( File.Exists( path ) )
{
using( StreamReader sr = new StreamReader( path ) )
{
while( sr.Peek() >= 0 )
{
char c = ( char )sr.Read();
if( Char.IsNumber( c ) )
Console.Write( c );
}
}
}
}
catch (Exception e)
{
Console.WriteLine("The process failed: {0}", e.ToString());
}
}
}