We Keep Coding

How can I reduce memory usage when parse json in c#

I'm trying to parse huge json file to 2d array.
I can parse. But required memory is almost 10times.
My sample.json file has 100,000 rows, each with a different item.
If sample.json is 500MB this code need 5GB.
How can i reduce memory usage?
I use Newtonsoft.Json, .Net6.0
Read from json
static void Read()
{
List<Dictionary<string, string>> rows = new List<Dictionary<string, string>>();
string path = #"D:\small.json";
using (FileStream fsRead = File.Open(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
using (BufferedStream bsRead = new BufferedStream(fsRead))
using (StreamReader srRead = new StreamReader(bsRead))
{
string? line;
while ((line = srRead.ReadLine()) != null)
{
JObject jsonObject = JObject.Parse(line);
MakeRowData(jsonObject, out var row);
rows.Add(row);
}
}
}
Make row
private static void MakeRowData(JObject jsonData, out Dictionary<string, string> row)
{
Dictionary<string, string> output = new Dictionary<string, string>();
foreach (var item in jsonData)
{
int childSize = 0;
if (item.Value != null)
{
childSize = item.Value.Children().Count();
///if Item has child, explore deep
if (childSize > 0)
{
ExploreChild(item.Value, ref output);
}
///or not just add new item
else
{
string str = item.Value.ToString();
output[item.Key] = str ?? "";
}
}
}
row = output;
}
private static void ExploreChild(JToken jToken, ref Dictionary<string, string> row)
{
foreach (var item in jToken)
{
int childSize = item.Children().Count();
///if Item has child, explore deep
if (childSize > 0)
{
ExploreChild(item, ref row);
}
///or not just add new item
else
{
string path = jToken.Path.Replace('[', '(').Replace(']', ')');
string str = jToken.First.ToString();
row[path] = str?? "";
}
}
}
EDIT
Add Sample.json
It is set of json strings.
And Fields are not fixed.
Sample.json
{Field1:0,Field2:1,Field2:3}
{Field1:0,Field5:1,Field6:3}
{Field1:0,Field7:1,Field9:3}
{Field1:0,Field13:1,Field50:3,Field57:3}
...

You can try replacing the recursive exploring children with the iterative one. Something like this:
private static void MakeRowData(JObject jsonData, out Dictionary<string, string> row)
{
Dictionary<string, string> output = new Dictionary<string, string>();
foreach (var item in jsonData)
{
if (item.Value != null)
{
///if Item has child, explore deep
if (item.Value.HasValues)
{
var queue = new Queue<JToken>();
queue.Enqueue(item.Value);
while (queue.Any())
{
var currItem = queue.Dequeue();
if (currItem.HasValues)
{
foreach(var child in item)
queue.Enqueue(child);
}
else
{
// add item without children to row here
}
}
}
///or not just add new item
else
{
string str = item.Value.ToString();
output[item.Key] = str ?? "";
}
}
}
row = output;
}
Recursive calls, unless it is a tail recursion, keep the stack of a method they were called from. This can lead to extensive memory usage.

Inserting data to .CSV file at the same time using foreach

I am new here and actually very new to c#.
In a nutshell, I am using c# via Visual Studio, I am calling a data from a database and I want to save these data in a .csv file. The problem now is that I want to save these data on two columns at the same time.
My code do write them in a file but shifted not on the right rows.
Dictionary<string, string> elementNames = new Dictionary<string, string>();
Dictionary<string, string> elementTypes = new Dictionary<string, string>();
var nodes = webservice.nepService.GetAllElementsOfElementType(webservice.ext, "Busbar", ref elementNames, ref elementTypes);
Dictionary<string, string> nodeResults = new Dictionary<string, string>();
Dictionary<string, string> nodeResults1 = new Dictionary<string, string>();
foreach (var nodename in elementNames.Values)
{
var nodeRes = webservice.nepService.GetResultElementByName(webservice.ext, nodename, "Busbar", -1, "LoadFlow", null);
var Uvolt = GetXMLAttribute(nodeRes, "U");
nodeResults.Add(nodename, Uvolt);
var Upercentage = GetXMLAttribute(nodeRes, "Up");
nodeResults1.Add(nodename, Upercentage);
StringBuilder strBldr = new StringBuilder();
string outputFile = #"C:\Users\12.csv";
string separator = ",";
foreach (var res in nodeResults)
{
strBldr.AppendLine($"{res.Key}{separator}{res.Value}");
}
foreach (var res1 in nodeResults1)
{
strBldr.AppendLine($"{separator}{separator}{res1.Value}");
}
File.WriteAllText(outputFile, strBldr.ToString());
}
this is the output of the previous code:
https://ibb.co/T4trQC3
I want these shifted values to move up beside the other values like that:
https://ibb.co/4S25v0h
Thank you

if you look to the code you are using AppendLine
strBldr.AppendLine($"{separator}{separator}{res1.Value}");
and if you want to append on same line just use Append
strBldr.Append($"{separator}{separator}{res1.Value}");
EDITED:
in linq you can use Zip function to zip to lists
// using System.Linq;
var results = Results.Zip(Results1, (firstList, secondList) => firstList.Key + "," + firstList.Value + "," + secondList.Value);
Edit Full example
public static IDictionary<string, string> Results { get; set; }
public static IDictionary<string, string> Results1 { get; set; }
private static void Main(string[] args)
{
StringBuilder strBldr = new StringBuilder();
string outputFile = #"D:\12.csv";
Results = new Dictionary<string, string>()
{
{"N1", "20"},
{"N2", "0.399992"},
{"N3", "0.369442"},
{"N4", "0.369976"}
};
Results1 = new Dictionary<string, string>()
{
{"N1", "100"},
{"N2", "99.9805"},
{"N3", "92.36053"},
{"N4", "92.49407"}
};
IEnumerable<string> results = Results.Zip(Results1,
(firstList, secondList) => firstList.Key + "," + firstList.Value + "," + secondList.Value);
foreach (string res1 in results)
{
strBldr.AppendLine(res1);
}
File.WriteAllText(outputFile, strBldr.ToString());
}
for faster code you can try this
HashSet<Tuple<string, string, string>> values = new HashSet<Tuple<string, string, string>>();
var nodes = webservice.nepService.GetAllElementsOfElementType(webservice.ext, "Busbar", ref elementNames, ref elementTypes);
foreach (var nodename in elementNames.Values)
{
var nodeRes = webservice.nepService.GetResultElementByName(webservice.ext, nodename, "Busbar", -1, "LoadFlow", null);
var Uvolt = GetXMLAttribute(nodeRes, "U");
var Upercentage = GetXMLAttribute(nodeRes, "Up");
values.Add(Tuple.Create(nodename, Uvolt, Upercentage));
}
var output = string.Join("\n", values.ToList().Select(tuple => $"{tuple.Item1},{tuple.Item2},{tuple.Item3}").ToList());
string outputFile = #"C:\Users\12.csv";
File.WriteAllText(outputFile, output);

if the rowCount for Results and Results1 are same and the keys are in the same order, try:
for (int i = 0; i < Results.Count; i++)
strBldr.AppendLine($"{Results[i].Key}{separator}{Results[i].Value}{separator}{Results1[i].Value}");
Or, if the rows are not in the same order, try:
foreach (var res in Results)
strBldr.AppendLine($"{res.Key}{separator}{res.Value}{separator}{Results1.Single(x => x.Key == res.Key).Value}");

Extracting a dictionary from sparse csv file

I have a sparsely populated excel file I want to extract two columns into a dictionary in C#. I have tried the following. This fails when it reads the blank lines. Is there a cleaner way to achieve the same. I don't care about any other values here. Just a mapping of AR ID to AR Type would do.
public class Table
{
private Dictionary<string, string> _ARID_ARTypeValues = new Dictionary<string, string>();
private string _arId;
public Table(string arId)
{
_arId = arId;
}
public void AddValue(string key, string value)
{
_ARID_ARTypeValues.Add(key, value);
}
}
public static IDictionary ParseCsvFile(StreamReader reader)
{
Dictionary<string, Table> tables = new Dictionary<string, Table>();
// First line contains column names.
var columnNames = reader.ReadLine().Split(',');
for (int i = 1; i < columnNames.Length; ++i)
{
var columnName = columnNames[i];
var ntable = new Table(columnName);
if ((columnName == "AR ID") || (columnName == "AR Type"))
{
tables.Add(columnName, ntable);
}
}
var line = reader.ReadLine();
while (line != null)
{
var columns = line.Split(',');
for (int j = 1; j < columns.Length; ++j)
{
var table = tables[columnNames[j]];
table.AddValue(columns[0], columns[j]);
}
line = reader.ReadLine();
}
return tables;
}

I would just use a CSV library, like CsvHelper and read the csv file with that.
Dictionary<string, string> arIdToArTypeMapping = new Dictionary<string, string>();
using (var sr = File.OpenText("test.csv"))
{
var csvConfiguration = new CsvConfiguration
{
SkipEmptyRecords = true
};
using (var csvReader = new CsvReader(sr, csvConfiguration))
{
while (csvReader.Read())
{
string arId = csvReader.GetField("AR ID");
string arType = csvReader.GetField("AR Type");
if (!string.IsNullOrEmpty(arId) && !string.IsNullOrEmpty(arType))
{
arIdToArTypeMapping.Add(arId, arType);
}
}
}
}

You can use Cinchoo ETL - an open source library, to read the csv and convert them to dictionary as simple as with few lines of code shown below
using (var parser = new ChoCSVReader("Dict1.csv")
.WithField("AR_ID", 7)
.WithField("AR_TYPE", 8)
.WithFirstLineHeader(true)
.Configure(c => c.IgnoreEmptyLine = true)
)
{
var dict = parser.ToDictionary(item => item.AR_ID, item => item.AR_TYPE);
foreach (var kvp in dict)
Console.WriteLine(kvp.Key + " " + kvp.Value);
}
Hope this helps.
Disclaimer: I'm the author of this library.

C# Read text file and print out in to a table

I'm trying to read a text file and print out into a table.
I want the output to be this
But now I having different output
var column1 = new List<string>();
var column2 = new List<string>();
var column3 = new List<string>();
using (var rd = new StreamReader(#"C:\test.txt"))
{
while (!rd.EndOfStream)
{
var splits = rd.ReadLine().Split(';');
column1.Add(splits[0]);
column2.Add(splits[1]);
column3.Add(splits[2]);
}
}
Console.WriteLine("Date/Time \t Movie \t Seat");
foreach (var element in column1) Console.WriteLine(element);
foreach (var element in column2) Console.WriteLine(element);
foreach (var element in column3) Console.WriteLine(element);

You can use Linq to construct a convenient structure (e.g. List<String[]>) and then print out all the data wanted:
List<String[]> data = File
.ReadLines(#"C:\test.txt")
//.Skip(1) // <- uncomment this to skip caption if the csv has it
.Select(line => line.Split(';').Take(3).ToArray()) // 3 items only
.ToList();
// Table output (wanted one):
String report = String.Join(Environment.NewLine,
data.Select(items => String.Join("\t", items)));
Console.WriteLine(report);
// Column after column output (actual one)
Console.WriteLine(String.Join(Environment.NewLine, data.Select(item => item[0])));
Console.WriteLine(String.Join(Environment.NewLine, data.Select(item => item[1])));
Console.WriteLine(String.Join(Environment.NewLine, data.Select(item => item[2])));
EDIT: if you want to choose the movie, buy the ticket etc. elaborate the structure:
// Create a custom class where implement your logic
public class MovieRecord {
private Date m_Start;
private String m_Name;
private int m_Seats;
...
public MovieRecord(DateTime start, String name, int seats) {
...
m_Seats = seats;
...
}
...
public String ToString() {
return String.Join("\t", m_Start, m_Name, m_Seats);
}
public void Buy() {...}
...
}
And then convert to conventinal structure:
List<MovieRecord> data = File
.ReadLines(#"C:\test.txt")
//.Skip(1) // <- uncomment this to skip caption if the csv has it
.Select(line => {
String items[] = line.Split(';');
return new MovieRecord(
DateTime.ParseExact(items[0], "PutActualFormat", CultureInfo.InvariantCulture),
items[1],
int.Parse(items[2]));
}
.ToList();
And the table output will be
Console.Write(String.Join(Envrironment.NewLine, data));

Don't use Console.WriteLine if you want to add a "column". You should also use a single List<string[]> instead of multiple List<string>.
List<string[]> allLineFields = new List<string[]>();
using (var rd = new StreamReader(#"C:\test.txt"))
{
while (!rd.EndOfStream)
{
var splits = rd.ReadLine().Split(';');
allLineFields.Add(splits);
}
}
Console.WriteLine("Date/Time \t Movie \t Seat");
foreach(string[] line in allLineFields)
Console.WriteLine(String.Join("\t", line));
In general you should use a real csv parser if you want to parse a csv-file, not string methods or regex.
You could use the TextFieldParser which is the only one available in the framework directly:
var allLineFields = new List<string[]>();
using (var parser = new Microsoft.VisualBasic.FileIO.TextFieldParser(#"C:\test.txt"))
{
parser.Delimiters = new string[] { ";" };
parser.HasFieldsEnclosedInQuotes = false; // very useful
string[] lineFields;
while ((lineFields = parser.ReadFields()) != null)
{
allLineFields.Add(lineFields);
}
}
You need to add a reference to the Microsoft.VisualBasic dll to your project.
There are other available: Parsing CSV files in C#, with header

You could attempt to solve this in a more Object-Orientated manner, which might make it a bit easier for you to work with:
You can declare a simple class to represent a movie seat:
class MovieSeat
{
public readonly string Date, Name, Number;
public MovieSeat(string source)
{
string[] data = source.Split(';');
Date = data[0];
Name = data[1];
Number = data[2];
}
}
And then you can read in and print out the data in a few lines of code:
// Read in the text file and create a new MovieSeat object for each line in the file.
// Iterate over all MovieSeat objets and print them to console.
foreach(var seat in File.ReadAllLines(#"C:\test.txt").Select(x => new MovieSeat(x)))
Console.WriteLine(string.Join("\t", seat.Date, seat.Name, seat.Number));

Dictionaries in C#

This program is meant to read in a csv file and create a dictionary from it, which is then used to translate a word typed into a textbox (txtINPUT) and output the result to another textbox (txtOutput).
The program doesnt translate anything and always outputs "No translation found."
I've never used the dictionary class before so I dont know where the problem is coming from.
Thanks for any help you can give me.
Dictionary<string, string> dictionary;
private void CreateDictionary()
{
//Load file
List<string> list = new List<string>();
using (StreamReader reader = new StreamReader("dictionarylist.csv"))
{
string line;
while ((line = reader.ReadLine()) != null)
{
//Add to dictionary
dictionary = new Dictionary<string, string>();
string[] split = line.Split(',');
dictionary.Add(split[0], split[1]);
}
}
}
private void btnTranslate_Click(object sender, EventArgs e)
{
CreateDictionary();
string outputString = null;
if (dictionary.TryGetValue(txtInput.Text, out outputString))
{
txtOutput.Text = outputString;
}
else
{
txtOutput.Text = ("No translation found");
}
}

You are creating a new instance of a Dictionary each loop cycle, basically overwriting it each time you read a line. Move this line out of the loop:
// Instantiate a dictionary
var map = new Dictionary<string, string>();
Also why not load dictionary one time, you are loading it each button click, this is not efficient.
(>=.NET 3) The same using LINQ ToDictionary():
usign System.Linq;
var map = File.ReadAllLines()
.Select(l =>
{
var pair = l.Split(',');
return new { First = pair[0], Second = pair[1] }
})
.ToDictionary(k => k.First, v => v.Second);

In your while loop, you create a new dictionary every single pass!
You want to create one dictionary, and add all the entries to that:
while ((line = reader.ReadLine()) != null)
{
//Add to dictionary
dictionary = new Dictionary<string, string>(); /* DON'T CREATE NEW DICTIONARIES */
string[] split = line.Split(',');
dictionary.Add(split[0], split[1]);
}
You should do it more like this:
List<string> list = new List<string>();
dictionary = new Dictionary<string, string>(); /* CREATE ONE DICTIONARY */
using (StreamReader reader = new StreamReader("dictionarylist.csv"))
{
string line;
while ((line = reader.ReadLine()) != null)
{
string[] split = line.Split(',');
dictionary.Add(split[0], split[1]);
}
}