We Keep Coding

How can I reduce memory usage when parse json in c#

I'm trying to parse huge json file to 2d array.
I can parse. But required memory is almost 10times.
My sample.json file has 100,000 rows, each with a different item.
If sample.json is 500MB this code need 5GB.
How can i reduce memory usage?
I use Newtonsoft.Json, .Net6.0
Read from json
static void Read()
{
List<Dictionary<string, string>> rows = new List<Dictionary<string, string>>();
string path = #"D:\small.json";
using (FileStream fsRead = File.Open(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
using (BufferedStream bsRead = new BufferedStream(fsRead))
using (StreamReader srRead = new StreamReader(bsRead))
{
string? line;
while ((line = srRead.ReadLine()) != null)
{
JObject jsonObject = JObject.Parse(line);
MakeRowData(jsonObject, out var row);
rows.Add(row);
}
}
}
Make row
private static void MakeRowData(JObject jsonData, out Dictionary<string, string> row)
{
Dictionary<string, string> output = new Dictionary<string, string>();
foreach (var item in jsonData)
{
int childSize = 0;
if (item.Value != null)
{
childSize = item.Value.Children().Count();
///if Item has child, explore deep
if (childSize > 0)
{
ExploreChild(item.Value, ref output);
}
///or not just add new item
else
{
string str = item.Value.ToString();
output[item.Key] = str ?? "";
}
}
}
row = output;
}
private static void ExploreChild(JToken jToken, ref Dictionary<string, string> row)
{
foreach (var item in jToken)
{
int childSize = item.Children().Count();
///if Item has child, explore deep
if (childSize > 0)
{
ExploreChild(item, ref row);
}
///or not just add new item
else
{
string path = jToken.Path.Replace('[', '(').Replace(']', ')');
string str = jToken.First.ToString();
row[path] = str?? "";
}
}
}
EDIT
Add Sample.json
It is set of json strings.
And Fields are not fixed.
Sample.json
{Field1:0,Field2:1,Field2:3}
{Field1:0,Field5:1,Field6:3}
{Field1:0,Field7:1,Field9:3}
{Field1:0,Field13:1,Field50:3,Field57:3}
...

You can try replacing the recursive exploring children with the iterative one. Something like this:
private static void MakeRowData(JObject jsonData, out Dictionary<string, string> row)
{
Dictionary<string, string> output = new Dictionary<string, string>();
foreach (var item in jsonData)
{
if (item.Value != null)
{
///if Item has child, explore deep
if (item.Value.HasValues)
{
var queue = new Queue<JToken>();
queue.Enqueue(item.Value);
while (queue.Any())
{
var currItem = queue.Dequeue();
if (currItem.HasValues)
{
foreach(var child in item)
queue.Enqueue(child);
}
else
{
// add item without children to row here
}
}
}
///or not just add new item
else
{
string str = item.Value.ToString();
output[item.Key] = str ?? "";
}
}
}
row = output;
}
Recursive calls, unless it is a tail recursion, keep the stack of a method they were called from. This can lead to extensive memory usage.

Give each row in a list an ID

I'm trying to create a timetables.txt from other files, and I have this so far.
static void Main(string[] args)
{
string Routes = #"C:\Users\peepee poopoo\gtfs\routes.txt";
var column1 = new List<string>();
var column2 = new List<string>();
using (var rd = new StreamReader(Routes))
{
while (!rd.EndOfStream)
{
var splits = rd.ReadLine().Split(',');
column1.Add(splits[0]);
column2.Add(splits[1]);
}
}
// print column1
Console.WriteLine("Column 1:");
foreach (var element in column1)
Console.WriteLine(element);
// print column2
Console.WriteLine("Column 2:");
foreach (var element in column2)
Console.WriteLine(element);
}
However, I need the first column of every row in the list to have a number that just counts up from 1. How can I do this?

Just see the code written below and add it.
var dictionary = new Dictionart<string, string>();
using (var rd = new StreamReader(Routes))
{
while (!rd.EndOfStream)
{
var splits = rd.ReadLine().Split(',');
dictionary.Add(splits[0], splits[1]);
}
}
foreach(var item in dictionary)
{
Console.WriteLine(item.Key + "\t" + item.Value);
}

This was what worked:
string Routes = #"C:\Users\peepee poopoo\gtfs\routes.txt";
var dictionary = new Dictionary<string, string>();
using (var rd = new StreamReader(Routes))
{
while (!rd.EndOfStream)
{
var splits = rd.ReadLine().Split(',');
dictionary.Add(splits[0], splits[1]);
//dictionary.Add(splits[0], splits[1]);
}
}
foreach (var item in dictionary)
{
Console.WriteLine(item.Key + "\t" + item.Value);
}
The output then becomes
this

Sensible way to output content of dictionary as valid html tags?

I am trying to generate hreflang tags like so:
<link hreflang="en-DE" rel="alternate" href="/en-DE" />
I currently am putting together my dictionary as below, and I thought I could use a 2nd foreach loop to generate the tags with the key value pairs:
public static IHtmlString HrefLangLinks(this PageData currentPage)
{
var hrefLangTags = string.Empty;
var availablePageLanguages = currentPage.ExistingLanguages.Select(culture => culture.Name).ToArray();
Dictionary<string, string> langs = new Dictionary<string, string>();
var contentLoader = ServiceLocator.Current.GetInstance<IContentLoader>();
var urlResolver = ServiceLocator.Current.GetInstance<UrlResolver>();
foreach (string cultureName in availablePageLanguages)
{
var culturePage = contentLoader.Get<PageData>(currentPage.ContentGuid, new LanguageSelector(cultureName));
var culturePath = urlResolver.GetVirtualPath(culturePage.ContentLink, culturePage.Language.Name);
langs.Add(cultureName, culturePath.GetUrl());
foreach (KeyValuePair<string, string> entry in langs)
{
hrefLangTags += ("<link hreflang=\"{0}\" rel=\"alternate\" href=\"{1}\" >", langs.Keys, langs.Values);
}
}
return new HtmlString(hrefLangTags);
}
Is there a simple and elegant way to iterate through the dictionary and create my tags?

Why are you even using a dictionay? You could just iterate through "availablePageLanguages" and append the needed data to your "hrefLangTags".
So like this:
public static IHtmlString HrefLangLinks(this PageData currentPage)
{
var hrefLangTags = string.Empty;
var availablePageLanguages = currentPage.ExistingLanguages.Select(culture => culture.Name).ToArray();
var contentLoader = ServiceLocator.Current.GetInstance<IContentLoader>();
var urlResolver = ServiceLocator.Current.GetInstance<UrlResolver>();
foreach (string cultureName in availablePageLanguages)
{
var culturePage = contentLoader.Get<PageData>(currentPage.ContentGuid, new LanguageSelector(cultureName));
var culturePath = urlResolver.GetVirtualPath(culturePage.ContentLink, culturePage.Language.Name);
hrefLangTags += String.Format("<link hreflang=\"{0}\" rel=\"alternate\" href=\"{1}\" >", culturName, culturePath.GetUrl());
}
return new HtmlString(hrefLangTags);
}

Your method does not make use of the first principle of SOLID (Single responsibility). You want to return links based on data input. But you are generating the needed data in the same method that is responsible for converting it to a list of IHtmlStrings.
The code beneath is may not be compact but it's a bit easier to read because the nested foreach loop is removed. You could call the method below by chaining them like: CreateLanguageLinksList(CreateWorkingLanguageDictionary(currentPage)) or do it like so
var languageDictionary = CreateWorkingLanguageDictionary(currentPage);
var listOfIHtmlStrings = CreateLanguageLinksList(languageDictionary);
The code above is easier to read, so the cognetive load is less straining on the mind.
public Dictionary<string, string> CreateWorkingLanguageDictionary(this PageData currentPage)
{
var languageDictionary = new Dictionary<string, string>();
var contentLoader = ServiceLocator.Current.GetInstance<IContentLoader>();
var urlResolver = ServiceLocator.Current.GetInstance<UrlResolver>();
foreach (string cultureName in availablePageLanguages)
{
var culturePage = contentLoader.Get<PageData>(currentPage.ContentGuid, new LanguageSelector(cultureName));
var culturePath = urlResolver.GetVirtualPath(culturePage.ContentLink, culturePage.Language.Name);
languageDictionary.Add(cultureName, culturePath.GetUrl());
}
return languageDictionary;
}
public IList<IHtmlString> CreateLanguageLinksList(Dictionary<string, string> langs)
{
var htmlStringList = new List<IHtmlString>();
foreach (KeyValuePair<string, string> entry in langs)
{
htmlStringList.add(new HtmlString("<link hreflang=\"{0}\" rel=\"alternate\" href=\"{1}\" >", langs.Keys, langs.Values));
}
return htmlStringList;
}

Just for fun, you could do it in linq, it's a little more succinct:
public static IHtmlString HrefLangLinks(this PageData currentPage)
{
var contentLoader = ServiceLocator.Current.GetInstance<IContentLoader>();
var urlResolver = ServiceLocator.Current.GetInstance<UrlResolver>();
var links = currentPage.ExistingLanguages.Select(culture =>
{
var culturePage = contentLoader.Get<PageData>(currentPage.ContentGuid, new LanguageSelector(culture.Name));
var culturePath = urlResolver.GetVirtualPath(culturePage.ContentLink, culturePage.Language.Name);
return $"<link hreflang=\"{culture.Name}\" rel=\"alternate\" href=\"{culturePath.GetUrl()}\" >";
});
return new HtmlString(string.Join("",links));
}
Disclaimer: This is totally untested or verified.

Extracting a dictionary from sparse csv file

I have a sparsely populated excel file I want to extract two columns into a dictionary in C#. I have tried the following. This fails when it reads the blank lines. Is there a cleaner way to achieve the same. I don't care about any other values here. Just a mapping of AR ID to AR Type would do.
public class Table
{
private Dictionary<string, string> _ARID_ARTypeValues = new Dictionary<string, string>();
private string _arId;
public Table(string arId)
{
_arId = arId;
}
public void AddValue(string key, string value)
{
_ARID_ARTypeValues.Add(key, value);
}
}
public static IDictionary ParseCsvFile(StreamReader reader)
{
Dictionary<string, Table> tables = new Dictionary<string, Table>();
// First line contains column names.
var columnNames = reader.ReadLine().Split(',');
for (int i = 1; i < columnNames.Length; ++i)
{
var columnName = columnNames[i];
var ntable = new Table(columnName);
if ((columnName == "AR ID") || (columnName == "AR Type"))
{
tables.Add(columnName, ntable);
}
}
var line = reader.ReadLine();
while (line != null)
{
var columns = line.Split(',');
for (int j = 1; j < columns.Length; ++j)
{
var table = tables[columnNames[j]];
table.AddValue(columns[0], columns[j]);
}
line = reader.ReadLine();
}
return tables;
}

I would just use a CSV library, like CsvHelper and read the csv file with that.
Dictionary<string, string> arIdToArTypeMapping = new Dictionary<string, string>();
using (var sr = File.OpenText("test.csv"))
{
var csvConfiguration = new CsvConfiguration
{
SkipEmptyRecords = true
};
using (var csvReader = new CsvReader(sr, csvConfiguration))
{
while (csvReader.Read())
{
string arId = csvReader.GetField("AR ID");
string arType = csvReader.GetField("AR Type");
if (!string.IsNullOrEmpty(arId) && !string.IsNullOrEmpty(arType))
{
arIdToArTypeMapping.Add(arId, arType);
}
}
}
}

You can use Cinchoo ETL - an open source library, to read the csv and convert them to dictionary as simple as with few lines of code shown below
using (var parser = new ChoCSVReader("Dict1.csv")
.WithField("AR_ID", 7)
.WithField("AR_TYPE", 8)
.WithFirstLineHeader(true)
.Configure(c => c.IgnoreEmptyLine = true)
)
{
var dict = parser.ToDictionary(item => item.AR_ID, item => item.AR_TYPE);
foreach (var kvp in dict)
Console.WriteLine(kvp.Key + " " + kvp.Value);
}
Hope this helps.
Disclaimer: I'm the author of this library.

How to split a string in C#?

I am trying to parse the following string and get the result.
string test = "SiteA:Pages:1,SiteB:Pages:4,SiteA:Documents:6"
I am trying to get the following result after the split.
string SiteA = "Pages:1,Documents:6"
string SiteB = "Pages:4"
Here is my code but it doesn't seem to be working. How can I get all related "SiteA" and "SiteB"?
List<string> listItem = new List<string>();
string[] keyPairs = test.Split(',');
string[] item;
foreach (string keyPair in keyPairs)
{
item = keyPair.Split(':');
listItem.Add(string.Format("{0}:{1}", item[0].Trim(), item[1].Trim()));
}

I would use a Lookup for this:
string test = "SiteA:Pages:1,SiteB:Pages:4,SiteA:Documents:6";
var listItemsBySite = test.Split(',')
.Select(x => x.Split(':'))
.ToLookup(x => x[0],
x => string.Format("{0}:{1}",
x[1].Trim(),
x[2].Trim()));
You can then use it like this:
foreach (string item in listItemsBySite["SiteA"])
{
Console.WriteLine(item);
}

Here's my solution... pretty elegant in LINQ, you can use anonymous objects, Tuples, KeyValuePair, or your own custom class. I'm just using an anonymous type.
string test = "SiteA:Pages:1,SiteB:Pages:4,SiteA:Documents:6";
var results = test
.Split(',')
.Select(item => item.Split(':'))
.ToLookup(s => s[0], s => new { Key = s[1], Value = s[2] });
// This code just for display purposes
foreach (var site in results)
{
Console.WriteLine("Site: " + site.Key);
foreach (var value in site)
{
Console.WriteLine("\tKey: " + value.Key + " Value: " + value.Value);
}
}

Here is my code:
string test = "SiteA:Pages:1,SiteB:Pages:4,SiteA:Documents:6";
string[] data = test.Split(',');
Dictionary<string, string> dic = new Dictionary<string, string>();
for(int i = 0; i < data.Length; i++) {
int index = data[i].IndexOf(':');
string key = data[i].Substring(0, index);
string value = data[i].Substring(index + 1);
if(!dic.ContainsKey(key))
dic.Add(key, value);
else
dic[key] = string.Format("{0}, {1}", new object[] { dic[key], value });
}

Here is how I would do it:
SortedList<string, StringBuilder> listOfLists = new SortedList<string, StringBuilder>();
string[] keyPairs = test.Split(',');
foreach (string keyPair in keyPairs)
{
string[] item = keyPair.Split(':');
if (item.Length >= 3)
{
string nextValue = string.Format("{0}:{1}", item[1].Trim(), item[2].Trim());
if (listOfLists.ContainsKey(item[0]))
listOfLists[item[0]].Append(nextValue);
else
{
StringBuilder sb = new StringBuilder();
sb.Append(nextValue);
listOfLists.Add(item[0], sb);
}
}
}
foreach (KeyValuePair<string, StringBuilder> nextCollated in listOfLists)
System.Console.WriteLine(nextCollated.Key + ":" + nextCollated.Value.ToString());

This is what I would do (tested).
(However, does assume that all items will be correctly formatted).
And of course, it's not really optimized.
static void Main(string[] args)
{
string test = "SiteA:Pages:1,SiteB:Pages:4,SiteA:Documents:6";
Dictionary<String, List<String>> strings = new Dictionary<string, List<string>>();
String[] items = test.Split(',');
foreach (String item in items)
{
List<String> itemParts = item.Split(':').ToList();
String firstPart = itemParts[0];
itemParts.RemoveAt(0);
String secondPart = String.Join(":", itemParts);
if (!strings.ContainsKey(firstPart))
strings[firstPart] = new List<string>();
strings[firstPart].Add(secondPart);
}
// This is how you would consume it
foreach (String key in strings.Keys)
{
List<String> keyItems = strings[key];
Console.Write(key + ": ");
foreach (String item in keyItems)
Console.Write(item + " ");
Console.WriteLine();
}
}

Here's a solution using LINQ:
string test = "SiteA:Pages:1,SiteB:Pages:4,SiteA:Documents:6";
var dict = test
.Split(',')
.GroupBy(s => s.Split(':')[0])
.ToDictionary(g => g.Key,
g => string.Join(",",
g.Select(i => string.Join(":", i.Split(':').Skip(1)))
.ToArray()));

using System;
using System.Linq;
using System.Text.RegularExpressions;
class MyClass
{
static void Main(string[] args)
{
string test = "SiteA:Pages:1,SiteB:Pages:4,SiteA:Documents:6";
var sites = test.Split(',')
.Select(p => p.Split(':'))
.Select(s => new { Site = s[0], Key = s[1], Value = s[2] })
.GroupBy(s => s.Site)
.ToDictionary(g => g.Key, g => g.ToDictionary(e => e.Key, e => e.Value));
foreach (var site in sites)
foreach (var key in site.Value.Keys)
Console.WriteLine("Site {0}, Key {1}, Value {2}", site.Key, key, site.Value[key]);
// in your preferred format:
var SiteA = string.Join(",", sites["SiteA"].Select(p => string.Format("{0}:{1}", p.Key, p.Value)));
var SiteB = string.Join(",", sites["SiteB"].Select(p => string.Format("{0}:{1}", p.Key, p.Value)));
Console.WriteLine(SiteA);
Console.WriteLine(SiteB);
}
}