We Keep Coding

Iterate IDictionary<string, string> with dynamic nested JSON as value in C#

My application receives Kafka messages that contain a Dictionary<string,string> as one of the properties, and its values could be a nested (however dynamic) json string, and I need to iterate through this unknown json. I am struggling to find a logic and even the best data structure to do this iteration.
Examples of the dictionary (mocked data):
//could have complex nested json string as value
"reward":"{
'xp':'200',
'gp':'150',
'loot':'{
'item':'sword',
'rarity': 'low'
}'
}",
"achievement":"win_match"
// while other messages might be simple
"type":"generic_xp",
"percent":"100",
"status":"complete"
Serialized version of a real message:
"{\"player_stats\":\"{\\\"assist\\\":0,\\\"deaths\\\":0,\\\"kills\\\":0,\\\"team_index\\\":2}\",\"round_attr\":\"{\\\"max_player_count\\\":4,\\\"rdur\\\":0,\\\"round\\\":1,\\\"team_player_count\\\":{\\\"team_1\\\":1,\\\"team_2\\\":0},\\\"team_score\\\":0}\",\"custom\":\"{\\\"armor\\\":\\\"armor_pickup_lv2\\\",\\\"balance\\\":550,\\\"helmet\\\":\\\"helmet_pickup_lv2\\\",\\\"misc\\\":[{\\\"count\\\":48,\\\"item_id\\\":\\\"shotgun\\\"},{\\\"count\\\":120,\\\"item_id\\\":\\\"bullet\\\"},{\\\"count\\\":2,\\\"item_id\\\":\\\"health_pickup_combo_small\\\"},{\\\"count\\\":2,\\\"item_id\\\":\\\"health_pickup_health_small\\\"}],\\\"weapon_1\\\":\\\"mp_weapon_semipistol\\\",\\\"weapon_2\\\":\\\"mp_weapon_shotgun_pistol\\\"}\",\"gdur\":\"0\"}"
To complicate even more
Create a model class is not an option because this json is completely dynamic
Flatting the dictionary is not possible because the json may have duplicated key names, but under different hierarchy
I cant request to change the Kafka message
What I am trying to do
The end user will define rules that I need to check if I find a match. For instance, a rule could be reward.xp == 200 or reward.loot.rarity == high or status == complete. These rules will be defined by the user so it cant be hardcoded, however I can decide with data structure to use to save them. So for each Kafka message, I have to iterate through that dictionary and try to find a match with the rules.
What I have tried
I ve tried JsonConvert.Deserialize to object, dynamic, ExpandoObject and none could handle the nested json hierarchy. They just got the 1st level correct. Same result with JObject.Parse as well.

Parse the JSON using whatever parser you like (I used Newtonsoft.Json).
Then recursively visit the hierarchy and copy each property to a flat list using the full path to each property value as a key. You can then iterate that flat list.
Edit: Comment requested supporting arrays, so this version does.
https://dotnetfiddle.net/6ykHT0
using System;
using Newtonsoft.Json.Linq;
using System.Linq;
using System.Collections.Generic;
public class Program
{
public static void Main()
{
string json = #"{
'reward': {
'xp': '200',
'gp': '150',
'loot': {
'item': 'sword',
'rarity': 'low',
'blah': {
'socks': 5
}
},
'arrayofint': [1,2,3,4],
'arrayofobj': [
{
'foo': 'bar',
'stuff': ['omg!', 'what?!']
},
{
'foo': 'baz',
'stuff': ['a', 'b']
}
],
'arrayofarray': [
[1,2,3],
[4,5,6]
],
'arrayofheterogenousjunk': [
'a',
2,
{ 'objprop': 1 },
['staahp!']
]
},
'achievement': 'win_match'
}";
JObject data = JObject.Parse(json);
IList<string> nodes = flattenJSON(data);
Console.WriteLine(string.Join(Environment.NewLine, nodes));
}
private static IList<string> flattenJSON(JToken token)
{
return _flattenJSON(token, new List<string>());
}
private static IList<string> _flattenJSON(JToken token, List<string> path)
{
var output = new List<string>();
if (token.Type == JTokenType.Object)
{
// Output the object's child properties
output.AddRange(token.Children().SelectMany(x => _flattenJSON(x, path)));
}
else if (token.Type == JTokenType.Array)
{
// Output each array element
var arrayIndex = 0;
foreach (var child in token.Children())
{
// Append the array index to the end of the last path segment - e.g. someProperty[n]
var newPath = new List<string>(path);
newPath[newPath.Count - 1] += "[" + arrayIndex++ + "]";
output.AddRange(_flattenJSON(child, newPath));
}
}
else if (token.Type == JTokenType.Property)
{
var prop = token as JProperty;
// Insert the property name into the path
output.AddRange(_flattenJSON(prop.Value, new List<string>(path) { prop.Name }));
}
else
{
// Join the path segments delimited with periods, followed by the literal value
output.Add(string.Join(".", path) + " = " + token.ToString());
}
return output;
}
}
Output:
reward.xp = 200
reward.gp = 150
reward.loot.item = sword
reward.loot.rarity = low
reward.loot.blah.socks = 5
reward.arrayofint[0] = 1
reward.arrayofint[1] = 2
reward.arrayofint[2] = 3
reward.arrayofint[3] = 4
reward.arrayofobj[0].foo = bar
reward.arrayofobj[0].stuff[0] = omg!
reward.arrayofobj[0].stuff[1] = what?!
reward.arrayofobj[1].foo = baz
reward.arrayofobj[1].stuff[0] = a
reward.arrayofobj[1].stuff[1] = b
reward.arrayofarray[0][0] = 1
reward.arrayofarray[0][1] = 2
reward.arrayofarray[0][2] = 3
reward.arrayofarray[1][0] = 4
reward.arrayofarray[1][1] = 5
reward.arrayofarray[1][2] = 6
reward.arrayofheterogenousjunk[0] = a
reward.arrayofheterogenousjunk[1] = 2
reward.arrayofheterogenousjunk[2].objprop = 1
reward.arrayofheterogenousjunk[3][0] = staahp!
achievement = win_match
PREVIOUS VERSION (NO ARRAY SUPPORT)
This doesn't properly support arrays - it will output the contents of a property that is an array as the raw JSON - i.e. it won't traverse into the array.
https://dotnetfiddle.net/yZbwul
public static void Main()
{
string json = #"{
'reward': {
'xp': '200',
'gp': '150',
'loot': {
'item': 'sword',
'rarity': 'low',
'blah': {
'socks': 5
}
}
},
'achievement': 'win_match'
}";
JObject data = JObject.Parse(json);
IList<string> nodes = flattenJSON(data, new List<string>());
Console.WriteLine(string.Join(Environment.NewLine, nodes));
}
private static IList<string> flattenJSON(JObject obj, IList<string> path)
{
var output = new List<string>();
foreach (var prop in obj.Properties())
{
if (prop.Value.Type == JTokenType.Object)
{
output.AddRange(flattenJSON(prop.Value as JObject, new List<string>(path){prop.Name}));
}
else
{
var s = string.Join(".", new List<string>(path) { prop.Name }) + " = " + prop.Value.ToString();
output.Add(s);
}
}
return output;
}
Output:
reward.xp = 200
reward.gp = 150
reward.loot.item = sword
reward.loot.rarity = low
reward.loot.blah.socks = 5
achievement = win_match

c# use one variable value to set a second from a fixed list

I'm parsing a CSV file in a c# .net windows form app, taking each line into a class I've created, however I only need access to some of the columns AND the files being taken in are not standardized. That is to say, number of fields present could be different and the columns could appear in any column.
CSV Example 1:
Position, LOCATION, TAG, NAME, STANDARD, EFFICIENCY, IN USE,,
1, AFT-D3, P-D3101A, EQUIPMENT 1, A, 3, TRUE
2, AFT-D3, P-D3103A, EQUIPMENT 2, B, 3, FALSE
3, AFT-D3, P-D2301A, EQUIPMENT 3, A, 3, TRUE
...
CSV Example 2:
Position, TAG, STANDARD, NAME, EFFICIENCY, LOCATION, BACKUP, TESTED,,
1, P-D3101A, A, EQUIPMENT 1, 3, AFT-D3, FALSE, TRUE
2, P-D3103A, A, EQUIPMENT 2, 3, AFT-D3, TRUE, FALSE
3, P-D2301A, A, EQUIPMENT 3, 3, AFT-D3, FALSE, TRUE
...
As you can see, I will never know the format of the file I have to analyse, the only thing I know for sure is that it will always contain the few columns that I need.
My solution to this was to ask the user to enter the columns required and set as strings, the using their entry convert that to a corresponding integer that i could then use as a location.
string standardInpt = "";
string nameInpt = "";
string efficiencyInpt = "";
user would then enter a value from A to ZZ.
int standardLocation = 0;
int nameLocation = 0;
int efficiencyLocation = 0;
when the form is submitted. the ints get their final value by running through an if else... statement:
if(standard == "A")
{
standardLocation = 0;
}
else if(standard == "B")
{
standardLocation = 1;
}
...
etc running all the way to if VAR1 == ZZ and then the code is repeated for VAR2 and for VAR3 etc..
My class would partially look like:
class Equipment
{
public string Standard { get; set;}
public string Name { get; set; }
public int Efficiency { get; set; }
static Equipment FromLine(string line)
{
var data = line.split(',');
return new Equipment()
{
Standard = data[standardLocation],
Name = [nameLocation],
Efficiency = int.Parse(data[efficiencyLocation]),
};
}
}
I've got more code in there but i think this highlights where I would use the variables to set the indexes.
I'm very new to this and I'm hoping there has got to be a significantly better way to achieve this without having to write so much potentially excessive, repetitive If Else logic. I'm thinking some kind of lookup table maybe, but i cant figure out how to implement this, any pointers on where i could look?

You could make it automatic by finding the indexes of the columns in the header, and then use them to read the values from the correct place from the rest of the lines:
class EquipmentParser {
public IList<Equipment> Parse(string[] input) {
var result = new List<Equipment>();
var header = input[0].Split(',').Select(t => t.Trim().ToLower()).ToList();
var standardPosition = GetIndexOf(header, "std", "standard", "st");
var namePosition = GetIndexOf(header, "name", "nm");
var efficiencyPosition = GetIndexOf(header, "efficiency", "eff");
foreach (var s in input.Skip(1)) {
var line = s.Split(',');
result.Add(new Equipment {
Standard = line[standardPosition].Trim(),
Name = line[namePosition].Trim(),
Efficiency = int.Parse(line[efficiencyPosition])
});
}
return result;
}
private int GetIndexOf(IList<string> input, params string[] needles) {
return Array.FindIndex(input.ToArray(), needles.Contains);
}
}

You can use the reflection and attribute.
Write your samples in ,separated into DisplayName Attribute.
First call GetIndexes with the csv header string as parameter to get the mapping dictionary of class properties and csv fields.
Then call FromLine with each line and the mapping dictionary you just got.
class Equipment
{
[DisplayName("STND, STANDARD, ST")]
public string Standard { get; set; }
[DisplayName("NAME")]
public string Name { get; set; }
[DisplayName("EFFICIENCY, EFFI")]
public int Efficiency { get; set; }
// You can add any other property
public static Equipment FromLine(string line, Dictionary<PropertyInfo, int> map)
{
var data = line.Split(',').Select(t => t.Trim()).ToArray();
var ret = new Equipment();
Type type = typeof(Equipment);
foreach (PropertyInfo property in type.GetProperties())
{
int index = map[property];
property.SetValue(ret, Convert.ChangeType(data[index],
property.PropertyType));
}
return ret;
}
public static Dictionary<PropertyInfo, int> GetIndexes(string headers)
{
var headerArray = headers.Split(',').Select(t => t.Trim()).ToArray();
Type type = typeof(Equipment);
var ret = new Dictionary<PropertyInfo, int>();
foreach (PropertyInfo property in type.GetProperties())
{
var fieldNames = property.GetCustomAttribute<DisplayNameAttribute>()
.DisplayName.Split(',').Select(t => t.Trim()).ToArray();
for (int i = 0; i < headerArray.Length; ++i)
{
if (!fieldNames.Contains(headerArray[i])) continue;
ret[property] = i;
break;
}
}
return ret;
}
}

try this if helpful:
public int GetIndex(string input)
{
input = input.ToUpper();
char low = input[input.Length - 1];
char? high = input.Length == 2 ? input[0] : (char?)null;
int indexLow = low - 'A';
int? indexHigh = high.HasValue ? high.Value - 'A' : (int?)null;
return (indexHigh.HasValue ? (indexHigh.Value + 1) * 26 : 0) + indexLow;
}

You can use ASCII code for that , so no need to add if else every time
ex.
byte[] ASCIIValues = Encoding.ASCII.GetBytes(standard);
standardLocation = ASCIIValues[0]-65;

How do I process arrays within an array?

I have an array of arrays of arrays as JSON, a sample included below where
F1 = Feature #1
P1 = Point #1
X / Y = Coordinates
so F1P1X is the X-value of point #1 of feature #1.
[
[
[F1P1X,F1P1Y,null],
[F1PnX,F1PnY,null]
],
[
[F2P1X,F2P1Y,null],
[F2PnX,F2PnY,null]
],
[
[FnP1X,FnP1Y,null],
[FnPnX,FnPnY,null]
]
]
Here is the code I use to get the above JSON from a file:
string json = File.ReadAllText("ABC.json");
JObject obj = JObject.Parse(json);
JToken token = obj.SelectToken("$.features[?(#.name == 'X')]['XY']");
var paths = JToken.Parse(token.ToString()).SelectToken("XYZ");
Next, I need to build strings using the various arrays. How do I get the second-level array (the feature) so I can process its innermost arrays (points on the features)? The end will be List<string> where each string is a feature (second-level array in the JSON) and the innermost array are the points that make the feature. I can handle the string manipulation but first I need to get the arrays out of the JSON.

Good choice is Json.NET nuget package for work with JSON. I have create test method for you.
//Read your json file
string json = File.ReadAllText("ABC.json");
//deserialize
F1P1X[][][] yourArrayOfArraysOfArrays = JsonConvert.DeserializeObject<F1P1X[][][]>(json).ToList();
public class F1P1X
{
public string Feature { get; set; }
public string Point { get; set; }
public string Coordinates { get; set; }
}
public static void Test()
{
F1P1X[] test1Array = new[]
{
new F1P1X
{
Feature = "F1",
Point = "P1",
Coordinates = "X1"
},
new F1P1X
{
Feature = "F2",
Point = "P2",
Coordinates = "X2"
},
};
F1P1X[] test2Array = new[]
{
new F1P1X
{
Feature = "F3",
Point = "P3",
Coordinates = "X3"
},
new F1P1X
{
Feature = "F4",
Point = "P4",
Coordinates = "X4"
},
};
F1P1X[][] test = {test1Array, test2Array};
F1P1X[][] test2 = { test1Array, test2Array };
F1P1X[][][] top = {test, test2};
//array of arrays of arrays as JSON
string json = JsonConvert.SerializeObject(top);
List<F1P1X[][]> arrays = JsonConvert.DeserializeObject<F1P1X[][][]>(json).ToList();
foreach (F1P1X[][] item in arrays)
{
foreach (F1P1X[] f1P1X in item)
{
foreach (F1P1X p1X in f1P1X)
{
//do some magic
}
}
}
// or use linq
var result = arrays.SelectMany(x => x.SelectMany(y => y.Where(z => z.Coordinates == "X1")));
}
This linq statement returns list of features
List<string> result = arrays.SelectMany(x => x.SelectMany(y => y.Select(z => z.Feature))).ToList();

What is the easiest way to split columns from a txt file

I've been looking around a bit but haven't really found a good example with what I'm struggling right now.
I have a .txt file with a couple of columns as follows:
# ID,YYYYMMDD, COLD,WATER, OD, OP,
52,20120406, 112, 91, 20, 130,
53,20130601, 332, 11, 33, 120,
And I'm reading these from the file into a string[] array.
I'd like to split them into a list
for example
List results, and [0] index will be the first index of the columns
results[0].ID
results[0].COLD
etc..
Now I've been looking around, and came up with the "\\\s+" split
but I'm not sure how to go about it since each entry is under another one.
string[] lines = File.ReadAllLines(path);
List<Bus> results = new List<Bus>();
//Bus = class with all the vars in it
//such as Bus.ID, Bus.COLD, Bus.YYYYMMDD
foreach (line in lines) {
var val = line.Split("\\s+");
//not sure where to go from here
}
Would greatly appreciate any help!
Kind regards, Venomous.

I suggest using Linq, something like this:
List<Bus> results = File
.ReadLines(#"C:\MyFile.txt") // we have no need to read All lines in one go
.Skip(1) // skip file's title
.Select(line => line.Split(','))
.Select(items => new Bus( //TODO: check constructor's syntax
int.Parse(items[1]),
int.Parse(items[3]),
DateTime.ParseExact(items[2], "yyyyMMdd", CultureInfo.InvariantCulture)))
.ToList();

I would do
public class Foo
{
public int Id {get; set;}
public string Date {get; set;}
public double Cold {get; set;}
//...more
}
Then read the file
var l = new List<Foo>();
foreach (line in lines)
{
var sp = line.Split(',');
var foo = new Foo
{
Id = int.Parse(sp[0].Trim()),
Date = sp[1].Trim(),//or pharse the date to a date time struct
Cold = double.Parse(sp[2].Trim())
}
l.Add(foo);
}
//now l contains a list filled with Foo objects

I would probably keep a List of properties and use reflection to populate the object, something like this :
var columnMap = new[]{"ID","YYYYMMDD","COLD","WATER","OD","OP"};
var properties = columnMap.Select(typeof(Bus).GetProperty).ToList();
var resultList = new List<Bus>();
foreach(var line in lines)
{
var val = line.Split(',');
var adding = new Bus();
for(int i=0;i<val.Length;i++)
{
properties.ForEach(p=>p.SetValue(adding,val[i]));
}
resultList.Add(adding);
}
This is assuming that all of your properties are strings however

Something like this perhaps...
results.Add(new Bus
{
ID = val[0],
YYYYMMDD = val[1],
COLD = val[2],
WATER = val[3],
OD = val[4],
OP = val[5]
});
Keep in mind that all of the values in the val array are still strings at this point. If the properties of Bus are typed, you will need to parse them into the correct types e.g. assume ID is typed as an int...
ID = string.IsNullOrEmpty(val[0]) ? default(int) : int.Parse(val[0]),
Also, if the column headers are actually present in the file in the first line, you'll need to skip/disregard that line and process the rest.

Given that we have the Bus class with all the variables from your textfile:
class Bus
{
public int id;
public DateTime date;
public int cold;
public int water;
public int od;
public int op;
public Bus(int _id, DateTime _date, int _cold, int _water, int _od, int _op)
{
id = _id;
date = _date;
cold = _cold;
water = _water;
od = _od;
op = _op;
}
}
Then we can list them all in the results list like this:
List<Bus> results = new List<Bus>();
foreach (string line in File.ReadAllLines(path))
{
if (line.StartsWith("#"))
continue;
string[] parts = line.Replace(" ", "").Split(','); // Remove all spaces and split at commas
results.Add(new Bus(
int.Parse(parts[0]),
DateTime.ParseExact(parts[1], "yyyyMMdd", CultureInfo.InvariantCulture),
int.Parse(parts[2]),
int.Parse(parts[3]),
int.Parse(parts[4]),
int.Parse(parts[5])
));
}
And access the values as you wish:
results[0].id;
results[0].cold;
//etc.
I hope this helps.

I'm trying to fill my List<Class> from a text file using C#?

I have a class called cEspecie like this
public class cEspecie
{
private string name;
private int lifetime;
private int movility;
private int deadto;
private int type;
public string Name
{
get
{
return name;
}
set
{
name= value;
}
}
public int Type
{
get
{
return type;
}
set
{
type = value;
}
}
public int LifeTime
{
get
{
return lifetime;
}
set
{
lifetime= value;
}
}
public int Movility
{
get
{
return movility;
}
set
{
movility = value;
}
}
public int DeadTo
{
get
{
return deadto;
}
set
{
deadto = value;
}
}
}
I store some data in a list called
List<cEspecie> list = new List<cEspecie>() {
new cEspecie("Wolf", 100, 10, 0, 0)
new cEspecie("Rabiit", 100, 100, 1, 1),
new cEspecie("Lion", 200, 10, 2, 2),
new cEspecie("Tiger", 300, 10, 3, 3),
};
In one of the process of my program i store all the data inside a text file using this:
using (StreamWriter sr = new StreamWriter(#"../../Archives/TextFilecEspecie.txt"))
{
foreach (var item in list)
{
sr.WriteLine(item.Name);
sr.WriteLine(item.Type);
sr.WriteLine(item.Movility);
sr.WriteLine(item.LifeTime);
sr.WriteLine(item.DeadTo);
}
sr.Close();
}
the result inside "TextFilecEspecie.txt" was this:
Wolf
100
10
0
0
Rabiit
100
100
1
1
Lion
200
10
2
2
Tiger
300
10
3
3
now my real, real problem is ... How can i get back the same data to store it in the same list? I'm using c# and wpf and i really dont find an answer.

Using LINQ, and the Buffer extension method from Microsoft's excellent Ix-Main package.
var species = File.ReadLines("file.txt")
.Buffer(5)
.Select(x => new cEspecie
{
Name = x[0],
Type = int.Parse(x[1]),
Movility = int.Parse(x[2]),
LifeTime = int.Parse(x[3]),
DeadTo = int.Parse(x[4])
});
Buffer(5) will group every 5 lines into an array.

This is what serialization and ISerializable are for.
Here is a quick tutorial that should make it fairly simple.
I might even recommend JSON.Net, which will write it in JSON format so you could more easily create/edit the objects in the file.

You do the same thing only backwards using StreamReader to read the file. I won't get into specifics since I'm guessing you can use the learning experience so consider this a point to the right direction and homework :)

you can do that this way
var list = new List<cEspecie>();
using (StreamReader reader = File.OpenText(filePath))
{
while(!reader.EndOfStream)
{
string name = reader.ReadLine();
int type = int.Parse(reader.ReadLine());
int movility = int.Parse(reader.ReadLine());
int lifeTime = int.Parse(reader.ReadLine());
int deadTo = int.Parse(reader.ReadLine());
list.Add(new cEspecie(name, type, movility, lifeTime, deadTo));
}
}
note that:
int.Parse throw error if the string is not a int, use TryParse the data may not be a int
this work only if the file has the correct number of lines

var list = File.ReadLines("path").Select(x => new cEspecie(x));
in cEspecie constructor you can initialize the object using input line.

Something like this is very simple and should work. Do error checking though.
List<cEspecie> list = new List<Persona>();
string[] readText = File.ReadAllLines(#"../../Archives/TextFilecEspecie.txt");
for(int i=0;i<readText.Length/5;i++)
{
list.Add(new Persona(readText[i*5], readText[i*5+1], readText[i*5+2], readText[i*5+3], readText[i*5+4]));
}