We Keep Coding

Remove all occurrences of a particular key from a JSON response in C#

I have a JSON string from which I want to eliminate all the occurrences of a given key.
JSON I have:
string requstBody =
{
"payLoad": [
{
"BaseVersionId_": 9,
"VersionId_": 10,
"AssetCollateralLink": [
{
"AssetId": 137,
"BaseVersionId_": 9,
"VersionId_": 10
},
{
"AssetId": 136,
"BaseVersionId_": 0,
"VersionId_": 1
}
],
"CollateralProvider": [],
"AdvCollateralAllocation": [
{
"LinkId": 91,
"IsDeleted_": false,
"BaseVersionId_": 1,
"VersionId_": 2
}
]
}
]
}
I want to eliminate keys "BaseVersionID_" and "VersionId_" as follows:
string requstBody =
{
"payLoad": [
{
"AssetCollateralLink": [
{
"AssetId": 137
},
{
"AssetId": 136
}
],
"CollateralProvider": [],
"AdvCollateralAllocation": [
{
"LinkId": 91,
"IsDeleted_": false
}
]
}
]
}
I used JObject.Remove(); as follows
JObject sampleObj1 = new JObject();
sampleObj1 = JsonHelper.JsonParse(requestBody);
sampleObj1.Remove("BaseVersionId_");
but able to remove the keys under payLoad Hierarchy only.
How do I remove all the occurrences of the Key.

The required properties can be removed from the Json, simply with Linq:
var jsonObj = JObject.Parse(requestBody);
jsonObj.SelectToken("payLoad[0]").SelectToken("AdvCollateralAllocation")
.Select(jt => (JObject)jt)
.ToList()
.ForEach(r =>
r
.Properties()
.ToList()
.ForEach(e =>
{
if (e.Name == "BaseVersionId_" || e.Name == "VersionId_")
e.Remove();
}));
The resultant jsonObj will be without the BaseVersionId_ and VersionId_ names as well as their values.

I'd use JsonPath as such:
var toRemove = jsonObject
.SelectTokens("$.payLoad.[*].AssetCollateralLink.[*]..BaseVersionId_")
.Concat(stuff.SelectTokens("$.payLoad.[*].AssetCollateralLink.[*]..VersionId_"))
.Concat(stuff.SelectTokens("$.payLoad.[*].AdvCollateralAllocation.[*]..VersionId_"))
.Concat(stuff.SelectTokens("$.payLoad.[*].AdvCollateralAllocation.[*]..BaseVersionId_"))
.ToList();
for (int i = toRemove.Count - 1; i >= 0; i--)
{
toRemove[i].Parent?.Remove();
}

Parse Json key/values of a key's value using JObject in C#

I am trying to get the sub key\values of a key's value. What I am trying to accomplish is to remove the elements that are empty or have "-" or have "N/A". I can not seem to figure out out to iterate over the values to search.
{
"name": {
"first": "Robert",
"middle": "",
"last": "Smith"
},
"age": 25,
"DOB": "-",
"hobbies": [
"running",
"coding",
"-"
],
"education": {
"highschool": "N/A",
"college": "Yale"
}
}
Code:
JObject jObject = JObject.Parse(response);
foreach (var obj in jObject)
{
Console.WriteLine(obj.Key);
Console.WriteLine(obj.Value);
}
I am trying to search "first":"Robert","middle":"","last":"Smith"

You can use Descendants method to get child tokens of type JProperty, then filter their values and print them or remove one by one
var properties = json.Descendants()
.OfType<JProperty>()
.Where(p =>
{
if (p.Value.Type != JTokenType.String)
return false;
var value = p.Value.Value<string>();
return string.IsNullOrEmpty(value);
})
.ToList();
foreach (var property in properties)
property.Remove();
Console.WriteLine(json);
Gives you the following result (with "middle": "" property removed)
{
"name": {
"first": "Robert",
"last": "Smith"
},
"age": 25,
"DOB": "-",
"hobbies": [
"running",
"coding",
"-"
],
"education": {
"highschool": "N/A",
"college": "Yale"
}
}
You can also add more conditions to return statement, like return string.IsNullOrEmpty(value) || value.Equals("-"); to remove "DOB": "-" property as well

You can recursively iterate JObject properties:
private static void IterateProps(JObject o)
{
foreach (var prop in o.Properties())
{
Console.WriteLine(prop.Name);
if (prop.Value is JObject)
{
IterateProps((JObject)prop.Value);
}
else
{
Console.WriteLine(prop.Value);
}
}
}

How to find total number of lowest level of elements in a JSON object?

I have a Json as below
{
"name": "xxxx",
"type": "ccc",
"properties": {
"serialNumber": {
"value": "24-66292"
},
"documentLinks": {
"productManuals": {
"54868484": {
"url": {
"value": "xxxx"
},
"productName": {
"value": "R02400"
}
}
},
"keystringFiles": {
"60050588": {
"url": {
"value": "http://se-s-0010052.de.abb.com/stage/wc/!control.controller?action=view_document&doc_id=60050588"
},
"name": {
"value": "24-66292_160.kxt"
},
"fileSize": {
"value": 0.87
},
"addedDate": {
"value": "2012-01-19"
},
"addedBy": {
"value": "Loader"
}
}
}
}
},
"variables":{
"temperature" :{
"dataType": "number"
},
"modes" :{
"normal":{
"dataType": "string"
},
"fast":{
"dataType": "string"
}
}
}
}
I need to count the total number of elements in this excluding the root level elements. In this example it's like under "properties" below are the number of elements
serialNumber
documentLinks->productManuals->54868484->url
documentLinks->productManuals->54868484->productName
documentLinks->keystringFiles->60050588->url
documentLinks->keystringFiles->60050588->name
documentLinks->keystringFiles->60050588->fileSize
documentLinks->keystringFiles->60050588->addedDate
documentLinks->keystringFiles->60050588->addedBy
Under "variables"
temperature
modes->normal
modes->fast
Hence total number of elements is
8+3 = 11
I was trying multiple things as below, but I am not able to find the best logic which serves the purpose.
var ob = JObject.Parse(json);
var propCount = JObject.Parse(json).Root
.SelectTokens("properties")
.SelectMany(t => t.Children().OfType<JProperty>().Select(p => p.Name))
.ToArray();
var varCount = JObject.Parse(json).Root
.SelectTokens("variables")
.SelectMany(t => t.Children().OfType<JProperty>().Select(p => p.Name))
.ToArray();
int propCount = 0;
int variableCount = 0;
foreach (var prop in propCount)
{
propCount += ob["properties"][prop].Children().Count();
}
foreach (var variable in varCount)
{
variableCount += ob["variables"][variable].Children().Count();
}
var total = propCount + variableCount;

I think, based on what I understand, you have 11. You're missing properties->serialNumber in your count
private int GetLowestLevelCount(JToken obj)
{
int counter = 0;
if (obj.HasValues) // Checks if token has children.
{
foreach (var children in obj.Children())
{
counter += GetLowestLevelCount(children);
}
}
else // Lowest-level elem.
{
counter +=1;
}
return counter;
}
Use:
var ob = JObject.Parse(json);
var mainKeys = ob.Children().Select(x => ((JProperty)x).Name);
int lowestLevel = 0;
foreach (var mainKey in mainKeys)
{
var a = ob[mainKey];
if (a.HasValues) //Don't bother with top-level elements
{
foreach (var c in a.Children())
{
lowestLevel += GetLowestLevelCount(c);
}
}
}
Console.WriteLine(lowestLevel);

Remove fields from JSON dynamically using Json.Net

I have some JSON input, the shape of which I cannot predict, and I have to make some transformations (to call it something) so that some fields are not logged. For instance, if I have this JSON:
{
"id": 5,
"name": "Peter",
"password": "some pwd"
}
then after the transformation it should look like this:
{
"id": 5,
"name": "Peter"
}
The above sample is trivial, but the actual case is not so happy/easy. I will have some regular expressions and if any field(s) on the input JSON matches that, then it shouldn't be on the result. I will have to go recursively in case I have some nested objects. I've been seeing some stuff on LINQ to JSON but I have found nothing satisfying my needs.
Is there a way of doing this?
Note:
This is part of a logging library. I can use the JSON string if necessary or easier. The thing is that at some point in my logging pipeline I get the object (or string as required) and then I need to strip the sensitive data from it, such as passwords, but also any other client-specified data.

You can parse your JSON into a JToken, then use a recursive helper method to match property names to your regexes. Wherever there's a match, you can remove the property from its parent object. After all sensitive info has been removed, just use JToken.ToString() to get the redacted JSON.
Here is what the helper method might look like:
public static string RemoveSensitiveProperties(string json, IEnumerable<Regex> regexes)
{
JToken token = JToken.Parse(json);
RemoveSensitiveProperties(token, regexes);
return token.ToString();
}
public static void RemoveSensitiveProperties(JToken token, IEnumerable<Regex> regexes)
{
if (token.Type == JTokenType.Object)
{
foreach (JProperty prop in token.Children<JProperty>().ToList())
{
bool removed = false;
foreach (Regex regex in regexes)
{
if (regex.IsMatch(prop.Name))
{
prop.Remove();
removed = true;
break;
}
}
if (!removed)
{
RemoveSensitiveProperties(prop.Value, regexes);
}
}
}
else if (token.Type == JTokenType.Array)
{
foreach (JToken child in token.Children())
{
RemoveSensitiveProperties(child, regexes);
}
}
}
And here is a short demo of its use:
public static void Test()
{
string json = #"
{
""users"": [
{
""id"": 5,
""name"": ""Peter Gibbons"",
""company"": ""Initech"",
""login"": ""pgibbons"",
""password"": ""Sup3rS3cr3tP#ssw0rd!"",
""financialDetails"": {
""creditCards"": [
{
""vendor"": ""Viza"",
""cardNumber"": ""1000200030004000"",
""expDate"": ""2017-10-18"",
""securityCode"": 123,
""lastUse"": ""2016-10-15""
},
{
""vendor"": ""MasterCharge"",
""cardNumber"": ""1001200230034004"",
""expDate"": ""2018-05-21"",
""securityCode"": 789,
""lastUse"": ""2016-10-02""
}
],
""bankAccounts"": [
{
""accountType"": ""checking"",
""accountNumber"": ""12345678901"",
""financialInsitution"": ""1st Bank of USA"",
""routingNumber"": ""012345670""
}
]
},
""securityAnswers"":
[
""Constantinople"",
""Goldfinkle"",
""Poppykosh"",
],
""interests"": ""Computer security, numbers and passwords""
}
]
}";
Regex[] regexes = new Regex[]
{
new Regex("^.*password.*$", RegexOptions.IgnoreCase),
new Regex("^.*number$", RegexOptions.IgnoreCase),
new Regex("^expDate$", RegexOptions.IgnoreCase),
new Regex("^security.*$", RegexOptions.IgnoreCase),
};
string redactedJson = RemoveSensitiveProperties(json, regexes);
Console.WriteLine(redactedJson);
}
Here is the resulting output:
{
"users": [
{
"id": 5,
"name": "Peter Gibbons",
"company": "Initech",
"login": "pgibbons",
"financialDetails": {
"creditCards": [
{
"vendor": "Viza",
"lastUse": "2016-10-15"
},
{
"vendor": "MasterCharge",
"lastUse": "2016-10-02"
}
],
"bankAccounts": [
{
"accountType": "checking",
"financialInsitution": "1st Bank of USA"
}
]
},
"interests": "Computer security, numbers and passwords"
}
]
}
Fiddle: https://dotnetfiddle.net/KcSuDt

You can parse your JSON to a JContainer (which is either an object or array), then search the JSON hierarchy using DescendantsAndSelf() for properties with names that match some Regex, or string values that match a Regex, and remove those items with JToken.Remove().
For instance, given the following JSON:
{
"Items": [
{
"id": 5,
"name": "Peter",
"password": "some pwd"
},
{
"id": 5,
"name": "Peter",
"password": "some pwd"
}
],
"RootPasswrd2": "some pwd",
"SecretData": "This data is secret",
"StringArray": [
"I am public",
"This is also secret"
]
}
You can remove all properties whose name includes "pass.*w.*r.*d" as follows:
var root = (JContainer)JToken.Parse(jsonString);
var nameRegex = new Regex(".*pass.*w.*r.*d.*", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant);
var query = root.DescendantsAndSelf()
.OfType<JProperty>()
.Where(p => nameRegex.IsMatch(p.Name));
query.RemoveFromLowestPossibleParents();
Which results in:
{
"Items": [
{
"id": 5,
"name": "Peter"
},
{
"id": 5,
"name": "Peter"
}
],
"SecretData": "This data is secret",
"StringArray": [
"I am public",
"This is also secret"
]
}
And you can remove all string values that include the substring secret by doing:
var valueRegex = new Regex(".*secret.*", RegexOptions.IgnoreCase);
var query2 = root.DescendantsAndSelf()
.OfType<JValue>()
.Where(v => v.Type == JTokenType.String && valueRegex.IsMatch((string)v));
query2.RemoveFromLowestPossibleParents();
var finalJsonString = root.ToString();
Which when applied after the first transform results in:
{
"Items": [
{
"id": 5,
"name": "Peter"
},
{
"id": 5,
"name": "Peter"
}
],
"StringArray": [
"I am public"
]
}
For convenience, I am using the following extension methods:
public static partial class JsonExtensions
{
public static TJToken RemoveFromLowestPossibleParent<TJToken>(this TJToken node) where TJToken : JToken
{
if (node == null)
return null;
JToken toRemove;
var property = node.Parent as JProperty;
if (property != null)
{
// Also detach the node from its immediate containing property -- Remove() does not do this even though it seems like it should
toRemove = property;
property.Value = null;
}
else
{
toRemove = node;
}
if (toRemove.Parent != null)
toRemove.Remove();
return node;
}
public static IEnumerable<TJToken> RemoveFromLowestPossibleParents<TJToken>(this IEnumerable<TJToken> nodes) where TJToken : JToken
{
var list = nodes.ToList();
foreach (var node in list)
node.RemoveFromLowestPossibleParent();
return list;
}
}
Demo fiddle here.

Create JSON from 2 strings from Database (Latency and Formatting Issues)

I have created a JSON file from JArray in MVC application from 2 coloumns of my Database. But there are 2 issues:
1. The format of the file. It has extra brackets for each object.
[
[
{
"Code": "1",
"Name": "ASSETS"
}
],
[
{
"Code": "2",
"Name": "LIABILITIES"
}
],
[
{
"Code": "3",
"Name": "CAPITAL"
}
],
[
{
"Code": "4",
"Name": "REVENUE"
}
]
]
I want it as:
[
{
"Code": "1",
"Name": "ASSETS"
},
{
"Code": "2",
"Name": "LIABILITIES"
},
{
"Code": "3",
"Name": "CAPITAL"
},
{
"Code": "4",
"Name": "REVENUE"
}
]
I have loaded the values in JArray from Database and then add it in another JArray. I am using 2 for loops. 1 for making JArray for 5000 entries. And the second for returning the next valid primary key from Database. But the problem is that, it takes more than 15 minutes to process the loops and return the JSON file. Why is there so much latency? And how can I make it fast. Here is the code.
int idd =0;
JArray Array = new JArray();
for (int b = 0; b<5000; b++)
{
idd = dbid(idd);
IEnumerable<MST> accList = new List<MST>
{
new MST
{
S1 = db.MSTs.Find(idd).S1,
S2 = db.MSTs.Find(idd).S2
}
};
JArray Arrayone = new JArray(
accList.Select(p => new JObject
{
{ "Code", p.S1 },
{ "Name", p.S2 },
})
);
Array.Add(Arrayone);
}
string jsonfile = JsonConvert.SerializeObject(Array,Formatting.Indented);
string path = #"C:\Users\Awais\Desktop\accounts.json";
System.IO.File.WriteAllText(path, jsonfile);
return View(v);
}
public int dbid(int id)
{
decimal i = db.MSTs.Max(a => a.N100);
MST m = new MST();
for (; id <= i; id++)
{
m = db.MSTs.Find(++id);
if (m == null)
continue;
else
{
break;
}
}
return id;
}
When I tried the first loop for 100 entries, it took about 60 seconds to return the file.

By modeling your JSON with anonymous type, you could've done it like this:
var array = (from coa in db.MSTs
select new { Code = coa.S2, Name = coa.S1 }).ToArray();
string jsonfile = JsonConvert.SerializeObject(array, Formatting.Indented);
string path = #"C:\Users\Awais\Desktop\accounts.json";
System.IO.File.WriteAllText(path, jsonfile);
It took me 319 ms for 100 000 objects.

Try:
public class RootObject
{
public string Code { get; set; }
public string Name { get; set; }
}
var o = new List<RootObject>();
for (var i = 0; i < 100; ++i)
{
o.Add(new RootObject
{
Code = "foo",
Name = "bar"
});
}
var v = JsonConvert.SerializeObject(o);
Took around 274 milliseconds for my list to Serialize.

I have removed all loops and instead try to solve it in query and making a single array.
var k = (from coa in db.MSTs
select new { S2 = coa.S2, S1 = coa.S1 }).ToList().
Select(x => new MST { S2 = x.S2, S1 = x.S1 }).ToList();
JArray Arrayone = new JArray(
k.Select(p => new JObject
{
{ "Code", p.S1 },
{ "Name", p.S2 },
})
);
string jsonfile = JsonConvert.SerializeObject(Arrayone,Formatting.Indented);
string path = #"C:\Users\Awais\Desktop\accounts.json";
System.IO.File.WriteAllText(path, jsonfile);
It solves both problems. i.e. Brackets formatting issue and latency. Now with this code it works in less than 5 seconds