We Keep Coding

How do I generate new JSON files that may contain a single quote from an old JSON file

I've written some code which has to read through 5,000 JSON files and replace a number which appears twice in each file. It's set to create a new json file and delete the old one, with that, other data must be transferred into the new file too, which is not a problem, however we have a list called attributes that contain some text that use single quotes (for punctuation.) - however when the new json file is written, the text with the single quotes is changed and the single quote is replaced with \u0027
Example: old file contains:"0.png" and "Hen's Secret Bra" which in the new file is changing to "5000.png" correct, and "Hen\u0027s Secret Bra" incorrect. How do I transfer the single quote over properly?
Here is my code:
using System.Text.Json;
using System.Web;
internal class Program
{
private static void Main(string[] args)
{
//get all the json files
var jsonPath = #"/Users/jasonnienaber/Desktop/nft/cef/official/fatJSONselfFIX/test";
var jsonFiles = Directory.GetFiles(jsonPath, "*.json");
//loop through each file and process according to specs
foreach(var jsonFile in jsonFiles)
{
var json = File.ReadAllText(jsonFile);
var sample = JsonSerializer.Deserialize<Sample>(json);
var sampleNew = new SampleNew();
var intCounter = 0;
var newIntCounter = intCounter+5000;
var Counter = intCounter.ToString();
var newCounter = newIntCounter.ToString();
sampleNew.image = sample.image.Replace(Counter, newCounter);
sampleNew.name = sample.name.Replace(Counter, newCounter);
sampleNew.description = sample.description;
sampleNew.external_url = sample.external_url;
sampleNew.attributes = sample.attributes;
// serialize JSON to a string and then write string to a file
File.Delete(jsonFile);
File.WriteAllText(jsonFile, JsonSerializer.Serialize(sampleNew));
intCounter++;
}
}
public class Attribute
{
public string trait_type { get; set; }
public string value { get; set; }
}
public class Sample
{
// public string dna { get; set; }
public string image { get; set; }
public string name { get; set; }
public string description { get; set; }
public string external_url { get; set; }
public List<Attribute> attributes { get; set; }
public string compiler { get; set; }
}
public class SampleNew
{
public string image { get; set; }
public string name { get; set; }
public string description { get; set; }
public string external_url { get; set; }
//public List<Attribute> attributes { get; set; }
public List<Attribute> attributes {get; set;}
}
}
The "Hen's Secret Bra" is within attributes

Like Jon Skeet mentioned, it's valid to use \u0027, however, you can change the serializer encoding options:
using System.Text.Json;
var obj = new
{
Name = "John's awesome object",
Age = 31
};
var options = new JsonSerializerOptions
{
Encoder = System.Text.Encodings.Web.JavaScriptEncoder.UnsafeRelaxedJsonEscaping
};
// writes obj to json file
File.WriteAllText("obj.json", JsonSerializer.Serialize(obj, options));
Outputs
{"Name":"John's awesome object","Age":31}
So in your code, you can do as follows:
var options = new JsonSerializerOptions
{
Encoder = System.Text.Encodings.Web.JavaScriptEncoder.UnsafeRelaxedJsonEscaping
};
//loop through each file and process according to specs
foreach (var jsonFile in jsonFiles)
{
var json = File.ReadAllText(jsonFile);
var sample = JsonSerializer.Deserialize<Sample>(json);
var sampleNew = new SampleNew();
var intCounter = 0;
var newIntCounter = intCounter + 5000;
var Counter = intCounter.ToString();
var newCounter = newIntCounter.ToString();
sampleNew.image = sample.image.Replace(Counter, newCounter);
sampleNew.name = sample.name.Replace(Counter, newCounter);
sampleNew.description = sample.description;
sampleNew.external_url = sample.external_url;
sampleNew.attributes = sample.attributes;
// serialize JSON to a string and then write string to a file
File.Delete(jsonFile);
// use serializer options
File.WriteAllText(jsonFile, JsonSerializer.Serialize(sampleNew, options));
intCounter++;
}
See UnsafeRelaxedJsonEscaping

It is better to use Newtonsoft.Json to avoid any problems like this one.
But one of the thing you can try to fix an issue is to use a serialize options
var options = new JsonSerializerOptions
{
Encoder = System.Text.Encodings.Web.JavaScriptEncoder.UnsafeRelaxedJsonEscaping
};
but it is unsafe
Another option is more awkward but more safe, since it affects only one symbol, not any as in the first case
var attr= new Attribute{trait_type="trait_type", value="Hen's Secret Bra"};
attr.value=attr.value.Replace("'","==="); //or you can try something else if you don't like ===
var json=System.Text.Json.JsonSerializer.Serialize(attr);
json=json.Replace("===","'");
File.WriteAllText(jsonFile, json);
json = File.ReadAllText(jsonFile);
result
{"trait_type":"trait_type","value":"Hen's Secret Bra"}

How can I parse some JSON dynamically without knowing JSON values?

So I am using TDAmeritrade API to receive stock data with a C# Winforms program on Visual Studio. It takes the user input stock symbol and searches for the info. I am using HttpClient and Newtonsoft.Json and have been able to successfully perform the GET request and receive a JSON string back, but I do not know how to get all of the information I need out of it.
Here is the JSON:
https://drive.google.com/file/d/1TpAUwjyqrHArEXGXMof_K1eQe0hFoaw5/view?usp=sharing
Above is the JSON string sent back to me then formatted. My goal is to record information for each price in "callExpDateMap.2021-02-19:11" and "callExpDateMap.2021-03-19:39". The problem is that for each different stock, the dates that show up in "callExpDateMap" are going to be different.
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
var response = await client.GetAsync(url);
var info = await response.Content.ReadAsStringAsync();
dynamic config = JsonConvert.DeserializeObject<dynamic>(info, new ExpandoObjectConverter());
return config;
This is the code I have right now. I know the last for statement is not correct. How can I parse to the specific sections I want (callExpDateMap.expirationdate.StrikePrice) and get the information needed from each without knowing the dates and Strike prices beforehand? Is there a way to innumerate it and search through the JSON as if it were all a bunch of arrays?

The code below is perhaps not the most elegant nor complete, but I think it will get you going. I would start by using the JObject.Parse() from the Newtonsoft.Json.Linq namespace and take it from there.
JObject root = JObject.Parse(info);
string symbol = root["symbol"].ToObject<string>();
foreach (JToken toplevel in root["callExpDateMap"].Children())
{
foreach (JToken nextlevel in toplevel.Children())
{
foreach (JToken bottomlevel in nextlevel.Children())
{
foreach (JToken jToken in bottomlevel.Children())
{
JArray jArray = jToken as JArray;
foreach (var arrayElement in jArray)
{
InfoObject infoObject = arrayElement.ToObject<InfoObject>();
Console.WriteLine(infoObject.putCall);
Console.WriteLine(infoObject.exchangeName);
Console.WriteLine(infoObject.multiplier);
}
}
}
}
}
public class InfoObject
{
public string putCall { get; set; }
public string symbol { get; set; }
public string description { get; set; }
public string exchangeName { get; set; }
// ...
public int multiplier { get; set; }
// ...
}

This is official documentation of Newtonsoft method you are trying to use.
https://www.newtonsoft.com/json/help/html/Overload_Newtonsoft_Json_JsonConvert_DeserializeObject.htm
If an API's method returns different json propeties and you cannot trust it's property names all the times, then you can try using a deserialize method that returns .Net object, for example: JsonConvert.DeserializeObject Method (String)
https://www.newtonsoft.com/json/help/html/M_Newtonsoft_Json_JsonConvert_DeserializeObject.htm
That method's signature is this:
public static Object DeserializeObject(string value)
Parameter is: value of type json string.
Return Value is: Object of type object.
If you do not want an Object, then you can of course use a .Net type you have. Such as this method:
JsonConvert.DeserializeObject Method (String)
Any property that you have in both (the .net type and json object) will get populated. If .net type has properties that do not exist in json object, then those will be ignored. If json object has properties that do not exist in.net, then those will be ignored too.
Here's an example of a .Net type
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using Newtonsoft.Json.Serialization;
using System.Collections.Generic;
using System.IO;
using System.Text;
namespace MyNameSpace
{
public class TDAmeritradeStockData
{
[JsonProperty("symbol")]
public string Symbol { get; set; }
[JsonProperty("status")]
public string Status { get; set; }
[JsonProperty("callExpDateMap")]
public object CallExpDateMap { get; set; }
//...
//...
public CallExpDateMapType[] CallExpDateMapList { get; set; }
}
public class CallExpDateMapType
{
[JsonProperty("expirationdate")]
public string Expirationdate { get; set; }
[JsonProperty("StrikePrice")]
public List<StrikePriceType> StrikePriceList { get; set; }
}
public class StrikePriceType
{
public string StrikePrice { get; set; }
public List<StrikePricePropertiesType> StrikePricePropertiesList { get; set; }
}
public class StrikePricePropertiesType
{
[JsonProperty("putCall")]
public string PutCall { get; set; }
[JsonProperty("symbol")]
public string Symbol { get; set; }
[JsonProperty("description")]
public string Description { get; set; }
[JsonProperty("exchangeName")]
public string ExchangeName { get; set; }
[JsonProperty("bid")]
public double Bid { get; set; }
[JsonProperty("ask")]
public double Ask { get; set; }
//...
//...
}
[TestClass]
public class TestTestTest
{
[TestMethod]
public void JsonTest()
{
var jsondata = ReadFile("data.json");
var model = JsonConvert.DeserializeObject<TDAmeritradeStockData>(jsondata);
JObject jObject = (JObject)model.CallExpDateMap;
var count = ((JObject)model.CallExpDateMap).Count;
model.CallExpDateMapList = new CallExpDateMapType[count];
var jToken = (JToken)jObject.First;
for (var i = 0; i < count; i++)
{
model.CallExpDateMapList[i] = new CallExpDateMapType
{
Expirationdate = jToken.Path,
StrikePriceList = new List<StrikePriceType>()
};
var nextStrikePrice = jToken.First.First;
while (nextStrikePrice != null)
{
var nextStrikePriceProperties = nextStrikePrice;
var srikePriceList = new StrikePriceType
{
StrikePrice = nextStrikePriceProperties.Path,
StrikePricePropertiesList = JsonConvert.DeserializeObject<List<StrikePricePropertiesType>>(nextStrikePrice.First.ToString())
};
model.CallExpDateMapList[i].StrikePriceList.Add(srikePriceList);
nextStrikePrice = nextStrikePrice.Next;
}
jToken = jToken.Next;
}
Assert.IsNotNull(model);
}
private string ReadFile(string fileName)
{
using (var fileStream = new FileStream(fileName, FileMode.Open, FileAccess.Read))
{
var data = new StringBuilder();
using (var streamReader = new StreamReader(fileStream))
{
while (!streamReader.EndOfStream) data.Append(streamReader.ReadLine());
streamReader.Close();
}
fileStream.Close();
return data.ToString();
}
}
}
}

Converting JSON array

I am attempting to use the Newtonsoft JSON library to parse a JSON string dynamically using C#. In the JSON is a named array. I would like to remove the square brackets from this array and then write out the modified JSON.
The JSON now looks like the following. I would like to remove the square bracket from the ProductDescription array.
{
"Product": "123",
"to_Description": [
{
"ProductDescription": "Product 1"
}
]
}
Desired result
{
"Product": "123",
"to_Description":
{
"ProductDescription": "Product 1"
}
}
I believe I can use the code below to parse the JSON. I just need some help with making the modification.
JObject o1 = JObject.Parse(File.ReadAllText(#"output.json"));

The to_Description property starts off as List<Dictionary<string,string>> and you want to take the first element from the List.
So, given 2 classes
public class Source
{
public string Product {get;set;}
public List<Dictionary<string,string>> To_Description{get;set;}
}
public class Destination
{
public string Product {get;set;}
public Dictionary<string,string> To_Description{get;set;}
}
You could do it like this:
var src = JsonConvert.DeserializeObject<Source>(jsonString);
var dest = new Destination
{
Product = src.Product,
To_Description = src.To_Description[0]
};
var newJson = JsonConvert.SerializeObject(dest);
Note: You might want to check there really is just 1 item in the list!
Live example: https://dotnetfiddle.net/vxqumd

You do not need to create classes for this task. You can modify your object like this:
// Load the JSON from a file into a JObject
JObject o1 = JObject.Parse(File.ReadAllText(#"output.json"));
// Get the desired property whose value is to be replaced
var prop = o1.Property("to_Description");
// Replace the property value with the first child JObject of the existing value
prop.Value = prop.Value.Children<JObject>().FirstOrDefault();
// write the changed JSON back to the original file
File.WriteAllText(#"output.json", o1.ToString());
Fiddle: https://dotnetfiddle.net/M83zv3

I have used json2csharp to convert the actual and desired output to classes and manipulated the input json.. this will help in the maintenance in future
First defined the model
public class ToDescription
{
public string ProductDescription { get; set; }
}
public class ActualObject
{
public string Product { get; set; }
public List<ToDescription> to_Description { get; set; }
}
public class ChangedObject
{
public string Product { get; set; }
public ToDescription to_Description { get; set; }
}
Inject the logic
static void Main(string[] args)
{
string json = "{\"Product\": \"123\", \"to_Description\": [ { \"ProductDescription\": \"Product 1\" } ]} ";
ActualObject actualObject = JsonConvert.DeserializeObject<ActualObject>(json);
ChangedObject changedObject = new ChangedObject();
changedObject.Product = actualObject.Product;
changedObject.to_Description = actualObject.to_Description[0];
string formattedjson = JsonConvert.SerializeObject(changedObject);
Console.WriteLine(formattedjson);
}

Why not:
public class EntityDescription
{
public string ProductDescription { get; set; }
}
public class Entity
{
public string Product { get; set; }
}
public class Source : Entity
{
[JsonProperty("to_Description")]
public EntityDescription[] Description { get; set; }
}
public class Target : Entity
{
[JsonProperty("to_Description")]
public EntityDescription Description { get; set; }
}
var raw = File.ReadAllText(#"output.json");
var source = JsonConvert.DeserializeObject<Source>(raw);
var target = new Target { Product = source.Product, Description = source.Description.FirstOrDefault() };
var rawResult = JsonConvert.SerializeObject(target);
Update For dynamic JSON
var jObject = JObject.Parse(File.ReadAllText(#"output.json"));
var newjObject = new JObject();
foreach(var jToken in jObject) {
if(jToken.Value is JArray) {
List<JToken> l = jToken.Value.ToObject<List<JToken>>();
if(l != null && l.Count > 0) {
newjObject.Add(jToken.Key, l.First());
}
} else {
newjObject.Add(jToken.Key, jToken.Value);
}
}
var newTxt = newjObject.ToString();

Converting list of objects to json array

I have a List of class objects that have email address and status data members. I am trying to convert these to a json, making sure to have the "operations" word on the array.
This is my class:
class MyClass
{
public string email {get; set; }
public string status { get; set; }
}
This is my current code (not building):
List<MyClass> data = new List<MyClass>();
data = MagicallyGetData();
string json = new {
operations = new {
JsonConvert.SerializeObject(data.Select(s => new {
email_address = s.email,
status = s.status
}))
}
};
This is the JSON I am trying to get:
{
"operations": [
{
"email_address": "email1#email.com",
"status": "good2go"
},
{
"email_address": "email2#email.com",
"status": "good2go"
},...
]
}
EDIT1
I should mention that the data I am getting for this comes from a DB. I am de-serializing a JSON from the DB and using the data in several different ways, so I cannot change the member names of my class.

I believe this will give you what you want. You will have to change your class property names if possible.
Given this class
class MyClass
{
public string email_address { get; set; }
public string status { get; set; }
}
You can add the objects to a list
List<MyClass> data = new List<MyClass>()
{
new MyClass(){email_address = "e1#it.io", status = "s1"}
, new MyClass(){ email_address = "e2#it.io", status = "s1"}
};
Using an anonymous-type you can assign data to the property operations
var json = JsonConvert.SerializeObject(new
{
operations = data
});

class MyClass
{
public string email_address { get; set; }
public string status { get; set; }
}
List<MyClass> data = new List<MyClass>() { new MyClass() { email_address = "email1#email.com", status = "good2go" }, new MyClass() { email_address = "email2#email.com", status = "good2go" } };
//Serialize
var json = JsonConvert.SerializeObject(data);
//Deserialize
var jsonToList = JsonConvert.DeserializeObject<List<MyClass>>(json);

You can try with something like this:
using System.Web.Script.Serialization;
var jsonSerialiser = new JavaScriptSerializer();
var json = jsonSerialiser.Serialize(data);

Here is the simple code
JArray.FromObject(objList);

Schema dependent class to schemaless document using MongoDb and C#

Let us suppose we have a document to store our client which has fixed and extra fields.
So here goes our sample class for the client:
public class Client
{
public string Name{ get; set; }
public string Address{ get; set; }
public List<ExtraField> ExtraFields{ get; set; } //these fields are extra ones
}
In extra field class we have something like this:
public class ExtraField
{
public string Key{ get; set; }
public string Type { get; set; }
public string Value { get; set; }
}
If I use standard driver's behaviour for serialization I would get smth like this:
{{Name:VName, Address:VAddress, ExtraFields:[{Key:VKey,Type:VType,
Value:VValue},...]}, document2,...,documentn}
While I would like to have something like this:
{{Name:VName, Address:VAddress, VKey:VValue,...}, document2,...,documentn}
This would improve the search performance and is generally the point of document orientation.
How can I customize the serialization to such a way?

Here is the way I solved it (it works fine) and solved the issue.
using System;
using System.Collections.Generic;
using System.Linq; using System.Text;
using MongoDB.Bson;
using MongoDB.Bson.Serialization;
using MongoDB.Bson.Serialization.Serializers;
namespace TestDataGeneration {
public class FieldsWrapper : IBsonSerializable
{
public List<DataFieldValue> DataFieldValues { get; set; }
public object Deserialize(MongoDB.Bson.IO.BsonReader bsonReader, Type nominalType, IBsonSerializationOptions options)
{
if (nominalType != typeof(FieldsWrapper)) throw new ArgumentException("Cannot deserialize anything but self");
var doc = BsonDocument.ReadFrom(bsonReader);
var list = new List<DataFieldValue>();
foreach (var name in doc.Names)
{
var val = doc[name];
if (val.IsString)
list.Add(new DataFieldValue {LocalIdentifier = name, Values = new List<string> {val.AsString}});
else if (val.IsBsonArray)
{
DataFieldValue df = new DataFieldValue {LocalIdentifier = name};
foreach (var elem in val.AsBsonArray)
{
df.Values.Add(elem.AsString);
}
list.Add(df);
}
}
return new FieldsWrapper {DataFieldValues = list};
}
public void Serialize(MongoDB.Bson.IO.BsonWriter bsonWriter, Type nominalType, IBsonSerializationOptions options)
{
if (nominalType != typeof (FieldsWrapper))
throw new ArgumentException("Cannot serialize anything but self");
bsonWriter.WriteStartDocument();
foreach (var dataFieldValue in DataFieldValues)
{
bsonWriter.WriteName(dataFieldValue.LocalIdentifier);
if (dataFieldValue.Values.Count != 1)
{
var list = new string[dataFieldValue.Values.Count];
for (int i = 0; i < dataFieldValue.Values.Count; i++)
list[i] = dataFieldValue.Values[i];
BsonSerializer.Serialize(bsonWriter, list);
}
else
{
BsonSerializer.Serialize(bsonWriter, dataFieldValue.Values[0]);
}
}
bsonWriter.WriteEndDocument();
}
} }

Essentially you just need to implement two methods yourself. First one to serialize an object as you want and second to deserialize an object from db to your Client class back:
1 Seialize client class:
public static BsonValue ToBson(Client client)
{
if (client == null)
return null;
var doc = new BsonDocument();
doc["Name"] = client.Name;
doc["Address"] = client.Address;
foreach (var f in client.ExtraFields)
{
var fieldValue = new BsonDocument();
fieldValue["Type"] = f.Type;
fieldValue["Value"] = f.Value;
doc[f.Key] = fieldValue;
}
return doc;
}
2 Deserialize client object:
public static Client FromBson(BsonValue bson)
{
if (bson == null || !bson.IsBsonDocument)
return null;
var doc = bson.AsBsonDocument;
var client = new Client
{
ExtraFields = new List<ExtraField>(),
Address = doc["Address"].AsString,
Name = doc["Name"].AsString
};
foreach (var name in doc.Names)
{
var val = doc[name];
if (val is BsonDocument)
{
var fieldDoc = val as BsonDocument;
var field = new ExtraField
{
Key = name,
Value = fieldDoc["Value"].AsString,
Type = fieldDoc["Type"].AsString
};
client.ExtraFields.Add(field);
}
}
return client;
}
3 Complete test example:
I've added above two method to your client class.
var server = MongoServer.Create("mongodb://localhost:27020");
var database = server.GetDatabase("SO");
var clients = database.GetCollection<Type>("clients");
var client = new Client() {Id = ObjectId.GenerateNewId().ToString()};
client.Name = "Andrew";
client.Address = "Address";
client.ExtraFields = new List<ExtraField>();
client.ExtraFields.Add(new ExtraField()
{
Key = "key1",
Type = "type1",
Value = "value1"
});
client.ExtraFields.Add(new ExtraField()
{
Key = "key2",
Type = "type2",
Value = "value2"
});
//When inseting/saving use ToBson to serialize client
clients.Insert(Client.ToBson(client));
//When reading back from the database use FromBson method:
var fromDb = Client.FromBson(clients.FindOneAs<BsonDocument>());
4 Data structure in a database:
{
"_id" : ObjectId("4e3a66679c66673e9c1da660"),
"Name" : "Andrew",
"Address" : "Address",
"key1" : {
"Type" : "type1",
"Value" : "value1"
},
"key2" : {
"Type" : "type2",
"Value" : "value2"
}
}
BTW: Take a look into serialization tutorial as well.