What I have
I have templates that are stored in a database, and JSON data that gets converted into a dictionary in C#.
Example:
Template: "Hi {FirstName}"
Data: "{FirstName: 'Jack'}"
This works easily with one level of data by using a regular expression to pull out anything within {} in the template.
What I want
I would like to be able to go deeper in the JSON than the first layer.
Example:
Template: "Hi {Name: {First}}"
Data: "{Name: {First: 'Jack', Last: 'Smith'}}"
What approach should I be taking? (and some guidance on where to start with your pick)
A regular expression
Not use JSON in the template (in favor of xslt or something similar)
Something else
I'd also like to be able to loop through data in the template, but I have no idea at all where to start with that one!
Thanks heaps
You are in luck! SmartFormat does exactly as you describe. It is a lightweight, open-source string formatting utility.
It supports named placeholders:
var template = " {Name:{Last}, {First}} ";
var data = new { Name = new { First="Dwight", Last="Schrute" } };
var result = Smart.Format(template, data);
// Outputs: " Schrute, Dwight " SURPRISE!
It also supports list formatting:
var template = " {People:{}|, |, and} ";
var data = new { People = new[]{ "Dwight", "Michael", "Jim", "Pam" } };
var result = Smart.Format(template, data);
// Outputs: " Dwight, Michael, Jim, and Pam "
You can check out the unit tests for Named Placeholders and List Formatter to see plenty more examples!
It even has several forms of error-handling (ignore errors, output errors, throw errors).
Note: the named placeholder feature uses reflection and/or dictionary lookups, so you can deserialize the JSON into C# objects or nested Dictionaries, and it will work great!
Here is how I would do it:
Change your template to this format Hi {Name.First}
Now create a JavaScriptSerializer to convert JSON in Dictionary<string, object>
JavaScriptSerializer jss = new JavaScriptSerializer();
dynamic d = jss.Deserialize(data, typeof(object));
Now the variable d has the values of your JSON in a dictionary.
Having that you can run your template against a regex to replace {X.Y.Z.N} with the keys of the dictionary, recursively.
Full Example:
public void Test()
{
// Your template is simpler
string template = "Hi {Name.First}";
// some JSON
string data = #"{""Name"":{""First"":""Jack"",""Last"":""Smith""}}";
JavaScriptSerializer jss = new JavaScriptSerializer();
// now `d` contains all the values you need, in a dictionary
dynamic d = jss.Deserialize(data, typeof(object));
// running your template against a regex to
// extract the tokens that need to be replaced
var result = Regex.Replace(template, #"{?{([^}]+)}?}", (m) =>
{
// Skip escape values (ex: {{escaped value}} )
if (m.Value.StartsWith("{{"))
return m.Value;
// split the token by `.` to run against the dictionary
var pieces = m.Groups[1].Value.Split('.');
dynamic value = d;
// go after all the pieces, recursively going inside
// ex: "Name.First"
// Step 1 (value = value["Name"])
// value = new Dictionary<string, object>
// {
// { "First": "Jack" }, { "Last": "Smith" }
// };
// Step 2 (value = value["First"])
// value = "Jack"
foreach (var piece in pieces)
{
value = value[piece]; // go inside each time
}
return value;
});
}
I didn't handle exceptions (e.g. the value couldn't be found), you can handle this case and return the matched value if it wasn't found. m.Value for the raw value or m.Groups[1].Value for the string between {}.
Have you thought of using Javascript as your scripting language? I had great success with Jint, although the startup cost is high. Another option is Jurassic, which I haven't used myself.
If you happen to have a Web Application, using Razor maybe an idea, see here.
Using Regex or any sort of string parsing can certainly work for trivial things, but can get painful when you want logic or even just basic hierarchies. If you deserialize your JSON into nested Dictionaries, you can build a parser relatively easily:
// Untested and error prone, just to illustrate the concept
var parts = "parentObj.childObj.property".split('.');
Dictionary<object,object> current = yourDeserializedObject;
foreach(var key in parts.Take(parts.Length-1)){
current = current[key];
}
var value = current[parts.Last()];
Just whatever you do, don't do XSLT. Really, if XSLT is the answer then the question must have been really desperate :)
Why not us nvelocity or something?
Related
Does anyone know how to convert the below nested JSON to CSV via CHOETL (An ETL framework for .NET)? Thank you!
I'm using this code but it will only return the first equipment record.
CODE:
{
using (var json = new ChoJSONReader("./test.json"))
{
csv.Write(json.Cast<dynamic>().Select(i => new
{
EquipmentId = i.GpsLocation.Equipment[0].EquipmentId,
InquiryValue = i.GpsLocation.Equipment[0].InquiryValue,
Timestamp = i.GpsLocation.Equipment[0].Timestamp
}));
}
}
JSON:
"GpsLocation": {
"Equipment": [
{
"EquipmentId": "EQ00001",
"InquiryValue": [
"IV00001"
],
"Timestamp": "2020-01-01 01:01:01.01",
},
{
"EquipmentId": "EQ00002",
"InquiryValue": [
"IV00002"
],
"Timestamp": "2020-01-01 01:01:01.01"
}
]
}
}````
As others suggest, the issue is you are only looking at the first element of the array.
It appears that the easiest way to control what you serialise into CSV is by correctly defining your source objects from JSON. JSON Path expressions come in pretty handy.
What I ended up doing here is query all JSON to return an array of Equipment objects regardless of where they are in the hierarchy (which means you may need to filter it a bit better depending on your full JSON).
Then it's pretty easy to define each field based on JSON path and just pass the result to CSVWriter.
Also check out some gotchas that I outlined in the respective comment lines.
void Main()
{
var jsonString = "{\"GpsLocation\":{\"Equipment\":[{\"EquipmentId\":\"EQ00001\",\"InquiryValue\":[\"IV00001\"],\"Timestamp\":\"2020-01-01 01:01:01.01\"},{\"EquipmentId\":\"EQ00002\",\"InquiryValue\":[\"IV00002\"],\"Timestamp\":\"2020-01-01 01:01:01.01\"}]}}";
var jsonReader = new StringReader(jsonString);
var csvWriter = new StringWriter(); // outputs to string, comment out if you want file output
//var csvWriter = new StreamWriter(".\\your_output.csv"); // writes to a file of your choice
using (var csv = new ChoCSVWriter(csvWriter))
using (var json = new ChoJSONReader(jsonReader)
.WithJSONPath("$..Equipment[*]", true) // firstly you scope the reader to all Equipment objects. take note of the second parameter. Apparently you need to pass true here as otherwise it just won't return anythig
.WithField("EquipmentId", jsonPath: "$.EquipmentId", isArray: false) // then you scope each field in the array to what you want it to be. Since you want scalar values, pass `isArray: false` for better predictability
.WithField("InquiryValue", jsonPath: "$.InquiryValue[0]", isArray: false) // since your InquiryValue is actually an array, you want to obtain first element here. if you don't do this, fields names and values would go askew
.WithField("Timestamp", jsonPath: "$.Timestamp", fieldType: typeof(DateTime), isArray: false)) // you can also supply field type, otherwise it seems to default to `string`
{
csv.WithFirstLineHeader().Write(json);
}
Console.WriteLine(csvWriter.GetStringBuilder().ToString()); // comment this out if writing to file - you won't need it
}
Update summary:
Pivoted to update the code to rely on JSON Path scoping - this seems to allow for field name manipulation with pretty low effort
Looking at your comment, you could probably simplify your file writer a little bit - use StreamWriter instead of StringWriter - see updated code for example
Here is the working sample of producing CSV from your JSON
string json = #"{
""GpsLocation"": {
""Equipment"": [
{
""EquipmentId"": ""EQ00001"",
""InquiryValue"": [
""IV00001""
],
""Timestamp"": ""2020-02-01 01:01:01.01"",
},
{
""EquipmentId"": ""EQ00002"",
""InquiryValue"": [
""IV00002""
],
""Timestamp"": ""2020-01-01 01:01:01.01""
}
]
}
}";
StringBuilder csv = new StringBuilder();
using (var r = ChoJSONReader.LoadText(json)
.WithJSONPath("$.GpsLocation.Equipment")
.WithField("EquipmentId")
.WithField("InquiryValue", jsonPath: "InquiryValue[0]", fieldType: typeof(string))
.WithField("Timestamp", fieldType: typeof(DateTime))
)
{
using (var w = new ChoCSVWriter(csv)
.WithFirstLineHeader())
w.Write(r);
}
Console.WriteLine(csv.ToString());
Output:
EquipmentId,InquiryValue,Timestamp
EQ00001,IV00001,2/1/2020 1:01:01 AM
EQ00002,IV00002,1/1/2020 1:01:01 AM
Sample fiddle: https://dotnetfiddle.net/hJWtqH
Your code is sound, but the issue is that you're only writing the first variable in the array by using i.GpsLocation.Equipment[0]. Instead, try looping over everything by putting it into a for loop, and changing the [0] to your iterating variable inside of said loop.
Can one store the template of a string in a variable and use interpolation on it?
var name = "Joe";
var template = "Hi {name}";
I then want to do something like:
var result = $template;
The reason is my templates will come from a database.
I guess that these strings will have always the same number of parameters, even if they can change. For example, today template is "Hi {name}", and tomorrow could be "Hello {name}".
Short answer: No, you cannot do what you have proposed.
Alternative 1: use the string.Format method.
You can store in your database something like this:
"Hi {0}"
Then, when you retrieve the string template from the db, you can write:
var template = "Hi {0}"; //retrieved from db
var name = "Joe";
var result = string.Format(template, name);
//now result is "Hi Joe"
With 2 parameters:
var name2a = "Mike";
var name2b = "John";
var template2 = "Hi {0} and {1}!"; //retrieved from db
var result2 = string.Format(template2, name2a, name2b);
//now result2 is "Hi Mike and John!"
Alternative 2: use a placeholder.
You can store in your database something like this:
"Hi {name}"
Then, when you retrieve the string template from the db, you can write:
var template = "Hi {name}"; //retrieved from db
var name = "Joe";
var result = template.Replace("{name}", name);
//now result is "Hi Joe"
With 3 parameters:
var name2a = "Mike";
var name2b = "John";
var template2 = "Hi {name2a} and {name2b}!"; //retrieved from db
var result2 = template2
.Replace("{name2a}", name2a)
.Replace("{name2b}", name2b);
//now result2 is "Hi Mike and John!"
Pay attention at which token you choose for your placeholders. Here I used surrounding curly brackets {}. You should find something that is unlikely to cause collisions with the rest of your text. And that depends entirely on your context.
This can be done as requested using dynamic compilation, such as through the Microsoft.CodeAnalysis.CSharp.Scripting package. For example:
var name = "Joe";
var template = "Hi {name}";
var result = await CSharpScript.EvaluateAsync<string>(
"var name = \"" + name + "\"; " +
"return $\"" + template + "\";");
Note that this approach is slow, and you'd need to add more logic to handle escaping of quotes (and injection attacks) within strings, but the above serves as a proof-of-concept.
No you can't do that since it needs name value at the time string is created (compile time). Consider using String.Format or String.Replace instead.
I just had the same need in my app so will share my solution using String.Replace(). If you're able to use LINQ then you can use the Aggregate method (which is a reducing function, if you're familiar with functional programming) combined with a Dictionary that provides the substitutions you want.
string template = "Hi, {name} {surname}";
Dictionary<string, string> substitutions = new Dictionary<string, string>() {
{ "name", "Joe" },
{ "surname", "Bloggs" },
};
string result = substitutions.Aggregate(template, (args, pair) =>
args.Replace($"{{{pair.Key}}}", pair.Value)
);
// result == "Hi, Joe Bloggs"
This works by starting with the template and then iterating over each item in the substitution dictionary, replacing the occurrences of each one. The result of one Replace() call is fed into the input to the next, until all substitutions are performed.
The {{{pair.Key}}} bit is just to escape the { and } used to find a placeholder.
This is pretty old now, but as I've just come across it it's new to me!
It's a bit overkill for what you need, but I have used Handlebars.NET for this sort of thing.
You can create quite complex templates and merge in hierarchical data structures for the context. There's rules for looping and conditional sections, partial template compositing and even helper function extension points. It also handles many data types gracefully.
There's way too much to go into here, but a short example to illustrate...
var source = #"Hello {{Guest.FirstName}}{{#if Guest.Surname}} {{Guest.Surname}}{{/if}}!";
var template = Handlebars.Compile(source);
var rec = new {
Guest = new { FirstName = "Bob", Surname = null }
};
var resultString = template(rec);
In this case the surname will only be included in the output if the value is not null or empty.
Now admittedly this is more complicated for users than simple string interpolation, but remember that you can still just use {{fieldName}} if you want to, just that you can do lots more as well.
This particular nuGet is a port of HandlebarsJs so it has a high degree of compatibility. HandlebarsJs is itself a port of Mustache - there are direct dotNet ports of Mustache but IMHO HandlebarsNET is the business.
I'm trying to get all of the "last" values from this JSON here:
{"btc":{
"usd": {
"bitfinex": {
"last": "1191.60",
"volume": "1.99324e+7"
},
"bitstamp": {
"last": "1193.06",
"volume": "8.73693e+6"
},
"btce": {
"last": "1174.27",
"volume": "6.03521e+6"
}
}
}
But for some reason I can only access "btc" and "usd". I can't get anything out of it including the "last" values. Here is the code i'm using:
private string GetPrice()
{
WebClient wc = new WebClient();
var data = wc.DownloadString("http://preev.com/pulse/units:btc+usd/sources:bitfinex+bitstamp+btce");
JObject o = JObject.Parse(data);
string response = o["btc"].ToString();
return response;
}
If I change it to:
o["last"].ToString();
It just doesn't return anything. Can someone please provide me with a solution? I also tried making a key/value dict out of it and looping over each pair. Did not work.
The JObject structure is similar to a class with properties, so the first-level indexer ["btc"] returns another object that you have to query for its own properties ["usd"]
You can also opt for using JObject.SelectToken, generally not a bad idea. Other answers have shown how to chain the indexers but that's hard to read and maintain. Instead you can do:
jObj.SelectToken("btc[0].usd[0].bitstamp[0].last").ToString();
Further you can use the power of this syntax for other queries:
// a list o all the 'last' values
jObj.SelectTokens("btc.usd.*.last").Select(t=>t.ToString()).ToList();
Another advantage, if you're building a more complex system, is that you could put the queries in a config file or attributes etc to make them more manageable or deploy logic changes without rebuilding.
Yet another approach would be to build your own class structure and deserialize your json into it, so you have strongly typed values (double instead of string for the values for example)
public class btc {
public usd usd {get;set;}
}
public class usd....
var btcLoaded = JsonConvert.DeserializeObject<btc>(jsonString);
var lastBitstamp = btc.usd.bitstamp.last;
Use: o["btc"]["usd"]["bitfinex"]["last"].ToString() to get the 'last' value of 'bitfinex'.
After parsing the JSON whenever you index the o variable you are indexing from the root of the JSON. In order to access nested properties like 'last' you will need to index into the next level of the JSON as such:
var bitfinex = o["btc"]["usd"]["bitfinex"]["last"].ToString();
var bitstamp = o["btc"]["usd"]["bitstamp"]["last"].ToString();
var btce = o["btc"]["usd"]["btce"]["last"].ToString();
To reduce the repetition you could iterate over the properties under the btc.usd field.
if u want to all last values use this..
decimal[] lastValues = obj.SelectTokens("$..last").ToArray()
.Select(a => a.Parent.ToObject<decimal>()).ToArray();
if u want to dictionary, use this..
var dictionary = obj["btc"]["usd"].Select(a =>
new
{
Key = ((JProperty)a).Name,
Value = a.First["last"].ToObject<decimal>()
})
.ToDictionary(a => a.Key, a => a.Value);
I'm working with an API with an example output of the following
{
"id": 12345678
"photo-url-1280": "http://68.media.tumblr.com/5076e813bdc463409f59172b2c152487/tumblr_ocy8peMYvo1uur5s9o1_1280.png",
"photo-url-500": "http://67.media.tumblr.com/5076e813bdc463409f59172b2c152487/tumblr_ocy8peMYvo1uur5s9o1_500.png",
"photo-url-400": "http://68.media.tumblr.com/5076e813bdc463409f59172b2c152487/tumblr_ocy8peMYvo1uur5s9o1_400.png",
"photo-url-250": "http://67.media.tumblr.com/5076e813bdc463409f59172b2c152487/tumblr_ocy8peMYvo1uur5s9o1_250.png",
"photo-url-100": "http://67.media.tumblr.com/5076e813bdc463409f59172b2c152487/tumblr_ocy8peMYvo1uur5s9o1_100.png",
"photo-url-75": "http://67.media.tumblr.com/5076e813bdc463409f59172b2c152487/tumblr_ocy8peMYvo1uur5s9o1_75sq.png",
}
Those last items are very related, so I'd like to move them into their own object.
As of now, getting a specified image would be super messy. Getting the 400px version might look like this -
myPhotoObject.Photo400
However, if I was able to move these URLs into an object of their own, I could more cleanly call it like the following:
myPhotoObject.Photo.400
or even introduce more friendly methods
myPhoto.Photo.GetClosestTo(451);
There is a pattern in the URLs here - after an _ character, the identifying size is shown e.g. ..._1280.png and ..._500.png
Would the best way to get these be to write a getter property that just appends a list of known sizes to a URL? Would this way be any less/more efficient than using purely the JSON.net converters?
I suggest to parse your JSON into a dictionary first, as described here, and then manually copy it to a more structured object:
Dictionary<string, string> dict = JsonConvert.DeserializeObject<Dictionary<string, string>>(json);
var myImageObject = new { Id = "", Images = new Dictionary<int, string>()};
foreach(var key in dict.Keys) {
if(key == "id") {
myImageObject.Id = dict[key]
}
else {
var imageSize = ParseForImageSize(key);
myImageObject.Images.Add(imageSize, dict[key])
}
}
I have two arrays that need to be mapped. In code
var result = "[placeholder2] Hello my name is [placeholder1]";
var placeholder = { "[placeholder1]", "[placeholder2]", "[placeholder3]", "[placeholder4]" };
var placeholderValue = { "placeholderValue3", "placeholderValue2", "placeholderValue3" };
Array.ForEach(placeholder , i => result = result.Replace(i, placeholderValue));
given i, placeholderValue needs to be set in an intelligent way. I can implement a switch statement. The cyclomatic complexity would be unacceptable with 30 elements or so. What is a good pattern, extension method or otherwise means to achieve my goal?
I skipped null checks for simplicity
string result = "[placeholder2] Hello my name is [placeholder1]";
var placeHolders = new Dictionary<string, string>() {
{ "placeholder1", "placeholderValue1" },
{ "placeholder2", "placeholderValue2" }
};
var newResult = Regex.Replace(result,#"\[(.+?)\]",m=>placeHolders[m.Groups[1].Value]);
The smallest code change would be to just use a for loop, rather than a ForEach or, in your case, a ForEach taking a lambda. With a for loop you'll have the index of the appropriate value in the placehoderValue array.
The next improvement would be to make a single array of an object holding both a placeholder and it's value, rather than two 'parallel' arrays that you need to keep in sync.
Even better than that, and also even simpler to implement, is to just have a Dictionary with the key being a placeholder and the value being the placeholder value. This essentially does the above suggestion for you through the use of the KeyValuePair class (so you don't need to make your own).
At that point the pseudocode becomes:
foreach(key in placeholderDictionary) replace key with placeholderDictionary[key]
I think you want to use Zip to combine the placeholders with their values.
var result = "[placeholder2] Hello my name is [placeholder1]";
var placeholder = new[] { "[placeholder1]", "[placeholder2]", "[placeholder3]", "[placeholder4]" };
var placeholderValue = new[] { "placeholderValue1", "placeholderValue2", "placeholderValue3", "placeholderValue4" };
var placeHolderPairs = placeholder.Zip(placeholderValue, Tuple.Create);
foreach (var pair in placeHolderPairs)
{
result = result.Replace(pair.Item1, pair.Item2);
}