We Keep Coding

Couchbase Lite 2 + JsonConvert

The following code sample writes a simple object to a couchbase lite (version 2) database and reads all objects afterwards. This is what you can find in the official documentation here
This is quite a lot of manual typing since every property of every object must be transferred to the MutableObject.
class Program
{
static void Main(string[] args)
{
Couchbase.Lite.Support.NetDesktop.Activate();
const string DbName = "MyDb";
var db = new Database(DbName);
var item = new Item { Name = "test", Value = 5 };
// Serialization HERE
var doc = new MutableDocument();
doc.SetString("Name", item.Name);
doc.SetInt("Value", item.Value);
db.Save(doc);
using (var qry = QueryBuilder.Select(SelectResult.All())
.From(DataSource.Database(db)))
{
foreach (var result in qry.Execute())
{
var resultItem = new Item
{
// Deserialization HERE
Name = result[DbName].Dictionary.GetString("Name"),
Value = result[DbName].Dictionary.GetInt("Value")
};
Console.WriteLine(resultItem.Name);
}
}
Console.ReadKey();
}
class Item
{
public string Name { get; set; }
public int Value { get; set; }
}
}
From my research Couchbase lite uses JsonConvert internally, so there might be a way to simplify all that with the help of JsonConvert.
Anything like:
var json = JsonConvert.SerializeObject(item);
var doc = new MutableDocument(json); // No overload to provide raw JSON
or maybe
var data = JsonConvert.SerializeToDict(item); // JsonConvert does not provide this
var doc = new MutableDocument(data);
Is there or is this some kind of optimization and the preferred approach is by intend?

People ask about this quite often, but Couchbase Lite does not actually store JSON strings in the database. They are stored in a different format so this would not give the benefit that you think (the JSON would need to be reparsed and then broken down into the other format). I'd been pushing for a way to serialize classes directly instead of going through dictionary objects (which seems like the ultimate goal here) but our priority is on things that enterprise clients want and this doesn't seem to be one of them. Note that for it to make it in, it needs to be implemented in C# Java and Objective-C / Swift.

I don't know about JsonConvert but there seems to be a constructor that takes IDictionary<string, object> as argument. So I would try something like this (brain-compiled):
MutableDocument CreateDocument(object data)
{
if (data == null) return null;
var propertyValues = new Dictionary<string, object>();
foreach (var property in data.GetType().GetProperties(BindingFlags.Public | BindingFlags.Instance))
{
propertyValues[property.Name] = property.GetValue(data);
}
return new MutableDocument(propertyValues);
}
See if this works.

Change the property names in a JSON object before conversion to XML

I am trying to convert some JSON to XML, but before that I need to change some properties to make a successful conversion.
Some of the properties in the JSON structure start with numbers, and when I try to make the conversion to XML I get an error because XML does not admit tags that start with numbers.
So, one solution that works for me is to change those property names that start with numbers by adding a prefix to the property.
I have been trying to do something like this:
public string ChangeNumericalPropertyNames(JsonReader reader)
{
JObject jo = JObject.Load(reader);
foreach (JProperty jp in jo.Properties())
{
if (Regex.IsMatch(jp.Name, #"^\d"))
{
string name = "n" + jp.Name;
//Logic to set changed name
}
}
return "Here I want to return the entire json string with changed names";
}
When I try this:
jp.Name = name;
Visual studio says that is not possible because jp.Name is read only.
Does anybody know how to achieve this solution?

Since the property name is read only, you'll need to replace the whole property. You can use the Replace method to do this:
if (Regex.IsMatch(jp.Name, #"^\d"))
{
string name = "n" + jp.Name;
jp.Replace(new JProperty(name, jp.Value));
}
However, this will lead to another problem-- since you are trying to modify the Properties collection of the JObject while iterating over it, Json.Net will throw an InvalidOperationException. To get around this, you must copy the the properties to a separate list and iterate over that instead. You can do that using the ToList() method in your foreach like this:
foreach (JProperty jp in jo.Properties().ToList())
Finally, to convert the updated JObject back to JSON, just use ToString(). Putting it all together we have:
public static string ChangeNumericalPropertyNames(JsonReader reader)
{
JObject jo = JObject.Load(reader);
foreach (JProperty jp in jo.Properties().ToList())
{
if (Regex.IsMatch(jp.Name, #"^\d"))
{
string name = "n" + jp.Name;
jp.Replace(new JProperty(name, jp.Value));
}
}
return jo.ToString();
}
Fiddle: https://dotnetfiddle.net/rX4Jyy
The above method will only handle a simple JSON object with properties all on one level. You indicated in your comment that your actual JSON is not flat, but hierarchical. In order to replace all of the numeric property names in a hierarchical structure, you'll need to make your method recursive, like this:
public static string ChangeNumericalPropertyNames(JsonReader reader)
{
JObject jo = JObject.Load(reader);
ChangeNumericalPropertyNames(jo);
return jo.ToString();
}
public static void ChangeNumericalPropertyNames(JObject jo)
{
foreach (JProperty jp in jo.Properties().ToList())
{
if (jp.Value.Type == JTokenType.Object)
{
ChangeNumericalPropertyNames((JObject)jp.Value);
}
else if (jp.Value.Type == JTokenType.Array)
{
foreach (JToken child in jp.Value)
{
if (child.Type == JTokenType.Object)
{
ChangeNumericalPropertyNames((JObject)child);
}
}
}
if (Regex.IsMatch(jp.Name, #"^\d"))
{
string name = "n" + jp.Name;
jp.Replace(new JProperty(name, jp.Value));
}
}
}
Fiddle: https://dotnetfiddle.net/qeZK1C

C# Trying to mask child values in a dynamic object

At the moment I'm adding functionality to our service that will take in an object that is about to be logged to trace and mask any sensitive fields that are included in the object.
The issue is that we can get objects with different layers. The code I have written so far only handles a parent field and a single child field and uses a nasty embedded for loop implementation to do it.
In the event that we have a third embedded layer of fields in an object we want to log, this wouldn't be able to handle it at all. There has to be a more efficient way of handling generic parsing of a dynamic object, but so far it's managed to avoid me.
The actual code that deserializes and then masks field sin the object looks like this:
private string MaskSensitiveData(string message)
{
var maskedMessage = JsonConvert.DeserializeObject<dynamic>(message);
LoggingProperties.GetSensitiveFields();
for (int i = 0; i < LoggingProperties.Fields.Count(); i++)
{
for (int j = 0; j < LoggingProperties.SubFields.Count(); j++)
{
if (maskedMessage[LoggingProperties.Fields[i]] != null)
{
if (maskedMessage[LoggingProperties.Fields[i]][LoggingProperties.SubFields[j]] != null)
{
maskedMessage[LoggingProperties.Fields[i]][LoggingProperties.SubFields[j]] = MaskField(LoggingProperties.SubFieldLengths[j]);
}
}
}
}
return maskedMessage.ToString(Formatting.None);
}
And it works off of a LoggingProperties class that looks like this:
public static class LoggingProperties
{
// Constants indicating the number of fields we need to mask at present
private const int ParentFieldCount = 2;
private const int SubFieldCount = 4;
// Constant representing the character we are using for masking
public const char MaskCharacter = '*';
// Parent fields array
public static string[] Fields = new string[ParentFieldCount];
// Subfields array
public static string[] SubFields = new string[SubFieldCount];
// Array of field lengths, each index matching the subfield array elements
public static int[] SubFieldLengths = new int[SubFieldCount];
public static void GetSensitiveFields()
{
// Sensitive parent fields
Fields[0] = "Parent1";
Fields[1] = "Parent2";
// Sensitive subfields
SubFields[0] = "Child1";
SubFields[1] = "Child2";
SubFields[2] = "Child3";
SubFields[3] = "Child4";
// Lengths of sensitive subfields
SubFieldLengths[0] = 16;
SubFieldLengths[1] = 16;
SubFieldLengths[2] = 20;
SubFieldLengths[3] = 3;
}
}
}
The aim was to have a specific list of fields for the masking method to look out for that could be expanded or contracted along with our systems needs.
The nested loop method though just seems a bit roundabout to me. Any help is appreciated.
Thanks!
UPDATE:
Here's a small example of a parent and child record that would be in the message prior to the deserialize call. For this example say I'm attempting to mask the currency ID (So in properties the fields could be set like this: Parent1 = "Amounts" and Child1 = "CurrencyId"):
{
"Amounts":
{
"Amount":20.0,
"CurrencyId":826
}
}
An example of a problem would then be if the Amount was divided into pounds and pence:
{
"Amounts":
{
"Amount":
{
"Pounds":20,
"Pence":0
},
"CurrencyId":826
}
}
This would another layer and yet another embedded for loop...but with that I would be making it overly complex and difficult if the next record in a message had only two layers.
Hope this clarifies a few things =]

Okay, I've really tried but I couldn't figure out an elegant way. Here's what I did:
The first try was using reflection but since all the objects are of type JObject / JToken, I found no way of deciding whether a property is an object or a value.
The second try was (and still is, if you can figure out a good way) more promising: parsing the JSON string into a JObject with var data = JObject.Parse(message) and enumerating its properties in a recursive method like this:
void Mask(data)
{
foreach (JToken token in data)
{
if (token.Type == JTokenType.Object)
{
// It's an object, mask its children
Mask(token.Children());
}
else
{
// Somehow mask it but I couldn't figure out to do it with JToken
// Pseudocode, it doesn't actually work:
if (keysToMask.Contains(token.Name))
token.Value = "***";
}
}
}
Since it doesn't work with JTokens, I've tried the same with JProperties and it works for the root object, but there's a problem: although you can see if a given JProperty is an object, you can not select its children object, JProperty.Children() gives JToken again and I found no way to convert it to a JProperty. If anyone knows how to achieve it, please post it.
So the only way I found is a very dirty one: using regular expressions. It's all but elegant - but it works.
// Make sure the JSON is well formatted
string formattedJson = JObject.Parse(message).ToString();
// Define the keys of the values to be masked
string[] maskedKeys = {"mask1", "mask2"};
// Loop through each key
foreach (var key in maskedKeys)
{
string original_pattern = string.Format("(\"{0}\": )(\"?[^,\\r\\n]+\"?)", key);
string masked_pattern = "$1\"censored\"";
Regex pattern = new Regex(original_pattern);
formatted_json = pattern.Replace(formatted_json, masked_pattern);
}
// Parse the masked string
var maskedMessage = JsonConvert.DeserializeObject<dynamic>(formatted_json);
Assuming this is your input:
{
"val1" : "value1",
"val2" : "value2",
"mask1" : "to be masked",
"prop1" : {
"val3" : "value3",
"val1" : "value1",
"mask2" : "to be masked too",
"prop2" : {
"val1" : "value 1 again",
"mask1" : "this will also get masked"
}
}
}
This is what you get:
{
"val1": "value1",
"val2": "value2",
"mask1": "censored",
"prop1": {
"val3": "value3",
"val1": "value1",
"mask2": "censored",
"prop2": {
"val1": "value 1 again",
"mask1": "censored"
}
}
}

Parsing json files of different schemas to different tables json.net

I'm receiving json files of different schema and have to dump them in sql data base.
The json files have the schema
{'type':'abc','data':{'column1':'x','column2':'y',.........}}
Corresponding to each type of schema I have a strongly typed class named similar to the type but with word 'Table' attached..
eg. 'abcTable' which has only the schema of json.data (column1, column2, ...)
So, what I can do is do a dynamic deserializing of the main json and then based on the type value do a strongly typed json parsing of the corresponding data
dynamic jsondata = JsonConvert.DeserializeObject<dynamic>(json);
if (jsonata.type=='abc')
{
var abcobj = JsonConvert.DeserializeObject<abcTable>(jsondata.data);
}
Here I'm deserializing the object twice, so don't look like the right way of doing..
Also I have 25+ such schemas and a similar number of classes/tables
So, I will be using a lot of if / else if /else statements...
I would like to understand if there are other better ways of solving what I'm trying to do..
Any help is sincerely appreciated..
Thanks

As usually, JObject is your friend:
var parsed = JObject.Parse(json);
var type = parsed.Value<string>("type");
if (type == "abc")
{
var abcObject = parsed["data"].ToObject<abcTable>();
}
I order to avoid many ifs, you can use the follwing pattern:
public interface ITableType
{
bool Match(string type);
void Handle(JToken jsonTable);
}
public AbcTableHandler: ITableType
{
public bool Match(string type)
{
return type == "abc";
}
public void Handle(JToken jsonTable)
{
var abcTable = jsonTable.ToObject<abcTable>();
// other code
}
}
usage:
var handlers = new[] { new AbcTableHandler() };
// ...
var parsed = JObject.Parse(json);
var type = parsed.Value<string>("type");
var handler = handlers.SingleOfDefault(h => h.Match(type));
if (handler == null)
throw new InvalidOperationException("Cannot find handler for " + type);
handler.Handle(parsed["data"]);
EDIT:
Adding multiple handlers:
var handlers = new ITableType[] { new AbcTableHandler(), new OtherHandler, etc.. };
or
var handlers = new List<ITableType>();
handlers.Add(new AbcTableHandler());
handlers.Add(new OtherHandler());

A clean way of generating QueryString parameters for web requests

I came across a problem in my current application that required fiddling with the query string in a base Page class (which all my pages inherit from) to solve the problem. Since some of my pages use the query string I was wondering if there is any class that provides clean and simple query string manipulation.
Example of code:
// What happens if I want to future manipulate the query string elsewhere
// (e.g. maybe rewrite when the request comes back in)
// Or maybe the URL already has a query string (and the ? is invalid)
Response.Redirect(Request.Path + "?ProductID=" + productId);

Use HttpUtility.ParseQueryString, as someone suggested (and then deleted).
This will work, because the return value from that method is actually an HttpValueCollection, which inherits NameValueCollection (and is internal, you can't reference it directly). You can then set the names/values in the collection normally (including add/remove), and call ToString -- which will produce the finished querystring, because HttpValueCollection overrides ToString to reproduce an actual query string.

I was hoping to find a solution built into the framework but didn't. (those methods that are in the framework require to much work to make it simple and clean)
After trying several alternatives I currently use the following extension method: (post a better solution or comment if you have one)
public static class UriExtensions
{
public static Uri AddQuery(this Uri uri, string name, string value)
{
string newUrl = uri.OriginalString;
if (newUrl.EndsWith("&") || newUrl.EndsWith("?"))
newUrl = string.Format("{0}{1}={2}", newUrl, name, value);
else if (newUrl.Contains("?"))
newUrl = string.Format("{0}&{1}={2}", newUrl, name, value);
else
newUrl = string.Format("{0}?{1}={2}", newUrl, name, value);
return new Uri(newUrl);
}
}
This extension method makes for very clean redirection and uri manipulation:
Response.Redirect(Request.Url.AddQuery("ProductID", productId).ToString());
// Will generate a URL of www.google.com/search?q=asp.net
var url = new Uri("www.google.com/search").AddQuery("q", "asp.net")
and will work for the following Url's:
"http://www.google.com/somepage"
"http://www.google.com/somepage?"
"http://www.google.com/somepage?OldQuery=Data"
"http://www.google.com/somepage?OldQuery=Data&"

Note that whatever route you use, you should really encode the values - Uri.EscapeDataString should do that for you:
string s = string.Format("http://somesite?foo={0}&bar={1}",
Uri.EscapeDataString("&hehe"),
Uri.EscapeDataString("#mwaha"));

What I usually do is just rebuild the querystring. Request has a QueryString collection.
You can iterator over that to get the current (unencoded) parameters out, and just join them together (encoding as you go) with the appropriate separators.
The advantage is that Asp.Net has done the original parsing for you, so you don't need to worry about edge cases such as trailing & and ?s.

I find my way for easy manipulating with get parameters.
public static string UrlFormatParams(this string url, string paramsPattern, params object[] paramsValues)
{
string[] s = url.Split(new string[] {"?"}, StringSplitOptions.RemoveEmptyEntries);
string newQueryString = String.Format(paramsPattern, paramsValues);
List<string> pairs = new List<string>();
NameValueCollection urlQueryCol = null;
NameValueCollection newQueryCol = HttpUtility.ParseQueryString(newQueryString);
if (1 == s.Length)
{
urlQueryCol = new NameValueCollection();
}
else
{
urlQueryCol = HttpUtility.ParseQueryString(s[1]);
}
for (int i = 0; i < newQueryCol.Count; i++)
{
string key = newQueryCol.AllKeys[i];
urlQueryCol[key] = newQueryCol[key];
}
for (int i = 0; i < urlQueryCol.Count; i++)
{
string key = urlQueryCol.AllKeys[i];
string pair = String.Format("{0}={1}", key, urlQueryCol[key]);
pairs.Add(pair);
}
newQueryString = String.Join("&", pairs.ToArray());
return String.Format("{0}?{1}", s[0], newQueryString);
}
Use it like
"~/SearchInHistory.aspx".UrlFormatParams("t={0}&s={1}", searchType, searchString)

Check This!!!
// First Get The Method Used by Request i.e Get/POST from current Context
string method = context.Request.HttpMethod;
// Declare a NameValueCollection Pair to store QueryString parameters from Web Request
NameValueCollection queryStringNameValCollection = new NameValueCollection();
if (method.ToLower().Equals("post")) // Web Request Method is Post
{
string contenttype = context.Request.ContentType;
if (contenttype.ToLower().Equals("application/x-www-form-urlencoded"))
{
int data = context.Request.ContentLength;
byte[] bytData = context.Request.BinaryRead(context.Request.ContentLength);
queryStringNameValCollection = context.Request.Params;
}
}
else // Web Request Method is Get
{
queryStringNameValCollection = context.Request.QueryString;
}
// Now Finally if you want all the KEYS from QueryString in ArrayList
ArrayList arrListKeys = new ArrayList();
for (int index = 0; index < queryStringNameValCollection.Count; index++)
{
string key = queryStringNameValCollection.GetKey(index);
if (!string.IsNullOrEmpty(key))
{
arrListKeys.Add(key.ToLower());
}
}