I'm receiving json files of different schema and have to dump them in sql data base.
The json files have the schema
{'type':'abc','data':{'column1':'x','column2':'y',.........}}
Corresponding to each type of schema I have a strongly typed class named similar to the type but with word 'Table' attached..
eg. 'abcTable' which has only the schema of json.data (column1, column2, ...)
So, what I can do is do a dynamic deserializing of the main json and then based on the type value do a strongly typed json parsing of the corresponding data
dynamic jsondata = JsonConvert.DeserializeObject<dynamic>(json);
if (jsonata.type=='abc')
{
var abcobj = JsonConvert.DeserializeObject<abcTable>(jsondata.data);
}
Here I'm deserializing the object twice, so don't look like the right way of doing..
Also I have 25+ such schemas and a similar number of classes/tables
So, I will be using a lot of if / else if /else statements...
I would like to understand if there are other better ways of solving what I'm trying to do..
Any help is sincerely appreciated..
Thanks
As usually, JObject is your friend:
var parsed = JObject.Parse(json);
var type = parsed.Value<string>("type");
if (type == "abc")
{
var abcObject = parsed["data"].ToObject<abcTable>();
}
I order to avoid many ifs, you can use the follwing pattern:
public interface ITableType
{
bool Match(string type);
void Handle(JToken jsonTable);
}
public AbcTableHandler: ITableType
{
public bool Match(string type)
{
return type == "abc";
}
public void Handle(JToken jsonTable)
{
var abcTable = jsonTable.ToObject<abcTable>();
// other code
}
}
usage:
var handlers = new[] { new AbcTableHandler() };
// ...
var parsed = JObject.Parse(json);
var type = parsed.Value<string>("type");
var handler = handlers.SingleOfDefault(h => h.Match(type));
if (handler == null)
throw new InvalidOperationException("Cannot find handler for " + type);
handler.Handle(parsed["data"]);
EDIT:
Adding multiple handlers:
var handlers = new ITableType[] { new AbcTableHandler(), new OtherHandler, etc.. };
or
var handlers = new List<ITableType>();
handlers.Add(new AbcTableHandler());
handlers.Add(new OtherHandler());
Related
Is there a more efficient way of converting dynamo db data into concrete types? For example, when I query the data everything is in:
List<Dictionary<string, AttributeValue>>
Is it possible to easily convert the type without having to loop through each item and doing this all manually?
For example I am doing:
return items.Select(item => new Connection
{
ConnectionId = Guid.Parse(item["connectionId"].S),
ClientId = item["clientId"].S,
ProviderId = item["providerId"].S,
Scopes = item["scopes"].SS.ToArray(),
CredentialsId = item["credentialsId"].S,
Evidences = ToEvidences(item["consentEvidences"].L)
})
.ToList();
This then returns a list of my type Connection however I am explicitly mapping each field. Is there an easier way or a helper library that can do the mapping?
I think you'll have luck with the higher-level .NET Document model. It presents more natural data types.
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DotNetSDKMidLevel.html
The easiest way I have found is to use the Document.FromAttributeMap function to convert it to a Document object and then convert it again to the .NET type using the DynamoDBContext.FromDocument method as shown below.
public async Task<IEnumerable<WeatherForecast>> GetAll(string cityName)
{
var queryRequest = new QueryRequest()
{
TableName = nameof(WeatherForecast),
KeyConditionExpression = "CityName = :cityName",
ExpressionAttributeValues = new Dictionary<string, AttributeValue>()
{
{":cityName", new AttributeValue(cityName)},
}
};
var response = await _dynamoDbClient.QueryAsync(queryRequest);
return response.Items.Select(a =>
{
var doc = Document.FromAttributeMap(a);
return _dynamoDbContext.FromDocument<WeatherForecast>(doc);
});
}
I'm having trouble converting JSON data to a dynamic type definition. I've looked at JObject, JsonConvert serialize/deserialize and nothing works. The closest thing to making this work is JsonConvert.DeserializeAnonymousType but this requires a definition that matches the JSON. In my case, the JSON is quite complicated so I really need to convert an instance of the JSON to a fully anonymous, dynamic type.
dynamic rawjson = #"{ 'tags': { 'abcd' : '12345' },'properties': { 'desired': { 'PropOne' : '2345', 'PropTwo' : '6789' } } }";
#region reference object
dynamic reference = new
{
SomeName = $"xxxx",
initialTwin = new
{
tags = new { abcd = 12345 },
properties = new
{
desired = new
{
PropOne = "2345",
PropTwo = "6789"
}
}
}
};
#endregion
dynamic anonobject = JsonConvert.DeserializeAnonymousType(rawjson, reference.initialTwin);
dynamic testobject = new
{
SomeName = $"xxxx",
initialTwin = $"{anonobject}"
};
I need the "testobject" in the code above to look exactly like the "reference" object. Using the DesializeAnonymousType gets me very close but the definition would be very hard to create and maintain.
How can I get the same results working from an instance of JSON data without typing the definition for DeserializeAnonyousType? Is that possible?
I have tested the way below and it worked for me:
var anonobject = JsonConvert.DeserializeObject<ExpandoObject>(rawjson);
dynamic testobject = new
{
SomeName = $"xxxx",
initialTwin = anonobject
};
But since this is a dynamic object, you need to know the properties you may want to use.
Example:
Console.WriteLine(testobject.initialTwin.tags.abcd);
The following code sample writes a simple object to a couchbase lite (version 2) database and reads all objects afterwards. This is what you can find in the official documentation here
This is quite a lot of manual typing since every property of every object must be transferred to the MutableObject.
class Program
{
static void Main(string[] args)
{
Couchbase.Lite.Support.NetDesktop.Activate();
const string DbName = "MyDb";
var db = new Database(DbName);
var item = new Item { Name = "test", Value = 5 };
// Serialization HERE
var doc = new MutableDocument();
doc.SetString("Name", item.Name);
doc.SetInt("Value", item.Value);
db.Save(doc);
using (var qry = QueryBuilder.Select(SelectResult.All())
.From(DataSource.Database(db)))
{
foreach (var result in qry.Execute())
{
var resultItem = new Item
{
// Deserialization HERE
Name = result[DbName].Dictionary.GetString("Name"),
Value = result[DbName].Dictionary.GetInt("Value")
};
Console.WriteLine(resultItem.Name);
}
}
Console.ReadKey();
}
class Item
{
public string Name { get; set; }
public int Value { get; set; }
}
}
From my research Couchbase lite uses JsonConvert internally, so there might be a way to simplify all that with the help of JsonConvert.
Anything like:
var json = JsonConvert.SerializeObject(item);
var doc = new MutableDocument(json); // No overload to provide raw JSON
or maybe
var data = JsonConvert.SerializeToDict(item); // JsonConvert does not provide this
var doc = new MutableDocument(data);
Is there or is this some kind of optimization and the preferred approach is by intend?
People ask about this quite often, but Couchbase Lite does not actually store JSON strings in the database. They are stored in a different format so this would not give the benefit that you think (the JSON would need to be reparsed and then broken down into the other format). I'd been pushing for a way to serialize classes directly instead of going through dictionary objects (which seems like the ultimate goal here) but our priority is on things that enterprise clients want and this doesn't seem to be one of them. Note that for it to make it in, it needs to be implemented in C# Java and Objective-C / Swift.
I don't know about JsonConvert but there seems to be a constructor that takes IDictionary<string, object> as argument. So I would try something like this (brain-compiled):
MutableDocument CreateDocument(object data)
{
if (data == null) return null;
var propertyValues = new Dictionary<string, object>();
foreach (var property in data.GetType().GetProperties(BindingFlags.Public | BindingFlags.Instance))
{
propertyValues[property.Name] = property.GetValue(data);
}
return new MutableDocument(propertyValues);
}
See if this works.
Are there any libraries in .Net to help compare and find differences between two json objects? I've found some solutions available for JavaScript, but nothing interesting for C#. The point of my question is to create json with changes marked in some way, based on the comparison. So that the user could see where the changes are.
using Microsoft.XmlDiffPatch;
using Newtonsoft.Json;
Convert each json to xml and use MS XmlDiff libary. Available on nuget. Differences are given in another xml doc which here I write to the console. This is suitable for unit testing for example.
public bool CompareJson(string expected, string actual)
{
var expectedDoc = JsonConvert.DeserializeXmlNode(expected, "root");
var actualDoc = JsonConvert.DeserializeXmlNode(actual, "root");
var diff = new XmlDiff(XmlDiffOptions.IgnoreWhitespace |
XmlDiffOptions.IgnoreChildOrder);
using (var ms = new MemoryStream())
using (var writer = new XmlTextWriter(ms, Encoding.UTF8))
{
var result = diff.Compare(expectedDoc, actualDoc, writer);
if (!result)
{
ms.Seek(0, SeekOrigin.Begin);
Console.WriteLine(new StreamReader(ms).ReadToEnd());
}
return result;
}
}
I have used different JSON objects than those in your example but it will apply to your case correctly.
private static string GetJsonDiff(string action, string existing, string modified, string objectType)
{
// convert JSON to object
JObject xptJson = JObject.Parse(modified);
JObject actualJson = JObject.Parse(existing);
// read properties
var xptProps = xptJson.Properties().ToList();
var actProps = actualJson.Properties().ToList();
// find differing properties
var auditLog = (from existingProp in actProps
from modifiedProp in xptProps
where modifiedProp.Path.Equals(existingProp.Path)
where !modifiedProp.Value.ToString().Equals(existingProp.Value.ToString())
select new AuditLog
{
Field = existingProp.Path,
OldValue = existingProp.Value.ToString(),
NewValue = modifiedProp.Value.ToString(),
Action = action, ActionBy = GetUserName(),
ActionDate = DateTime.UtcNow.ToLongDateString(),
ObjectType = objectType
}).ToList();
return JsonConvert.SerializeObject(auditLog);
}
I think your best bet is to use JSON.NET to create two JSON objects, then recursively loop through the tree, comparing each node to see if it exists and is equal while you go.
I think the best way to go here is to create objects using newtonsoft json.http://www.nuget.org/packages/newtonsoft.json/
So, you will have two objects of the same type, which you can easily compare and mark the differences.
private IEnumerable<JProperty> JSONCompare(string expectedJSON, string actualJSON)
{
// convert JSON to object
JObject xptJson = JObject.Parse(expectedJSON);
JObject actualJson = JObject.Parse(actualJSON);
// read properties
var xptProps = xptJson.Properties().ToList();
var actProps = actualJson.Properties().ToList();
// find missing properties
var missingProps = xptProps.Where(expected => actProps.Where(actual => actual.Name == expected.Name).Count() == 0);
return missingProps;
}
I am trying to deserialize an instance of the following class from a JSON string using JavaScriptSerializer:
public class Filter
{
public HashSet<int> DataSources { get; set; }
}
Here is the code I am trying out:
Filter f = new Filter();
f.DataSources = new HashSet<int>(){1,2};
string json = (new JavaScriptSerializer()).Serialize(f);
var g= (new JavaScriptSerializer()).Deserialize<Filter>(json);
It errors out with the following message:
Object of type 'System.Collections.Generic.List1[System.Int32]'
cannot be converted to type
'System.Collections.Generic.HashSet1[System.Int32]'.
Apparently, the serializer is unable to distinguish between a list and set from JSON representation. What is the solution to this?
Note : I would prefer avoiding the use of external libraries due to constraints at work.
What is the solution to this?
Use Json.Net. This code works...
Filter f = new Filter();
f.DataSources = new HashSet<int>() { 1, 2 };
string json = JsonConvert.SerializeObject(f);
var g = JsonConvert.DeserializeObject<Filter>(json);
EDIT
DataContractJsonSerializer seems to work too...
DataContractJsonSerializer dcjs = new DataContractJsonSerializer(typeof(Filter));
var g2 = dcjs.ReadObject(new MemoryStream(Encoding.UTF8.GetBytes(json))) as Filter;
Here's my simple NOT so great solution, but it works.
var dataList = new JavaScriptSerializer().Deserialize<List<int>>(returnData);
var data = new HashSet<int>(dataList);