At the moment I'm adding functionality to our service that will take in an object that is about to be logged to trace and mask any sensitive fields that are included in the object.
The issue is that we can get objects with different layers. The code I have written so far only handles a parent field and a single child field and uses a nasty embedded for loop implementation to do it.
In the event that we have a third embedded layer of fields in an object we want to log, this wouldn't be able to handle it at all. There has to be a more efficient way of handling generic parsing of a dynamic object, but so far it's managed to avoid me.
The actual code that deserializes and then masks field sin the object looks like this:
private string MaskSensitiveData(string message)
{
var maskedMessage = JsonConvert.DeserializeObject<dynamic>(message);
LoggingProperties.GetSensitiveFields();
for (int i = 0; i < LoggingProperties.Fields.Count(); i++)
{
for (int j = 0; j < LoggingProperties.SubFields.Count(); j++)
{
if (maskedMessage[LoggingProperties.Fields[i]] != null)
{
if (maskedMessage[LoggingProperties.Fields[i]][LoggingProperties.SubFields[j]] != null)
{
maskedMessage[LoggingProperties.Fields[i]][LoggingProperties.SubFields[j]] = MaskField(LoggingProperties.SubFieldLengths[j]);
}
}
}
}
return maskedMessage.ToString(Formatting.None);
}
And it works off of a LoggingProperties class that looks like this:
public static class LoggingProperties
{
// Constants indicating the number of fields we need to mask at present
private const int ParentFieldCount = 2;
private const int SubFieldCount = 4;
// Constant representing the character we are using for masking
public const char MaskCharacter = '*';
// Parent fields array
public static string[] Fields = new string[ParentFieldCount];
// Subfields array
public static string[] SubFields = new string[SubFieldCount];
// Array of field lengths, each index matching the subfield array elements
public static int[] SubFieldLengths = new int[SubFieldCount];
public static void GetSensitiveFields()
{
// Sensitive parent fields
Fields[0] = "Parent1";
Fields[1] = "Parent2";
// Sensitive subfields
SubFields[0] = "Child1";
SubFields[1] = "Child2";
SubFields[2] = "Child3";
SubFields[3] = "Child4";
// Lengths of sensitive subfields
SubFieldLengths[0] = 16;
SubFieldLengths[1] = 16;
SubFieldLengths[2] = 20;
SubFieldLengths[3] = 3;
}
}
}
The aim was to have a specific list of fields for the masking method to look out for that could be expanded or contracted along with our systems needs.
The nested loop method though just seems a bit roundabout to me. Any help is appreciated.
Thanks!
UPDATE:
Here's a small example of a parent and child record that would be in the message prior to the deserialize call. For this example say I'm attempting to mask the currency ID (So in properties the fields could be set like this: Parent1 = "Amounts" and Child1 = "CurrencyId"):
{
"Amounts":
{
"Amount":20.0,
"CurrencyId":826
}
}
An example of a problem would then be if the Amount was divided into pounds and pence:
{
"Amounts":
{
"Amount":
{
"Pounds":20,
"Pence":0
},
"CurrencyId":826
}
}
This would another layer and yet another embedded for loop...but with that I would be making it overly complex and difficult if the next record in a message had only two layers.
Hope this clarifies a few things =]
Okay, I've really tried but I couldn't figure out an elegant way. Here's what I did:
The first try was using reflection but since all the objects are of type JObject / JToken, I found no way of deciding whether a property is an object or a value.
The second try was (and still is, if you can figure out a good way) more promising: parsing the JSON string into a JObject with var data = JObject.Parse(message) and enumerating its properties in a recursive method like this:
void Mask(data)
{
foreach (JToken token in data)
{
if (token.Type == JTokenType.Object)
{
// It's an object, mask its children
Mask(token.Children());
}
else
{
// Somehow mask it but I couldn't figure out to do it with JToken
// Pseudocode, it doesn't actually work:
if (keysToMask.Contains(token.Name))
token.Value = "***";
}
}
}
Since it doesn't work with JTokens, I've tried the same with JProperties and it works for the root object, but there's a problem: although you can see if a given JProperty is an object, you can not select its children object, JProperty.Children() gives JToken again and I found no way to convert it to a JProperty. If anyone knows how to achieve it, please post it.
So the only way I found is a very dirty one: using regular expressions. It's all but elegant - but it works.
// Make sure the JSON is well formatted
string formattedJson = JObject.Parse(message).ToString();
// Define the keys of the values to be masked
string[] maskedKeys = {"mask1", "mask2"};
// Loop through each key
foreach (var key in maskedKeys)
{
string original_pattern = string.Format("(\"{0}\": )(\"?[^,\\r\\n]+\"?)", key);
string masked_pattern = "$1\"censored\"";
Regex pattern = new Regex(original_pattern);
formatted_json = pattern.Replace(formatted_json, masked_pattern);
}
// Parse the masked string
var maskedMessage = JsonConvert.DeserializeObject<dynamic>(formatted_json);
Assuming this is your input:
{
"val1" : "value1",
"val2" : "value2",
"mask1" : "to be masked",
"prop1" : {
"val3" : "value3",
"val1" : "value1",
"mask2" : "to be masked too",
"prop2" : {
"val1" : "value 1 again",
"mask1" : "this will also get masked"
}
}
}
This is what you get:
{
"val1": "value1",
"val2": "value2",
"mask1": "censored",
"prop1": {
"val3": "value3",
"val1": "value1",
"mask2": "censored",
"prop2": {
"val1": "value 1 again",
"mask1": "censored"
}
}
}
Related
I have a dynamic object which basically holds an AvroRecord. AvroRecord class details here.
I can assign values to the properties statically but I was wondering if this could be done dynamically. I have looked at the forum question here ,here and also here. But none of these work for me.
This is the static code that works.
var serializer = AvroSerializer.CreateGeneric(Schema);
var rootSchema = serializer.WriterSchema as RecordSchema;
dynamic counterpartRow = new AvroRecord(rootSchema);
counterpartRow.CounterpartID = Row.CounterpartID
counterpartRow.CounterpartFirstDepositDate = Row.CounterpartFirstDepositDate
The Row is an object of the InputBuffer class of SSIS and it holds all the columns coming from the upstream data source.
The schema variable used above is an avro schema, which is something like this.
Schema = #"{
""type"":""record"",
""name"":""Microsoft.Hadoop.Avro.Specifications.Counterparts"",
""fields"":
[
{ ""name"":""CounterpartID"", ""type"":""int"" },
{ ""name"":""CounterpartFirstDepositDate"", ""type"":[""string"",""null""] },
{ ""name"":""CounterpartFirstTradeDate"",""type"":[""string"",""null""] },
{ ""name"":""ClientSegmentReportingID"",""type"":""int"" },
{ ""name"":""ClientSegmentReportingName"", ""type"":[""string"",""null""] },
{ ""name"":""ContractID"", ""type"":""int""},
{ ""name"":""ContractFirstDepositDate"", ""type"":[""string"",""null""]},
{ ""name"":""ContractFirstTradeDate"",""type"":[""string"",""null""] },
{ ""name"":""ContractClosingOffice"",""type"":[""string"",""null""] },
{ ""name"":""LeadCreationDate"", ""type"":[""string"",""null""] },
{ ""name"":""ContractCountryOfResidence"", ""type"":[""string"",""null""]}
]
}";
I have tried something like the earlier forum links suggested like
counterpartRow.GetType().GetField("CounterpartID").SetValue(Row, Row.CounterpartID, null);
and also the other method (which apparently should work for for dynamic type), but even that does not.
foreach (string propertyName in GetPropertyKeysForDynamic(counterpartRow.Schema.Fields()))
{
string propertyValue = counterpartRow[propertyName];
}
and the function defined like this.
public List<string> GetPropertyKeysForDynamic(dynamic dynamicToGetPropertiesFor)
{
var jObject = (JObject)JToken.FromObject(dynamicToGetPropertiesFor);
Dictionary<string, object> values = jObject.ToObject<Dictionary<string, object>>();
List<string> toReturn = new List<string>();
foreach (string key in values.Keys)
{
toReturn.Add(key);
}
return toReturn;
}
The dictionary above returns blank.
the Row mentioned above is an object of InputBuffer class (auto generated class in SSIS).
which is something like this.
public class Input0Buffer: ScriptBuffer
{
public Input0Buffer(PipelineBuffer Buffer, int[] BufferColumnIndexes, OutputNameMap OutputMap)
: base(Buffer, BufferColumnIndexes, OutputMap)
{
}
public Int32 CounterpartID
{
get
{
return Buffer.GetInt32(BufferColumnIndexes[0]);
}
}
------more properties
If you see my original static code, I am trying to dynamically generate the assignment. I have already dynamically generated the schema (instead of the static definition I have given above). So, the only piece left to generate dynamically is the assignment of the assignment. An idea could be that I generate the string but how do I then execute that string? That is only if there is no way to achieve this.
If I understand correct your question I think you can try something like this
string _schema = "your avro schema"
RecordSchema _record = (RecordSchema)Avro.Schema.Parse(_schema);
GenericRecord _generic_record = new GenericRecord(payload_record);
for (int ii = 0; ii < _record.Fields.Count; ii++)
{
_generic_record.Add(_record.Fields[ii].Name, Raw.TheFiledYouNeed);
}
apologies if I'm doing something wrong, this is my first post.
I'm currently working with C# and want to save a bunch of data out to a JSON file and load it back, but I'm having trouble figuring out how to get it in the following format.
// Primary ID
001
{
// Secondary ID
01
{
// Tertiary ID
01
{
string: "this is some information.",
int: 9371
}
}
// Secondary ID
02
{
// Tertiary ID
01
{
string: "blah blah blah.",
int: 2241
}
}
}
I'd essentially like to be able to call up information with a particular set of IDs for example 001-02-01 which would return a string ("blah blah blah.") and an int (2241).
The reason I want to go about it like this instead of just having one longer ID is so that when the JSON file becomes very large, I'm hoping to be able to speed up the search for information by passing each ID in turn.
If that makes no sense and it would be equally as fast to just pass in one longer ID and not be bothered by this whole nested ID segments concept then please let me know!
If, however what I'm thinking is correct and it would help the speed of finding particular data by structuring it out like this, how would I go about doing that? With nested C# classes in arrays?
The most simple way and efficient way would be to have all data as same type. Currently, you seem to go for each object is of type of the given id:
{
"01":{},
"02" :{}
}
this will not go too well if trying to use a serializable class.
I would recommend the following:
{
"items" : [
{"id":"01" }, { "id":"02" },...
]
}
Then you can serialize/deserialize easily with
[Serializable]
public class Item
{
public string id = null;
}
[Serializable]
public class RootObject
{
public List<Item> items = null;
}
and then in Unity:
void Start(){
string str = GetJson(); // However you get it
RootObject ro = JsonUtility.FromJson<RootObject>(str);
}
if you want to speed up the fetching and your collection is large, convert to dictionary.
Dictionary<string, Item> dict = null;
void Start(){
string str = GetJson(); // However you get it
RootObject ro = JsonUtility.FromJson<RootObject>(str);
this.dict = new Dictionary<string,Item>();
foreach(Item item in ro.items){
Item temp = temp;
this.dict.Add(item.Id, temp);
}
ro = null;
}
Now you can access real fast.
Item GetItem(string id)
{
if(string.IsNullOrEmpty(id) == true){ return null; }
Item item = null;
this.dict.TryGetValue(id, out item);
return item;
}
If you end up storing millions of records in your file and want to start doing something more performant it would be easier to switch to a decent document database like MongoDB rather than trying to reinvent the wheel.
Worry about writing good standard code before worrying about performance problems that don't yet exist.
The following example is not in your language of choice but it does explain that JSON and arrays of 1,000,000 objects can be searched very quickly:
const getIncidentId = () => {
let id = Math.random().toString(36).substr(2, 6).toUpperCase().replace("O", "0")
return `${id.slice(0, 3)}-${id.slice(3)}`
}
console.log("Building array of 1,000,000 objects")
const littleData = Array.from({ length: 1000000 }, (v, k) => k + 1).map(x => ({ cells: { Number: x, Id: getIncidentId() } }))
console.log("Getting list of random Ids for array members [49, 60, 70000, 700000, 999999]")
const randomIds = ([49, 60, 70000, 700000, 999999]).map(i => littleData[i].cells.Id)
console.log(randomIds)
console.log("Finding each array item that contains a nested Id property in the randomIds list.")
const foundItems = littleData.filter(i => randomIds.includes(i.cells.Id))
console.log(foundItems)
I'm currently working on an ASP.NET Web API .NET Framework 4.7.2. I try to alter some JSON data in my service class. I try to add a new object after every 2nd object in my JArray.
I thought about manipulating the JSON data, rather then concrete objects because the received data will most likely be dynamic data. I'm using the library JObject, but I'm getting some error without any real exception messages.
My received JSON structure looks like that:
{ "data" : [
{"IsPlaceholder": 0, "Name" : "Test1", "Size" : 2 },
{"IsPlaceholder": 0, "Name" : "Test2", "Size" : 3 },
{"IsPlaceholder": 0, "Name" : "Test3", "Size" : 1 }
]}
My service class looks like that:
public class MyService : IMyService
{
public async Task<JObject> UpdateInformationAsync(JObject coolData)
{
// Random placeholder, new placeholder object after 2nd
var placeholder = JObject.FromObject(new PlaceholderVm());
var cnt = 0;
foreach (JObject c in coolData["data"] as JArray)
{
if (cnt % 2 == 0)
{
coolData["data"][cnt].AddAfterSelf(placeholder);
}
cnt++;
}
return coolData;
}
}
My placeholder view model looks like that:
public class PlaceholderVm
{
public int IsPlaceholder => 1;
public string Name => "Placeholder";
public float Size { get; set; } = 0;
}
When I try to add a placeholderVm to my JArray, it works fine the first time, but on the 2nd iteration it throws an error without exception message.
Do you know how I can add a new JObject on nth position to my JArray?
This is because you are mutating the underlying collection while looping through it in the foreach. This is the reason you'll often times see folks initialize a new List<T> when doing operations like this, to avoid this error.
It actually yields this exception :
Run-time exception (line 21): Collection was modified; enumeration operation may not execute.
System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
The easiest way around it is to simply create a new collection and place things where you want them to go. In your case, this may look like :
var jObject = new JObject();
JArray jArray = new JArray();
foreach (JObject c in coolData["data"] as JArray)
{
jArray.Add(c);
if (cnt % 2 == 0)
{
jArray[jArray.Count - 1].AddAfterSelf(placeholder);
}
cnt++;
}
jObject.Add("data", jArray);
Here's a .NET Fiddle
I have the following JSON which has to be converted to URL parameters for a GET request.
An example is given here, however due to the complexity of this object, there can be multiple line_items_attributes each with the given values as shown, I'm having difficulties passing on the correct one.
I've also tried to just serialize the JSON object and pass on that value but that did not solve the issue either.
{
"purchase_invoice":
{
"date":"14/04/2015",
"due_date":"14/04/2015",
"contact_id":500,
"contact_name":"TestContact",
"reference":"TestReference",
"line_items_attributes":[
{
"unit_price":10.00,
"quantity":1,
"description":"TestLineItemAttDesc",
"tax_code_id":1,
"ledger_account_id":501,
"tax_rate_percentage":19.0,
"tax_amount":1.60
}]
}
}
I've been searching for a while now but without much luck. Any insights are appreciated and most welcome!
This is calling an API which does not support the incoming data in JSON format, so doing this server-side or changing the web service to support data in JSON format is not possible.
x-www-form-urlencoded content is, essentially, a flat sequence of key/value tuples, and as explained in this answer to How do I use FormUrlEncodedContent for complex data types? by Tomalak, there is no canonical way to transform a hierarchical, nested key/value structure into a flat one.
Nevertheless, from the accepted answer to this question, this example from the Stripe API, and the question mentioned above, it seems that it is common to flatten parameters inside complex nested objects by surrounding their keys in brackets and appending them to the topmost key like so:
{
{ "purchase_invoice[date]", "14/04/2015" }
{ "purchase_invoice[due_date]", "14/04/2015" }
{ "purchase_invoice[contact_id]", "500" }
{ "purchase_invoice[contact_name]", "TestContact" }
{ "purchase_invoice[reference]", "TestReference" }
{ "purchase_invoice[line_items_attributes][0][unit_price]", "10" }
{ "purchase_invoice[line_items_attributes][0][quantity]", "1" }
{ "purchase_invoice[line_items_attributes][0][description]", "TestLineItemAttDesc" }
{ "purchase_invoice[line_items_attributes][0][tax_code_id]", "1" }
{ "purchase_invoice[line_items_attributes][0][ledger_account_id]", "501" }
{ "purchase_invoice[line_items_attributes][0][tax_rate_percentage]", "19" }
{ "purchase_invoice[line_items_attributes][0][tax_amount]", "1.6" }
}
If this is what you want, you can generate such key/value pairs with json.net using the following extension methods:
public static partial class JsonExtensions
{
public static string ToUrlEncodedQueryString(this JContainer container)
{
return container.ToQueryStringKeyValuePairs().ToUrlEncodedQueryString();
}
public static IEnumerable<KeyValuePair<string, string>> ToQueryStringKeyValuePairs(this JContainer container)
{
return container.Descendants()
.OfType<JValue>()
.Select(v => new KeyValuePair<string, string>(v.ToQueryStringParameterName(), (string)v));
}
public static string ToUrlEncodedQueryString(this IEnumerable<KeyValuePair<string, string>> pairs)
{
return string.Join("&", pairs.Select(p => HttpUtility.UrlEncode(p.Key) + "=" + HttpUtility.UrlEncode(p.Value)));
//The following works but it seems heavy to construct and await a task just to built a string:
//return new System.Net.Http.FormUrlEncodedContent(pairs).ReadAsStringAsync().Result;
//The following works and eliminates allocation of one intermediate string per pair, but requires more code:
//return pairs.Aggregate(new StringBuilder(), (sb, p) => (sb.Length > 0 ? sb.Append("&") : sb).Append(HttpUtility.UrlEncode(p.Key)).Append("=").Append(HttpUtility.UrlEncode(p.Value))).ToString();
//Answers from https://stackoverflow.com/questions/3865975/namevaluecollection-to-url-query that use HttpUtility.ParseQueryString() are wrong because that class doesn't correctly escape the keys names.
}
public static string ToQueryStringParameterName(this JToken token)
{
// Loosely modeled on JToken.Path
// https://github.com/JamesNK/Newtonsoft.Json/blob/master/Src/Newtonsoft.Json/Linq/JToken.cs#L184
// By https://github.com/JamesNK
if (token == null || token.Parent == null)
return string.Empty;
var positions = new List<string>();
for (JToken previous = null, current = token; current != null; previous = current, current = current.Parent)
{
switch (current)
{
case JProperty property:
positions.Add(property.Name);
break;
case JArray array:
case JConstructor constructor:
if (previous != null)
positions.Add(((IList<JToken>)current).IndexOf(previous).ToString(CultureInfo.InvariantCulture)); // Don't localize the indices!
break;
}
}
var sb = new StringBuilder();
for (var i = positions.Count - 1; i >= 0; i--)
{
var name = positions[i];
// TODO: decide what should happen if the name contains the characters `[` or `]`.
if (sb.Length == 0)
sb.Append(name);
else
sb.Append('[').Append(name).Append(']');
}
return sb.ToString();
}
}
Then if you have a JSON string, you can parse it into a LINQ-to-JSON JObject and generate the query string like so:
var obj = JObject.Parse(jsonString);
var queryString = obj.ToUrlEncodedQueryString();
Alternatively, if you have some hierarchical data model POCO, you can generate your JObject from the model using JObject.FromObject():
var obj = JObject.FromObject(myModel);
var queryString = obj.ToUrlEncodedQueryString();
Demo fiddle here.
So the final URL would be easy to compute using any URL Encoding mechanism. In C#, we could do the following:
string json = "...";
string baseUrl = "http://bla.com/somepage?myJson="
string urlWithJson = baseUrl + System.Net.WebUtility.UrlEncode(json)
Is there any way you can POST the data or otherwise send a request body instead? It would seem slightly easier/cleaner.
Sounds like you need something which is x-www-form-urlencoded.
From your example, it would look like this:
purchase_invoice%5Bdate%5D=14%2F04%2F2015&purchase_invoice%5Bdue_date%5D=14%2F04%2F2015&purchase_invoice%5Bcontact_id%5D=500&purchase_invoice%5Bcontact_name%5D=TestContact&purchase_invoice%5Breference%5D=TestReference&purchase_invoice%5Bline_items_attributes%5D%5B0%5D%5Bunit_price%5D=10&purchase_invoice%5Bline_items_attributes%5D%5B0%5D%5Bquantity%5D=1&purchase_invoice%5Bline_items_attributes%5D%5B0%5D%5Bdescription%5D=TestLineItemAttDesc&purchase_invoice%5Bline_items_attributes%5D%5B0%5D%5Btax_code_id%5D=1&purchase_invoice%5Bline_items_attributes%5D%5B0%5D%5Bledger_account_id%5D=501&purchase_invoice%5Bline_items_attributes%5D%5B0%5D%5Btax_rate_percentage%5D=19&purchase_invoice%5Bline_items_attributes%5D%5B0%5D%5Btax_amount%5D=1.6
The best reference for this encoding that I'm aware of is the undocumented jQuery.param method on the jQuery JavaScript library.
I need my wcf app to return a complex type
==============================
case 1 is simply 2 fields. no need to waste anyones time w/ that
==============================
//case 2 - 1 dim array
if the return type looks like this
public class TestResult
{
public string Key = "myKey";
public string Message = "myMessage";
public string[] fields;
}
and the implementation (snippet) looks like this
r.fields = new string[] { "name", "varchar(32)", "bob" };
return r;
I get what I expect
{"GetStringArrayResult":{"Key":"myKey",
"Message":"myMessage",
"fields":["name","varchar(32)","bob"]}}
==============================
//case 3 - 2 dim array
if the return class looks like this
public class TestResult
{
public string Key = "myKey";
public string Message = "myMessage";
public string[][] fields;
}
and the implementation looks like this
string[] a1 = { "fname", "varchar(32)", "bob"};
string[] a2 = { "lname", "varchar(50)", "smithersonsonson" };
string[] a3 = { "age", "varchar(32)", "40" };
r.fields = new string [][]{a1,a2,a3};
return r;
then i get what i expect in the browser
{"GetStringArrayResult":{"Key":"myKey",
"Message":"myMessage",
"fields":[
["fname","varchar(32)","bob"],
["lname","varchar(50)","smithersonsonson"],
["age","varchar(32)","40"]
]
}
}
==============================
you will notice that I am not JSONifying anything. I simply return r and it gets
JSONP'ed up by wcf.
I learned about this the hard way as I tried to JSON it and then WCF double JSON'ed it which really makes the result ugly. user error.fine.
==============================
so the next step is to have the complex object so we dont have all these arrays hanging around.
//case 4 - A single embedded object
so if i define a simple class like this
public class fieldd
{
public string fieldName = "";
public string datatype = "";
public string value = "";
public fieldd(string _fn, string _dt, string _v)
{
fieldName = _fn;
datatype = _dt;
value = _v;
}
}
and if my result type looks like this
public class TestResult
{
public string Key = "myKey";
public string Message = "myMessage";
public fieldd field ;
}
and my implementation looks like this
r.field = new fieldd("name", "varchar(32)", "bob");
//return r;
google reports
ERR_CONNECTION_RESET
ie9 reports
•Internet connectivity has been lost.
•The website is temporarily unavailable.
What I expected was this
{"GetStringArrayResult":{"Key":"myKey",
"Message":"myMessage",
"field":{
"fieldName":"name".
"datatype":"varchar(32)",
"value":"bob"
}
}
}
================================
I feel I have pretty fairly demonstrated that wcf is having problems building JSON for the embedded object. It does fine w/ arrays.
any thoughts?
is there some reason a complex object cant be returned?
can i turn off wcf's jsonp implementation and do my own?
I wish the answer was more complicated...
After trying many things, I tried taking out the constructor. Success.
After trying many more things... objects have to have a constructor with a blank signature. You can have other constructors, just also have a blank one.
Failing to have a constructor w/ a blank signature will cause the jsonp logic to crash. This happens after the cursor has exited your code and is swimming in the WCF code somewhere.
i have returned to this problem many times over the course of a couple months.
an exception w/ a nice message would have helped, but it is the life...
Greg