I'm currently working on an ASP.NET Web API .NET Framework 4.7.2. I try to alter some JSON data in my service class. I try to add a new object after every 2nd object in my JArray.
I thought about manipulating the JSON data, rather then concrete objects because the received data will most likely be dynamic data. I'm using the library JObject, but I'm getting some error without any real exception messages.
My received JSON structure looks like that:
{ "data" : [
{"IsPlaceholder": 0, "Name" : "Test1", "Size" : 2 },
{"IsPlaceholder": 0, "Name" : "Test2", "Size" : 3 },
{"IsPlaceholder": 0, "Name" : "Test3", "Size" : 1 }
]}
My service class looks like that:
public class MyService : IMyService
{
public async Task<JObject> UpdateInformationAsync(JObject coolData)
{
// Random placeholder, new placeholder object after 2nd
var placeholder = JObject.FromObject(new PlaceholderVm());
var cnt = 0;
foreach (JObject c in coolData["data"] as JArray)
{
if (cnt % 2 == 0)
{
coolData["data"][cnt].AddAfterSelf(placeholder);
}
cnt++;
}
return coolData;
}
}
My placeholder view model looks like that:
public class PlaceholderVm
{
public int IsPlaceholder => 1;
public string Name => "Placeholder";
public float Size { get; set; } = 0;
}
When I try to add a placeholderVm to my JArray, it works fine the first time, but on the 2nd iteration it throws an error without exception message.
Do you know how I can add a new JObject on nth position to my JArray?
This is because you are mutating the underlying collection while looping through it in the foreach. This is the reason you'll often times see folks initialize a new List<T> when doing operations like this, to avoid this error.
It actually yields this exception :
Run-time exception (line 21): Collection was modified; enumeration operation may not execute.
System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
The easiest way around it is to simply create a new collection and place things where you want them to go. In your case, this may look like :
var jObject = new JObject();
JArray jArray = new JArray();
foreach (JObject c in coolData["data"] as JArray)
{
jArray.Add(c);
if (cnt % 2 == 0)
{
jArray[jArray.Count - 1].AddAfterSelf(placeholder);
}
cnt++;
}
jObject.Add("data", jArray);
Here's a .NET Fiddle
Related
apologies if I'm doing something wrong, this is my first post.
I'm currently working with C# and want to save a bunch of data out to a JSON file and load it back, but I'm having trouble figuring out how to get it in the following format.
// Primary ID
001
{
// Secondary ID
01
{
// Tertiary ID
01
{
string: "this is some information.",
int: 9371
}
}
// Secondary ID
02
{
// Tertiary ID
01
{
string: "blah blah blah.",
int: 2241
}
}
}
I'd essentially like to be able to call up information with a particular set of IDs for example 001-02-01 which would return a string ("blah blah blah.") and an int (2241).
The reason I want to go about it like this instead of just having one longer ID is so that when the JSON file becomes very large, I'm hoping to be able to speed up the search for information by passing each ID in turn.
If that makes no sense and it would be equally as fast to just pass in one longer ID and not be bothered by this whole nested ID segments concept then please let me know!
If, however what I'm thinking is correct and it would help the speed of finding particular data by structuring it out like this, how would I go about doing that? With nested C# classes in arrays?
The most simple way and efficient way would be to have all data as same type. Currently, you seem to go for each object is of type of the given id:
{
"01":{},
"02" :{}
}
this will not go too well if trying to use a serializable class.
I would recommend the following:
{
"items" : [
{"id":"01" }, { "id":"02" },...
]
}
Then you can serialize/deserialize easily with
[Serializable]
public class Item
{
public string id = null;
}
[Serializable]
public class RootObject
{
public List<Item> items = null;
}
and then in Unity:
void Start(){
string str = GetJson(); // However you get it
RootObject ro = JsonUtility.FromJson<RootObject>(str);
}
if you want to speed up the fetching and your collection is large, convert to dictionary.
Dictionary<string, Item> dict = null;
void Start(){
string str = GetJson(); // However you get it
RootObject ro = JsonUtility.FromJson<RootObject>(str);
this.dict = new Dictionary<string,Item>();
foreach(Item item in ro.items){
Item temp = temp;
this.dict.Add(item.Id, temp);
}
ro = null;
}
Now you can access real fast.
Item GetItem(string id)
{
if(string.IsNullOrEmpty(id) == true){ return null; }
Item item = null;
this.dict.TryGetValue(id, out item);
return item;
}
If you end up storing millions of records in your file and want to start doing something more performant it would be easier to switch to a decent document database like MongoDB rather than trying to reinvent the wheel.
Worry about writing good standard code before worrying about performance problems that don't yet exist.
The following example is not in your language of choice but it does explain that JSON and arrays of 1,000,000 objects can be searched very quickly:
const getIncidentId = () => {
let id = Math.random().toString(36).substr(2, 6).toUpperCase().replace("O", "0")
return `${id.slice(0, 3)}-${id.slice(3)}`
}
console.log("Building array of 1,000,000 objects")
const littleData = Array.from({ length: 1000000 }, (v, k) => k + 1).map(x => ({ cells: { Number: x, Id: getIncidentId() } }))
console.log("Getting list of random Ids for array members [49, 60, 70000, 700000, 999999]")
const randomIds = ([49, 60, 70000, 700000, 999999]).map(i => littleData[i].cells.Id)
console.log(randomIds)
console.log("Finding each array item that contains a nested Id property in the randomIds list.")
const foundItems = littleData.filter(i => randomIds.includes(i.cells.Id))
console.log(foundItems)
I'm receiving a JSON files that contains a list of files each with a number representing the total number of lines it contains. The file names can change over time. Does JSON.NET provide a mechanism to handle this scenario?
In the sample JSON below, I am specifically interested in items under noOfProductsByFile. The list of files is variable and I would need to consume them through some type of dynamic object.
{
"noOfProductsByFileType" : {
"rdi_product_stuff" : 41228,
"rdi_product_junk" : 62519,
"rdi_product_otherStuff" : 165023,
"rdi_product_bobsJunk" : 1289
},
"startTime" : "20160907050000",
"endTime" : "20160907052713",
"timeTaken" : "27 minutes and 13 seconds",
"noOfProductsBySecurityType" : {
"AGENCIES" : 41228,
"ASTBK" : 50991,
"TSYCURV" : 78
},
"noOfProductsByFile" : {
"rdi_product_stuff_1_of_1.json" : 41228,
"rdi_product_junk_1_of_2.json" : 60219,
"rdi_product_junk_2_of_2.json" : 2300,
"rdi_product_myStuff_1_of_2.json" : 147690,
"rdi_product_myStuff_2_of_2.json" : 17333,
"rdi_product_test_1_of_1.json" : 1289
},
"noOfProducts" : 1925914
}
There are a few options to use. Here's one using the ExpandObject
dynamic dynData = JsonConvert.DeserializeObject<ExpandoObject>(jsonString, new ExpandoObjectConverter());
You can then reference the fields as you would a normal object:
dynData.noOfProductsByFileType.rdi_product_stuff
If you need to handle fields that may or may not be present, ExpandoObjects can be cast to a IDictionary and refer to fields as follows:
((IDictionary<string,object>)dynData)["noOfProductsByFileType"]
Assign to a IDictionary type and iterate through properties or test if a property exists
var dictData = (IDictionary<string,object>)dynData;
var noOfProductsByFileTypeDict = (IDictionary<string,object>)dynData.noOfProductsByFileType; // objects within objects
Test For Property
dictData.containsKey("testprop"); // test for a property - testprop
or Iterate
foreach (KeyValuePair<string, int> pair in noOfProductsByFileTypeDict) ... //iterate
Yes. There are a number of ways to handle this. One simple way is to use a JObject.
var jsonString = "...";
var jObject = JObject.Parse(jsonString);
foreach (JProperty e in jObject["noOfProductsByFile"])
{
var file = e.Name;
var noOfProducts = e.Value;
Debug.WriteLine(file);
Debug.WriteLine(noOfProducts);
}
I have a User class that accumulates lots of DataTime entries in some List<DateTime> Entries field.
Occasionally, I need to get last 12 Entries (or less, if not reached to 12). It can get to very large numbers.
I can add new Entry object to dedicated collection, but then I have to add ObjectId User field to refer the related user.
It seems like a big overhead, for each entry that holds only a DateTime, to add another field of ObjectId. It may double the collection size.
As I occasionally need to quickly get only last 12 entries of 100,000 for instance, I cannot place these entries in a per-user collection like:
class PerUserEntries {
public ObjectId TheUser;
public List<DateTime> Entries;
}
Because it's not possible to fetch only N entries from an embedded array in a mongo query, AFAIK (if I'm wrong, it would be very gladdening!).
So am I doomed to double my collection size or is there a way around it?
Update, according to #profesor79's answer:
If your answer works, that will be perfect! but unfortunately it fails...
Since I needed to filter on the user entity as well, here is what I did:
With this data:
class EndUserRecordEx {
public ObjectId Id { get; set; }
public string UserName;
public List<EncounterData> Encounters
}
I am trying this:
var query = EuBatch.Find(u => u.UserName == endUser.UserName)
.Project<BsonDocument>(
Builders<EndUserRecordEx>.Projection.Slice(
u => u.Encounters, 0, 12));
var queryString = query.ToString();
var requests = await query.ToListAsync(); // MongoCommandException
This is the query I get in queryString:
find({ "UserName" : "qXyF2uxkcESCTk0zD93Sc+U5fdvUMPow" }, { "Encounters" : { "$slice" : [0, 15] } })
Here is the error (the MongoCommandException.Result):
{
{
"_t" : "OKMongoResponse",
"ok" : 0,
"code" : 9,
"errmsg" : "Syntax error, incorrect syntax near '17'.",
"$err" : "Syntax error, incorrect syntax near '17'."
}
}
Update: problem identified...
Recently, Microsoft announced their DocumentDB protocol support for MongoDB. Apparently, it doesn't support yet all projection operators. I tried it with mLab.com, and it works.
You can use PerUserEntries as this is a valuable document structure.
To get part of that array we need to add projection to query, so we can get only x elements and this is done server side.
Please see snippet below:
static void Main(string[] args)
{
// To directly connect to a single MongoDB server
// or use a connection string
var client = new MongoClient("mongodb://localhost:27017");
var database = client.GetDatabase("test");
var collection = database.GetCollection<PerUserEntries>("tar");
var newData = new PerUserEntries();
newData.Entries = new List<DateTime>();
for (var i = 0; i < 1000; i++)
{
newData.Entries.Add(DateTime.Now.AddSeconds(i));
}
collection.InsertOne(newData);
var list =
collection.Find(new BsonDocument())
.Project<BsonDocument>
(Builders<PerUserEntries>.Projection.Slice(x => x.Entries, 0, 3))
.ToList();
Console.ReadLine();
}
public class PerUserEntries
{
public List<DateTime> Entries;
public ObjectId TheUser;
public ObjectId Id { get; set; }
}
At the moment I'm adding functionality to our service that will take in an object that is about to be logged to trace and mask any sensitive fields that are included in the object.
The issue is that we can get objects with different layers. The code I have written so far only handles a parent field and a single child field and uses a nasty embedded for loop implementation to do it.
In the event that we have a third embedded layer of fields in an object we want to log, this wouldn't be able to handle it at all. There has to be a more efficient way of handling generic parsing of a dynamic object, but so far it's managed to avoid me.
The actual code that deserializes and then masks field sin the object looks like this:
private string MaskSensitiveData(string message)
{
var maskedMessage = JsonConvert.DeserializeObject<dynamic>(message);
LoggingProperties.GetSensitiveFields();
for (int i = 0; i < LoggingProperties.Fields.Count(); i++)
{
for (int j = 0; j < LoggingProperties.SubFields.Count(); j++)
{
if (maskedMessage[LoggingProperties.Fields[i]] != null)
{
if (maskedMessage[LoggingProperties.Fields[i]][LoggingProperties.SubFields[j]] != null)
{
maskedMessage[LoggingProperties.Fields[i]][LoggingProperties.SubFields[j]] = MaskField(LoggingProperties.SubFieldLengths[j]);
}
}
}
}
return maskedMessage.ToString(Formatting.None);
}
And it works off of a LoggingProperties class that looks like this:
public static class LoggingProperties
{
// Constants indicating the number of fields we need to mask at present
private const int ParentFieldCount = 2;
private const int SubFieldCount = 4;
// Constant representing the character we are using for masking
public const char MaskCharacter = '*';
// Parent fields array
public static string[] Fields = new string[ParentFieldCount];
// Subfields array
public static string[] SubFields = new string[SubFieldCount];
// Array of field lengths, each index matching the subfield array elements
public static int[] SubFieldLengths = new int[SubFieldCount];
public static void GetSensitiveFields()
{
// Sensitive parent fields
Fields[0] = "Parent1";
Fields[1] = "Parent2";
// Sensitive subfields
SubFields[0] = "Child1";
SubFields[1] = "Child2";
SubFields[2] = "Child3";
SubFields[3] = "Child4";
// Lengths of sensitive subfields
SubFieldLengths[0] = 16;
SubFieldLengths[1] = 16;
SubFieldLengths[2] = 20;
SubFieldLengths[3] = 3;
}
}
}
The aim was to have a specific list of fields for the masking method to look out for that could be expanded or contracted along with our systems needs.
The nested loop method though just seems a bit roundabout to me. Any help is appreciated.
Thanks!
UPDATE:
Here's a small example of a parent and child record that would be in the message prior to the deserialize call. For this example say I'm attempting to mask the currency ID (So in properties the fields could be set like this: Parent1 = "Amounts" and Child1 = "CurrencyId"):
{
"Amounts":
{
"Amount":20.0,
"CurrencyId":826
}
}
An example of a problem would then be if the Amount was divided into pounds and pence:
{
"Amounts":
{
"Amount":
{
"Pounds":20,
"Pence":0
},
"CurrencyId":826
}
}
This would another layer and yet another embedded for loop...but with that I would be making it overly complex and difficult if the next record in a message had only two layers.
Hope this clarifies a few things =]
Okay, I've really tried but I couldn't figure out an elegant way. Here's what I did:
The first try was using reflection but since all the objects are of type JObject / JToken, I found no way of deciding whether a property is an object or a value.
The second try was (and still is, if you can figure out a good way) more promising: parsing the JSON string into a JObject with var data = JObject.Parse(message) and enumerating its properties in a recursive method like this:
void Mask(data)
{
foreach (JToken token in data)
{
if (token.Type == JTokenType.Object)
{
// It's an object, mask its children
Mask(token.Children());
}
else
{
// Somehow mask it but I couldn't figure out to do it with JToken
// Pseudocode, it doesn't actually work:
if (keysToMask.Contains(token.Name))
token.Value = "***";
}
}
}
Since it doesn't work with JTokens, I've tried the same with JProperties and it works for the root object, but there's a problem: although you can see if a given JProperty is an object, you can not select its children object, JProperty.Children() gives JToken again and I found no way to convert it to a JProperty. If anyone knows how to achieve it, please post it.
So the only way I found is a very dirty one: using regular expressions. It's all but elegant - but it works.
// Make sure the JSON is well formatted
string formattedJson = JObject.Parse(message).ToString();
// Define the keys of the values to be masked
string[] maskedKeys = {"mask1", "mask2"};
// Loop through each key
foreach (var key in maskedKeys)
{
string original_pattern = string.Format("(\"{0}\": )(\"?[^,\\r\\n]+\"?)", key);
string masked_pattern = "$1\"censored\"";
Regex pattern = new Regex(original_pattern);
formatted_json = pattern.Replace(formatted_json, masked_pattern);
}
// Parse the masked string
var maskedMessage = JsonConvert.DeserializeObject<dynamic>(formatted_json);
Assuming this is your input:
{
"val1" : "value1",
"val2" : "value2",
"mask1" : "to be masked",
"prop1" : {
"val3" : "value3",
"val1" : "value1",
"mask2" : "to be masked too",
"prop2" : {
"val1" : "value 1 again",
"mask1" : "this will also get masked"
}
}
}
This is what you get:
{
"val1": "value1",
"val2": "value2",
"mask1": "censored",
"prop1": {
"val3": "value3",
"val1": "value1",
"mask2": "censored",
"prop2": {
"val1": "value 1 again",
"mask1": "censored"
}
}
}
How can I get names of all the keys in a MongoDB collection using c#.
I am using mongocsharpdriver.
I am able to get all the records using
var collection1 = database1.GetCollection("Games").FindAll();
Now I need the key names to display/use it. I need key names of collection that I have fetched.
e.g. If I have collection which
{ "_id" : ObjectId("c3"), "GameID" : 20, "GameName" : "COD5", "Cost" : 100}
{ "_id" : ObjectId("c4"), "GameID" : 21, "GameName" : "NFS", "Publisher" : "EA"}
{ "_id" : ObjectId("c5"), "GameID" : 22, "GameName" : "CS", "Cost" : 200}
So I should get list of keys like GameID, GameName, Cost, Publisher.
I also went through MongoDB Get names of all keys in collection but was not able to implement it, didnot understood it & got problem with mapreduce.
Inspired from the link in your question:
string map = #"function() {
for (var key in this) { emit(key, null); }
}";
string reduce = #"function(key, stuff) { return null; }";
string finalize = #"function(key, value){
return key;
}";
MapReduceArgs args = new MapReduceArgs();
args.FinalizeFunction = new BsonJavaScript(finalize);
args.MapFunction = new BsonJavaScript(map);
args.ReduceFunction = new BsonJavaScript(reduce);
var results = collection1.MapReduce(args);
foreach (BsonValue result in results.GetResults().Select(item => item["_id"]))
{
Console.WriteLine(result.AsString);
}
A very inefficient, but simple way to do this is
HashSet<string> keys = new HashSet<string>();
foreach (var rover in collection1.FindAll())
{
rover.Names.ToList().ForEach(p => keys.Add(p));
}
Keep in mind that finding the set of keys, no matter how it's implemented, will always have to iterate the entire collection, so it will be terribly slow on larger collections.
It makes sense to use Map/Reduce for this problem on larger collections, because that avoids all the data transfer and deserialization overhead that is incurred by the solution I posted above, but you should generally try to avoid doing something like this at all. At least, don't do it where it's needed synchronously.
If you somehow need to know the set of all fields quickly, you're better off keeping track of the fields during writes and store the list of fields in a separate collection somewhere.