We Keep Coding

Deserializing Json with unexpected characters in property

My application processes a Json that contains a list of objects. The properties of these objects are not completely known to me.
Example, in the following Json, only property "Id" is known to my application:
{"msgs": [
{
"Id": "Id1",
"A": "AAA"
},
{
"Id": "Id2",
"B": "BBB"
},
{
"Id": "Id3",
"C": "CCC"
}
]}
I want to parse these messages and extract the Id of each message. This has been working fine with the following code:
public class RootElem
{
[BsonElement("msgs")]
public List<JToken> Records { get; set; }
}
then read
var rootElem = JsonConvert.DeserializeObject<RootElem>(JSON_DATA);
Once I have my rootElem, I can iterate over each record in "Records" and extract the Id.
The problem is that sometime, some of the records will contain unexpected characters.
Example:
{
"Id": "Id2",
"B": "THIS CONTAINS UNEXPECTED DOUBLE QUOTE " WHAT SHOULD I DO?"
}
I tried adding error handling settings, but that didn't work:
var rootElem = JsonConvert.DeserializeObject<RootElem>(data, new JsonSerializerSettings
{
Error = HandleDeserializationError
});
private static void HandleDeserializationError(object sender, ErrorEventArgs errorArgs)
{
var currentError = errorArgs.ErrorContext.Error.Message;
Console.WriteLine(currentError);
errorArgs.ErrorContext.Handled = true;
}
I want to be able to access the rest of the records in the list, even if one/some of them contain these invalid chars. So I'm ok if "B" value of the 2nd record is return as null, or if the 2nd record is null altogether.
How can I achieve this?
p.s. I'm using Newtonsoft Json but I'm open to using other libraries.

The issue here is that what your application is receiving is not valid JSON, hence the errors you are seeing. The input should be properly escaped prior to being submitted to this method. If this is behind an HTTP API, the appropriate response would be a 400 as the request is not in a valid format. If you really need to work around this it's possible you could implement your own JsonConverter class and decorate the converting class with it, but this may not be possible as the JSON itself is invalid and the JsonReader is going to choke when it hits it.

C# MongoDB Driver: Can't find the way to run complex query for AnyIn filter in MongoDB

I have a document like this:
{
"id": "xxxxxxxxxxxx",
"groupsAuthorized": [
"USA/California/SF",
"France/IDF/Paris"
]
}
And I have an user that has a list of authorized groups, like for example the following:
"groups": [
"France/IDF",
"USA/NY/NYC"
]
What I'm trying to achieve is to retrieve all documents in the database that the user is authorized to retrieve, essentially I want to be able to check in the list "groupsAuthorized" if one of the group contains a subset of an element of the other list "groups" contained in my user authorizations
using the following values:
my document:
{
"id": "xxxxxxxxxxxx",
"groupsAuthorized": [
"USA/California/SF",
"France/IDF/Paris"
]
}
my user permissions:
"groups": [
"France/IDF",
"USA/NY/NYC"
]
the user should be able to retrieve this document as the string "France/IDF" is correctly contained in the string "France/IDF/Paris", however, if the values would've been like this:
my document:
{
"id": "xxxxxxxxxxxx",
"groupsAuthorized": [
"USA/California/SF",
"France/IDF"
]
}
my user permissions:
"groups": [
"France/IDF/Paris",
"USA/NY/NYC"
]
it should not work, because my user is only authorized to view documents from France/IDF/Paris and USA/NY/NYC and none of the string inside of the authorizedGroups of my document contains those sequences
I've tried to use a standard LINQ query to achieve this which is fairly simple:
var userAuthorizedGroups = new List<string> { "France/IDF/Paris", "USA/NY/NYC" };
var results = collection.AsQueryable()
.Where(entity => userAuthorizedGroups
.Any(userGroup => entity.authorizedGroups
.Any(entityAuthorizedGroup => entityAuthorizedGroup.Contains(userGroup))));
But i'm getting the famous unsupported filter exception that it seems lot of people is having, i've tried different options found on the internet like the following:
var userAuthorizedGroups = new List<string> { "France/IDF/Paris", "USA/NY/NYC" };
var filter = Builders<PartitionedEntity<Passport>>.Filter.AnyIn(i => i.authorizedGroups, userAuthorizedGroups);
var results = (await collection.FindAsync(filter)).ToList();
return results;
But the problem is this will only check if one of the element of the array is contained inside the other array, It will not correctly work for case like "France/IDF" that should correctly match "France/IDF/Paris" because "France/IDF" string is contained inside the "France/IDF/Paris" string inside of my document
I'm getting a bit clueless on how to achieve this using the mongodb C# driver, i'm starting to think that I should just pull all documents to client and do the filtering manually but that would be quite messy
Has anyone an Idea on this subject ?

i'm starting to think that I should just pull all documents to client and do the filtering manually but that would be quite messy
don't do it :)
One place you can start with is here. It describes all the LINQ operators that are supported by the MongoDB .NET driver. As you can see .Contains() isn't mentioned there which means you can't use it and you'll get an arror in the runtime but it does not mean that there's no way to do what you're trying to achieve.
The operator closest to contains you can use is $indexOfBytes which returns -1 if there's no match and the position of a substring otherwise. Also since you need to match an array against another array you need two pairs of $map and $anyElementTrue to do exactly what .NET's .Any does.
Your query (MongoDB client) can look like this:
db.collection.find({
$expr: {
$anyElementTrue: {
$map: {
input: "$groupsAuthorized",
as: "group",
in: {
$anyElementTrue: {
$map: {
input: ["France/IDF/Paris", "USA/NY/NYC"],
as: "userGroup",
in: { $ne: [ -1, { $indexOfBytes: [ "$$userGroup", "$$group" ] } ] }
}
}
}
}
}
}
})
Mongo Playground,
You can run the same query from .NET using BsonDocument class which takes a string (JSON) and converts into a query:
var query = BsonDocument.Parse(#"{
$expr: {
$anyElementTrue:
{
$map:
{
input: '$groupsAuthorized',
as: 'group',
in: {
$anyElementTrue:
{
$map:
{
input: ['France/IDF/Paris', 'USA/NY/NYC'],
as: 'userGroup',
in: { $ne: [-1, { $indexOfBytes: ['$$userGroup', '$$group'] } ] }
}
}
}
}
}
}
}");
var result = col.Find(query).ToList();

JSON.NET - getting particular properties inside objects from Jarray

There is a series of JSON objects called "issues", which each have one or more "issue links", which have the following format:
// an issue link
{
"id": "000000",
"self": "some link",
"type": {
"id": "0000",
"name": "some name",
"inward": "is met by",
"outward": "meets",
"self": "some link"
},
"outwardIssue": {
"id": "000000",
"key": "the required key",
"self": "some link",
"fields": {
// the remainder is not applicable
}
}
}
}
These "issue links" have been extracted as follows. Create a JArray for the JSON for the "issue" itself, and extract the child JObjects:
public void Deserialize(dynamic jsonObject)
{
// get the issue links
if (jsonObject["fields"]["issuelinks"]!=null)
{
JArray issueLinksArray = jsonObject["fields"]["issuelinks"];
var issueLinkObjects = issueLinksArray.Children();
foreach (var issueLink in issueLinkObjects)
{
// now need the "key" in the "outwardIssue" for this object, if the value of "inward" is "is met by".
}
}
}
How to go about extracting the value of the second property "key" of "outwardIssue"?

Not sure whether I fully understand but following excerpt gets u the value (or null if condition not met) like this.
var key = issueLink["type"]["inward"].ToString()=="is met by" ? issueLink["outwardIssue"]["key"]: null;
Hint: Try to avoid dynamic.
Nowadays loops can in certain conditions considered old-school. Think LINQ: The problem can be divided into smaller probs and distributed across multiple lines (think-steps).
The additional variables might improve readability. As the project grows, for loops are prone to span more and more lines. So if you just need requested values following might be of interest:
var inwardLinks = issueLinkObject.Where(i=>i["type"]["inward"].ToString()=="is met by");
var keys = inwardLinks.Select(iwl=>iwl["outwardIssue"]["key"]);

Multiple Filters and Updates in One Mongo Query [duplicate]

I have a collection which elements can be simplified to this:
{tags : [1, 5, 8]}
where there would be at least one element in array and all of them should be different. I want to substitute one tag for another and I thought that there would not be a problem. So I came up with the following query:
db.colll.update({
tags : 1
},{
$pull: { tags: 1 },
$addToSet: { tags: 2 }
}, {
multi: true
})
Cool, so it will find all elements which has a tag that I do not need (1), remove it and add another (2) if it is not there already. The problem is that I get an error:
"Cannot update 'tags' and 'tags' at the same time"
Which basically means that I can not do pull and addtoset at the same time. Is there any other way I can do this?
Of course I can memorize all the IDs of the elements and then remove tag and add in separate queries, but this does not sound nice.

The error is pretty much what it means as you cannot act on two things of the same "path" in the same update operation. The two operators you are using do not process sequentially as you might think they do.
You can do this with as "sequential" as you can possibly get with the "bulk" operations API or other form of "bulk" update though. Within reason of course, and also in reverse:
var bulk = db.coll.initializeOrderedBulkOp();
bulk.find({ "tags": 1 }).updateOne({ "$addToSet": { "tags": 2 } });
bulk.find({ "tags": 1 }).updateOne({ "$pull": { "tags": 1 } });
bulk.execute();
Not a guarantee that nothing else will try to modify,but it is as close as you will currently get.
Also see the raw "update" command with multiple documents.

If you're removing and adding at the same time, you may be modeling a 'map', instead of a 'set'. If so, an object may be less work than an array.
Instead of data as an array:
{ _id: 'myobjectwithdata',
data: [{ id: 'data1', important: 'stuff'},
{ id: 'data2', important: 'more'}]
}
Use data as an object:
{ _id: 'myobjectwithdata',
data: { data1: { important: 'stuff'},
data2: { important: 'more'} }
}
The one-command update is then:
db.coll.update(
'myobjectwithdata',
{ $set: { 'data.data1': { important: 'treasure' } }
);
Hard brain working for this answer done here and here.

Starting in Mongo 4.4, the $function aggregation operator allows applying a custom javascript function to implement behaviour not supported by the MongoDB Query Language.
And coupled with improvements made to db.collection.update() in Mongo 4.2 that can accept an aggregation pipeline, allowing the update of a field based on its own value,
We can manipulate and update an array in ways the language doesn't easily permit:
// { "tags" : [ 1, 5, 8 ] }
db.collection.updateMany(
{ tags: 1 },
[{ $set:
{ "tags":
{ $function: {
body: function(tags) { tags.push(2); return tags.filter(x => x != 1); },
args: ["$tags"],
lang: "js"
}}
}
}]
)
// { "tags" : [ 5, 8, 2 ] }
$function takes 3 parameters:
body, which is the function to apply, whose parameter is the array to modify. The function here simply consists in pushing 2 to the array and filtering out 1.
args, which contains the fields from the record that the body function takes as parameter. In our case, "$tag".
lang, which is the language in which the body function is written. Only js is currently available.

In case you need replace one value in an array to another check this answer:
Replace array value using arrayFilters

Comparing JSON response of

Good morning everyone. I have a service which returns me a JSON response (something like below):
{
"sessionid": "AQIC5wM2LY4SfcytTIcteNkTtCVrE8A-AS7VR*",
"Customers": [
{
"id": "4193942846",
"firstname": "Anto",
"lastname": "Paul",
"customertype": "ph",
"companyCode": "ABCD",
},
{
"id": "4193942236",
"firstname": "Dimple",
"lastname": "Paul",
"customertype": "ph",
"companyCode": "AB",
}
],
"Status": "ACTIVE",
"serviceStatus": "SUCCESS",
"Addresses": {
"Address": [
{
"type": "M",
"addr1": "11011, main st",
"addr2": "Apt. 2",
"zipcode": "11011"
}
]
}
}
The above structure varies based on the input I pass to the service. So, I cant contruct one class to deserialize the response. I need to compare (attribute-attribute comparison) this response to a response I already have with me (in a different place).
I tried to do it with dynamic class in C# but no luck so far. Could someone share a better,working approach? Thank you.

You can use JToken.DeepEquals like this:
var response = JObject.Parse(responseJson);
var goldenStandard = JObject.Parse(goldenStandardJson);
if (JToken.DeepEquals(response, goldenStandard))
{
// the two JSONs have the same data
}

Try Json.Net. It supports dynamic structures.
Here is a tutorial:
http://www.codeproject.com/Tips/631586/Dynamic-types-with-JSON-NET

Use this:
https://jsonutil.codeplex.com/
var obj1 = JSONSerializer.Deserialize(jsontext1);
var obj2 = JSONSerializer.Deserialize(jsontext2);
bool Compare(object obj1, object obj2)
{
//if(obj1 is JSONObject && obj2 is JSONObject)
// => typecase and use jsonObj1.Members to iterate over members and compare values recursively
//if JSONArray, then iterate over items and compare
//if anything else... i.e. primitive then compare directly
//else return false;
}

If you are using this JSON response one time, then you can use a dynamic JSON NET types.
But if you are using this JSON structure response some times, then it is preferable to make this response to a c# classes (objects), so you will have a very easy access to it's properties, you just have your c# objects and they have properties and you can simply approach to each field you wish (with intellisense). By the way, it is very easy to copy a JSON response to a C# classes (something like copy paste). Here is how to do that:
http://blogs.msdn.com/b/webdev/archive/2012/12/18/paste-json-as-classes-in-asp-net-and-web-tools-2012-2-rc.aspx