We Keep Coding

How query specific item from json object which has dictionary?

How to query the below json object collection using SQL/LINQ, to get 'ordername' which has item(s) ONLY from particular 'factory' i.e., factory1?
Note: json object collections are nothing but cosmos db collection
I know below json structure can be improved, by replacing dictionary with array but that change not feasible at the moment.
{
id:123,
ordername:order1,
itemdictionary: {
item1: {
id: "item1",
name: "milk",
manufacture:
{
name:factory1,
location:location1
}
},
item2: {
id: "item2",
name: "curd",
manufacture:
{
name:factory2,
location:location2
}
}
}
orderamt:"5$"
}
{
id:1234,
ordername:order2,
itemdictionary: {
item1: {
id: "item1";
name: "honey",
manufacture:
{
name:factory3,
location:location3
}
},
item2: {
id: "item2",
name: "milk",
manufacture:
{
name:factory1,
location:location1
}
}
}
orderamt:"7$"
}
c# representation:
public class OrderModel
{
public string OrderAmt{ get; set; }
public string Id{ get; set; }
public Dictionary<string,ItemDescription> ItemDictionary{ get; set; }
public class ItemDescription
{
public string Id { get; set; }
public string Name{ get; set; }
public Manufacture Manufacture{ get; set; }
}
}
public class Manufacture
{
public string Location{ get; set; }
public string Name{ get; set; }
}
Query tried, returning all records without applying filter but need to apply filter like (manufacture.name == "factory1") :
var query = dbClient.CreateDocumentQuery<OrderModel>(collectionUri,
feedOptions).Where(w => w.ItemDictionary.Values.Where(i =>
i.manufacture.name == "factory1") != null).AsDocumentQuery();

Firstly, like you mentioned yourself, this dictionary is a rather inefficient model for your needs. You really should plan to migrate to a better model at some point, it may be easier (and cheaper) than constantly hassling with issues like you are having now.
I don't think you can do it with a clean indexed SQL query alone. What you could try:
Append factory summary to model
If you do have some tools set up to upgrade models, then you could normalize the used factories to /factories[] or similar. This extra field could be indexed and queried trivially and efficiently.
Note that adding a field to cosmosDB will not break your existing applications. It's cheap and simple as long as you have a way of enforcing this new array will be kept in sync on insert/update (ex via some pipe on data layer) + upgrading any old documents via your schema upgrade tools.
Prefilter in server, final filter in client
If you really-really can't touch the model at all then best you can do, is work with partial filter on cosmosDB side and apply final filter in client. Depending on data, this can be costly and messy.
For example:
Set up index for first item:
/itemdictionary/item1/manufacture/name
SQL query:
SELECT *
FROM c
where (c.itemdictionary.item1.manufacture.name = #factory)
and ( not is_defined(c.itemdictionary.item2.manufacture.name) or c.itemdictionary.item2.manufacture.name = #factory)
and ( not is_defined(c.itemdictionary.item3.manufacture.name) or c.itemdictionary.item3.manufacture.name = #factory)
and ( not is_defined(c.itemdictionary.item4.manufacture.name) or c.itemdictionary.item4.manufacture.name = #factory)
In client after SQL query has returned the results, eliminate results with other factories. That is execute .AsDocumentQuery() before applying the predicate .Where(w => w.ItemDictionary.All(i => i.manufacture.name == "factory1")) on fetched results.
This is inefficient if orders got many items on average and the chosen N items do not limit the results sufficiently. Note that you don't have to choose prefilter indexes sequentially: depending on data it may make more sense to precheck [1,2, 10, 20] or similar. You can also play with N to get the best out of prefilter for the average case.
Filter in client with UDF
I usually hate UDFs due to their development cost and maintenance pains, but it is an option to script the filter with JS (which can do anything) and do all the filtering on server side.
WIth UDF you do save on bandwidth, but do note though:
UDF is mixing your querying logic to data layer. All clients of this data will share a single UDF version on server.
You still need to prefilter with an indexable property to avoid a costly full-scan.
UDF queries are RU-costly. Full-scan + UDF is usually a no-go.

C# getting MongoDB BsonDocument to return as JSON

I asked a question a couple of days ago to collect data from MongoDB as a tree.
MongoDB create an array within an array
I am a newbie to MongoDB, but have used JSON quite substantially. I thought using a MongoDB to store my JSON would be a great benefit, but I am just experiencing immense frustration.
I am using .NET 4.5.2
I have tried a number of ways to return the output from my aggregate query to my page.
public JsonResult GetFolders()
{
IMongoCollection<BsonDocument> collection = database.GetCollection<BsonDocument>("DataStore");
PipelineDefinition<BsonDocument, BsonDocument> treeDocs = new BsonDocument[]
{
// my query, which is a series of new BsonDocument
}
var documentGroup = collection.Aggregate(treeDocs).ToList();
// Here, I have tried to add it to a JsonResult Data,
// as both documentGroup alone and documentGroup.ToJson()
// Also, loop through and add it to a List and return as a JsonResult
// Also, attempted to serialise, and even change the JsonWriterSettings.
}
When I look in the Immediate Window at documentGroup, it looks exactly like Json, but when I send to browser, it is an escaped string, with \" surrounding all my keys and values.
I have attempted to create a model...
public class FolderTree
{
public string id { get; set; }
public string text { get; set; }
public List<FolderTree> children { get; set; }
}
then loop through the documentGroup
foreach(var docItem in documentGroup)
{
myDocs.Add(BsonSerializer.Deserialize<FolderTree>(docItem));
}
but Bson complains that it cannot convert int to string. (I have to have text and id as a string, as some of the items are strings)
How do I get my MongoDB data output as Json, and delivered to my browser as Json?
Thanks for your assistance.
========= EDIT ===========
I have attempted to follow this answer as suggested by Yong Shun below, https://stackoverflow.com/a/43220477/4541217 but this failed.
I had issues, that the "id" was not all the way through the tree, so I changed the folder tree to be...
public class FolderTree
{
//[BsonSerializer(typeof(FolderTreeObjectTypeSerializer))]
//public string id { get; set; }
[BsonSerializer(typeof(FolderTreeObjectTypeSerializer))]
public string text { get; set; }
public List<FolderTreeChildren> children { get; set; }
}
public class FolderTreeChildren
{
[BsonSerializer(typeof(FolderTreeObjectTypeSerializer))]
public string text { get; set; }
public List<FolderTreeChildren> children { get; set; }
}
Now, when I look at documentGroup, I see...
[0]: {Plugins.Models.FolderTree}
[1]: {Plugins.Models.FolderTree}
To be fair to sbc in the comments, I have made so many changes to get this to work, that I can't remember the code I had that generated it.
Because I could not send direct, my json result was handled as...
JsonResult json = new JsonResult();
json.Data = documentGroup;
//json.Data = JsonConvert.SerializeObject(documentGroup);
json.JsonRequestBehavior = JsonRequestBehavior.AllowGet;
return json;
Note, that I also tried to send it as...
json.Data = documentGroup.ToJson();
json.Data = documentGroup.ToList();
json.Data = documentGroup.ToString();
all with varying failures.
If I leave as documentGroup, I get {Current: null, WasFirstBatchEmpty: false, PostBatchResumeToken: null}
If I do .ToJson(), I get "{ \"_t\" : \"AsyncCursor`1\" }"
If I do .ToList(), I get what looks like Json in json.Data, but get an error of Unable to cast object of type 'MongoDB.Bson.BsonInt32' to type 'MongoDB.Bson.BsonBoolean'.
If I do .ToString(), I get "MongoDB.Driver.Core.Operations.AsyncCursor`1[MongoDB.Bson.BsonDocument]"
=========== EDIT 2 =================
As this way of extracting the data from MongoDB doesn't want to work, how else can I make it work?
I am using C# MVC4. (.NET 4.5.2)
I need to deliver json to the browser, hence why I am using a JsonResult return type.
I need to use an aggregate to collect from MongoDB in the format I need it.
My Newtonsoft.Json version is 11.0.2
My MongoDB.Driver is version 2.11.1
My method is the simplest it can be.
What am I missing?

How to save/pass MongoDB UpdateDefinition for logging and later use?

I am stumped on how to save/pass MongoDB UpdateDefinition for logging and later use
I have created general functions for MongoDB in Azure use on a collection for get, insert, delete, update that work well.
The purpose is to be able to have a standard, pre-configured way to interact with the collection. For update especially, the goal is to be able to flexibly pass in an appropriate UpdateDefinition where that business logic is done elsewhere and passed in.
I can create/update/set/combine the UpdateDefinition itself, but when i try to log it by serializing it, it shows null:
JsonConvert.SerializeObject(updateDef)
When I try to log it, save it to another a class or pass it to another function it displays null:
public class Account
{
[BsonElement("AccountId")]
public int AccountId { get; set; }
[BsonElement("Email")]
public string Email { get; set; }
}
var updateBuilder = Builders<Account>.Update;
var updates = new List<UpdateDefinition<Account>>();
//just using one update here for brevity - purpose is there could be 1:many depending on fields updated
updates.Add(updateBuilder.Set(a => a.Email, email));
//Once all the logic and field update determinations are made
var updateDef = updateBuilder.Combine(updates);
//The updateDef does not serialize to string, it displays null when logging.
_logger.LogInformation("{0} - Update Definition: {1}", actionName, JsonConvert.SerializeObject(updateDef));
//Class Created for passing the Account Update Information for Use by update function
public class AccountUpdateInfo
{
[BsonElement("AccountId")]
public int AccountId { get; set; }
[BsonElement("Update")]
public UpdateDefinition<Account> UpdateDef { get; set; }
}
var acct = new AccountUpdateInfo();
acctInfo.UpdateDef = updateDef
//This also logs a null value for the Update Definition field when the class is serialized.
_logger.LogInformation("{0} - AccountUpdateInfo: {1}", actionName, JsonConvert.SerializeObject(acct));
Any thoughts or ideas on what is happening? I am stumped on why I cannot serialize for logging or pass the value in a class around like I would expect

give this a try:
var json = updateDef.Render(
BsonSerializer.SerializerRegistry.GetSerializer<Account>(),
BsonSerializer.SerializerRegistry)
.AsBsonDocument
.ToString();
and to turn a json string back to an update definition (using implicit operator), you can do:
UpdateDefinition<Account> updateDef = json;
this is off the top of my head and untested. the only thing i'm unsure of (without an IDE) is the .Document.ToString() part above.

Use Guid as ID in MongoDb for C#

I'm trying to use Guid datatype as Id in my Poco object "Parameter". However, while I'm able to write files to the database I can't read from it.
This is the import function writing table headers from a csv file into the database. First line of the csv file are parameters and second line the units those parameters are measured in. All other lines contain actual values and are stored in another collection as BsonDocument. The csv files are dynamic and need to be selectable via combobox, which is why the parameters are written in their own collection.
IMongoCollection<Parameter> parameterCollection = this.MongoDatabase.GetCollection<Parameter>("Parameters");
columnNames.Select((columnName, index) => new Parameter() { Name = columnName, Unit = columnUnits[index] })
.ToList()
.ForEach(parameter =>
{
parameterCollection.UpdateOne(Builders<Parameter>.Filter.Eq("Name", parameter.Name),
Builders<Parameter>.Update.Set("Unit", parameter.Unit),
new UpdateOptions()
{
IsUpsert = true
});
});
This is the Parameter class:
public class Parameter
{
[BsonId]
public Guid Id { get; set; }
public string Name { get; set; }
public string Unit { get; set; }
}
Here's the method trying to read the data from the document:
public List<Parameter> GetParameters()
{
return this.MongoDatabase.GetCollection<Parameter>("Parameters")
.Find(Builders<Parameter>.Filter.Empty)
.ToList();
}
This results in the following error message:
"SystemFormatException: 'An error occurred while deserializing the Id property of class TimeSeriesInterface.DTO.Parameter: Cannot deserialize a 'Guid' from BsonType 'ObjectId'.'
I also tried this attribute: [BsonId(IdGenerator = typeof(GuidGenerator))]
I'm unable to find any help besides those two attributes. They seem to solve it for everybody else, but I still keep getting this error.
I may add that the import and read functions are parts of different classes each calling their own new MongoClient().GetDatabase(MongoDatabaseRepository.DatabaseName); but when I use ObjectId as data type I do get the data so I don't think that's the issue.
Why not use ObjectId as data type? We have an extra project for database access and I do not wish to add the mongodb assembly all over the place just because other projects use the POCOs and require a reference for that pesky little ObjectId.
EDIT:
This is the mapping used within the constructor after suggestion by AlexeyBogdan (beforehand it was simply the call to AutoMap()):
public MongoDatabaseRepository(string connectionString)
{
this.MongoDbClient = new MongoClient();
this.MongoDatabase = this.MongoDbClient.GetDatabase(MongoDatabaseRepository.DatabaseName);
BsonClassMap.RegisterClassMap<Parameter>(parameterMap =>
{
parameterMap.AutoMap();
parameterMap.MapIdMember(parameter => parameter.Id);
});
}

Instead of [BsonId] I recommend you to use this mapping
BsonClassMap.RegisterClassMap<Type>(cm =>
{
cm.AutoMap();
cm.MapIdMember(c => c.Id);
});
I Hope it helps you.

Most efficient way to convert a object to another (Model to ViewModel)

Suppose I have a model with 20 fields, and in my index page, I want to list all models that are stored in my database.
In index page, instead of listing all fields of the model, I only to list 3 fields.
So, I make two class:
class CompleteModel {
public int Id { get; set; }
public string Field01 { get; set; }
public string Field02 { get; set; }
public string Field03 { get; set; }
public string Field04 { get; set; }
public string Field05 { get; set; }
...
public string Field20 { get; set; }
}
now, in my Controller, I can use:
await _context.CompleteModel.ToListAsync();
but I feel that it does not seem to be the right way to do it, because I'm getting all fields and using only 3 fields.
So, I made this code:
class ViewModel {
public string Field02 { get; set; }
public string Field04 { get; set; }
public string Field08 { get; set; }
}
var result = _context.CompleteModel.Select(
x => new {
x.Field02,
x.Field04,
x.Field08
}).ToListAsync();
var listResults = new List<IndexViewModel>();
if (result != null)
{
listResults.AddRange(results.Select(x => new IndexViewModel
{
Field02 = x.Field02,
Field04 = x.Field04,
Field08 = x.Field08
}));
}
I think this is a lot of code to do this.
First, I selected all the fields that I want, then, copied everything to another object.
There's a "more directly" way to do the same thing?
Like:
_context.CompleteModel.Select(x => new IndexViewModel { Field02, Field04, Field08 });

You could use AutoMapper to reduce the boiler plate so you're not manually copying field values over.
If you include the AutoMapper NuGet package then you'd need to have the following in your startup somewhere to configure it for your classes:
Mapper.Initialize(cfg => cfg.CreateMap<CompleteModel, ViewModel>());
You could then do something like the following:
var results = await _context.CompleteModel.ToListAsync();
var viewModelResults = results.Select(Mapper.Map<ViewModel>).ToList();
There are a lot of configuration options for the package so do take a look at the documentation to see if it suits your needs and determine the best way to use it if it does.

In my view this is one of the weaknesses of over abstraction and layering. The VM contains the data that is valuable to your application within the context of use (screen, process etc). The data model contains all the data that could be stored that might be relevant. At some point you need to match the two.
Use EF Projection to fetch only the data you need from the database into projected data model classes (using the EF POCO layer to define the query, but not to store the resultant data).
Map the projected classes onto your VM, if there is a naieve mapping, using Automapper or similar. However unless you are just writing CRUD screens a simple field by field mapping is of little value; the data you fetch from your data store via EF is in its raw, probably relational form. The data required by your VM is probably not going to fit that form very neatly (again, unless you are doing a simple CRUD form), so you are going to need to add some value by coding the relationship between the data store and the View Model.
I think concentrating on the count of lines of code would lead to the wrong approach. I think you can look at that code and ask "is it adding any value". If you can delegate the task to Automapper, then great; but your VM isn't really pulling its weight other than adding some validation annotation if you can consistently delegate the task of data model to VM data copying.