I am trying to get my head around a specific problem to decide whether to take the plunge in converting some personal projects to MongoDb after completing a basic course last week. What I am trying to achieve is a representation of my data based on grouping and then ultimately selecting specific parts of that group to create a new projection which shows my final result. In the code presently, we do the grouping and then do a sub-select to create the final dataset, I am hoping this can be done in a single hit.
Example document
{
"_id": {
"$oid": "600d88b0d7016d5675cd59bd"
},
"DeviceId": {
"$oid": "600d729764ea780882ac559b"
},
"UserId": {
"$oid": "600b660eff59aab915985b1d"
},
"Date": {
"$date": {
"$numberLong": "1611499696095"
}
},
"Records": [
{
"Count": {
"$numberInt": "10"
},
"Test1": {
"Inconclusive": null,
"Passed": true,
"Failed": null
},
"Test2": {
"Inconclusive": null,
"Passed": true,
"Failed": null
}
},
{
"Count": {
"$numberInt": "15"
},
"Test1": {
"Inconclusive": true,
"Passed": null,
"Failed": null
},
"Test2": {
"Inconclusive": null,
"Passed": true,
"Failed": null
}
},
{
"Count": {
"$numberInt": "15"
},
"Test1": {
"Inconclusive": true,
"Passed": null,
"Failed": null
},
"Test2": {
"Inconclusive": null,
"Passed": null,
"Failed": true
}
}
]
}
Ultimately, what I am trying to get is this as close to this as possible;
{
"DeviceId": "600d729764ea780882ac559b",
"Test1Inconclusive": 30,
"Test1Passed": 10,
"Test1Failed": 0,
"Test2Inconclusive": 0,
"Test2Passed": 25,
"Test2Failed": 15
}
So far, all I have managed to get is the data grouped and it is at this point in the existing code (Entity Framework/SQL server) that I would use Linq to pull out the SUM'd values.
[{
$match: {
UserId: ObjectId('600b660eff59aab915985b1d')
}
}, {
$unwind: {
path: '$Records'
}
}, {
$group: {
_id: {
DeviceId: '$DeviceId',
Test1Inconclusive: '$Records.Test1.Inconclusive',
Test1Passed: '$Records.Test1.Passed',
Test1Failed: '$Records.Test1.Failed',
Test2Inconclusive: '$Records.Test2.Inconclusive',
Test2Passed: '$Records.Test2.Passed',
Test2Failed: '$Records.Test2.Failed',
},
Count: {
$sum: '$Records.Count'
}
}
}, {}]
I am not sure if it is possible to do what I want, and if so how the do the next projection step while performing a subselect of this grouped data. It might even be that my approach is flawed from the start, so feel free to change it completely.
Bonus internet points if you can also give me the MongoDb C# syntax for doing the same (on a MongoCollection)
Following on from the initial version by #turivishal, the answer below worked;
db.collection.aggregate([
{
$match: {
UserId: ObjectId("600b660eff59aab915985b1d")
}
},
{
$unwind: {
path: "$Records"
}
},
{
$group: {
_id: "$DeviceId",
Test1Inconclusive: {
$sum: {
$cond: [
{
$eq: [
"$Records.Test1.Inconclusive",
true
]
},
"$Records.Count",
0
]
}
},
Test1Passed: {
$sum: {
$cond: [
{
$eq: [
"$Records.Test1.Passed",
true
]
},
"$Records.Count",
0
]
}
},
Test1Failed: {
$sum: {
$cond: [
{
$eq: [
"$Records.Test1.Failed",
true
]
},
"$Records.Count",
0
]
}
},
Test2Inconclusive: {
$sum: {
$cond: [
{
$eq: [
"$Records.Test2.Inconclusive",
true
]
},
"$Records.Count",
0
]
}
},
Test2Passed: {
$sum: {
$cond: [
{
$eq: [
"$Records.Test2.Passed",
true
]
},
"$Records.Count",
0
]
}
},
Test2Failed: {
$sum: {
$cond: [
{
$eq: [
"$Records.Test2.Failed",
true
]
},
"$Records.Count",
0
]
}
},
Count: {
$sum: "$Records.Count"
}
}
}
])
Related
I want to create a chart but I can't send the data from a list. I am using C#. I need to send the values of X and Y from a query that I perform and the values change. I need to use this library because it allows to create simple graphs.
The value of X has to be of type date and the value of Y is double
{
"type": "line",
"data": {
"datasets": [
{
"label": "Dataset with string point data",
"backgroundColor": "rgba(255, 99, 132, 0.5)",
"borderColor": "rgb(255, 99, 132)",
"fill": false,
"data": [
{
"x": "2020-06-14T09:15:34-07:00",
"y": 75
},
{
"x": "2020-06-16T09:15:34-07:00",
"y": -53
},
{
"x": "2020-06-18T09:15:34-07:00",
"y": 31
},
{
"x": "2020-06-19T09:15:34-07:00",
"y": 6
}
]
}
]
},
"options": {
"responsive": true,
"title": {
"display": true,
"text": "Chart.js Time Point Data"
},
"scales": {
"xAxes": [{
"type": "time",
"display": true,
"scaleLabel": {
"display": true,
"labelString": "Date"
},
"ticks": {
"major": {
"enabled": true
}
}
}],
"yAxes": [{
"display": true,
"scaleLabel": {
"display": true,
"labelString": "value"
}
}]
}
}
}
I have documents like this:
[
// 1
{
"_id": ObjectId("573f3944a75c951d4d6aa65e"),
"Source": "IGN",
"Family": [
{
"Countries": [
{
"uid": 17,
"name": "Japan",
}
]
}
]
},
// 2
{
"_id": ObjectId("573f3d41a75c951d4d6aa65f"),
"Source": "VG",
"Family": [
{
"Countries": [
{
"uid": 17,
"name": "USA"
}
]
}
]
},
// 3
{
"_id": ObjectId("573f4367a75c951d4d6aa660"),
"Source": "NRK",
"Family": [
{
"Countries": [
{
"uid": 17,
"name": "Germany"
}
]
}
]
},
// 4
{
"_id": ObjectId("573f4571a75c951d4d6aa661"),
"Source": "VG",
"Family": [
{
"Countries": [
{
"uid": 10,
"name": "France"
}
]
}
]
},
// 5
{
"_id": ObjectId("573f468da75c951d4d6aa662"),
"Source": "IGN",
"Family": [
{
"Countries": [
{
"uid": 14,
"name": "England"
}
]
}
]
}
]
I want to return only the documents with source equals 'Countries.uid' equal 17
so I have in the end :
[
{
"_id": ObjectId("573f3944a75c951d4d6aa65e"),
"Source": "IGN",
"Family": [
{
"Countries": [
{
"uid": 17,
"name": "Japan",
}
]
}
]
},
{
"_id": ObjectId("573f3d41a75c951d4d6aa65f"),
"Source": "VG",
"Family": [
{
"Countries": [
{
"uid": 17,
"name": "USA"
}
]
}
]
},
{
"_id": ObjectId("573f4367a75c951d4d6aa660"),
"Source": "NRK",
"Family": [
{
"Countries": [
{
"uid": 17,
"name": "Germany"
}
]
}
]
}
]
How can I do this with the official C# MongoDB driver?
Tried this :
public List<Example> getLinkedCountry(string porduitId)
{
var filter = Builders<Example>.Filter.AnyIn("Family.Countries.uid", porduitId);
var cursor = await _certificats.FindAsync(filter);
var docs = cursor.ToList();
return docs;
}
Unfortunately, I think my filter is wrong.
Is there a way to find all the documents by accessing the nested list by id and retrieving it?
Solution 1
Use ElemMatch instead of AnyIn.
var filter = Builders<Example>.Filter.ElemMatch(
x => x.Family,
y => y.Countries.Any(z => z.uid == porduitId));
Output
Solution 2
If you are unconfident with MongoDB .Net Driver syntax, you can convert the query as BsonDocument via MongoDB Compass (Export to language feature).
var filter = new BsonDocument("Family.Countries.uid", porduitId);
Just to expand on #Yong Shun 's answer,
if you just want to return the list of nested documents and not all of it, you have a few options.
Using project
var filter = Builders<Example>.Filter.ElemMatch(
x => x.Family,
y => y.Countries.Any(z => z.uid == porduitId));
var project = Builders<Example>.Project.ElemMatch(
x => x.Family,
y => y.Countries.Any(z => z.uid == porduitId)
);
var examples = await collection.filter(filter).Project<Example>(project).toListAsync();
Using the aggregate pipeline
var filter = Builders<Example>.Filter.ElemMatch(
x => x.Family,
y => y.Countries.Any(z => z.uid == porduitId));
var project = Builders<ServiceProvider>.Projection.Expression(
x => x.Faimily.Where(y => y.uid == porduitId)
);
var result = await collection
.Aggregate()
.Match(filter)
.Project(project)
.ToListAsync(); //Here result is a list of Iterable<Countries>
I have a lot of data stored in the following format (I simplified the data to explain the problem).
What I need is:
group all the data by "Action Id" field
calculate the difference between max and min values of "Created Time" for each group (from the previous action)
order the results by the calculated field ("Action duration" - difference between max and min)
I use NEST (C#) to query the ElasticSearch. I think that if you can help me with native Elastic query it also will be very helpful, I'll translate it to C#.
Thank you.
Case your mappings looks like that:
PUT /index
{
"mappings": {
"doc": {
"properties": {
"ActionId": {
"type": "text",
"fielddata": true
},
"CreatedDate":{
"type": "date"
},
"SubActionName":{
"type": "text",
"fielddata": true
}
}
}
}
}
Your elasticsearch query should look like that:
GET index/_search
{
"size": 0,
"aggs": {
"actions": {
"terms": {
"field": "ActionId"
},
"aggs": {
"date_created": {
"date_histogram": {
"field": "CreatedDate",
"interval": "hour"
},
"aggs": {
"the_max": {
"max": {
"field": "CreatedDate"
}
},
"the_min": {
"min": {
"field": "CreatedDate"
}
},
"diff_max_min": {
"bucket_script": {
"buckets_path": {
"max": "the_max",
"min": "the_min"
},
"script": "params.max - params.min"
}
}
}
}
}
}
}
}
You can read more about Pipeline Aggregetions here
Hope that helps
class MyObject{
public string Name{get;set;}
public List<string> Tags{get;set;}
}
/*Create mapping */
client.Map<MyObject>(m =>
m.Properties(props =>
props.String(s =>
s.Name(p => p.Name)
.Path(MultiFieldMappingPath.Full)
.Index(FieldIndexOption.NotAnalyzed)
.Fields(f =>
f.String(ps =>
ps.Name(p => p.Name.Suffix("searchable"))
.Index(FieldIndexOption.Analyzed)
)
)
)
)
);
How to index NotAnalyzed for field Tags same field Name
I want to search exactly one phrase in the Tags field
Example: I want search "elastic search" to find out which object contains exactly that word in the Tags field
Obj1:
{
"Name":"Object 1",
"Tags":["elastic search","how to code"]
}
Obj2:
{
"Name":"Object 2",
"Tags":["elastic","c#"]
}
Obj3:
{
"Name":"Object 2",
"Tags":["learn elastic search","learn C#"]
}
===> Result: Obj 1
Based on your request I will create a test1 index.
PUT test1/doc/1
{
"Name": "Object 1",
"Tags": [
"elastic search",
"how to code"
]
}
PUT test1/doc/2
{
"Name":"Object 2",
"Tags":["elastic","c#"]
}
So I will write query to fetch the exact term elastic search as you mentioned in your example.
GET test1/doc/_search
{
"query": {
"term": {
"Tags.keyword":
"elastic search"
}
}
}
So the result is for the below query is
curl -XGET "http://localhost:9200/test1/doc/_search"
{
"took": 0,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 1,
"hits": [
{
"_index": "test1",
"_type": "doc",
"_id": "2",
"_score": 1,
"_source": {
"Name": "Object 2",
"Tags": [
"elastic",
"c#"
]
}
},
{
"_index": "test1",
"_type": "doc",
"_id": "1",
"_score": 1,
"_source": {
"Name": "Object 1",
"Tags": [
"elastic search",
"how to code"
]
}
}
]
}
}
So now the query to fetch documents based on your field.
curl -XGET "http://localhost:9200/test1/doc/_search" -H 'Content-Type: application/json' -d'
{
"query": {
"term": {
"Tags.keyword":
"elastic search"
}
}
}'
And the result is
{
"took": 0,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 0.2876821,
"hits": [
{
"_index": "test1",
"_type": "doc",
"_id": "1",
"_score": 0.2876821,
"_source": {
"Name": "Object 1",
"Tags": [
"elastic search",
"how to code"
]
}
}
]
}
}
Hope it works. Let me know if you are still facing any issues.
We have documents saving to MongoDb. The problem is that one of our sub-documents has an Id property that is getting returned as _id, which is causing serialize/deserialize issues with the C# driver due to how it interprets Id fields (see http://mongodb.github.io/mongo-csharp-driver/2.0/reference/bson/mapping/)
I would like to rename the property from Id to SetId, but our data is fairly dynamic and simple field rename solutions that I've seen elsewhere do not apply. Here's an example of some heavily edited simple data:
{
"Id": "5a6238dbccf20b38b0db6cf2",
"Title": "Simple Document",
"Layout": {
"Name": "Simple Document Layout",
"Tabs": [
{
"Name": "Tab1",
"Sections": [
{
"Name": "Tab1-Section1",
"Sets": [
{
"Id": 1
}
]
}
]
}
]
}
}
Compare with more complex data:
{
"Id": "5a6238dbccf20b38b0db6abc",
"Title": "Complex Document",
"Layout": {
"Name": "Complex Document Layout",
"Tabs": [
{
"Name": "Tab1",
"Sections": [
{
"Name": "Tab1-Section1",
"Sets": [
{
"Id": 1
}
]
},
{
"Name": "Tab1-Section2",
"Sets": [
{
"Id": 1
}
]
}
]
},
{
"Name": "Tab2",
"Sections": [
{
"Name": "Tab2-Section1",
"Sets": [
{
"Id": 1
}
]
}
]
},
{
"Name": "Tab3",
"Sections": [
{
"Name": "Tab3-Section1",
"Sets": [
{
"Id": 1
},
{
"Id": 2
}
]
}
]
}
]
}
}
Note that the Set.Id field can be on multiple tabs on multiple sections with multiple sets. I just don't know how to approach a query to handle renaming data at all these levels.
I took #Veerum's advice and did a manual iteration over the collection with something like this:
myCol = db.getCollection('myCol');
myCol.find({ "Layout.Tabs.Sections.Sets._id": {$exists: true} }).forEach(function(note) {
for(tab = 0; tab != note.Layout.Tabs.length; ++tab) {
for(section = 0; section != note.Layout.Tabs[tab].Sections.length; ++section) {
for(set = 0; set != note.Layout.Tabs[tab].Sections[section].Sets.length; ++set) {
note.Layout.Tabs[tab].Sections[section].Sets[set].SetId = NumberInt(note.Layout.Tabs[tab].Sections[section].Sets[set]._id);
delete note.Layout.Tabs[tab].Sections[section].Sets[set]._id
}
}
}
myCol.update({ _id: note._id }, note);
});
Perhaps there is a more efficient way, but we are still on Mongo v3.2 and it seems to work well.