We Keep Coding

Navigating or transforming JSON with Linq

Let consider this JSON
{
"data": "014",
"theme": "COLORADO CASUAL",
"family": "2163",
"category": "00",
"compo_groups": [
{
"title": "HEAD024",
"values": [
{
"perc": "100",
"desc": "COMP036"
}
]
},
{
"title": "HEAD035",
"values": [
{
"perc": "100",
"desc": "COMP042"
},
{
"perc": "50",
"desc": "COMP043"
}
]
}
],
"product_name": "D812",
"supplier_code": "1011"
}
I need to check that all my compositions are exactly 100pc. In this JSON I have 2 group of composition. The first one is correct. I have one element to 100pc. The second one is composed by 2 elements and total is 150pc. This is an error.
I need to write a code in C# that detect the error. I can write most part of this code. I just don't know how to transform this JSON in list of values I can manage with LinQ.

Assuming you are using a recent version of .NET (e.g. .NET6) then you can use the built-in System.Text.Json libraries to parse your JSON input. If you need to use other parts of the JSON, I would recommend deserialising in to concrete C# classes so you get proper validation, IntelliSense and all that good stuff.
However, if you simply want to check those percentages you can use the STJ library directly, something like this for example:
// Load JSON
var json = "{...}";
var doc = JsonDocument.Parse(json);
// Manually walk the document to get the values you need and summarise
var result = doc.RootElement
.GetProperty("compo_groups")
.EnumerateArray()
.Select(a => new
{
Title = a.GetProperty("title").ToString(),
Percentage = a.GetProperty("values")
.EnumerateArray()
.Select(v => double.Parse(v.GetProperty("perc").ToString()))
.Sum()
});
And you can iterate over that result like this:
foreach(var value in result)
{
Console.WriteLine($"Title '{value.Title}' has a percentage of {value.Percentage}");
}
Which will output this:
Title 'HEAD024' has a percentage of 100
Title 'HEAD035' has a percentage of 150

you don't need any classes to get the data you want
using System.Text.Json;
List<string> titles = JsonNode.Parse(json)["compo_groups"].AsArray()
.Select(x => x["values"].AsArray())
.Where(v => v.Select(x =>
Convert.ToInt32(x["perc"].GetValue<string>())).Sum() > 100)
.Select(v => v.Parent["title"].GetValue<string>())
.ToList(); // result ["HEAD035"]
or
using Newtonsoft.Json;
List<string> titles = JObject.Parse(json)["compo_groups"]
.Select(x => x["values"])
.Where(v => v.Select(x => (int)x["perc"]).Sum() > 100)
.Select(v => v.Parent.Parent)
.Select(p=> (string) p["title"]) // here you can select any data you need
.ToList(); // result ["HEAD035"]

Elasticsearch could not search on string field

I am trying to use NEST to create search query dynamically based on user's input.
I want to add multiple filter in Filter with Term but string field searching is not possible and I cannot find any solution.
Code for example is that, this code try to search on string field an it is not working
var response = await _elasticClient.SearchAsync<CustomerAddressInfo>(p => p
.Query(q => q
.Bool(b => b
.Filter(f => f.Term(t => t.Field(p => p.AccountAddressId).Value(type.AccountAddressId)))
)
)
);
And the other search simple is with integer field and it is working with success
var response = await _elasticClient.SearchAsync<CustomerAddressInfo>(p => p
.Query(q => q
.Bool(b => b
.Filter(f => f.Term(t => t.Field(p => p.CreateUnitId).Value(type.CreateUnitId)))
)
)
);
But; if I search data on string field with Match keyword, again it is successfull on search
var response = await _elasticClient.SearchAsync<CustomerAddressInfo>(p => p
.Query(q => q
.Match(m => m
.Field(f => f.AccountAddressId)
.Query(type.AccountAddressId)
)
)
);
And the question is, how can I give multiple search criteria with Match query method or how can I seach on string field by Term query method on elastic

I am not familiar with NEST, but to search on multiple fields using match query or term query, you can refer following example :
Bool query is used to combine one or more clauses, to know more refer this
Avoid using the term query for text fields.
By default, Elasticsearch changes the values of text fields as part of
analysis. This can make finding exact matches for text field values
difficult.
To search text field values, use the match query instead.
Index Mapping
{
"mappings": {
"properties": {
"name": {
"type": "text"
},
"cost": {
"type": "long"
}
}
}
}
Index data:
{
"name":"apple",
"cost":"40"
}
{
"name":"apple",
"cost":"55"
}
Search Query: Multiple Search criteria with match
{
"query": {
"bool": {
"must": [
{ "match": { "name": "apple" }},
{ "match": { "cost": 40 }}
]
}
}
}
Search on-field by term query
{
"query": {
"bool" : {
"must" :[
{"term" : { "name" : "apple" }},
{"term": { "cost":40 }}
]
}
}
}
Search Result:
"hits": [
{
"_index": "my-index",
"_type": "_doc",
"_id": "3",
"_score": 1.1823215,
"_source": {
"name": "apple",
"cost": "40"
}
}
]

Hey i do not get the whole requirements of yours. But if you want to add multiple condition on filter then you can do like below.
QueryContainer qSs = null;
foreach(var query in queries) // let say queries is list of yours search item
{
qSs &= new TermQuery { Field = "your_field_name", Value = query };
}
var searchResults = await _elasticClient.SearchAsync<CustomerAddressInfo>(s => s
.Query(q => q
.Bool(b => b.Filter(qSs) )
)
);

Complex query filtering of object arrays in MongoDb (C# Driver)

Let's say I have this document.
"Id" : "lot1",
"Fruits" : [
[{ "Id": "fruit1", "Name": "apple"}, { "Id": "fruit2", "Name": "carrot"}]
[{ "Id": "fruit3", "Name": "banana"}]
]
Q: How can I query the Fruits array by a list of fruits names ?
I have tried like this:
var fruitNames = new List<string>(){ "apple", "banana" };
var builder = Builders<Lot>.Filter;
var filter = builder.AnyIn(l => l.Fruits.Select(f => f.Name), fruitNames); //TAKE 1
var filter = builder.Where(l => l.Fruits.Select(f => f.Name).Any(f => fruitNames.Contains(f)));//TAKE 2
var filter = builder.AnyIn("Fruits.Name", fruitNames );//TAKE 3
var results = mongoContext.Lots.Find(filter).ToList();
I have tried this in 3 different ways with no success.

Can you try this? It is a little hack but i am pretty sure it will work:
var filter= Builders<Lot>.Filter.ElemMatch(y => y.fruits, x => fruitNames.Contains(x.name));

Elasticsearch Date Histogram report with Terms aggregation

I'm trying Nest plugin for querying elastic search data. I have a yearly job count report based on a field. Currently I have used the date Histogram report for this and below is the elastic query.
POST insight/_search
{
"size": "0",
"query": {
"filtered": {
"query": {
"query_string": {
"query": "(onet.familycode: 11)"
}
}
}
},
"aggregations": {
"jobcounts_by_year": {
"date_histogram": {
"field": "jobdate",
"interval": "year",
"format": "yyyy"
},
"aggregations": {
"count_by_occupation_family": {
"terms": {
"field": "onet.family"
}
}
}
}
}
}
Equivalent Nest query
result = ElasticSearchClient.Instance.Search<Job>(s => s.Size(0)
.Query(query => query
.Filtered(filtered => filtered
.Query(q => q
.QueryString(qs => qs.Query(queryString)))))
.Aggregations(a => a
.DateHistogram("jobcounts_by_year", dt => dt
.Field(ElasticFields.JobDate)
.Interval("year")
.Format("yyyy")
.Aggregations(a1 => a1
.Terms("top_agg", t => t
.Field(criteria.GroupBy.GetElaticSearchTerm())
.Exclude("NA|Unknown|Not available")
.Size(Constants.DataSizeToCompare)))
)));
Everything works well, but now the problem is iterating over the result to get values, For normal aggregation I'm currently doing it like below
data = result.Aggs.Terms("top_agg").Items.Select(item =>
new JobReportResult
{
Group = item.Key,
Count = item.DocCount
}).ToList();
But it seems Nest doesn't support buckets with in Date Histogram buckets.
If i tried like below I'm getting null reference exception.
result.Aggs.DateHistogram("jobcounts_by_year").Terms("top_agg")
It seems we have to use something like below.The d2 now has IAggregation
var d1 = result.Aggs.DateHistogram("jobcounts_by_year").Items;
var d2 =(TermsAggregator)d1[0].Aggregations["top_agg"];
But the Aggregation property is not exposing any values.
I'm stuck here. Can someone let me know how can I access buckets inside DateHistogram Buckets using NEST
Regards,

Try this
var dateHistogram = searchResponse.Aggs.DateHistogram("jobcounts_by_year");
foreach (var item in dateHistogram.Items)
{
var bucket = item.Terms("top_agg");
}
Hope this helps.

Searching using NEST does not return results, when querying on certain fields

I'm developing an .NET application using Elastic Search. I used ES River to index the data.
Results (in Sense) look kinda like this:
{
"_index": "musicstore",
"_type": "songs",
"_id": "J2k-NjXjRa-mgWKAq0RMlw",
"_score": 1,
"_source": {
"songID": 42,
"tempo": "andante",
"metrum": "3/4 E8",
"intonation": "F",
"title": "Song",
"archiveSongNumber": "3684",
"Year": 2000,
"Place": "London"
}
},
To access the indexed data I use NEST queries similar to this:
var result = ElasticClient.Search<Song>(s => s.Query(q => q.Term(p => p.title, "Song")));
I'm having a problem that the query doesn't return any results, when I search for a certain field.
For instance when I search for a title, songID, tempo or archiveSongNumber the query works fine and it returns the same results as Sense, but when I search for Year, Place, metrum, etc. the query doesn't return any results, but it should (Sense does and it should).
Queries like these work (and return the right results):
var result = ElasticClient.Search<Song>(s => s.Query(q => q.Term(p => p.title, "Song")));
var result = ElasticClient.Search<Song>(s => s.Query(q => q.Term(p => p.songID, 42)));
var result = ElasticClient.Search<Song>(s => s.Query(q => q.Term(p => p.archiveSongNumber , "3684")));
Queries like these don't return any results (but they should):
var result = ElasticClient.Search<Song>(s => s.Query(q => q.Term(p => p.Place, "London")));
var result = ElasticClient.Search<Song>(s => s.Query(q => q.Term(p => p.Year, 2000)));
What am I doing wrong? Did I mess up when I was indexing data?
Update:
Mapping looks like this:
{
"musicstore": {
"mappings": {
"songs": {
"properties": {
"Year": {
"type": "long"
},
"Place": {
"type": "string"
},
"archiveSongNumber": {
"type": "string"
},
"songID": {
"type": "long"
},
"intonation": {
"type": "string"
},
"metrum": {
"type": "string"
},
"title": {
"type": "string"
},
"tempo": {
"type": "string"
}
}
}
}
}
}
Update 2:
ES river request looks like this:
PUT /_river/songs_river/_meta
{
"type":"jdbc",
"jdbc": {
"driver":"com.microsoft.sqlserver.jdbc.SQLServerDriver",
"url":"jdbc:sqlserver://ip_address:1433;databaseName=database",
"user":"user",
"password":"password",
"strategy":"simple",
"poll":"300s",
"autocommit":true,
"fetchsize":10,
"max_retries":3,
"max_retries_wait":"10s",
"index":"musicstore",
"type":"songs",
"analysis": {
"analyzer" :{
"whitespace" :{
"type" : "whitespace",
"filter":"lowercase"
}
}
},
"sql":"some_sql_query"
}
}
ES client configuration looks like this:
private static ElasticClient ElasticClient
{
get
{
Uri localhost = new Uri("http://localhost:9200");
var setting = new ConnectionSettings(localhost);
setting.SetDefaultIndex("musicstore").MapDefaultTypeNames(d => d.Add(typeof(Song), "songs"));
setting.SetConnectionStatusHandler(c =>
{
if (!c.Success)
throw new Exception(c.ToString());
});
return new ElasticClient(setting);
}
}

From looking at your mapping, the issue here is most likely that all of your fields are being analyzed when indexing, but you are using term queries with NEST, which are not analyzed, meaning they will only find exact matches. If you don't explicitly specify an analyzer in your mappings, Elasticsearch defaults to the standard analyzer.
When you perform a search in Elasticsearch using a query string, like you're doing in Sense: GET _search/?q=Place:London, a query string query is what's being run by Elasticsearch, which is different than a term query.
From your examples though, it doesn't look like you are actually using query string syntax. You probably want a match query instead:
client.Search<Song>(s => s
.Query(q => q
.Match(m => m
.OnField(p => p.Place)
.Query("London")
)
)
);
If you do however want a query string query like the one you're performing with Sense, than you can use QueryString:
client.Search<Song>(s => s
.Query(q => q
.QueryString(qs => qs
.OnFields(p => p.Place)
.Query("London")
)
)
);
Hope that helps. I suggest checking out the getting started guide, specifically the section on exact values vs. full text.

Add "keyword" suffix to your Term Field :
var result = ElasticClient.Search<Song>(s => s
.Query(q => q
.Term(p => p
.Field(x => x.Year.Suffix("keyword")).Value(2000))));
Try it, it will work!