Let consider this JSON
{
"data": "014",
"theme": "COLORADO CASUAL",
"family": "2163",
"category": "00",
"compo_groups": [
{
"title": "HEAD024",
"values": [
{
"perc": "100",
"desc": "COMP036"
}
]
},
{
"title": "HEAD035",
"values": [
{
"perc": "100",
"desc": "COMP042"
},
{
"perc": "50",
"desc": "COMP043"
}
]
}
],
"product_name": "D812",
"supplier_code": "1011"
}
I need to check that all my compositions are exactly 100pc. In this JSON I have 2 group of composition. The first one is correct. I have one element to 100pc. The second one is composed by 2 elements and total is 150pc. This is an error.
I need to write a code in C# that detect the error. I can write most part of this code. I just don't know how to transform this JSON in list of values I can manage with LinQ.
Assuming you are using a recent version of .NET (e.g. .NET6) then you can use the built-in System.Text.Json libraries to parse your JSON input. If you need to use other parts of the JSON, I would recommend deserialising in to concrete C# classes so you get proper validation, IntelliSense and all that good stuff.
However, if you simply want to check those percentages you can use the STJ library directly, something like this for example:
// Load JSON
var json = "{...}";
var doc = JsonDocument.Parse(json);
// Manually walk the document to get the values you need and summarise
var result = doc.RootElement
.GetProperty("compo_groups")
.EnumerateArray()
.Select(a => new
{
Title = a.GetProperty("title").ToString(),
Percentage = a.GetProperty("values")
.EnumerateArray()
.Select(v => double.Parse(v.GetProperty("perc").ToString()))
.Sum()
});
And you can iterate over that result like this:
foreach(var value in result)
{
Console.WriteLine($"Title '{value.Title}' has a percentage of {value.Percentage}");
}
Which will output this:
Title 'HEAD024' has a percentage of 100
Title 'HEAD035' has a percentage of 150
you don't need any classes to get the data you want
using System.Text.Json;
List<string> titles = JsonNode.Parse(json)["compo_groups"].AsArray()
.Select(x => x["values"].AsArray())
.Where(v => v.Select(x =>
Convert.ToInt32(x["perc"].GetValue<string>())).Sum() > 100)
.Select(v => v.Parent["title"].GetValue<string>())
.ToList(); // result ["HEAD035"]
or
using Newtonsoft.Json;
List<string> titles = JObject.Parse(json)["compo_groups"]
.Select(x => x["values"])
.Where(v => v.Select(x => (int)x["perc"]).Sum() > 100)
.Select(v => v.Parent.Parent)
.Select(p=> (string) p["title"]) // here you can select any data you need
.ToList(); // result ["HEAD035"]
Related
I'm using Nest to build a query. I have a class called Event which has a property called Location and a property called Address (a string). So the indices look something like this:
"id" : "something1",
"location": {
"address": "The Club",
}
"id" : "something2",
"location": {
"address": "The Hole",
}
I want to create a query where the user types "The Club" (the location variable) and it only retrieves the first item that has "The Club" as the address
This is my current query
var skip = (page - 1) * rows;
var searchResponse = await _elasticClient.SearchAsync<Event>(s => s
.From(skip)
.Size(rows)
.Query(q => q
.QueryString(qs => qs
.Fields(fields => fields
.Field(f => f.Location.Address)
)
.Analyzer("keyword")
.Query("*" + (location ?? string.Empty) + "*")
)
));
Currently, the query retrieves both objects since it finds "The". Do I need to add something to my index previously or what should I do?
For the record I added this when creating the index but it has no effect
client.Indices.Create(eventsIndex, c => c.Map<Event>(m => m
.Properties(p => p
.Text(t => t
.Name(n => n.Location.Address)
.Analyzer("keyword"))
)
));
I have two suggestions:
1 - Add the stop filter (default language english) that way you won't produce tokens like the term "the", all stopwords will be removed. Now you will only match the term "hotel", the term "the" will be filtered out.
POST _analyze
{
"tokenizer": "standard",
"filter": ["stop"],
"text": ["the hotel"]
}
Token
{
"tokens": [
{
"token": "hotel",
"start_offset": 4,
"end_offset": 9,
"type": "<ALPHANUM>",
"position": 1
}
]
}
2 - Use the default_operator AND in Query String query.
Using this Json string (cannot be altered because I receive it from an external source)
{
"IsValid": true,
"Result": [
{
"PartNumber": "ABC",
"Id": "x123"
},
{
"PartNumber": "DEF",
"Id": "y456"
},
{
"PartNumber": "GHI",
"Id": "z789"
}
]
}
What I need is a list of PartNumber/SupplyId like this one
"ABC", "x123"
"DEF", "y456"
"GHI", "z789"
I'm just doing it using a foreach approach, my question is: may I do it using something like
var props = obj.Descendants()
.OfType<JProperty>() .Where(p => p.Name== "PartNumber" || p.Name == "SupplyId")
... and now?
thank you
You can access child tokens of Result array to get PartNumber and Id values, then map them to list of tuples or anonymous types using Select method
var result = obj["Result"]?.Children()
.Select(t => (t["PartNumber"]?.Value<string>(), t["Id"]?.Value<string>()))
.ToList();
Or
var result = obj["Result"]?.Children()
.Select(t => new { PartNumber = t["PartNumber"]?.Value<string>(), Id = t["Id"]?.Value<string>() })
.ToList();
If I understood your request correctly you can do something like this:
var jObj = JsonConvert.DeserializeObject<JObject>(json);
var result = jObj["Result"]
.Children()
.SelectMany(c => new[] { c["PartNumber"].ToString(), c["Id"].ToString()})
.ToList()
result will contain flat list of part numbers and ids.
I'm new to LINQ and I need to write a LINQ query that returns each project's grade, called notet also the average of all notes.
Here is the query I have:
`var query = _context.Set<EvaluationResult>()
.Include(x => x.RatingLevel)
.Include(x => x.Skill)
.Include(x => x.Evaluation)
.ThenInclude(y => y.Project)
.ThenInclude(z => z.ProjectRatingLevels)
.ThenInclude(a => a.RatingLevel)
.Include(y => y.Evaluation.Project)
.ThenInclude(y => y.Degree)
.Where(x => x.Evaluation.Project.DegreeId == QueriedDegreeId)
.GroupBy(i => new { project = i.Evaluation.Project })
.Select(g => new
{
project = g.Select(y => y.Evaluation.Project.Label)
.Distinct().FirstOrDefault(),
note = Math.Round(((g.Sum(y => (double)y.Skill.Weight * (double)y.RatingLevel.Rate) /
g.Sum(y => (double)y.RatingLevel.Rate)) * 100) /
(double)g.Key.project.ProjectRatingLevels
.Select(z => z.RatingLevel.Rate)
.Max(), 2, MidpointRounding.AwayFromZero)
});
Here is the result:
[
{
"project": "Projet 1",
"note": 42.86
},
{
"project": "Projet 2",
"note": 41.67
},
{
"project": "Projet 3",
"note": 46.67
}
]
What I want is to add another value, average, which is just the Average of all "note" values, like so (the asterisks are just for emphasis):
[
{
"project": "Projet 1",
"note": 42.86,
**"average": 43.73**
},
{
"project": "Projet 2",
"note": 41.67,
**"average": 43.73**
},
{
"project": "Projet 3",
"note": 46.67,
**"average": 43.73**
}
]
MY PROBLEM
I'm stuck trying to calculate an Average of all the returned notes. I have no idea how to proceed. I tried to add an average key in my Select after note and project that reused the note query, like this:
average = g.Average(Math.Round(((g.Sum(... etc
But this gives me type errors: CS1929 'IGrouping<<anonymous type: Project project>, EvaluationResult>' does not contain a definition for average 'Average'. So I'm at an utter loss of what to do.
Can someone point me in the right direction? Am I missing something, like the need to use a Key or something?
Is your goal to do it all in one query ?
The only possible Resultset, with Aggregate and the Aggregated Items, is a GroupBy.
If you want to aggregate all, you want only one group, so you have to have a fictive key,the same one for each item
So Append:
.GroupBy(x => 1) /* makes one group for all, all have the same key */
.Select(g => new { average = g.Average(x => x.notes), items = g.Select(x => x)});
But this is really Forcing the SQL-Server to do the average. You Manifest the items in your memory anyway. So you can also take your existing Resultset, manifested with ToList or ToArray and just compute
var result = <yourquery>.ToList();
double average = result.Average(x => x.notes);
The only difference is, this is done on your CPU, not on the SQL-Server.
I'm trying Nest plugin for querying elastic search data. I have a yearly job count report based on a field. Currently I have used the date Histogram report for this and below is the elastic query.
POST insight/_search
{
"size": "0",
"query": {
"filtered": {
"query": {
"query_string": {
"query": "(onet.familycode: 11)"
}
}
}
},
"aggregations": {
"jobcounts_by_year": {
"date_histogram": {
"field": "jobdate",
"interval": "year",
"format": "yyyy"
},
"aggregations": {
"count_by_occupation_family": {
"terms": {
"field": "onet.family"
}
}
}
}
}
}
Equivalent Nest query
result = ElasticSearchClient.Instance.Search<Job>(s => s.Size(0)
.Query(query => query
.Filtered(filtered => filtered
.Query(q => q
.QueryString(qs => qs.Query(queryString)))))
.Aggregations(a => a
.DateHistogram("jobcounts_by_year", dt => dt
.Field(ElasticFields.JobDate)
.Interval("year")
.Format("yyyy")
.Aggregations(a1 => a1
.Terms("top_agg", t => t
.Field(criteria.GroupBy.GetElaticSearchTerm())
.Exclude("NA|Unknown|Not available")
.Size(Constants.DataSizeToCompare)))
)));
Everything works well, but now the problem is iterating over the result to get values, For normal aggregation I'm currently doing it like below
data = result.Aggs.Terms("top_agg").Items.Select(item =>
new JobReportResult
{
Group = item.Key,
Count = item.DocCount
}).ToList();
But it seems Nest doesn't support buckets with in Date Histogram buckets.
If i tried like below I'm getting null reference exception.
result.Aggs.DateHistogram("jobcounts_by_year").Terms("top_agg")
It seems we have to use something like below.The d2 now has IAggregation
var d1 = result.Aggs.DateHistogram("jobcounts_by_year").Items;
var d2 =(TermsAggregator)d1[0].Aggregations["top_agg"];
But the Aggregation property is not exposing any values.
I'm stuck here. Can someone let me know how can I access buckets inside DateHistogram Buckets using NEST
Regards,
Try this
var dateHistogram = searchResponse.Aggs.DateHistogram("jobcounts_by_year");
foreach (var item in dateHistogram.Items)
{
var bucket = item.Terms("top_agg");
}
Hope this helps.
I'm developing an .NET application using Elastic Search. I used ES River to index the data.
Results (in Sense) look kinda like this:
{
"_index": "musicstore",
"_type": "songs",
"_id": "J2k-NjXjRa-mgWKAq0RMlw",
"_score": 1,
"_source": {
"songID": 42,
"tempo": "andante",
"metrum": "3/4 E8",
"intonation": "F",
"title": "Song",
"archiveSongNumber": "3684",
"Year": 2000,
"Place": "London"
}
},
To access the indexed data I use NEST queries similar to this:
var result = ElasticClient.Search<Song>(s => s.Query(q => q.Term(p => p.title, "Song")));
I'm having a problem that the query doesn't return any results, when I search for a certain field.
For instance when I search for a title, songID, tempo or archiveSongNumber the query works fine and it returns the same results as Sense, but when I search for Year, Place, metrum, etc. the query doesn't return any results, but it should (Sense does and it should).
Queries like these work (and return the right results):
var result = ElasticClient.Search<Song>(s => s.Query(q => q.Term(p => p.title, "Song")));
var result = ElasticClient.Search<Song>(s => s.Query(q => q.Term(p => p.songID, 42)));
var result = ElasticClient.Search<Song>(s => s.Query(q => q.Term(p => p.archiveSongNumber , "3684")));
Queries like these don't return any results (but they should):
var result = ElasticClient.Search<Song>(s => s.Query(q => q.Term(p => p.Place, "London")));
var result = ElasticClient.Search<Song>(s => s.Query(q => q.Term(p => p.Year, 2000)));
What am I doing wrong? Did I mess up when I was indexing data?
Update:
Mapping looks like this:
{
"musicstore": {
"mappings": {
"songs": {
"properties": {
"Year": {
"type": "long"
},
"Place": {
"type": "string"
},
"archiveSongNumber": {
"type": "string"
},
"songID": {
"type": "long"
},
"intonation": {
"type": "string"
},
"metrum": {
"type": "string"
},
"title": {
"type": "string"
},
"tempo": {
"type": "string"
}
}
}
}
}
}
Update 2:
ES river request looks like this:
PUT /_river/songs_river/_meta
{
"type":"jdbc",
"jdbc": {
"driver":"com.microsoft.sqlserver.jdbc.SQLServerDriver",
"url":"jdbc:sqlserver://ip_address:1433;databaseName=database",
"user":"user",
"password":"password",
"strategy":"simple",
"poll":"300s",
"autocommit":true,
"fetchsize":10,
"max_retries":3,
"max_retries_wait":"10s",
"index":"musicstore",
"type":"songs",
"analysis": {
"analyzer" :{
"whitespace" :{
"type" : "whitespace",
"filter":"lowercase"
}
}
},
"sql":"some_sql_query"
}
}
ES client configuration looks like this:
private static ElasticClient ElasticClient
{
get
{
Uri localhost = new Uri("http://localhost:9200");
var setting = new ConnectionSettings(localhost);
setting.SetDefaultIndex("musicstore").MapDefaultTypeNames(d => d.Add(typeof(Song), "songs"));
setting.SetConnectionStatusHandler(c =>
{
if (!c.Success)
throw new Exception(c.ToString());
});
return new ElasticClient(setting);
}
}
From looking at your mapping, the issue here is most likely that all of your fields are being analyzed when indexing, but you are using term queries with NEST, which are not analyzed, meaning they will only find exact matches. If you don't explicitly specify an analyzer in your mappings, Elasticsearch defaults to the standard analyzer.
When you perform a search in Elasticsearch using a query string, like you're doing in Sense: GET _search/?q=Place:London, a query string query is what's being run by Elasticsearch, which is different than a term query.
From your examples though, it doesn't look like you are actually using query string syntax. You probably want a match query instead:
client.Search<Song>(s => s
.Query(q => q
.Match(m => m
.OnField(p => p.Place)
.Query("London")
)
)
);
If you do however want a query string query like the one you're performing with Sense, than you can use QueryString:
client.Search<Song>(s => s
.Query(q => q
.QueryString(qs => qs
.OnFields(p => p.Place)
.Query("London")
)
)
);
Hope that helps. I suggest checking out the getting started guide, specifically the section on exact values vs. full text.
Add "keyword" suffix to your Term Field :
var result = ElasticClient.Search<Song>(s => s
.Query(q => q
.Term(p => p
.Field(x => x.Year.Suffix("keyword")).Value(2000))));
Try it, it will work!