We Keep Coding

Match exact phrase using elasticsearch Nest

I'm using Nest to build a query. I have a class called Event which has a property called Location and a property called Address (a string). So the indices look something like this:
"id" : "something1",
"location": {
"address": "The Club",
}
"id" : "something2",
"location": {
"address": "The Hole",
}
I want to create a query where the user types "The Club" (the location variable) and it only retrieves the first item that has "The Club" as the address
This is my current query
var skip = (page - 1) * rows;
var searchResponse = await _elasticClient.SearchAsync<Event>(s => s
.From(skip)
.Size(rows)
.Query(q => q
.QueryString(qs => qs
.Fields(fields => fields
.Field(f => f.Location.Address)
)
.Analyzer("keyword")
.Query("*" + (location ?? string.Empty) + "*")
)
));
Currently, the query retrieves both objects since it finds "The". Do I need to add something to my index previously or what should I do?
For the record I added this when creating the index but it has no effect
client.Indices.Create(eventsIndex, c => c.Map<Event>(m => m
.Properties(p => p
.Text(t => t
.Name(n => n.Location.Address)
.Analyzer("keyword"))
)
));

I have two suggestions:
1 - Add the stop filter (default language english) that way you won't produce tokens like the term "the", all stopwords will be removed. Now you will only match the term "hotel", the term "the" will be filtered out.
POST _analyze
{
"tokenizer": "standard",
"filter": ["stop"],
"text": ["the hotel"]
}
Token
{
"tokens": [
{
"token": "hotel",
"start_offset": 4,
"end_offset": 9,
"type": "<ALPHANUM>",
"position": 1
}
]
}
2 - Use the default_operator AND in Query String query.

Nest search query returning different results from the Elastic query DSL

I have an ElasticSearch index called challenges, which contains objects of type Challenge.
When I execute the following filter query in the Kibana console, it returns the 9 results, which are correct.
GET challenges/_search
{
"query": {
"bool": {
"filter": [
{
"term": {
"type": "Orphan"
}
}
]
}
}
}
However, the following query from the Nest client, returns zero hits:
var challenges = await _client.SearchAsync<Challenge>(s => s
.Query(q => +q
.Term(t => t.Type, Models.Enums.ChallengeType.Orphan)
)
);
I've also tried the following variation, to no avail:
var challenges = await _client.SearchAsync<Challenge>(s => s
.Query(q => q
.Bool(b => b
.Filter(f => f
.Term(t => t
.Field(f => f.Type)
.Value(challengeType)
)
)
)
)
);
The type property on which I'm filtering, is an enum with the following values:
public enum ChallengeType
{
SixDimensions,
Intro,
Normal,
UserCreated,
Orphan,
Youmate
}
and is stored as a keyword in the index.
An example object that is actually in the index:
{
"id": "3bce0ce1-9676-4858-b165-1442a443bf5a",
"icon": "water-bottle.png",
"index": 0,
"default-time": "09:00",
"default-days": [
"Saturday",
"Monday",
"Wednesday"
],
"default-repetitions": 3,
"category": "A",
"title": {
"Persian": "آب خوردن"
},
"dimension": "Physical",
"type": "Orphan",
"id-package": "00000000-0000-0000-0000-000000000000",
"intro-pages": [ ],
"date-created": "2020-10-14T12:39:21.8427517+03:30",
"notify": true,
"template": 0,
"belongs-to-user": "00000000-0000-0000-0000-000000000000",
"active": false
}
Do you have any suggestions as to why the results from the console differ from when it is executed from the Nest client?

The enum should be marked with StringEnumAttribute to serialize as a string
[StringEnum]
public enum ChallengeType
{
SixDimensions,
Intro,
Normal,
UserCreated,
Orphan,
Youmate
}
Without this, the enum will be serialized as its underlying integer value.

Aggregation with a term and a date range

I'm very new to ElasticSearch, and I'm trying to make an aggregation, but can't seem to get it right.
I have some data in an ElasticSearch index that looks like this:
{
"customerId": "example_customer",
"request": {
"referer": "https://example.org",
}
"#timestamp": "2020-09-29T14:14:00.000Z"
}
My mapping:
{
"mappings": {
"properties": {
"customerId": { "type": "keyword" },
"request": {
"properties": {
"referer": { "type": "keyword" }
}
}
}
}
}
And I'm trying to get the referers that appear the most frequently for a specific customer in a date range. I could make the filter for the customer like this:
var result = await _client.SearchAsync<InsightRecord>(s =>
s.Aggregations(
a => a
.Filter("customer", customer =>
customer.Filter(q => q.Term(ir => ir.CustomerId, customerId)))
.Terms("top_referer", ts => ts.Field("request.referer"))
)
);
return result.Aggregations.Terms("top_referer").Buckets
.Select(bucket => new TopReferer { Url = bucket.Key, Count = bucket.DocCount ?? 0})
Now I want to narrow this down to a specific time range. This is what I have so far:
var searchDescriptor = s.Aggregations(a =>
a.Filter("customer", customer =>
customer.Filter(q =>
q.Bool(b =>
b.Must(
f2 => f2.DateRange(date => date.GreaterThanOrEquals(from).LessThanOrEquals(to)),
f1 => f1.Term(ir => ir.CustomerId, customerId)
)
)
)
)
.Terms("top_referers", ts => ts.Field("request.referer"))
);
The problem is that the date filter doesn't get included in the query, it translates to this JSON:
{
"aggs": {
"customer": {
"filter": {
"bool": {
"filter": [{
"term": {
"customerId": {
"value": "example_customer"
}
}
}
]
}
}
},
"top_referers": {
"terms": {
"field": "request.referer"
}
}
}
}
I tried ordering them differently, but it didn't help. It's always the customer filter that will appear in the JSON, and the date range is skipped. I also saw some people using a query combined with an aggregation, but I feel like I should be able to do this using the aggregation alone. Is this possible? What am I doing wrong in my query that the range doesn't show up in the JSON?

The issue is that the date range query does not specify a field
f2 => f2.DateRange(date => date.GreaterThanOrEquals(from).LessThanOrEquals(to)),
Add a field to this query
f2 => f2.DateRange(date => date
.Field("#timestamp")
.GreaterThanOrEquals(from)
.LessThanOrEquals(to)
),
In addition, for the filter aggregation to apply to the terms aggregation, the terms aggregation needs to be a sub aggregation of the filter aggregation. So it would be something like
var customerId = "foo";
var from = "now-365d";
var to = "now";
var result = await _client.SearchAsync<InsightRecord>(s => s
.Aggregations(a => a
.Filter("customer", customer => customer
.Filter(q => q
.Bool(b => b
.Must(
f2 => f2.DateRange(date => date
.Field("#timestamp")
.GreaterThanOrEquals(from)
.LessThanOrEquals(to)
),
f1 => f1.Term(ir => ir.CustomerId, customerId)
)
)
)
.Aggregations(aa => aa
.Terms("top_referers", ts => ts.Field("request.referer"))
)
)
)
);
Rather than specifying the date range and term queries using a filter aggregation though, I'd be inclined to specify them as the query to the search request. The query is taken into account when calculating aggregations. A filter aggregation is useful when you want to query on a dataset but run an aggregation only on a subset of the dataset e.g. if you were searching across all customers but then wanted to run an aggregation only on a subset of the customers. In practice, for this particular example, the outcome should be the same whether the query is specified as the query part of a search request, or as a filter aggregation with the terms aggregation as a sub aggregation, but the former is perhaps a little easier to get the results from.
Specified as the query would look something like
var result = await _client.Search<InsightRecord>(s => s
.Query(q => q
.Bool(b => b
.Must(
f2 => f2.DateRange(date => date
.Field("#timestamp")
.GreaterThanOrEquals(from)
.LessThanOrEquals(to)
),
f1 => f1.Term(ir => ir.CustomerId, customerId)
)
)
)
.Aggregations(aa => aa
.Terms("top_referers", ts => ts.Field("request.referer"))
)
);
Further, there's a couple more things we can do
Since we're only interested in the results of the terms aggregation and not the search hits, we can specify .Size(0) to not bother returning search hits on the response.
The bool query can be more succinctly expressed by combining queries with operator overloading, and since both queries are predicates (a document either matches the query or it doesn't), we can specify both in a filter clause to omit scoring. The final query then is something like
var result = await _client.SearchAsync<InsightRecord>(s => s
.Size(0)
.Query(q => +q
.DateRange(date => date
.Field("#timestamp")
.GreaterThanOrEquals(from)
.LessThanOrEquals(to)
) && +q
.Term(ir => ir.CustomerId, customerId)
)
.Aggregations(aa => aa
.Terms("top_referers", ts => ts.Field("request.referer"))
)
);
which generates the query
{
"aggs": {
"top_referers": {
"terms": {
"field": "request.referer"
}
}
},
"query": {
"bool": {
"filter": [{
"range": {
"#timestamp": {
"gte": "now-365d",
"lte": "now"
}
}
}, {
"term": {
"customerId": {
"value": "foo"
}
}
}]
}
},
"size": 0
}
The terms aggregation buckets can be accessed as expressed in your question.

Querying a subfield in DocumentDB to sort and get the latest date

To add/expound on my recent question
Below are DocumentDB collections: "delivery"
{
"doc": [
{
"docid": "15",
"deliverynum": "123",
"text": "txxxxxx",
"date": "2019-07-18T12:37:58Z"
},
{
"docid": "17",
"deliverynum": "999",
"text": "txxxxxx",
"date": "2018-07-18T12:37:58Z"
}
],
"id": "123",
"cancelled": false
},
{
"doc": [
{
"docid": "16",
"deliverynum": "222",
"text": "txxxxxx",
"date": "2019-07-18T12:37:58Z"
},
{
"docid": "17",
"deliverynum": "999",
"text": "txxxxxx",
"date": "2019-07-20T12:37:58Z"
}
],
"id": "124",
"cancelled": false
}
I need to search the deliverynum=999 w/ the latest date to get the "id", which in the case above is "124" because it has the latest "date" in the "doc" w/ deliverynum=999.
I was going to do:
var list = await collection.Find(filter).Project(projection).ToListAsync();
then do a LINQ to sort, but the problem here is my projection change the list from my Model Class to BsonDocument even if my projection included all the fields.
Was looking for a way to either get just needed "id" or get the single document.

i believe the following will do the trick. (if i understood your requirement correctly)
var result = collection.Find(x => x.docs.Any(d => d.deliverynum == 999))
.Sort(Builders<Record>.Sort.Descending("docs.date"))
.Limit(1)
.Project(x=>x.Id) //remove this to get back the whole record
.ToList();
update: strongly typed solution
var result = collection.AsQueryable()
.Where(r => r.docs.Any(d => d.deliverynum == 999))
.SelectMany(r => r.docs, (r, d) => new { r.Id, d.date })
.OrderByDescending(x => x.date)
.Take(1)
.Select(x => x.Id)
.ToArray();

Searching using NEST does not return results, when querying on certain fields

I'm developing an .NET application using Elastic Search. I used ES River to index the data.
Results (in Sense) look kinda like this:
{
"_index": "musicstore",
"_type": "songs",
"_id": "J2k-NjXjRa-mgWKAq0RMlw",
"_score": 1,
"_source": {
"songID": 42,
"tempo": "andante",
"metrum": "3/4 E8",
"intonation": "F",
"title": "Song",
"archiveSongNumber": "3684",
"Year": 2000,
"Place": "London"
}
},
To access the indexed data I use NEST queries similar to this:
var result = ElasticClient.Search<Song>(s => s.Query(q => q.Term(p => p.title, "Song")));
I'm having a problem that the query doesn't return any results, when I search for a certain field.
For instance when I search for a title, songID, tempo or archiveSongNumber the query works fine and it returns the same results as Sense, but when I search for Year, Place, metrum, etc. the query doesn't return any results, but it should (Sense does and it should).
Queries like these work (and return the right results):
var result = ElasticClient.Search<Song>(s => s.Query(q => q.Term(p => p.title, "Song")));
var result = ElasticClient.Search<Song>(s => s.Query(q => q.Term(p => p.songID, 42)));
var result = ElasticClient.Search<Song>(s => s.Query(q => q.Term(p => p.archiveSongNumber , "3684")));
Queries like these don't return any results (but they should):
var result = ElasticClient.Search<Song>(s => s.Query(q => q.Term(p => p.Place, "London")));
var result = ElasticClient.Search<Song>(s => s.Query(q => q.Term(p => p.Year, 2000)));
What am I doing wrong? Did I mess up when I was indexing data?
Update:
Mapping looks like this:
{
"musicstore": {
"mappings": {
"songs": {
"properties": {
"Year": {
"type": "long"
},
"Place": {
"type": "string"
},
"archiveSongNumber": {
"type": "string"
},
"songID": {
"type": "long"
},
"intonation": {
"type": "string"
},
"metrum": {
"type": "string"
},
"title": {
"type": "string"
},
"tempo": {
"type": "string"
}
}
}
}
}
}
Update 2:
ES river request looks like this:
PUT /_river/songs_river/_meta
{
"type":"jdbc",
"jdbc": {
"driver":"com.microsoft.sqlserver.jdbc.SQLServerDriver",
"url":"jdbc:sqlserver://ip_address:1433;databaseName=database",
"user":"user",
"password":"password",
"strategy":"simple",
"poll":"300s",
"autocommit":true,
"fetchsize":10,
"max_retries":3,
"max_retries_wait":"10s",
"index":"musicstore",
"type":"songs",
"analysis": {
"analyzer" :{
"whitespace" :{
"type" : "whitespace",
"filter":"lowercase"
}
}
},
"sql":"some_sql_query"
}
}
ES client configuration looks like this:
private static ElasticClient ElasticClient
{
get
{
Uri localhost = new Uri("http://localhost:9200");
var setting = new ConnectionSettings(localhost);
setting.SetDefaultIndex("musicstore").MapDefaultTypeNames(d => d.Add(typeof(Song), "songs"));
setting.SetConnectionStatusHandler(c =>
{
if (!c.Success)
throw new Exception(c.ToString());
});
return new ElasticClient(setting);
}
}

From looking at your mapping, the issue here is most likely that all of your fields are being analyzed when indexing, but you are using term queries with NEST, which are not analyzed, meaning they will only find exact matches. If you don't explicitly specify an analyzer in your mappings, Elasticsearch defaults to the standard analyzer.
When you perform a search in Elasticsearch using a query string, like you're doing in Sense: GET _search/?q=Place:London, a query string query is what's being run by Elasticsearch, which is different than a term query.
From your examples though, it doesn't look like you are actually using query string syntax. You probably want a match query instead:
client.Search<Song>(s => s
.Query(q => q
.Match(m => m
.OnField(p => p.Place)
.Query("London")
)
)
);
If you do however want a query string query like the one you're performing with Sense, than you can use QueryString:
client.Search<Song>(s => s
.Query(q => q
.QueryString(qs => qs
.OnFields(p => p.Place)
.Query("London")
)
)
);
Hope that helps. I suggest checking out the getting started guide, specifically the section on exact values vs. full text.

Add "keyword" suffix to your Term Field :
var result = ElasticClient.Search<Song>(s => s
.Query(q => q
.Term(p => p
.Field(x => x.Year.Suffix("keyword")).Value(2000))));
Try it, it will work!