We Keep Coding

ElasticSearch Mapping With NEST 6.6.0

I'm new with ElasticSearch and encountered some problem while mapping my documents into ES Index.
My document structure is
public class DocumentItem
{
public string Id { get; set; }
public DocumentType DocType { get; set; }
public Dictionary<string, string> Props { get; set; } = new Dictionary<string, string>();
}
And here's my mapping
var indexResponseFiles = dClient.CreateIndex(sedIndex, c => c
.InitializeUsing(indexConfig)
.Mappings(m => m
.Map<DocumentItem>(mp => mp.AutoMap()
)
));
As u see i'm trying to map a DICTIONARY type. In every document the keys of dictionary are different.
My goal is to set my custom analyzer to all text values of dictionary. I have no idea how to do this.

Dynamic templates feature will help you here. You can configure the dynamic template for all string fields below props object which will create a mapping for such fields with certain analyzer.
Here is the example with creating text fields with english analyzer
var createIndexResponse = await client.CreateIndexAsync("index_name",
c => c.Mappings(m => m
.Map<Document>(mm => mm.DynamicTemplates(dt => dt
.DynamicTemplate("props_fields", t => t
.PathMatch("props.*")
.MatchMappingType("string")
.Mapping(dm => dm.Text(text => text.Analyzer("english"))))))));
and here is the mapping after indexing following document
var document = new Document { Id = "1", Name = "name"};
document.Props.Add("field1", "value");
var indexDocument = await client.IndexDocumentAsync(document);
mapping
{
"index_name": {
"mappings": {
"document": {
"dynamic_templates": [
{
"props_fields": {
"path_match": "props.*",
"match_mapping_type": "string",
"mapping": {
"analyzer": "english",
"type": "text"
}
}
}
],
"properties": {
"id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"props": {
"properties": {
"field1": {
"type": "text",
"analyzer": "english"
}
}
}
}
}
}
}
}
Hope that helps.

Elasticsearch NEST PUT Mapping to Add Field / Property

I am using the NEST Suggest.Completion query to provide suggestive search. I already have data in the index and wanted to add a new field "IsActive" to allow me to disable certain documents from appearing in the suggest.
I thought the NEST Map<> method would add the new field to all existing documents in the index when run, but it does not. Is there some way to make it work like that?
I am using Elasticsearch 6.8.0.
My Object with the new field
[ElasticsearchType(
IdProperty = "search"
)]
public class SearchCompletion
{
public string search { get; set; }
/// <summary>
/// Use this field for aggregations and sorts
/// </summary>
[Keyword]
public string search_keyword { get; set; }
public bool isActive { get; set; } // <---- This is the new field
/// <summary>
/// To use for sorting results
/// since you can't sort by the Completionfield.Weight
/// property for some reason
/// </summary>
public int weight { get; set; }
public CompletionField suggest { get; set; }
}
Method to Re-Apply Mapping
public static void MapSearchCompletions(ElasticClient client, string index)
{
var mapResponse = client.Map<SearchCompletion>(m => m
.Index(index)
.AutoMap()
); //re-apply the index mapping
}
The PUT request
PUT /local.project.tests.searchcompletions/searchcompletion/_mapping
{
"properties": {
"search": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"search_keyword": {
"type": "keyword"
},
"isActive": {
"type": "boolean"
},
"weight": {
"type": "integer"
},
"suggest": {
"type": "completion"
}
}
}
The result of querying the index after the mapping
GET /local.project.tests.searchcompletions/searchcompletion/_search
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "local.project.tests.searchcompletions",
"_type": "searchcompletion",
"_id": "the",
"_score": 1,
"_source": {
"search": "the",
"search_keyword": "the",
"weight": 1,
"suggest": {
"input": [
"the",
"the"
],
"weight": 1
}
}
}
]
}
}

Yes, updating mapping won't change existing documents. To do so, you can use update_by_query API.
var updateByQueryResponse = await client.UpdateByQueryAsync<Document>(u => u
.Query(q => q.MatchAll())
.Script("ctx._source.isActive = true")
.Refresh());
Here is the full example:
class Program
{
public class Document
{
public int Id { get; set; }
public bool IsActive { get; set; }
}
static async Task Main(string[] args)
{
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var connectionSettings = new ConnectionSettings(pool);
connectionSettings.DefaultIndex("documents");
var client = new ElasticClient(connectionSettings);
var deleteIndexResponse = await client.Indices.DeleteAsync("documents");
var createIndexResponse = await client.Indices.CreateAsync("documents", d => d
.Map(m => m.AutoMap<Document>()));
var indexDocument = await client.IndexDocumentAsync(new Document {Id = 1});
var refreshAsync = client.Indices.RefreshAsync();
var putMappingResponse = await client.MapAsync<Document>(m => m
.AutoMap());
var updateByQueryResponse = await client.UpdateByQueryAsync<Document>(u => u
.Query(q => q.MatchAll())
.Script("ctx._source.isActive = true")
.Refresh());
var response = await client.GetAsync<Document>(1);
Console.WriteLine(response.Source.IsActive);
}
}
Prints:
True
Hope that helps.

C# NEST elasticsearch source filtering returning null for most of the fields

I am new to Elasticsearch with .NET using NEST.
I am trying to execute a simple search to match all and interested with only few properties. I am not able to get the values for almost all the fields in the source. all shows as null value
The index already exists in elasticsearch.
I have a class representing the type.
public class DocType
{
public long CommunicationsDate { get; set; }
public string ControlNumber { get; set; }
public string Kind { get; set; }
public string PrimaryCommuncationsName { get; set; }
public float RiskScore { get; set; }
}
and my mapping is:
PUT relativity
{
"mappings": {
"doctype": {
"properties": {
"comms_date": {
"type": "date"
},
"control_number": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"kind": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"primary_comms_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"risk_score": {
"type": "float"
}
}
}
}
}
The following query returns Hits count correctly but the values are null except for the Kind property. Not sure what am i doing wrong here. Is this because the property names are different in c# class or something else?
return await _connection.Get<ElasticClient>().SearchAsync<DocType>(s =>
{
var searchDescriptor = s.Index("relativity")
.Type("DocType")
.Size(100)
.Source(sf => sf
.Includes(i => i
.Fields(
f => f.ControlNumber,
f => f.PrimaryCommuncationsName,
f => f.RiskScore,
f => f.Kind,
f => f.CommunicationsDate
)
)
);
}

Property need to have the same name in order to nest to map it correctly with your es index.
You can use attribute in your c# class file to change the mappings if you want to have different name on c# side.
You can use fluent mapping too.
https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/attribute-mapping.html
Hope it's help
++

How to search/query/get/sort from ElasticSearch using NEST 1.x with C#

I send time for this problem 2 days but can't fix it. Please advise.
class Test_Market
{
public string x_ric { get; set; }
public string date { get; set; }
public string timestamp { get; set; }
public string rdn_code { get; set; }
public Dictionary<string, string> fids { get; set; }
}
Example JSON in ES.
"hits": {
"total": 1,
"max_score": 0.6931472,
"hits": [
{
"_index": "test-market-b",
"_type": "test_market",
"_id": "AWTKqb0e8MENF1jasgCD",
"_score": 0.6931472,
"_source": {
"x_ric": "ABC",
"date": "07-24-2018",
"timestamp": "07-24-2018 05:01:17",
"rdn_code": "XYZ",
"fids": {
"BID": "555",
"ASK": "555",
"BIDSIZE": "555",
"ASKSIZE": "555"
}
}
}
]
}
I have no problem about add data to Elasticsearch, but face the issue about sort/query/search.
This code is no problem.
var node = new Uri("URL");
var ESsettings = new ConnectionSettings(node);
var client = new ElasticClient(ESsettings);
var response = client.Search<Test_Market>(s => s
.Index("test-market-b")
.Type("test_market").Size(10)
);
I can get value of timestamp using
var myData = new List<object>();
foreach (var hit in response.Documents)
{
myData.Add(hit.timestamp);
}
Below are I got it no value.
Sorting to find one is the latest timestamp.
var responseXX = client.Search<Test_Market>(s => s
.Index("test-market-b")
.Type("test_market")
.Size(1)
.Sort(sort => sort.OnField( t=>t.timestamp).Descending())
);
Querying
var result = client.Search<Test_Market>(s => s
.Query(q=>q
.Term("date","07-24-2018")
)
);
UPDATED !!!! I can fix this my problem now.
Query AND Sort
var responseXX = client.Search<dynamic>(s => s
.Query(q => q
.Match(m => m.OnField("x_ric").Query("AAA")))
.Sort(sort => sort.OnField("timestamp").Descending())
);
var products = responseXX.Documents.ToList();
First of all, post api in ES before insert data into ES.
PUT test-market-b
{
"mappings": {
"test_market": {
"properties": {
"date": {
"type": "date",
"format": "MM-dd-yyyy"
},
"timestamp": {
"type": "date",
"format": "MM-dd-yyyy HH:mm:ss"
}
}
}
}
}

Nest renders range boundaries as string instead of double

I am querying ElasticSearch with Nest, with this code:
var mustTerms = new List<Func<QueryDescriptor<ElasticCheckIn>, Nest.QueryContainer>>();
var dmaxPrice = maxPrice.HasValue ? (double?)maxPrice.Value : 100000d;
var dminPrice = minPrice.HasValue ? (double?)minPrice.Value : 0d;
mustTerms.Add(mt => mt.Range(rd => rd.OnField("price").LowerOrEquals(dmaxPrice).GreaterOrEquals(dminPrice)));
Func<QueryDescriptor<ElasticCheckIn>, Nest.QueryContainer> queryFunc = qd => qd
.FunctionScore(fsq => fsq
.Query(fsqd => fsqd
.Bool(bqd => bqd
.Must(mustTerms.ToArray())
.Should(shouldTerms.ToArray())))
.Functions(fsd => fsd
.Linear("createDate", fsdfd => fsdfd
.Scale("1d")
.Decay(0.5d)
.Origin(DateTime.UtcNow.ToString("O")))));
Func<SearchDescriptor<ElasticCheckIn>, SearchDescriptor<ElasticCheckIn>> searchFunc = q => q
.Index(_indexName)
.Type(_typeName)
.Query(queryFunc)
.Size(limit);
This code produces query:
{
"query": {
"function_score": {
"functions": [
{
"linear": {
"createDate": {
"origin": "2015-03-16T12:48:14.2660667Z",
"scale": "1d",
"decay": 0.5
}
}
}
],
"query": {
"bool": {
"must": [
{
"range": {
"price": {
"gte": "29.97",
"lte": "67.5"
}
}
}
]
}
}
}
}
}
As you see, Nest renders the double values in the range part as strings:
{
"range": {
"price": {
"gte": "29.97",
"lte": "67.5"
}
}
}
This results in:
NumberFormatException[For input string: "29.97"]
When I correct the double values by hand, elasticsearch correctly return the query results. How can I get Nest to render the double values correctly?
Update:
Mapping for this index:
{
"myindice": {
"mappings": {
"mytype": {
"properties": {
"createDate": {
"type": "date",
"format": "dateOptionalTime"
},
"id": {
"type": "string"
},
/* other fields */
"price": {
"type": "long"
}
/* other fields */
}
}
}
}
}
We did not map the price field as long specifically. Some of the price fields on the docs are integers, some doubles.

This may be a bug in NEST using the syntax that you've choosen, you could create a new issue in their github repo. A more standard syntax would be to create a C# POCO to represent your ES data. something like this:
[ElasticType]
public class Type1
{
public string Id { get; set; }
public DateTime CreationDate { get; set; }
public double Price { get; set; }
}
With that, your query syntax become much, much simpler:
var result2 = ElasticClient.Search<Type1>(s=>s
.Query(q=> q
.Filtered(f=>f
.Filter(ff=> ff
.Range(r=>r
.OnField(rf=>rf.Price)
.GreaterOrEquals(29.97)
.LowerOrEquals(67.5)
)
)
)
)
);