I am querying ElasticSearch with Nest, with this code:
var mustTerms = new List<Func<QueryDescriptor<ElasticCheckIn>, Nest.QueryContainer>>();
var dmaxPrice = maxPrice.HasValue ? (double?)maxPrice.Value : 100000d;
var dminPrice = minPrice.HasValue ? (double?)minPrice.Value : 0d;
mustTerms.Add(mt => mt.Range(rd => rd.OnField("price").LowerOrEquals(dmaxPrice).GreaterOrEquals(dminPrice)));
Func<QueryDescriptor<ElasticCheckIn>, Nest.QueryContainer> queryFunc = qd => qd
.FunctionScore(fsq => fsq
.Query(fsqd => fsqd
.Bool(bqd => bqd
.Must(mustTerms.ToArray())
.Should(shouldTerms.ToArray())))
.Functions(fsd => fsd
.Linear("createDate", fsdfd => fsdfd
.Scale("1d")
.Decay(0.5d)
.Origin(DateTime.UtcNow.ToString("O")))));
Func<SearchDescriptor<ElasticCheckIn>, SearchDescriptor<ElasticCheckIn>> searchFunc = q => q
.Index(_indexName)
.Type(_typeName)
.Query(queryFunc)
.Size(limit);
This code produces query:
{
"query": {
"function_score": {
"functions": [
{
"linear": {
"createDate": {
"origin": "2015-03-16T12:48:14.2660667Z",
"scale": "1d",
"decay": 0.5
}
}
}
],
"query": {
"bool": {
"must": [
{
"range": {
"price": {
"gte": "29.97",
"lte": "67.5"
}
}
}
]
}
}
}
}
}
As you see, Nest renders the double values in the range part as strings:
{
"range": {
"price": {
"gte": "29.97",
"lte": "67.5"
}
}
}
This results in:
NumberFormatException[For input string: "29.97"]
When I correct the double values by hand, elasticsearch correctly return the query results. How can I get Nest to render the double values correctly?
Update:
Mapping for this index:
{
"myindice": {
"mappings": {
"mytype": {
"properties": {
"createDate": {
"type": "date",
"format": "dateOptionalTime"
},
"id": {
"type": "string"
},
/* other fields */
"price": {
"type": "long"
}
/* other fields */
}
}
}
}
}
We did not map the price field as long specifically. Some of the price fields on the docs are integers, some doubles.
This may be a bug in NEST using the syntax that you've choosen, you could create a new issue in their github repo. A more standard syntax would be to create a C# POCO to represent your ES data. something like this:
[ElasticType]
public class Type1
{
public string Id { get; set; }
public DateTime CreationDate { get; set; }
public double Price { get; set; }
}
With that, your query syntax become much, much simpler:
var result2 = ElasticClient.Search<Type1>(s=>s
.Query(q=> q
.Filtered(f=>f
.Filter(ff=> ff
.Range(r=>r
.OnField(rf=>rf.Price)
.GreaterOrEquals(29.97)
.LowerOrEquals(67.5)
)
)
)
)
);
Related
Need to get 10 items of each group. Order By Id decending and Group By Category
public class Item
{
public string Id { get; set ; }
public string Name { get ; set ; }
public string Category { get ; set ; }
}
I tryied something like this.
var data = Collection.AsQueryable()
.OrderByDescending(o=> o.Id)
.GroupBy(x => x.Category)
.Select(g => new { GroupName = g.Key, Items =
g.Take(10).ToList() });
But got exception like this
System.NotSupportedException: Specified method is not supported. at
MongoDB.Driver.Linq.Processors.AccumulatorBinder.GetAccumulatorArgument(Expression
node)
Use Aggregation Framework instead. MongoDB LINQ provider is built on top of it.
You can get 10 items of each group ordered by Id descending with the following query:
db.items.aggregate([
{ "$sort": { "Id": -1 } },
{ "$group": { "_id": "$Category", "Items": { "$push": "$$ROOT" } } },
{ "$project": { "_id": 0, "GroupName": "$_id", "Items": { "$slice": ["$Items", 10] } } }
])
C# code will be as follows:
// models
public class Item
{
public string Id { get; set; }
public string Name { get; set; }
public string Category { get; set; }
}
public class ItemsGroup
{
public string GroupName { get; set; }
public Item[] Items { get; set; }
}
// query
var collection = db.GetCollection<Item>("Items");
IAggregateFluent<ItemsGroup> result = collection.Aggregate()
.SortByDescending(o => o.Id)
.Group(BsonDocument.Parse("{ _id: '$Category', Items: { '$push': '$$ROOT'}}"))
.Project<ItemsGroup>(BsonDocument.Parse("{ _id: 0, GroupName: '$_id', Items: { $slice: ['$Items', 10]}}"));
List<ItemsGroup> groups = result.ToList();
This query, however, may have a problem. If there are thousands of items for each group, I guess the group stage will keep them all in the Items temporary array, while we just want 10.
If the number of keys being grouped is not large then it would be better to do $lookup for each group instead of pushing everything into one array and then $slicing it. This can be achieved by the following query:
aggregate([
{ "$group": { "_id": "$Category" } },
{ "$lookup": {
"from": "Items",
"let": { "c": "$_id" },
"as": "Items"
"pipeline": [
{ "$match": { "$expr": { "$eq": ["$Category", "$$c"] } } },
{ "$sort": { "Id": -1 } },
{ "$limit": 10 }
],
}
},
{ "$project": { "_id": 0, "GroupName": "$_id", "Items": 1 } }
])
In C# code, it would be like:
BsonArray pipeline = new BsonArray
{
BsonDocument.Parse("{ $match: { $expr: { $eq: ['$Category', '$$c']} } }"),
BsonDocument.Parse("{ $sort: { Id: -1 } }"),
BsonDocument.Parse("{ $limit: 10 }")
};
BsonDocument lookup = new BsonDocument("$lookup",
new BsonDocument("from", "Items")
.Add("let", new BsonDocument("c", "$_id"))
.Add("pipeline", pipeline)
.Add("as", "Items")
);
IAggregateFluent<ItemsGroup> result = collection.Aggregate()
.Group(BsonDocument.Parse("{ _id: '$Category' }"))
.AppendStage<object>(lookup)
.Project<ItemsGroup>("{ _id: 0, GroupName: '$_id', Items: 1 }");
List<ItemsGroup> groups = result.ToList();
I have a table of logs records and I want to conduct a simple search by date.
For example, I wanted to search all the queries before 01.06.2019 00:00:00 (mm.DD.yyyy hh:mm:ss) and I wrote this query:
var query = client.Search<SearchEventDto>(s => s
.AllIndices()
.AllTypes()
.Query(q => q
.MatchAll() && +q
.DateRange(r =>r
.Field(f => f.timestamp)
.LessThanOrEquals(new DateTime(2019,06,01, 0, 0, 0))
)
)
);
My Dto looks like this:
public class SearchEventDto : IDto
{
[KendoColumn(Hidden = true, Editable = true)]
public string id { get; set; }
[KendoColumn(Order = 2, DisplayName = "Level")]
public string level { get; set; }
[KendoColumn(Order = 4, DisplayName = "Message")]
public string message { get; set; }
[KendoColumn(Hidden = true)]
public string host { get; set; }
[KendoColumn(Order = 3, DisplayName = "Source")]
public string src { get; set; }
[KendoColumn(Order = 1, DisplayName = "Timestamp", UIType = UIType.DateTime)]
public DateTime timestamp { get; set; }
[KendoColumn(Hidden = true)]
public DateTime time { get; set; }
}
Unfortunately, it is returning all the records without filtering anything.
Where am I going wrong in this?
Thanks in advance!
PS: ES version: 6.7.0, NEST: 6.8
PS: I have integrated the logs with Nlog. So, now every day it inserts a new index with the date as the name. Here is a mapping for 219-06-28 (I am using the #timestamp):
{
"logstash-2019-06-28": {
"mappings": {
"logevent": {
"properties": {
"#timestamp": {
"type": "date"
},
"host": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"level": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"src": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"time": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}
I'll post what we have figured out in comments as an answer as I think there are couple things which could be improved to increase performance and readability.
Solution:
Query from the question was using .Field(f => f.timestamp) which was translated by NEST to use timestamp field not #timestamp. Simple change to .Field("#timestamp") would resolve the problem as this is the proper field name in index mapping.
{
"logstash-2019-06-28": {
"mappings": {
"logevent": {
"properties": {
"#timestamp": {
"type": "date"
},
..
}
}
}
}
}
We could also mark timestamp property with PropertyName attribute to tell NEST to use #timestamp as a name instead of timestamp
public class SearchEventDto : IDto
{
[KendoColumn(Order = 1, DisplayName = "Timestamp", UIType = UIType.DateTime)]
[PropertyName("#timestamp")]
public DateTime timestamp { get; set; }
}
and query
var query = client.Search<SearchEventDto>(s => s
.AllIndices()
.AllTypes()
.Query(q => q
.MatchAll() && +q
.DateRange(r =>r
.Field(f => f.timestamp)
.LessThanOrEquals(new DateTime(2019,06,01, 0, 0, 0))
)
)
);
would be just working as well.
Improvements:
Query only specific indices:
var query = client.Search<SearchEventDto>(s => s
.AllIndices()
.AllTypes()
..
By using AllIndices() we are telling elasticsearch to try to gather documents from all of the indices, we could change it a little bit to query only indices with logs data:
var query = client.Search<SearchEventDto>(s => s
.Index("logstash-*")
.Type("logevent")
..
Use filter context for date range filter:
.Query(q => q.Bool(b => b.Filter(f => f.DateRange(..))))
This way your query should be faster as it doesn't care about calculating search relevance score. You can read more about it here.
Hope that helps.
I am new to Elasticsearch with .NET using NEST.
I am trying to execute a simple search to match all and interested with only few properties. I am not able to get the values for almost all the fields in the source. all shows as null value
The index already exists in elasticsearch.
I have a class representing the type.
public class DocType
{
public long CommunicationsDate { get; set; }
public string ControlNumber { get; set; }
public string Kind { get; set; }
public string PrimaryCommuncationsName { get; set; }
public float RiskScore { get; set; }
}
and my mapping is:
PUT relativity
{
"mappings": {
"doctype": {
"properties": {
"comms_date": {
"type": "date"
},
"control_number": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"kind": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"primary_comms_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"risk_score": {
"type": "float"
}
}
}
}
}
The following query returns Hits count correctly but the values are null except for the Kind property. Not sure what am i doing wrong here. Is this because the property names are different in c# class or something else?
return await _connection.Get<ElasticClient>().SearchAsync<DocType>(s =>
{
var searchDescriptor = s.Index("relativity")
.Type("DocType")
.Size(100)
.Source(sf => sf
.Includes(i => i
.Fields(
f => f.ControlNumber,
f => f.PrimaryCommuncationsName,
f => f.RiskScore,
f => f.Kind,
f => f.CommunicationsDate
)
)
);
}
Property need to have the same name in order to nest to map it correctly with your es index.
You can use attribute in your c# class file to change the mappings if you want to have different name on c# side.
You can use fluent mapping too.
https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/attribute-mapping.html
Hope it's help
++
I send time for this problem 2 days but can't fix it. Please advise.
class Test_Market
{
public string x_ric { get; set; }
public string date { get; set; }
public string timestamp { get; set; }
public string rdn_code { get; set; }
public Dictionary<string, string> fids { get; set; }
}
Example JSON in ES.
"hits": {
"total": 1,
"max_score": 0.6931472,
"hits": [
{
"_index": "test-market-b",
"_type": "test_market",
"_id": "AWTKqb0e8MENF1jasgCD",
"_score": 0.6931472,
"_source": {
"x_ric": "ABC",
"date": "07-24-2018",
"timestamp": "07-24-2018 05:01:17",
"rdn_code": "XYZ",
"fids": {
"BID": "555",
"ASK": "555",
"BIDSIZE": "555",
"ASKSIZE": "555"
}
}
}
]
}
I have no problem about add data to Elasticsearch, but face the issue about sort/query/search.
This code is no problem.
var node = new Uri("URL");
var ESsettings = new ConnectionSettings(node);
var client = new ElasticClient(ESsettings);
var response = client.Search<Test_Market>(s => s
.Index("test-market-b")
.Type("test_market").Size(10)
);
I can get value of timestamp using
var myData = new List<object>();
foreach (var hit in response.Documents)
{
myData.Add(hit.timestamp);
}
Below are I got it no value.
Sorting to find one is the latest timestamp.
var responseXX = client.Search<Test_Market>(s => s
.Index("test-market-b")
.Type("test_market")
.Size(1)
.Sort(sort => sort.OnField( t=>t.timestamp).Descending())
);
Querying
var result = client.Search<Test_Market>(s => s
.Query(q=>q
.Term("date","07-24-2018")
)
);
UPDATED !!!! I can fix this my problem now.
Query AND Sort
var responseXX = client.Search<dynamic>(s => s
.Query(q => q
.Match(m => m.OnField("x_ric").Query("AAA")))
.Sort(sort => sort.OnField("timestamp").Descending())
);
var products = responseXX.Documents.ToList();
First of all, post api in ES before insert data into ES.
PUT test-market-b
{
"mappings": {
"test_market": {
"properties": {
"date": {
"type": "date",
"format": "MM-dd-yyyy"
},
"timestamp": {
"type": "date",
"format": "MM-dd-yyyy HH:mm:ss"
}
}
}
}
}
I've a json query coming from gui. It has filtered query parts in it. I want to extend filtered query with date range filter How can I do that using nest. I tried a method (QueryTest method) like below, but failed.
public SearchDescriptor<dynamic> GetSearchDescriptor(string queryJson = null)
{
var searchDesc = new SearchDescriptor<dynamic>();
if (string.IsNullOrEmpty(queryJson))
{
return searchDesc;
}
searchDesc = _nestClient.Serializer.Deserialize<SearchDescriptor<dynamic>>(new MemoryStream(Encoding.UTF8.GetBytes(queryJson)));
return searchDesc;
}
public string ConvertToJsonQuery(SearchDescriptor<dynamic> searchDescriptor)
{
return Encoding.UTF8.GetString(_nestClient.Serializer.Serialize(searchDescriptor, SerializationFormatting.Indented));
}
public void QueryTest()
{
var query = #"{
""query"": {
""filtered"": {
""filter"": {
""bool"": {
""must"": [
{
""term"": {
""display_name"": ""NBLEYLEK$""
}
}
]
}
}
}
}
}";
var searchDescriptor = GetSearchDescriptor(query).AllTypes();
var queryLastA = ConvertToJsonQuery(searchDescriptor);
searchDescriptor.Query(x => x.Filtered(y => y.Filter(z => z.Range(t => t.OnField("sign_time").Greater(DateTime.Now)))));
var queryLastB = ConvertToJsonQuery(searchDescriptor);
}
If I look at values of queryLastA and queryLastB, I see as:
queryLastA =>
{
"query": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"term": {
"display_name": "NBLEYLEK$"
}
}
]
}
}
}
}
}
queryLastB =>
{
"query": {
"filtered": {
"filter": {
"range": {
"sign_time": {
"gt": "2015-08-25T16:13:56.694"
}
}
}
}
}
}
The desired query is :
{
"query": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"term": {
"display_name": "NBLEYLEK$"
}
},
{
"range": {
"sign_time": {
"gt": "2015-08-25T16:13:56.694"
}
}
}
]
}
}
}
}
}
I expect from nest to append new filter under old one, but it ignores old filter (compare queryLastB with queryLastA). Any advice ?
Currently in nest if we give new filter, it overrides the old filter. I had the same problem. I did string operations on query and converted it into c# class and applied filter from scratch using nest. This worked out for me.