I am using the NEST Suggest.Completion query to provide suggestive search. I already have data in the index and wanted to add a new field "IsActive" to allow me to disable certain documents from appearing in the suggest.
I thought the NEST Map<> method would add the new field to all existing documents in the index when run, but it does not. Is there some way to make it work like that?
I am using Elasticsearch 6.8.0.
My Object with the new field
[ElasticsearchType(
IdProperty = "search"
)]
public class SearchCompletion
{
public string search { get; set; }
/// <summary>
/// Use this field for aggregations and sorts
/// </summary>
[Keyword]
public string search_keyword { get; set; }
public bool isActive { get; set; } // <---- This is the new field
/// <summary>
/// To use for sorting results
/// since you can't sort by the Completionfield.Weight
/// property for some reason
/// </summary>
public int weight { get; set; }
public CompletionField suggest { get; set; }
}
Method to Re-Apply Mapping
public static void MapSearchCompletions(ElasticClient client, string index)
{
var mapResponse = client.Map<SearchCompletion>(m => m
.Index(index)
.AutoMap()
); //re-apply the index mapping
}
The PUT request
PUT /local.project.tests.searchcompletions/searchcompletion/_mapping
{
"properties": {
"search": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"search_keyword": {
"type": "keyword"
},
"isActive": {
"type": "boolean"
},
"weight": {
"type": "integer"
},
"suggest": {
"type": "completion"
}
}
}
The result of querying the index after the mapping
GET /local.project.tests.searchcompletions/searchcompletion/_search
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "local.project.tests.searchcompletions",
"_type": "searchcompletion",
"_id": "the",
"_score": 1,
"_source": {
"search": "the",
"search_keyword": "the",
"weight": 1,
"suggest": {
"input": [
"the",
"the"
],
"weight": 1
}
}
}
]
}
}
Yes, updating mapping won't change existing documents. To do so, you can use update_by_query API.
var updateByQueryResponse = await client.UpdateByQueryAsync<Document>(u => u
.Query(q => q.MatchAll())
.Script("ctx._source.isActive = true")
.Refresh());
Here is the full example:
class Program
{
public class Document
{
public int Id { get; set; }
public bool IsActive { get; set; }
}
static async Task Main(string[] args)
{
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var connectionSettings = new ConnectionSettings(pool);
connectionSettings.DefaultIndex("documents");
var client = new ElasticClient(connectionSettings);
var deleteIndexResponse = await client.Indices.DeleteAsync("documents");
var createIndexResponse = await client.Indices.CreateAsync("documents", d => d
.Map(m => m.AutoMap<Document>()));
var indexDocument = await client.IndexDocumentAsync(new Document {Id = 1});
var refreshAsync = client.Indices.RefreshAsync();
var putMappingResponse = await client.MapAsync<Document>(m => m
.AutoMap());
var updateByQueryResponse = await client.UpdateByQueryAsync<Document>(u => u
.Query(q => q.MatchAll())
.Script("ctx._source.isActive = true")
.Refresh());
var response = await client.GetAsync<Document>(1);
Console.WriteLine(response.Source.IsActive);
}
}
Prints:
True
Hope that helps.
Related
I'm trying to write my own small API. I found an article on how to implement an API with Mongodb on ASP.NET Core MVC: article. The problem is that when creating a GET request, the object is incorrectly serialized. As I understand it, the standard Json serializer cannot serialize objects of a custom data type.
Here is the file with the model that I am reading from the database. Everything comes from the database correctly, but when I try to send a response, a file with an empty List<Weekday> arrives.
[Serializable]
public class Group
{
[BsonId]
[BsonRepresentation(BsonType.ObjectId)]
public string? Id { get; set; }
public string groupName { get; set; }
public Schedule schedule { get; set; }
public Group()
{
groupName = "UNKNOWN";
schedule = new Schedule();
}
public Group(string groupName)
{
this.groupName = groupName;
schedule = new Schedule();
}
}
[Serializable]
public class Schedule
{
public List<Weekday> week = new List<Weekday>();
}
[Serializable]
public class Weekday
{
public List<DaysSchedule> daysSchedules { get; set; } = new List<DaysSchedule>();
public int dayNumber { get; set; } = 0;
}
public class DaysSchedule
{
public List<string> dates = new List<string>();
public List<Classes> classes = new List<Classes>();
}
public class Classes
{
[BsonElement("ordinal")]
public int ordinal { get; set; }
[BsonElement("name")]
public string name { get; set; }
[BsonElement("teacher")]
public string? teacher { get; set; }
[BsonElement("type")]
public string type { get; set; }
[BsonElement("location")]
public string location { get; set; }
public Classes()
{
ordinal = 0;
name = "UNKNOWN";
type = "UNDEFINED";
location = "UNDEFINED";
}
public Classes(int ordinal, string name, string teacher, string type, string location)
{
this.ordinal = ordinal;
this.name = name;
this.teacher = teacher;
this.type = type;
this.location = location;
}
}
Here is an example of a Json file that I expect. I did not insert the entire file, as it is very large. But the main point is that the Week list contains days of the week, each of which has certain schedule options.
{
"Id": "63ea85829903ec2ab03720d3",
"groupName": "М3О-225Бк-21",
"schedule": {
"week": [
{
"daysSchedules": [
{
"dates": [
"13.02.2023",
"13.02.2023"
],
"classes": [
{
"ordinal": 0,
"name": "Math",
"teacher": "Lyahner",
"type": "PZ",
"location": "64"
},
{
"ordinal": 1,
"name": "Programming",
"teacher": "Lyahner",
"type": "LK",
"location": "84"
},
{
"ordinal": 2,
"name": "OOP",
"teacher": "Lyahner",
"type": "LK",
"location": "29"
},
{
"ordinal": 3,
"name": "OOP",
"teacher": "Vestyak",
"type": "LK",
"location": "33"
}
]
},
{
"dates": [
"13.02.2023",
"13.02.2023"
],
"classes": [
{
"ordinal": 0,
"name": "Phisics",
"teacher": "Lyahner",
"type": "LK",
"location": "97"
},
{
"ordinal": 1,
"name": "Programming",
"teacher": "Lyahner",
"type": "LK",
"location": "78"
},
{
"ordinal": 2,
"name": "Programming",
"teacher": "Sukhno",
"type": "PZ",
"location": "91"
},
{
"ordinal": 3,
"name": "Programming",
"teacher": "Sukhno",
"type": "PZ",
"location": "32"
}
]
},
{
"dates": [
"13.02.2023",
"13.02.2023"
],
"classes": [
{
"ordinal": 0,
"name": "OOP",
"teacher": "Vestyak",
"type": "LK",
"location": "93"
},
{
"ordinal": 1,
"name": "Math",
"teacher": "Lyahner",
"type": "PZ",
"location": "72"
},
{
"ordinal": 2,
"name": "Math",
"teacher": "Vestyak",
"type": "LK",
"location": "70"
},
{
"ordinal": 3,
"name": "Phisics",
"teacher": "Vestyak",
"type": "LK",
"location": "42"
}
]
}
],
"dayNumber": 0
.......................................
Here's what Json I get:
{
"id": "63ea85829903ec2ab03720d3",
"groupName": "М3О-225Бк-21",
"schedule": {}
}
Here is the code of my Controller. Here I don't quite understand how the Http Get and Http Post attributes work. I understand that they configure routing and transfer control to a method for processing and sending a response. But where does the serialization of the Group object take place and how is it passed to the method? I tried to implement serialization explicitly by taking a string variable in the Get method and serializing the object using Newtonsoft.Json, it worked, but I don't think this option is correct and suits me. I would like to know what the problem is? And can I set custom serialization for the Group object so that I can pass and return it from methods with routing attributes?
using Newtonsoft.Json;
using Microsoft.AspNetCore.Mvc;
using ThreeplyWebApi.Services;
using ThreeplyWebApi.Models;
using System.Text.Json;
namespace ThreeplyWebApi.Controllers
{
[ApiController]
[Route("controller")]
public class GroupsController : ControllerBase
{
readonly private GroupsService _groupsService;
public GroupsController(GroupsService schedulesService)
{
_groupsService = schedulesService;
}
[HttpGet]
public async Task<List<Group>> Get() => await _groupsService.GetAsync();
[HttpGet("{groupName}")]
public async Task<ActionResult<Group>> Get(string groupName)
{
var group = await _groupsService.GetAsync(groupName);
if (group == null)
{
return NotFound();
}
return Ok(group);
}
[HttpPost]
public async Task<IActionResult> Post([FromBody]Group newGroup)
{
await _groupsService.CreateAsync(newGroup);
return CreatedAtAction(nameof(Get), new { groupName = newGroup.groupName }, newGroup);
}
}
}
I also provide the code of my service for accessing the database, the methods of which are called by the controller to access the database.
namespace ThreeplyWebApi.Services
{
public class GroupsService
{
private readonly IMongoCollection<Group> _groupsCollection;
public GroupsService(IOptions<GroupsDatabaseSettings> groupDatabaseSettings)
{
var MongoClient = new MongoClient(groupDatabaseSettings.Value.ConnectionString);
var MongoDatabase = MongoClient.GetDatabase(groupDatabaseSettings.Value.DatabaseName);
_groupsCollection = MongoDatabase.GetCollection<Group>(groupDatabaseSettings.Value.GroupsCollectionName);
}
public async Task<List<Group>> GetAsync()
{
return await _groupsCollection.Find<Group>(_ => true).ToListAsync();
}
public async Task<Group> GetAsync(string groupName)
{
return await _groupsCollection.Find(x => x.groupName == groupName).FirstOrDefaultAsync();
}
public async Task CreateAsync(Group group) {
await _groupsCollection.InsertOneAsync(group);
}
public async Task UpdateAsync(string groupName, Group updatedGroup) =>
await _groupsCollection.ReplaceOneAsync(x => x.groupName == groupName, updatedGroup);
public async Task RemoveAsync(string groupName) =>
await _groupsCollection.DeleteOneAsync(x => x.groupName == groupName);
}
}
{
"objects": [
{
"id": 123,
"tracking_datas": [
{
"id": 1,
"polygons": [1,3]
},
{
"id": 2,
"polygons": [3]
},
{
"id": 3,
"polygons": [1,2]
}
]
}
]
}
I have a json file as above. And there is a model that satisfies this json in my NetCore project. I want to get objects containing polygonIds that I have determined with the help of mongodb. How can I do this with c# mongo db?
For example, I have a reference array requiredPolygons: [1,2] and I want to get the data containing these polygon'ids in the tracking data of the objects in the json. The expected result is as follows.
{
"objects":
[
{
"id": 123,
"tracking_datas":[
{
"id": 1,
"polygons": [1,3]
},
{
"id": 3,
"polygons": [1,2]
}
]
}
]
}
public class Test
{
public ObjectId Id { get; set; }
public IEnumerable<Object> objects { get; set; }
[BsonExtraElements]
public BsonDocument UnmappedFields { get; set; } // I'm not sure why it's required, something wrong with mapping configuration,
// but it's a separate question
}
public class Object
{
public int id { get; set; }
public IEnumerable<TrackingData> tracking_datas { get; set; }
}
public class TrackingData
{
public int id { get; set; }
public IEnumerable<int> polygons { get; set; }
[BsonExtraElements]
public BsonDocument UnmappedFields { get; set; } // I'm not sure why it's required, something wrong with mapping configuration,
// but it's a separate question
}
var json = #"{
""objects"": [
{
""id"": 123,
""tracking_datas"": [
{
""id"": 1,
""polygons"": [1,3]
},
{
""id"": 2,
""polygons"": [3]
},
{
""id"": 3,
""polygons"": [1,2]
}
]
}
]
}";
var client = new MongoClient();
var db = client.GetDatabase("so_test");
var coll = db.GetCollection<BsonDocument>("coll");
coll.InsertOne(BsonDocument.Parse(json));
var ids = new[] { 1, 2 };
var typedColl = db.GetCollection<Test>("coll");
var result = typedColl
.Aggregate()
.Project(p =>
new Test
{
Id = p.Id,
objects = p.objects.Select(o =>
new Object
{
id = o.id,
tracking_datas = o.tracking_datas.Where(t => t.polygons.Any(p=>ids.Contains(p)))
})
}
)
.ToList();
Here you go:
db.collection.find({
"objects.tracking_datas.polygons": {
$in: [
1,
2
]
}
})
https://mongoplayground.net/p/MDlIV3YPkZB
I have a table of logs records and I want to conduct a simple search by date.
For example, I wanted to search all the queries before 01.06.2019 00:00:00 (mm.DD.yyyy hh:mm:ss) and I wrote this query:
var query = client.Search<SearchEventDto>(s => s
.AllIndices()
.AllTypes()
.Query(q => q
.MatchAll() && +q
.DateRange(r =>r
.Field(f => f.timestamp)
.LessThanOrEquals(new DateTime(2019,06,01, 0, 0, 0))
)
)
);
My Dto looks like this:
public class SearchEventDto : IDto
{
[KendoColumn(Hidden = true, Editable = true)]
public string id { get; set; }
[KendoColumn(Order = 2, DisplayName = "Level")]
public string level { get; set; }
[KendoColumn(Order = 4, DisplayName = "Message")]
public string message { get; set; }
[KendoColumn(Hidden = true)]
public string host { get; set; }
[KendoColumn(Order = 3, DisplayName = "Source")]
public string src { get; set; }
[KendoColumn(Order = 1, DisplayName = "Timestamp", UIType = UIType.DateTime)]
public DateTime timestamp { get; set; }
[KendoColumn(Hidden = true)]
public DateTime time { get; set; }
}
Unfortunately, it is returning all the records without filtering anything.
Where am I going wrong in this?
Thanks in advance!
PS: ES version: 6.7.0, NEST: 6.8
PS: I have integrated the logs with Nlog. So, now every day it inserts a new index with the date as the name. Here is a mapping for 219-06-28 (I am using the #timestamp):
{
"logstash-2019-06-28": {
"mappings": {
"logevent": {
"properties": {
"#timestamp": {
"type": "date"
},
"host": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"level": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"src": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"time": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}
I'll post what we have figured out in comments as an answer as I think there are couple things which could be improved to increase performance and readability.
Solution:
Query from the question was using .Field(f => f.timestamp) which was translated by NEST to use timestamp field not #timestamp. Simple change to .Field("#timestamp") would resolve the problem as this is the proper field name in index mapping.
{
"logstash-2019-06-28": {
"mappings": {
"logevent": {
"properties": {
"#timestamp": {
"type": "date"
},
..
}
}
}
}
}
We could also mark timestamp property with PropertyName attribute to tell NEST to use #timestamp as a name instead of timestamp
public class SearchEventDto : IDto
{
[KendoColumn(Order = 1, DisplayName = "Timestamp", UIType = UIType.DateTime)]
[PropertyName("#timestamp")]
public DateTime timestamp { get; set; }
}
and query
var query = client.Search<SearchEventDto>(s => s
.AllIndices()
.AllTypes()
.Query(q => q
.MatchAll() && +q
.DateRange(r =>r
.Field(f => f.timestamp)
.LessThanOrEquals(new DateTime(2019,06,01, 0, 0, 0))
)
)
);
would be just working as well.
Improvements:
Query only specific indices:
var query = client.Search<SearchEventDto>(s => s
.AllIndices()
.AllTypes()
..
By using AllIndices() we are telling elasticsearch to try to gather documents from all of the indices, we could change it a little bit to query only indices with logs data:
var query = client.Search<SearchEventDto>(s => s
.Index("logstash-*")
.Type("logevent")
..
Use filter context for date range filter:
.Query(q => q.Bool(b => b.Filter(f => f.DateRange(..))))
This way your query should be faster as it doesn't care about calculating search relevance score. You can read more about it here.
Hope that helps.
I am having trouble deserializing JSON received from HubSpot ContactList API.
I am using Restsharp and NewtonSoft, and I'm having real struggles understanding how to correctly define the required classes in order to deserialize the JSON string, which is below:
"contacts": [
{
"vid": 2251,
"portal-id": 5532227,
"is-contact": true,
"profile-url": "https://app.hubspot.com/contacts/5532227/contact/2251",
"properties": {
"firstname": {
"value": "Carl"
},
"lastmodifieddate": {
"value": "1554898386040"
},
"company": {
"value": "Cygnus Project"
},
"lastname": {
"value": "Swann"
}
},
"form-submissions": [],
"identity-profiles": [
{
"vid": 2251,
"saved-at-timestamp": 1553635648634,
"deleted-changed-timestamp": 0,
"identities": [
{
"type": "EMAIL",
"value": "cswann#cygnus.co.uk",
"timestamp": 1553635648591,
"is-primary": true
},
{
"type": "LEAD_GUID",
"value": "e2345",
"timestamp": 1553635648630
}
]
}
],
"merge-audits": []
},
{
"vid": 2301,
"portal-id": 5532227,
"is-contact": true,
"profile-url": "https://app.hubspot.com/contacts/5532227/contact/2301",
"properties": {
"firstname": {
"value": "Carlos"
},
"lastmodifieddate": {
"value": "1554886333954"
},
"company": {
"value": "Khaos Control"
},
"lastname": {
"value": "Swannington"
}
},
"identity-profiles": [
{
"vid": 2301,
"saved-at-timestamp": 1553635648733,
"deleted-changed-timestamp": 0,
"identities": [
{
"type": "EMAIL",
"value": "cswann#khaoscontrol.com",
"timestamp": 1553635648578,
"is-primary": true
},
{
"type": "LEAD_GUID",
"value": "c7f403ba",
"timestamp": 1553635648729
}
]
}
],
"merge-audits": []
}
],
"has-more": false,
"vid-offset": 2401
}
If I simply request the vid, I correctly get 2 vid's back. It's when I try to do the properties and that i get a fail.
Please help
Lets reduce the Json to the minimum to reproduce your error :
{
"vid": 2301,
"portal-id": 5532227,
"is-contact": true,
"profile-url": "https://app.hubspot.com/contacts/5532227/contact/2301",
"properties": {
"firstname": {
"value": "Carlos"
},
"lastmodifieddate": {
"value": "1554886333954"
},
"company": {
"value": "Khaos Control"
},
"lastname": {
"value": "Swannington"
}
}
}
And the appropriate class ContactListAPI_Result:
public partial class ContactListAPI_Result
{
[JsonProperty("vid")]
public long Vid { get; set; }
[JsonProperty("portal-id")]
public long PortalId { get; set; }
[JsonProperty("is-contact")]
public bool IsContact { get; set; }
[JsonProperty("profile-url")]
public Uri ProfileUrl { get; set; }
[JsonProperty("properties")]
public Dictionary<string, Dictionary<string, string>> Properties { get; set; }
}
public partial class ContactListAPI_Result
{
public static ContactListAPI_Result FromJson(string json)
=> JsonConvert.DeserializeObject<ContactListAPI_Result>(json);
//public static ContactListAPI_Result FromJson(string json)
// => JsonConvert.DeserializeObject<ContactListAPI_Result>(json, Converter.Settings);
}
public static void toto()
{
string input = #" {
""vid"": 2301,
""portal-id"": 5532227,
""is-contact"": true,
""profile-url"": ""https://app.hubspot.com/contacts/5532227/contact/2301"",
""properties"": {
""firstname"": {
""value"": ""Carlos""
},
""lastmodifieddate"": {
""value"": ""1554886333954""
},
""company"": {
""value"": ""Khaos Control""
},
""lastname"": {
""value"": ""Swannington""
}
}
}";
var foo = ContactListAPI_Result.FromJson(input);
}
But the Value of one property will be burrow in the sub dictionary, we can the project the object in a more usefull one :
public partial class ItemDTO
{
public long Vid { get; set; }
public long PortalId { get; set; }
public bool IsContact { get; set; }
public Uri ProfileUrl { get; set; }
public Dictionary<string, string> Properties { get; set; }
}
Adding the projection to the Class:
public ItemDTO ToDTO()
{
return new ItemDTO
{
Vid = Vid,
PortalId = PortalId,
IsContact = IsContact,
ProfileUrl = ProfileUrl,
Properties =
Properties.ToDictionary(
p => p.Key,
p => p.Value["value"]
)
};
}
Usage :
var result = foo.ToDTO();
Live Demo
Creating and managing class structure for big and nested key/value pair json is tedious task
So one approach is to use JToken instead.
You can simply parse your JSON to JToken and by querying parsed object, you will easily read the data that you want without creating class structure for your json
From your post it seems you need to retrieve vid and properties from your json so try below code,
string json = "Your json here";
JToken jToken = JToken.Parse(json);
var result = jToken["contacts"].ToObject<JArray>()
.Select(x => new
{
vid = Convert.ToInt32(x["vid"]),
properties = x["properties"].ToObject<Dictionary<string, JToken>>()
.Select(y => new
{
Key = y.Key,
Value = y.Value["value"].ToString()
}).ToList()
}).ToList();
//-----------Print the result to console------------
foreach (var item in result)
{
Console.WriteLine(item.vid);
foreach (var prop in item.properties)
{
Console.WriteLine(prop.Key + " - " + prop.Value);
}
Console.WriteLine();
}
Output:
I send time for this problem 2 days but can't fix it. Please advise.
class Test_Market
{
public string x_ric { get; set; }
public string date { get; set; }
public string timestamp { get; set; }
public string rdn_code { get; set; }
public Dictionary<string, string> fids { get; set; }
}
Example JSON in ES.
"hits": {
"total": 1,
"max_score": 0.6931472,
"hits": [
{
"_index": "test-market-b",
"_type": "test_market",
"_id": "AWTKqb0e8MENF1jasgCD",
"_score": 0.6931472,
"_source": {
"x_ric": "ABC",
"date": "07-24-2018",
"timestamp": "07-24-2018 05:01:17",
"rdn_code": "XYZ",
"fids": {
"BID": "555",
"ASK": "555",
"BIDSIZE": "555",
"ASKSIZE": "555"
}
}
}
]
}
I have no problem about add data to Elasticsearch, but face the issue about sort/query/search.
This code is no problem.
var node = new Uri("URL");
var ESsettings = new ConnectionSettings(node);
var client = new ElasticClient(ESsettings);
var response = client.Search<Test_Market>(s => s
.Index("test-market-b")
.Type("test_market").Size(10)
);
I can get value of timestamp using
var myData = new List<object>();
foreach (var hit in response.Documents)
{
myData.Add(hit.timestamp);
}
Below are I got it no value.
Sorting to find one is the latest timestamp.
var responseXX = client.Search<Test_Market>(s => s
.Index("test-market-b")
.Type("test_market")
.Size(1)
.Sort(sort => sort.OnField( t=>t.timestamp).Descending())
);
Querying
var result = client.Search<Test_Market>(s => s
.Query(q=>q
.Term("date","07-24-2018")
)
);
UPDATED !!!! I can fix this my problem now.
Query AND Sort
var responseXX = client.Search<dynamic>(s => s
.Query(q => q
.Match(m => m.OnField("x_ric").Query("AAA")))
.Sort(sort => sort.OnField("timestamp").Descending())
);
var products = responseXX.Documents.ToList();
First of all, post api in ES before insert data into ES.
PUT test-market-b
{
"mappings": {
"test_market": {
"properties": {
"date": {
"type": "date",
"format": "MM-dd-yyyy"
},
"timestamp": {
"type": "date",
"format": "MM-dd-yyyy HH:mm:ss"
}
}
}
}
}