Elasticsearch NEST and basic Search query - c#

I'm new to Elasticsearch and I'm using NEST. When I run my query in the browser (host/logstash-2019.03.17/_search?pretty) I get the following result:
{
"took" : 138,
"timed_out" : false,
"shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 10,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "logstash-2019.03.17",
"_type" : "logevent",
"_id" : "aa7djGkB1zvCMljS8jPd",
"_score" : 1.0,
"_source" : {
"#timestamp" : "2019-03-17T18:15:43.9506399Z",
"level" : "Info",
"message" : "Attempting to get results from ElasticSearch",
"logger" : "App.Api.Controllers.MyController"
}
}, OTHER HITS IN THE SAME FORMAT
However, when I'm trying to query the same index using ElasticClient i get the following exception:
Elasticsearch.Net.UnexpectedElasticsearchClientException: „Cannot
deserialize the current JSON object (e.g. {"name":"value"}) into type
'System.Int64' because the type requires a JSON primitive value (e.g.
string, number, boolean, null) to deserialize correctly. To fix this
error either change the JSON to a JSON primitive value (e.g. string,
number, boolean, null) or change the deserialized type so that it is a
normal .NET type (e.g. not a primitive type like integer, not a
collection type like an array or List) that can be deserialized
from a JSON object. JsonObjectAttribute can also be added to the type
to force it to deserialize from a JSON object. Path
'hits.total.value', line 1, position 115.”
I thought NEST is capable of autodeserializing JSON correctly on condition that it is provided with a class whose properties correspond to "_source" object fields. At least this is what you can infer from this tutorial.
Here is my POCO class follwed by the query which throws the exception:
public class Logevent
{
public string Id { get; set; }
public DateTime Timestamp { get; set; }
public string Level { get; set; }
public string Message { get; set; }
public string Logger { get; set; }
}
var client = new ElasticClient();
var searchResponse = client.Search<Logevent>(s => s.Index("logstash-2019.03.17").Query(q => q.Match(m => m.Field(f => f.Level).Query("message"))));
Could anyone explain what I'm doing wrong?

It looks like you're using a newer version of Elasticsearch (maybe one of the 7.0.0 prereleases?) where the total field is no longer just an Int64 value
"total" : {
"value" : 10,
"relation" : "eq"
}
NEST 6.x does not handle this, NEST 7.x will however, when it's released. For now, I would recommend using the latest Elasticsearch 6.x, currently 6.6.2.
Major versions of NEST are tied to major versions of Elasticsearch, so
NEST 5.x -> Elasticsearch 5.x
NEST 6.x -> Elasticsearch 6.x
etc.
But I would recommend keeping NEST up to date on minors within a major; we maintain backwards binary compatibility within a major to help with this.

Related

Extract c# object from json

I have a Json and I want to get it in my c# object.
var json = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
_ = JsonConvert.DeserializeObject<object>(json);
Here, I get the Json in the format of:
{{
"pipeline" : {
"url" : "url1",
"idP" : 1
},
"id": 1234,
"name" : "test1",
"state" : "inprogress",
"date" : "date"
}}
Now, from this JSON, I just want the id and idP.
How can I do that? Should I create a class with all the properties?
Can I please get a sample code?
If you just want id and idP, you don't need to create the classes and deserialize json. You can just parse it
var jsonParsed = JObject.Parse(json);
var id = (Int32)jsonParsed["id"]; //1234
var idP = (Int32) jsonParsed["pipeline"]["idP"]; //1
but you have to fix your json, by removing extra pair {}. You can make it manually if it is a typo. But if it is a bug, you can use this code, before parsing
json=json.Substring(1,json.Length-2);
or for example you can create one class
public class Pipeline
{
public string url { get; set; }
public int idP { get; set; }
}
and deserialize only one part of json
Pipeline pipeline = jsonParsed["pipeline"].ToObject<Pipeline>();

Return JSON from MongoDb without $date

I am using MongoDb with C# in a WebApi app. My data does not lend well to mapped data types due to the dynamic nature of the document stored. For example, here is an example of some data. Note the Data section with values that can either be arrays or a single string (Technician):
{
"_id" : "5a59129d16d5c42f7444b83d",
"CreatedDate" : "2018-01-09T20:30:19.455Z",
"Data" : {
"AlcoholTest" : [
{
"Technician" : [
"STT",
"BAT"
],
"TestReason" : "not well"
}
]
}
}
When I attempt to return the data like above, I get this instead:
{
"_id": {
"$oid": "5a59129d16d5c42f7444b83d"
},
"CreatedDate": {
"$date": 1515529819455
},
"Data": {
"AlcoholTest": [
{
"Technician": [
"STT",
"BAT"
],
"TestReason": "drunk"
}
]
}
}
Here is the code I'm using:
public object FindById(string id)
{
var filter = new BsonDocument { { "_id", ObjectId.Parse(id) } };
var result = _collection2.Find(filter);
var note = result.Any() ? result.First() : null;
var json = note.ToJson(new JsonWriterSettings{OutputMode = JsonOutputMode.Strict});
return JObject.Parse(json);
}
I can't just return the note object since Newtonsoft does not know how to convert those $data and $oid into valid types and returns a parsing error.
When I attempted to use MongoDb mapping classes in .NET, this is what my class looked like (I'm not including the BsonClassMap.RegisterClassMap stuff for simplicity):
public class Note
{
public string Id { get; set; }
public DateTime? CreatedDate { get; set; }
public IDictionary<string, IList<IDictionary<string, object>>> Data { get; set; }
}
When I tried this code, the Newtonsoft did not know how to handle the case where the object in IList<IDictionary<string, object>> could be an array or string and saved some rather nasty JArray and JObject data instead.
So here are my questions:
Is there a way to make the above C# code return identical JSON as what is represented in the Mongo database (or what the UI is sending and expecting to get back)?
Or is there a way to map my Data document using .NET types that would allow me to use MongoDb mapping classes that accepts either string values, arrays, or both?
I ended up using a different approach for save vs get. For saving, I converted a generic object to BsonDocument, and saved it as it. This resulted in the data format I was expecting (first data format my above question).
I used a strongly-typed collection to get data from the MongoDB so I could avoid the second data format from my question above. To get the data returned in the correct format, I changed this property from
public IDictionary<string, IList<IDictionary<string, IList<string>>>> Data { get; set; }
to
public IDictionary<string, object> Data { get; set; }
This gave me the correctly formatted data I needed. Having two MongoDb collections for getting vs saving was not ideal, but it eliminated my original hack to parse the dictionary BsonDocument into my strongly-typed POCO.

Json with dynamic properties - deserialize into C# object

I have the following JSON when user clicks save
tasks : {
updated : [
{ Id : 123, SomeField1 : 'new value', SomeField2 : 'new value', SomeField3 : 'new value' },
{ Id : 125, SomeField2 : 'new value' },
{ Id : 127, SomeField1 : 'new value', SomeField4 : 'new value' },
{ Id : 129, SomeField2 : 'new value', SomeField3 : 'new value' },
{ ................ }
],
removed : [
{ Id : 345 },
{ Id : 847 }
]
}
on the MVC server side (C#), I have a ViewModel and .NET deserializes this back to my viewmodel object.
in this example, This object has Id, SomeField1, SomeField2, SomeField3, SomeField4.
The problem I am having is that the client only sends the fields which were actually updated, so If the user never updated SomeField3 it wont be in the json and .NET for that array object will have a null as SomeeField3 ...
so i cant get record, update all the fields to what the viewmodel is and then call an update as it will set SomeField3 to null , which is not correct - there could be data in that field which the user just didn't touch in this case .. (in another case they may have deleted their text, which then the update would be valid..
I am not sure what is the best way to tackle this problem.
Looking forward to your suggestions.
I suggest you to post updated string in API action, then you can get your solution as :
Create dynamic property mapping function :
public static class DynamicToStatic
{
public static T ToStatic<T>(object source, T destination)
{
var entity = destination;
//source implements dictionary
var properties = source as IDictionary<string, object>;
if (properties == null)
return entity;
foreach (var entry in properties)
{
var propertyInfo = entity.GetType().GetProperty(entry.Key);
if (propertyInfo != null && entry.Value != null)//Check property and its values exist or not ,change only when source contains value
propertyInfo.SetValue(entity, entry.Value, null);
}
return entity;
}
}
Convert your request json to dynamic object and then map dynamic object to Your static class type model, Class type model initialized from your db record or any source as per your requirement.
//updatedJsonObjectString bound from request post data(JSONSTRINGIFY of post data)
dynamic source = Newtonsoft.Json.JsonConvert.DeserializeObject<dynamic>(updatedJsonObjectString);
Class1 model = new Class1();//mapped/filled by data call
var retUser = DynamicToStatic.ToStatic<Class1>(source, model);
if you are using Newton Json for Deserializing.
Newton Json DeserializeObject method has an overload which takes json string and JsonSerializerSettings as parameters. JsonSerializerSettings has NullValueHandling and MissingMemberHandling properties.
MissingMemberHandling: Gets or sets how missing members (e.g. JSON contains a property that isn't a member on the object) are handled
during deserialization.
NullValueHandling: Gets or sets how null values are handled during serialization and deserialization

C# MongoDB update object from oplog

What I have is a MongoDB which contains a collection of products. Lets say each product has the following structure:
public ObjectId Id { get; set; }
public string label { get; set; }
public List<IngredientsModel> ingredients { get; set; }
Now my C# program gets notified via the oplog if there was a change. This oplog looks like this:
{
"ts" : Timestamp(1425392023, 1),
"h" : NumberLong("-7452324865462273810"),
"v" : 2,
"op" : "u",
"ns" : "coll.products",
"o2" : { "_id" : ObjectId("54f5b87cd4c6959bd3ecf2d6") },
"o" : { "$set" : { "ingredients.1.ingredid" : "54f5c117d4c6959bd3ecf2e8" } }
}
When my application starts I create a list of products which I get from the MongoDB. I just use products.findAll() and the put them in a List<'products>
I have several clients connected to the MongoDB and every client gets notified via oplog.
Now I want to update my local lists without querying the MongoDB since I have all the information needed to update my local products already in the oplog.
My question is now: How to do that?
My approach is the following:
Deserialize the object to Json
var prodjson= Product[0].ToJson<ProductModel>();
Convert Json to BsonDocument
var prodbson= BsonSerializer.Deserialize<BsonDocument>(prodjson);
Apply the "o" key from the oplog to the prodbson
"o" : { "$set" : { "ingredients.1.ingredid" : "54f5c117d4c6959bd3ecf2e8" } }
NO IDEA

Assign JSON value to property in C#

From an API I receive a JSON-object that looks like this:
{
"wind" : {
"speed" : 7.31,
"deg" : 187.002
},
"rain" : {
"3h" : 0
},
"clouds" : {
"all" : 92
},
"coord" : {
"lon" : 139,
"lat" : 35
},
"dt" : 1369824698,
"id" : 1851632,
"cod" : 200,
"weather" : [
{
"id" : 804,
"main" : "clouds",
"icon" : "04n",
"description" : "overcast clouds"
}
],
"main" : {
"humidity" : 89,
"temp_max" : 292.04,
"temp_min" : 287.04,
"temp" : 289.5,
"pressure" : 1013
},
"sys" : {
"country" : "JP",
"sunrise" : 1369769524,
"sunset" : 1369821049
},
"name" : "Shuzenji"
}
I would like to assign two of these values to my class:
public class Weather {
public string Name { get; set; }
public string Temp { get; set; }
}
The name I can assign like this:
weather.Name = TheJSON.name.ToString();
But the temp is trickier, because it's nested inside the "main"-array. I´ve seen many examples on how to do this in Javascript but not so much in C#. Thanks!
Main is not an array. It is an object, so
TheJSON.main.temp.ToString()
The easiest way to work with JSON data is to deserialize them as C# objects and directly use them in your application. You can use a tool like JSON C# Class Generator to automatically generate the C# class from the JSON data. Once you have your C# classes generated, you can deserialize the JSON string using the JsonConvert.DeserializeObject(jsonText); The generated code requires Newtonsoft Json.NET which you can easily add as a NuGet package.
If you save your JSON content in D:\test.json, you can use the following code to access the values using the C# objects generated. The example below is to just give you an idea on the usage.
var json = File.ReadAllText(#"D:\test.json");
var weather = JsonConvert.DeserializeObject<Weather>(json);
Console.WriteLine(weather.Name);
Console.WriteLine(weather.Sys.Country);

Categories