Problems with Date format in elasticsearch.net and Nest (C#) - c#

I have the following C# model :
[ElasticType(Name = "myType")]
public class MyType
{
...
[ElasticProperty(Name = "ElasticId")]
[DataMember(Name = "ElasticId")]
public string ElasticId { get; set; }
...
[ElasticProperty(Name = "DateToBeUsed", Type = FieldType.Date, DateFormat = "date_hour_minute_second_millis")]
public string DateToBeUsed { get; set; }
...
}
The "date_hour_minute_second_millis" correspond to following format : yyyy-MM-dd’T'HH:mm:ss.SSS
(http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-date-format.html)
The mapping ES is done with Nest using "map" method and correspond to that :
"mappings": {
"myType": {
"properties": {
...,
"ElasticId": {
"type": "string"
},
...,
"DateToBeUsed": {
"type": "date",
"format": "date_hour_minute_second_millis"
},
...
}
}
}
I insert an document inside this index:
"_source": {
...,
"ElasticId": "2",
...,
"DateToBeUsed": "2012-05-21T09:51:34.073",
...
}
My problem is when I want to retrieve this object through Nest.
The value of DateToBeUsed is always formatted with the following format : MM/dd/yyyy HH:mm:ss
(ex : 05/21/2012 09:51:34)
(Using sense, the value is well formatted.)
1°) Is this normal?
I need to retrieve the same date format than the one I gave to ES.
(And I think it should be normal to have the same format as described in the mapping)
2°) Is there a "clean" solution to resolve this problem?
(Re-formatting the date after having retrieved the document is not a "clean" solution...)
Thanks for the answers!
Bye.

I've tried to reproduce what you're seeing, using the following code, but the date value is being returned in the Get call as expected:
string indexName = "so-27927069";
// --- create index ---
client.CreateIndex(cid => cid.Index(indexName));
Console.WriteLine("created index");
// --- define map ---
client.Map<MyType>(m => m
.Index(indexName)
.Type("myType")
.MapFromAttributes());
Console.WriteLine("set mapping");
// ---- index -----
client.Index<MyType>(
new MyType
{
DateToBeUsed = new DateTime(2012, 5, 21, 9, 51, 34, 73)
.ToString("yyyy-MM-ddThh:mm:ss.fff"),
ElasticId = "2"
},
i => i
.Index(indexName)
.Type("myType")
.Id(2)
);
Console.WriteLine("doc indexed");
// ---- get -----
var doc = client.Get<MyType>(i => i
.Index(indexName)
.Type("myType")
.Id(2)
);
Console.WriteLine();
Console.WriteLine("doc.Source.DateToBeUsed: ");
Console.WriteLine(doc.Source.DateToBeUsed);
Console.WriteLine();
Console.WriteLine("doc.RequestInformation.ResponseRaw: ");
Console.WriteLine(Encoding.UTF8.GetString(doc.RequestInformation.ResponseRaw));
I'm seeing the following result as output:
created index
set mapping
doc indexed
doc.Source.DateToBeUsed:
2012-05-21T09:51:34.073
doc.RequestInformation.ResponseRaw:
{"_index":"so-27927069","_type":"myType","_id":"2","_version":1,"found":true,"_source":{
"ElasticId": "2",
"DateToBeUsed": "2012-05-21T09:51:34.073"
}}
(Watching the traffic via Fiddler, I'm seeing an exact match between the ResponseRaw value and the payload of the response to the Get request.)
I'm on Elasticsearch version 1.5.2 and NEST version 1.6.0. (Maybe the issue you were seeing was fixed sometime in the interim....)

Related

parse Dynamic c# class from a json

I want to build a dynamic class with a given JSON.
Atm i parse the json with
dynamic customConfig = JsonConvert.DeserializeObject(configJson);
and its working fine with other json then the given BUT my problem is that the names of the properties (here valueOne and valueTwo are "dynamic", i get always others)
i Know if i know the names i can get e.g. the description by customConfig.config.valueOne.description
But what can i do to get e.g. the description by dont have the name of valueOne?
configJson=
"config": {
"valueOne":{
"description": "My first example value.",
"defaultValue": "Example 1",
"isRequired":false
},
"valueTwo":{
"description": "My second example value.",
"defaultValue": "Example 2",
"isRequired":false
},
},
What i tried was to get it in a loop but i dont get it to another class.
foreach (var param in customConfig.config)
{
foreach (var item in param)
{
Config.config.description[i] = item.description;
}
i++;
}
item.description gets the right description but why i cant save it in the other class (which is also dynamic)?
You might have some other underlying logic issues with your loop and what you're trying to accomplish there, but to answer your specific question, you might need to initialize your "config" and "data" members of your CustomConfigModel class. For example...
public class CustomConfigModel
{
public CustomConfigModel()
{
this.data = new ExpandoObject();
this.config = new ExpandoObject();
}
public dynamic data { get; set; }
public dynamic config { get; set; }
}
I was able to access the description you want, I think but it is in this form (my output) :
Token key: 'config.valueOne' ->>>> 'description' : 'My first example value.'
Token key: 'config.valueTwo' ->>>> 'description' : 'My first example value.'
If you know main item name , "config", before hand , you can parse and get rid of it so that you'd have values : "valueOne" or "valueTwo". As you can see from my example code, you can get description values by iterating. You can develop further functionality from this example. Please let me know if this works for you.
Here is the example code:
class Program
{
static void Main(string[] args)
{
string configJson = #"{
'config': {
'valueOne':{
'description': 'My first example value.',
'defaultValue': 'Example 1',
'isRequired':false
},
'valueTwo':{
'description': 'My second example value.',
'defaultValue': 'Example 2',
'isRequired':false
},
}
}";
JObject customConfig = (JObject)JsonConvert.DeserializeObject(configJson);
var children = customConfig.Children().Children();
List<JToken> subList = new List<JToken>();
foreach (JToken token in children.Values())
{
string key = token.Path;
subList.AddRange(token.Values());
JToken subitem = subList.Find(q => q.Path.Contains("description"));
string desc = ((JProperty)subitem).Value.ToString();
Console.WriteLine("Token key: '" + key + "' ->>>> 'description' : '" + desc+"'");
}
}
}
At
Config.config.description[i] = item.description;
is description[i] null and i get an excception but why

Filtering several Geo near operations in a query sends an error

I made a filter to do a Geospatial query on data from my Mongodb database and this filter works fine until I do an "And" operation on this filter with another Geospatial filter, but pointing to a different Property on the Collection I'm querying.
When I add this filter, I get an Exception stating:
Message: MongoDB.Driver.MongoCommandException : Command find failed: Too many geoNear expressions.
Here is the first filter,
var point = GeoJson.Point(GeoJson.Geographic(longitude: annEntityaAttr.CollectionLocation.Longitude,
latitude: annEntityaAttr.CollectionLocation.Latitude));
filter = Builders<AnnouncementEntity>.Filter.Near(a => a.CollectionLocation, point, annEntityaAttr.MaxDistanceFromLocationInKM * metersFor1KM);
Here is how I add the second filter:
var point = GeoJson.Point(GeoJson.Geographic(longitude: annEntityaAttr.DepositLocation.Longitude,
latitude: annEntityaAttr.DepositLocation.Latitude));
var secondFilter = Builders<AnnouncementEntity>.Filter.Near(a => a.DepositLocation, point, annEntityaAttr.MaxDistanceFromLocationInKM * metersFor1KM);
filter = filter & secondFilter;
Normaly, when applying & to two filters, it works, but in this case, it doesn't please does someone have a solution for this ?.
you can only do one geo query on a single collection at a time. if you must store two coordinate fields in a single document, you will have to issue two seperate geo queries and then intersect the results on the client side. also you'd need to create 2 geo indexes and specify the index key when querying. here's an example program that uses my library MongoDB.Entities to achieve what you need.
using MongoDB.Entities;
using MongoDB.Driver;
using System.Linq;
namespace StackOverflow
{
public class Program
{
public class Announcement : Entity
{
public Coordinates2D CollectionLocation { get; set; }
public Coordinates2D DepositLocation { get; set; }
public double DistanceMeters { get; set; }
}
static void Main(string[] args)
{
new DB("test");
DB.Index<Announcement>()
.Key(a => a.CollectionLocation, KeyType.Geo2DSphere)
.Create();
DB.Index<Announcement>()
.Key(a => a.DepositLocation, KeyType.Geo2DSphere)
.Create();
(new Announcement
{
DepositLocation = new Coordinates2D(48.8539241, 2.2913515),
CollectionLocation = new Coordinates2D(48.796964, 2.137456)
}).Save();
var searchPointA = new Coordinates2D(48.796964, 2.137456);
var queryA = DB.GeoNear<Announcement>(
NearCoordinates: searchPointA,
DistanceField: a => a.DistanceMeters,
IndexKey: "CollectionLocation",
MaxDistance: 10);
var searchPointB = new Coordinates2D(48.8539241, 2.2913515);
var queryB = DB.GeoNear<Announcement>(
NearCoordinates: searchPointB,
DistanceField: a => a.DistanceMeters,
IndexKey: "DepositLocation",
MaxDistance: 10);
var resultA = queryA.ToList();
var resultB = queryB.ToList();
var common = resultA.Where(a => resultB.Any(b => b.ID == a.ID)).ToArray();
}
}
}
the following two $geoNear queries will be issued to find the locations:
[
{
"$geoNear": {
"near": {
"type": "Point",
"coordinates": [
48.8539241,
2.2913515
]
},
"distanceField": "DistanceMeters",
"spherical": true,
"maxDistance": NumberInt("10"),
"key": "DepositLocation"
}
}
]
[
{
"$geoNear": {
"near": {
"type": "Point",
"coordinates": [
48.796964,
2.137456
]
},
"distanceField": "DistanceMeters",
"spherical": true,
"maxDistance": NumberInt("10"),
"key": "CollectionLocation"
}
}
]

Converting C# List into JSON specific format

I have a graph in my view which looks like this:
var hourlyGraph = Morris.Bar({
element: 'graph_bar',
data: [
#foreach (var item in ViewBag.HourlyGraph)
{
#:{device: '#item.Hour.ToString("D2"):00', geekbench:#item.Sales },
}
],
xkey: 'device',
ykeys: ['geekbench'],
labels: ['Sold'],
barRatio: 0.4,
barColors: ['#0A4D70', '#34495E', '#ACADAC', '#3498DB'],
xLabelAngle: 35,
hideHover: 'auto',
resize: true
});
This is a morris chart. Note how the data is set up here:
[
#foreach (var item in ViewBag.HourlyGraph)
{
#:{device: '#item.Hour.ToString("D2"):00', geekbench:#item.Sales },
}
]
And now I need to fill the chart with new data. In my Action I have created a list which contains 2 properties:
public int Hour {get;set;}
public int Sales {get;set;}
And they are stored into a list typed of:
var HourlyGraph = new List<HourlyGraph>();
Now I'd like to convert this list into a JSON format which would look something like this:
[
{device: '0', geekbench:5 },
{device: '1', geekbench:13 },
{device: '2', geekbench:25 },
{device: '3', geekbench:14 },
{device: '4', geekbench:16 },
]
Where value for device would be = hour, and geekbench = sales ...
How could I do this in C#?
With Json.Net and Linq it's easy:
string myJson =
JsonConvert.SerializeObject(mylist.Select(item=>
new {device=item.Hour, geekbench=item.Sales}));
You project an anonymous type with the fields and names that you'd like, and let Newtonsoft.Json do the rest.
since you are using mvc why not use return Json() it will convert the object to json string you can use it like
public ActionResult Myaction()
{
var HourlyGraph = new List<HourlyGraph>();
return Json(HourlyGraph.Select(x => new {Hour=x.Hour,Sales=x.Sales }));
}
You can achieve the desired output by using LINQ Anonymous Type
As per example following is class
public class HourlyGraph
{
public int Hour { get; set; }
public int Sales { get; set; }
}
Import Namespace System.Web.Script.Serialization which is a Microsoft's Inbuilt Class for dealing with JSON. You will need to refer additional assembly named System.Web.Extensions using 'Add References'.
using System.Web.Script.Serialization;
Declare List and Convert into customized JSON Format
var listHourlyGraph = new List<HourlyGraph>();
//Adding some Sample Values
listHourlyGraph.Add(new HourlyGraph() { Hour = 0, Sales = 5 });
listHourlyGraph.Add(new HourlyGraph() { Hour = 1, Sales = 10 });
//Declaring JavaScriptSerialzer Object
var serialzer = new JavaScriptSerializer();
//Using Serialize Method which returns back a JSON String
//We are using LINQ SELECT Method to create a new anonymous return type which contains our custom return types
string s = serialzer.Serialize(listHourlyGraph.Select(x => new { device = x.Hour, geekbench = x.Sales } ));
You will get the following output in 's' variable
[{"device":0,"geekbench":5},{"device":1,"geekbench":10}]
Note: If you want to get performance optimizations then you are better using Newtonsoft JSON Library instead of Microsoft's Default JSON Library.

Serialize list of dictionaries into accepted DataTable Ajax object

I have a web application in which I'm retrieving some data into bootstrap table, what i want to do now is to use jQuery DataTable instead of the current as it has too much useful features.
Currently I'm retrieving the data from the server side using OOP approach, where a class object represents a data row in a particular table, and this object includes a dictionary which stores column names and values.
What I'm doing now is that I'm retrieving these class objects and append each dictionary of each object in a List<Item> and then serialize this list using JavaScriptSerializer object, and this object returns the following JSON format:
[
{
"slno":"2",
"status_message":"Lights still flashing",
"crm_services_id":"1", "subject_id":"Lights are flashing",
"severity_id":"5",
"user_id":"husain.alhamali",
"status_id":"1"
},
{
"slno":"3",
"status_message":"lights working fine",
"crm_services_id":"2",
"subject_id":"Lights are flashing",
"severity_id":"3",
"user_id":"husain.alhamali",
"status_id":"2"
}
]
When i tried to use this object to fill my DataTable AJAX I've got an error says:
Invalid JSON response
I saw some examples of a valid JSON response that is acceptable to a DataTable which is as follow:
{
"data": [
[
"Tiger Nixon",
"System Architect",
"Edinburgh",
"5421",
"2011/04/25",
"$320,800"
],
[
"Garrett Winters",
"Accountant",
"Tokyo",
"8422",
"2011/07/25",
"$170,750"
]
}
Now my question is is there any tool or plugin that could re-format my JSON string into an acceptable format like the one above?
With this HTML:
<table id="example"></table>
This JS will create a table:
var data = [{
"slno": "2",
"status_message": "Lights still flashing",
"crm_services_id": "1",
"subject_id": "Lights are flashing",
"severity_id": "5",
"user_id": "husain.alhamali",
"status_id": "1"
}, {
"slno": "3",
"status_message": "lights working fine",
"crm_services_id": "2",
"subject_id": "Lights are flashing",
"severity_id": "3",
"user_id": "husain.alhamali",
"status_id": "2"
}];
function getColumns(){
for(var i = 0; i < data.length; i++){
let columnsArray = [];
var keys = Object.keys(data[i]);
for(k in Object.keys(data[i])){
if(data[i].hasOwnProperty(keys[k])){
columnsArray.push({
"data":keys[k]
});
}
}
return columnsArray;
}
}
$(document).ready(function() {
var table = $('#example').DataTable({
"columns": getColumns(),
"data": data
});
});
Working example. Hope that helps.
dataTable require json data in return from ajax response having following keys
1. data
2. draw
3. recordsTotal
4. recordsFiltered
Use this:
var data = list.Select(u => u.GetType()
.GetProperties()
.Select(p => p.GetValue(u, null)));
example
public class User
{
public int userId { get; set; }
public string name { get; set; }
}
public class Programm
{
static void Main()
{
var list = new List<User>();
list.Add(new User
{
userId = 1,
name = "name 1",
});
list.Add(new User
{
userId = 2,
name = "name 2",
});
var data = list.Select(u => u.GetType()
.GetProperties()
.Select(p => p.GetValue(u, null)));
Console.WriteLine(new JavaScriptSerializer().Serialize(new
{
data = data
}));
}
}
result
{
"data" : [
["1", "name 1"],
["2", "name 2"]
]
}

MongoDB Array Query

I'm new with Mongo DB and I'm trying to figure out how to do some more complex queries. I have a document that has a nested array of DateTime.
Here is my data:
{ "_id" : ObjectId("537d0b8c2c6b912520798b76"), "FirstName" : "Mary", "LastName" : "Johnson", "Age" : 27, "Phone" : "555 555-1212", "Dates" : [ISODate("2014-05-21T21:34:16.378Z"), ISODate("1987-01-05T08:00:00Z")] }
{ "_id" : ObjectId("537e4a7e2c6b91371c684a34"), "FirstName" : "Walter", "LastName" : "White", "Age" : 52, "Phone" : "800 123-4567", "Dates" : [ISODate("1967-12-25T08:00:00Z"), ISODate("2014-12-25T08:00:00Z")] }
What I want to do is return the document where the Dates array contains a value between a range. In my test case the range is 1/1/1987 and 1/10/1987 so I expect to get back the first document listed above (Mary Johnson) because 1/5/1987 is in that Dates array and falls between 1/1/1987 and 1/10/1987.
I'd like to be able to do this with both the MongoDB command line utility and the C# driver.
With the C# driver I tried the following LINQ query (based on an example in the MongoDB documentation):
DateTime beginRange = new DateTime(1987, 1, 1);
DateTime endRange = new DateTime(1987, 1, 10);
var result = from p in people.AsQueryable<Person>()
where p.Dates.Any(date => date > beginRange && date < endRange)
select p;
The above code throws an exception from within the C# Driver code:
Any is only supported for items that serialize into documents. The current serializer is DateTimeSerializer and must implement IBsonDocumentSerializer for participation in Any queries.
When I try and query the MongoDB database directly I tried the following:
db.People.find( {Dates: { $gt: ISODate("1987-01-01"), $lt: ISODate("1987-01-10") } } )
This query results in both of the documents coming back instead of just the one that has 1/5/1987 in its Dates array.
EDIT:
I figured out a way to do if from the C# driver. It isn't as clean as I would like but it is doable.
I figured that since there was a way to get what I wanted directly from the command utility that there must be a way to do if from the C# driver as well by just executing the same query from the C# driver.
string command = "{Dates : { $elemMatch : { $gt: ISODate(\"" + beginRange.ToString("yyyy-MM-dd") + "\"), $lt: ISODate(\"" + endRange.ToString("yyyy-MM-dd") + "\") } } } ";
var bsonDoc = BsonSerializer.Deserialize<BsonDocument>(command);
var queryDoc = new QueryDocument(bsonDoc);
MongoCursor<Person> p = people.Find(queryDoc);
C# Driver
Just as the exception suggests, you can't do what you want to do using the C# driver as long as your array is of a primitive type (DateTime) and not a genuine document.
From the MongoDB Linq Any description:
This will only function when the elements of the enumerable are serialized as a document. There is a server bug preventing this from working against primitives.
I guess you can create a document wrapper around a DateTime value so you can still do that:
var result = people.AsQueryable<Person>().Where(
person => person.Dates.Any(date =>
date.Value > beginRange && date.Value < endRange));
.
public class DocumentWrapper<T>
{
public ObjectId Id { get; private set; }
public T Value { get; private set; }
public DocumentWrapper(T value)
{
Id = ObjectId.GenerateNewId();
Value = value;
}
}
Native query
As to your query, it isn't actually the equivalent of the Linq query. That would be:
{
Dates :
{
$elemMatch :
{
$gt: ISODate("1987-01-01"),
$lt: ISODate("1987-01-10")
}
}
}
More on $elemMatch here

Categories