We Keep Coding

NEST - low level search doesn't return date

I have C# console application where I use NEST to index data and search in ElasticSearch.
Versions:
ElasticSearch 7.5.2
.NET 4.7.2
NEST 7.6.1
When I use NEST for search, everything is ok. But in some special cases, I would like to use NEST's low level search (because I can use sql command). But in low level search Timestamp is not obtained. Here is code sample:
class EsObject
{
public int Id { get; set; }
public string Name { get; set; }
public DateTime Timestamp { get; set; }
public override string ToString()
{
return $"Id = {Id}, Name = {Name}, Timestamp = {Timestamp.ToString("dd.MM.yyyy HH:mm:ss")}";
}
}
class Program
{
static void Main(string[] args)
{
var indexName = "es_objects";
//
// Create data objects
//
var obj = new EsObject()
{
Id = 1,
Name = "Object_1",
Timestamp = new DateTime(2020, 2, 12, 4, 55, 19)
};
Console.WriteLine($"Object created {obj}");
//
// Connect to ElasticSearch
//
Console.Write("Connecting to ElasticSearch... ");
var node = new Uri("http://localhost:9200");
var settings = new ConnectionSettings(node)
.DefaultIndex(indexName)
.DefaultMappingFor<EsObject>(x => x.IdProperty("Id"))
.DefaultFieldNameInferrer(s => s.ToLower()); ;
var esClient = new ElasticClient(settings);
Console.WriteLine("done");
//
// Index data
//
Console.Write("Indexing data... ");
var idxResp = esClient.IndexDocument(obj);
Console.WriteLine("done");
//
// Searching using NEST
//
Console.Write("Searching data using NEST ... ");
var result = esClient.Search<EsObject>(s => s
.Query(q => q
.Match(m => m
.Field(f => f.Name)
.Query(obj.Name))))
.Documents
.ToList();
Console.WriteLine("done");
Console.WriteLine($"Results: found {result.Count} items");
if (result.Count > 0)
Console.WriteLine($"Found item: {result[0]}");
//
// Searching using SQL
//
Console.Write("Searching data using low level ... ");
var request = new TranslateSqlRequest
{
Query = $"select * from {indexName} where name='{obj.Name}'"
};
var sqlResponse = esClient.LowLevel.Sql.Translate<StringResponse>(PostData.Serializable(request));
var resp = esClient.LowLevel.Search<SearchResponse<EsObject>>(indexName, sqlResponse.Body).Documents.ToList();
Console.WriteLine("done");
Console.WriteLine($"Results: found {resp.Count} items");
if (resp.Count > 0)
Console.WriteLine($"Found item: {resp[0]}");
Console.ReadKey();
}
}
When I run program, I get following results:
Object created Id = 1, Name = Object_1, Timestamp = 12.02.2020 04:55:19
Connecting to ElasticSearch... done
Indexing data... done
Searching data using NEST ... done
Results: found 1 items
Found item: Id = 1, Name = Object_1, Timestamp = 12.02.2020 04:55:19
Searching data using low level ... done
Results: found 1 items
Found item: Id = 1, Name = Object_1, Timestamp = 01.01.0001 00:00:00
Any idea how can I get proper date value when using low level search?

The query generated by the SQL Translate API looks like
{
"size" : 1000,
"query" : {
"term" : {
"name.keyword" : {
"value" : "Object_1",
"boost" : 1.0
}
}
},
"_source" : {
"includes" : [
"name"
],
"excludes" : [ ]
},
"docvalue_fields" : [
{
"field" : "id"
},
{
"field" : "timestamp",
"format" : "epoch_millis"
}
],
"sort" : [
{
"_doc" : {
"order" : "asc"
}
}
]
}
Crucially, source filtering is performed to return only the name field in the _source document, with id and timestamp being returned from docvalue_fields. To fetch these from the search response, you would need to do fetch them out of the fields for each hit
var resp = client.LowLevel.Search<SearchResponse<EsObject>>(indexName, sqlResponse.Body);
var documents = resp.Hits
.Select(h => {
h.Source.Id = h.Fields.ValueOf<EsObject, int>(f => f.Id);
h.Source.Timestamp = DateTimeOffset.FromUnixTimeMilliseconds(long.Parse(h.Fields.Value<string>("timestamp"))).DateTime;
return h.Source;
})
.ToList();
for reference, here's the search response JSON
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "es_objects",
"_type" : "_doc",
"_id" : "1",
"_score" : null,
"_source" : {
"name" : "Object_1"
},
"fields" : {
"id" : [
1
],
"timestamp" : [
"1581483319000"
]
},
"sort" : [
0
]
}
]
}
}
A search query that would do what you want
var resp = client.Search<EsObject>(s => s
.Query(q => q
.Term(f => f.Name.Suffix("keyword"), obj.Name)
)
.Sort(s => s.Ascending(SortSpecialField.DocumentIndexOrder))
);
var documents = resp.Documents.ToList();

MongoDB C# driver: how to use $in operator for nested embedded documents?

I am trying to use the $in operator using the MongoDB C# driver.
Here's an example document:
ObjectField:
{
"_id" : ObjectId("59421cd7f39cc1227cb7be55"),
"ObjectName" : "Customer",
"FieldName" : "Kind Customer",
"KeyValues" : [{
"Key" : "k11",
"Value" : "v11"
}, {
"Key" : "k21",
"Value" : "v21"
}]
}
And here's the first approach I tried:
var database = MongoDbClient.GetDatabase("Database");
var builder = Builders<ObjectField>.Filter;
var keys = new List<string>() { "k11", "k21" };
var filter = builder.Eq("ObjectName", "Customer") & builder.Eq("FieldName", "Kind Customer") & builder.In("KeyValues.Key", keys);
var projection = Builders<ObjectField>.Projection.Include("KeyValues.$");
var bsonDocuments = database.GetCollection<ObjectField>("ObjectField").Find(filter).Project(projection).ToList();
However, it returns only top one KeyValue from the list.
{
"_id": ObjectId("59421cd7f39cc1227cb7be55"),
"KeyValues": [{
"Key": "k11",
"Value": "v11"
}]
}
Here's the 2nd approach I tried:
var keys = new List<string>() { "k11", "k20" };
var result = (from x in database.GetCollection<ObjectField>("ObjectField").AsQueryable()
where x.KeyValues.Any(y => keys.Contains(y.Key))
&& x.ObjectName == "Customer" && x.FieldName == "Kind Customer"
select x).ToList();
In this case, it returned all nested records which are not included in keys.

var filter = Builders<XXXX>.Filter.In("_id", xxxx.Select(i => i.Id));
mongoClient.GetCollection<XXXX>("XXXX").DeleteMany(filter);

Aggregate field must be defined as an expression inside an object error

Trying to port below mongodb aggregation query to c# but getting the error -
Command aggregate failed: the group aggregate field 'ProductType' must
be defined as an expression inside an object.
Any thoughts on what is wrong with the c# code?
db.collectionName.aggregate([
{ $match: activeTrial},
{ $project: {_id:1, CustomerType: 1, ProductType: 1} },
{ $group: {_id: {"cust": "$CustomerType", "prod" : "$ProductType"}, count: {$sum: 1}}},
]).toArray()
collectionName.Aggregate()
.Match(filter)
.Project(x => new {x.CustomerType, x.SubscriptionId, x.ProductType})
.Group(x => new { x.ProductType, x.CustomerType }, g => new ActiveTrialsByProduct()
{
ProductType = g.Key.ProductType,
CustomerType = g.Key.CustomerType,
ActiveTrials = g.Count()
}).ToList();
Here is the collection..
{
"CustomerType" : "CustA",
"ProductType" : "ProdA",
}
{
"CustomerType" : "CustA",
"ProductType" : "ProdB",
}
{
"CustomerType" : "CustA",
"ProductType" : "ProdB",
}
{
"CustomerType" : "CustA",
"ProductType" : "ProdA",
}

The group method doesn't know about g.Key.ProductType so we have to project the elements of the key in the project method.
collectionName.Aggregate()
.Match(filter)
.Project(x => new {x.CustomerType, x.ProductName, x.SubscriptionId })
.Group(x => new { x.ProductName, x.CustomerType }, g => new
{
Id = g.Key,
ActiveTrials = g.Count()
})
.Project(x => new ActiveTrialsByProduct()
{
CustomerType = x.Id.CustomerType,
Product = x.Id.ProductName,
ActiveTrials = x.ActiveTrials
}).ToList();

Aggreagation on Mongo c# driver with not null field does not filter

I am doing an aggregation in Mongo and I cannot get the expected results;
I want to count those with a null column and those with not null but apparently
{ "$ne" : ["$RequestedOn", null] } is coming always as true.
I cannot really see where the problem is.
This is the data in the collection
/* 1 */
{
"_id" : ObjectId("56cf03445667a09b17f661ef"),
"Name" : "User1 Test1",
"AccountRef" : "AccountRef1",
"Voucher" : "Voucher1",
"Email" : "AccountRef1#server.ie",
"CampaignId" : ObjectId("56c752d439bac5655eec7fb7")
}
/* 2 */
{
"_id" : ObjectId("56cf034c5667a09b17f661f0"),
"Name" : "User2 Test2",
"AccountRef" : "AccountRef2",
"Voucher" : "Voucher2",
"Email" : "AccountRef1#server.ie",
"CampaignId" : ObjectId("56c752d439bac5655eec7fb7")
}
/* 3 */
{
"_id" : ObjectId("56cf03565667a09b17f661f1"),
"Name" : "User3 Test3",
"AccountRef" : "AccountRef3",
"Voucher" : "Voucher3",
"GuidLink" : "7f079244-d94f-5e4a-8096-59b6df3ef64f",
"Email" : "AccountRef1#server.ie",
"CampaignId" : ObjectId("56c752d439bac5655eec7fb7")
}
/* 4 */
{
"_id" : ObjectId("56cf035e5667a09b17f661f2"),
"Name" : "User4 Test4",
"AccountRef" : "AccountRef4",
"Voucher" : "Voucher4",
"Email" : "AccountRef1#server.ie",
"RequestedOn" : ISODate("2016-02-29T22:49:20.201Z"),
"CampaignId" : ObjectId("56c752d439bac5655eec7fb7")
}
And this is how I build the query
var wlUsers=Database.Collection;
var cs = wlUsers.AsQueryable()
.Where(wl => wl.CampaignId == id)
.Select(p=> new { id = p.CampaignId, p.RequestedOn})
.GroupBy(wl => wl.id)
.Select(g => new CampaignStatistics{
ExistingCustomers = g.Count(),
TotalOrdered = g.Sum(w => w.RequestedOn.HasValue ? 1 :0),
LastOrder = g.Max(w => w.RequestedOn),
FirstOrder = g.Min(w => w.RequestedOn),
Last24hOrders = g.Sum(w =>
w.RequestedOn > DateTime.Today.AddDays(-1)?1:0),
});
The result is
{
"ExistingCustomers" : 4,
"TotalOrdered" : 4,
"LastOrder" : ISODate("2016-02-29T22:49:20.201Z"),
"FirstOrder" : ISODate("2016-02-29T22:49:20.201Z"),
"Last24hOrders" : 4
}
But it should be
{
"ExistingCustomers" : 4,
"TotalOrdered" : 1,
"LastOrder" : ISODate("2016-02-29T22:49:20.201Z"),
"FirstOrder" : ISODate("2016-02-29T22:49:20.201Z"),
"Last24hOrders" : 0
}
So the totalOrdered has to be 1 and the 'Last24hOrders' = 0.
Also I am new on Mongo so I'll appreciate any tips advises on how to build these projections

I will suspect this line
TotalOrdered = g.Sum(w => w.RequestedOn.HasValue ? 1 :0),
try to change to:
TotalOrdered = g.Sum(w => (w.RequestedOn.HasValue && w.RequestedOn.Value > DateTime.Min) ? 1 :0)

thx profesor79
I kind of sorted that way. This is how I did it
var project = agg.Project(r =>
new{
id = r.CampaignId,
Last24H = r.RequestedOn.Value > DateTime.Today.AddDays(-1) ? 1 : 0,
HasRequested = (r.RequestedOn.Value > new DateTime() ? 1 : 0), //for some reason has value does not work (it is always true)
Date = r.RequestedOn});
var match = project.Match(p => p.id == id);
var group = match.Group(
r => r.id,
g => new CampaignStatistics{
ExistingCustomers = g.Count(),
TotalOrdered = g.Sum(p => p.HasRequested),
LastOrder = g.Max(w => w.Date),
FirstOrder = g.Min(w => w.Date),
Last24HOrders = g.Sum(p => p.Last24H)});

How to filter a nested document in a collection and update a filed?

I have a collection called Jobs.
[
{
_id : 1
content:[
{
_id : 1,
stuff: "A"
},
{
_id : 2,
stuff: "B"
}
]
},
{
_id : 2
content:[
{
_id : 1,
stuff: "X"
},
{
_id : 2,
stuff: "Y"
}
]
},
...
]
I would like to update the stuff field to be Z where job._id = 2 and content._id = 2.
I have tried something like this:
var collection = database.GetCollection("Jobs");
var query = Query.EQ("_id", jobId);
var job = collection.FindOneAs<Job>(query);
var con = job.content.FirstOrDefault(x => x._id == contentIndex);
if (con != null) con.stuff = "Z";
collection.Save(job);
So how use a single query something like this to update the nested doc?
collection.Update(Query.EQ("content._id", contentIndex), Update.Set("content.$.stuff", "Z"));
Thanks in advance!

In the shell, you could use the positional operator and update the element using a query like this:
db.jobs.update({"_id": 2, "content._id":2}, {"content.$.stuff", "Z"});
The same query can be translated to C# (which I'm not familiar with).