We Keep Coding

NEST Api SearchAfter return null in NEST but works in Kibana

We are using elastic search just for document search in our application so we don't have any one expert in it. I was able to use TermQuery, SimpleQueryStringQuery and MatchPhraseQuery successfully. But I found out in documentation that using From & Size for pagination is not good for production and Search After is recommended.
But my implementation return null. It is confusing for me what should be in <Project> parameter as shown in Nest API Object Initializer Syntax in docs here.
My code looks like this:
var request = new SearchRequest<ElasticSearchJsonObject._Source>
{
//Sort = new List<ISort>
//{
// new SortField { Field = Field<ElasticSearchJsonObject>(p=>)}
//},
SearchAfter = new List<object> {
},
Size = 20,
Query = query
};
Reality is I don't understand this. Over here ElasticSearchJsonObject._Source is the class to map returned results.
My documents are simple text documents and I only want documents sorted according to score so document Id is not relevant.
There was already a question like this on SO but I can't find it somehow.
Update
After looking at answer I updated my code and though query obtained does work. It return result in kibana but not in NEST.
This is the new updated code:
var request = new SearchRequest<ElasticSearchJsonObject.Rootobject>
{
Sort = new List<ISort>
{
new SortField { Field = "_id", Order = SortOrder.Descending}
},
SearchAfter = new List<object> {
"0fc3ccb625f5d95b973ce1462b9f7"
},
Size = 1,
Query = query
};
Over here I am using size=1 just for test as well as hard code _id value in SearchAfter.
The query generated by NEST is:
{
"size": 1,
"sort": [
{
"_id": {
"order": "desc"
}
}
],
"search_after": [
"0fc3ccb625f5d95b973ce1462b9f7"
],
"query": {
"match": {
"content": {
"query": "lahore",
"fuzziness": "AUTO",
"prefix_length": 3,
"max_expansions": 10
}
}
}
}
The response from the ES does say successful but no results are returned.
Results do return in Kibana
Query status is successful
But...
Total returned is 0 in NEST
Sort value is null in kibana I used TrackScores = true to solve this issue
Here is the debug information:
Valid NEST response built from a successful low level call on POST: /extract/_source/_search?typed_keys=true
# Audit trail of this API call:
- [1] HealthyResponse: Node: http://localhost:9200/ Took: 00:00:00.1002662
# Request:
{"size":1,"sort":[{"_id":{"order":"desc"}}],"search_after":["0fc3ccb625f5d95b973ce1462b9f7"],"query":{"match":{"content":{"query":"lahore","fuzziness":"AUTO","prefix_length":3,"max_expansions":10}}}}
# Response:
{"took":3,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}
So please tell me where I am wrong and what can be the problem and how to solve it.
Update 2:
Code in Controller:
Connection String:
var node = new Uri("http://localhost:9200");
var settings = new ConnectionSettings(node);
settings.DisableDirectStreaming();
settings.DefaultIndex("extract");
var client = new ElasticClient(settings);
Query:
var query = (dynamic)null;
query = new MatchQuery
{
Field = "content",
Query = content,
Fuzziness = Fuzziness.Auto,
PrefixLength = 3,
MaxExpansions = 10
};
Query Builder
var request = new SearchRequest<ElasticSearchJsonObject.Rootobject>
{
Sort = new List<ISort>
{
new SortField { Field = "_id", Order = SortOrder.Descending}
},
SearchAfter = new List<object> {
documentid //sent as parameter
},
Size = 1, //for testing 1 other wise 10
TrackScores = true,
Query = query
};
JSON Query
I use this code to get query I posted above. This query is then passed to kibana with GET <my index name>/_Search and there it works
var stream = new System.IO.MemoryStream();
client.SourceSerializer.Serialize(request, stream);
var jsonQuery = System.Text.Encoding.UTF8.GetString(stream.ToArray());
ES Response
string responseJson = "";
ElasticSearchJsonObject.Rootobject response = new ElasticSearchJsonObject.Rootobject();
var res = client.Search<object>(request);
if (res.ApiCall.ResponseBodyInBytes != null)
{
responseJson = System.Text.Encoding.UTF8.GetString(res.ApiCall.ResponseBodyInBytes);
try
{
response = JsonConvert.DeserializeObject<ElasticSearchJsonObject.Rootobject>(responseJson);
}
catch (Exception)
{
var model1 = new LoginSignUpViewModel();
return PartialView("_NoResultPage", model1);
}
}
This is where things go wrong. Above debug information was captured from response
ElasticSearchJsonObject
Some how I think problem might be here somewhere? The class is generated by taking response from NEST in Search request.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
namespace ESAPI
{
public class ElasticSearchJsonObject
{
public class Rootobject
{
public int took { get; set; }
public bool timed_out { get; set; }
public _Shards _shards { get; set; }
public Hits hits { get; set; }
}
public class _Shards
{
public int total { get; set; }
public int successful { get; set; }
public int skipped { get; set; }
public int failed { get; set; }
}
public class Hits
{
public int total { get; set; }
public float max_score { get; set; }
public Hit[] hits { get; set; }
}
public class Hit
{
public string _index { get; set; }
public string _type { get; set; }
public string _id { get; set; }
public float _score { get; set; }
public _Source _source { get; set; }
}
public class _Source
{
public string content { get; set; }
public Meta meta { get; set; }
public File file { get; set; }
public Path path { get; set; }
}
public class Meta
{
public string title { get; set; }
public Raw raw { get; set; }
}
public class Raw
{
public string XParsedBy { get; set; }
public string Originator { get; set; }
public string dctitle { get; set; }
public string ContentEncoding { get; set; }
public string ContentTypeHint { get; set; }
public string resourceName { get; set; }
public string ProgId { get; set; }
public string title { get; set; }
public string ContentType { get; set; }
public string Generator { get; set; }
}
public class File
{
public string extension { get; set; }
public string content_type { get; set; }
public DateTime last_modified { get; set; }
public DateTime indexing_date { get; set; }
public int filesize { get; set; }
public string filename { get; set; }
public string url { get; set; }
}
public class Path
{
public string root { get; set; }
public string _virtual { get; set; }
public string real { get; set; }
}
}
}
I am sure this can be used to get response.
Please note that in case of simple search this code works:
so for this query below my code is working:
var request = new SearchRequest
{
From = 0,
Size = 20,
Query = query
};

Using from/size is not recommended for deep pagination because of the amount of documents that need to be fetched from all shards for a deep page, only to be discarded when finally returning an overall ordered result set. This operation is inherent to the distributed nature of Elasticsearch, and is common to many distributed systems in relation to deep pagination.
With search_after, you can paginate forward through documents in a stateless fashion and it requires
the documents returned from the first search response are sorted (documents are sorted by _score by default)
passing the values for the sort fields of the last document in the hits from one search request as the values for "search_after": [] for the next request.
In the Search After Usage documentation, a search request is made with sort on NumberOfCommits descending, then by Name descending. The values to use for each of these sort fields are passed in SearchAfter(...) and are the values of Project.First.NumberOfCommits and Project.First.Name properties, respectively. This tells Elasticsearch to return documents that have values for the sort fields that correspond to the sort constraints for each field, and relate to the values supplied in the request. For example, sort descending on NumberOfCommits with a supplied value of 775 means that Elasticsearch should only consider documents with a value less than 775 (and to do this for all sort fields and supplied values).
If you ever need to dig further into any NEST documentation, click the "EDIT" link on the page:
which will take you to the github repository of the documentation, with the original asciidoc markdown for the page:
Within that page will be a link back to the original NEST source code from which the asciidoc was generated. In this case, the original file is SearchAfterUsageTests.cs in the 6.x branch

Properties of referenced class not showing during internal assignment

I have below ItemCountAmount.cs class.
public class ItemCountAmount
{
public int Count { get; set; }
public string Amount { get; set; }
}
and I am referencing it in one more class ChartData.cs as below:
public class ChartData
{
public ChartData()
{
Purchase = Stocks = Sales = new ItemCountAmount();
}
public string period { get; set; }
public ItemCountAmount Purchase { get; set; }
public ItemCountAmount Stocks { get; set; }
public ItemCountAmount Sales { get; set; }
}
Now when I try to assign the values for the ChartData properties, it isn't recognizing or available. Compile time exception will occur in VS.
var model = new ChartData
{
period = yearMonth,
Purchase.Count = await purchase.CountAsync(x => x.order_date.Month == mnth && x.order_date.Year == yr),
}
But Purchase.Count or Purchase.Amount isn't identified here where in if I try it as
var model=new ChartData();
model.Purchase.Count = .....;
model.Purchase.Amount = .....;
.......
it works fine, I mean, property Count and Amount is identified. I am still not getting why the first method is not able to identify the properties from ItemCountAmount.cs class which is referenced in ChartData. Any specific reasons for this?

I don't believe the C# object initialiser supports that syntax, however, you can just do this. I am not sure whether this would result in two constructions and assignemnts to Purchase though, you could break point that yourself and see.
var model = new ChartData
{
Purchase = new ItemCountAmount
{
count = await purchase.CountAsync(x => x.order_date.Month == mnth && x.order_date.Year == yr)
}
}

LINQ statement - WHERE clause for ENUM type

I followed MSDN tutorial about filtering functionality in ASP.NET MVC web app > http://www.asp.net/mvc/overview/getting-started/getting-started-with-ef-using-mvc/sorting-filtering-and-paging-with-the-entity-framework-in-an-asp-net-mvc-application
Unfortunately, I can't select an object which has an enum attriubute.
Used model is:
public Bus()
{
public int BusID { get; set; }
public string RegNum { get; set; }
public Status? Status { get; set; }
public string Latitude { get; set; }
public string Longitude { get; set; }
}
where the enum type is:
public enum Status
{
ON, OFF
}
According to the mentioned tutorial I implemented sorting method in BusController as shown below:
public ViewResult Index(string searchString)
{
var buses = from b in db.Buses select b;
if (!String.IsNullOrEmpty(searchString))
{
buses = buses.Where(b =>
b.RegNum.ToUpper().Contains(searchString.ToUpper())
);
}
return View(buses.ToList());
}
it works fine for RegNum filtering, but I can't select the Bus for the given status by searchstring value.
b.Status.Equals(searchstring) doesn't work.
I'd be grateful for any hints

That's because Status is not a string. You probably want to convert the searchString to a Status with
Status searchStatus = Enum.Parse(typeof(Status), searchString.ToUpper());
first then use that in your query (NOTE: Do not put the parsing code directly into the query). Or Enum.TryParse to avoid potential exceptions. Also note that what is saved in your DB with be an int, 0 for ON and 1 for OFF.

Try the following changes:
public class Bus
{
public int BusID { get; set; }
public string RegNum { get; set; }
public Status Status { get; set; }
public string Latitude { get; set; }
public string Longitude { get; set; }
}
Take Status as second parameter and filter on both. Sample code below.
public static IEnumerable<Bus> FilterBuses(string searchString, Status status)
{
//Setup some dummy data
var buses = new List<Bus>()
{
new Bus() { BusID = 12, RegNum = "Twelve", Status = Status.ON},
new Bus() { BusID = 13, RegNum = "Thirteen", Status = Status.OFF},
new Bus() { BusID = 20, RegNum = "Twenty", Status = Status.OFF}
};
IEnumerable<Bus> filteredList = new List<Bus>();
if (!String.IsNullOrEmpty(searchString))
{
filteredList = buses.Where<Bus>(b =>
b.RegNum.ToUpper().Contains(searchString.ToUpper()) &&
b.Status == status);
}
return filteredList.ToList();
}
Then, you can call the method as follows:
// Expect only Twelve bus
var result = LinqTestMethod.FilterBuses("Twelve", Status.ON);
// Expect no buses
result = LinqTestMethod.FilterBuses("Twelve", Status.OFF);
// Expect 2 buses -- Twelve and Twenty
result = LinqTestMethod.FilterBuses("T", Status.OFF);

ASP.NET Neo4jClient retrieving incorrect number

I have an ASP.NET Web Forms project which is using Readify-Neo4jClient and Neo4J Community 2.0.3, I'm getting a bug where a number stored in the database is changing its value when retrieved. Here is a picture of what’s in the database and what I can see in VS2013:
https://docs.google.com/file/d/0B6b_N7sDgjmvMVF5TFpaZXJmNFk/edit
The code to retrieve the user is as follows:
IEnumerable<SUser> FoundUsers = Neo4jGraphClient.Cypher.Match("(user:User)")
.Where((SUser user) => user.Email == UserName)
.Return(user => user.As<SUser>())
.Results;
Code to write to the database is as follows:
long DateTimeNow = DateTime.Now.Ticks;
SUser ss = new SUser
{
Id = UserCounter.SubmitAndCommitNewUser(),
DateOfBirth = DobDay.Text + "" + DobMonth.Text + "" + DobYear.Text,
Email = UserName.Text,
FirstName = FirstName.Text,
LastName = LastName.Text,
UserCreatedOn = DateTimeNow,
role = UType.ADMIN,
Status = UStatus.NEW
};
Neo4jReq.CreateSUser(ss);
......
public static SUser CrseateSUser(SUser NewUser)
{
//...
Neo4jGraphClient.Cypher
.Create("(user:User {NewUser})")
.WithParam("NewUser", NewUser)
.ExecuteWithoutResults();
existing = NewUser;
}
Class is as follows:
public class SUser
{
public int Id { get; set; }
public string FirstName { get; set; }
public string LastName { get; set; }
public string DateOfBirth { get; set; }
public string Email { get; set; }
public UType role { get; set; }
public UStatus Status { get; set; }
public string pass { get; set; }
public string VerificationGUID { get; set; }
public long UserCreatedOn { get; set; }
public string UserNotes { get; set; }
}
Any ideas on whats causing this?

Right - I've got this replicating, this looks like a bug(?) in the way the Neo4j browser shows the data (both the current and older webadmin), so the data stored in Neo4j is correct, but it's getting 'rounded' (in a sense anyway) in the display, if you run the 'Get' query in the browser you get the '00' ending, this also happens in the old web admin:
http://localhost:7474/webadmin/
if you run the query in the 'Data Browser'.
However, if you run the query in the Console (http://localhost:7474/webadmin/#/console/) you'll get the correct results. Neo4jClient is giving you the right results, it's the browser that is wrong in this case.

EF: Fetching Object containing a list as property

I have the following classes:
public class DbCatalogEntry
{
public int id { get; set; }
private string filename;
public long size { get; set; }
public int duration { get; set; }
public int height { get; set; }
public int width { get; set; }
public string vcodec { get; set; }
public List<CatalogAudioStreamEntry> audiostreams { get; set; }
public string imdbId { get; set; }
public int ownedByUser { get; set; }
public string mxHash { get; set; }
}
public class CatalogAudioStreamEntry
{
public int bitrate { get; set; }
public string codec { get; set; }
public string language { get; set; }
public int channels { get; set; }
public string features { get; set; }
}
This is a one to many relationship. I am trying to retrieve a list of DbCatalogEntry objects from the database and also filling the audiostreams in the same query.
I tried the following but it does not work:
var MovieEntryList =
(from vwCat in ctx.sp_GetMovieCatalog(getCurrentUserId())
select new DbCatalogEntry()
{
id = vwCat.id,
size = vwCat.size,
duration = vwCat.duration,
vcodec = vwCat.codec,
Filename = vwCat.filename,
imdbId = vwCat.imdb_id,
mxHash = vwCat.mxhash,
ownedByUser = (int)vwCat.owned,
width = vwCat.width,
height = vwCat.height,
audiostreams =
(from astr in ctx.audiostreamentry
where astr.movie_id == vwCat.id
select new CatalogAudioStreamEntry()
{
bitrate = astr.bitrate,
channels = astr.channels,
codec = astr.codec,
features = astr.features,
language = astr.language
}).ToList()
}).ToList();
Search revealed that you cannot put a ToList() into a linq to entity query as it cannot be converted converted. I read multiple suggestions about changing audiostreams to IEnumerable but was not able to get this to work either. Most attempts compiled fine but faild during runtime with Unable to create a constant value of type...
Can anyone point me to the correct direction to solve this issue? The important thing is that the round trips to the database must be kept at a minimum so it would not be possible to create a client-side subquery for each DbCatalogEntry to fill the list.
Update #1:
To provide more details: This is the model that was generated by EF from my database:
sp_GetMovieCatalog(getCurrentUserId()) is a stored function on the SQL server which accepts one parameter which is used to filter the results.
What I want to do is selecting from movieenty and load all associated rows from audiostreamenty.
This data should be used to create instances of DbCatalogEntry, so the result type would be List<DbCatalogEntry>.
Is that possible at all? Maybe there is a better/easier solution?

I believe, you have your problem, because trying to use same context for multiple queries
You can try it this way:
var MovieEntryList =
(from vwCat in ctx.sp_MovieCatalog
where vwCat.owned = currentUserId
select new DbCatalogEntry()
{
id = vwCat.id,
size = vwCat.size,
duration = vwCat.duration,
vcodec = vwCat.codec,
Filename = vwCat.filename,
imdbId = vwCat.imdb_id,
mxHash = vwCat.mxhash,
ownedByUser = (int)vwCat.owned,
width = vwCat.width,
height = vwCat.height,
audiostreams = vwCat.audiostreamentry.Select(astr=>
new CatalogAudioStreamEntry()
{
bitrate = astr.bitrate,
channels = astr.channels,
codec = astr.codec,
features = astr.features,
language = astr.language
}).ToList()
}).ToList();
Or this way:
var MovieEntryList = ctx.audiostreamentry.where(p=>p.movieCatolog.ownedByUser - currentUserId)
//.ToList() //if you call ToList() here futher will be LINQ to Object, and you want have most of problems
.GroupBy(p=>new {Id = p.movieCatolog.id, Owner = p.movieCatolog.p.movieCatolog.id})
.Select(p=> new DbCatalogEntry{
id = p.Key.Id,
ownedByUser = p.Key.Owner,
audiostrams = p.Select(x=>new CatalogAudioStreamEntry
{
bitrate = astr.bitrate,
channels = astr.channels,
codec = astr.codec,
features = astr.features,
language = astr.language
})
})