We Keep Coding

NEST Api SearchAfter return null in NEST but works in Kibana

We are using elastic search just for document search in our application so we don't have any one expert in it. I was able to use TermQuery, SimpleQueryStringQuery and MatchPhraseQuery successfully. But I found out in documentation that using From & Size for pagination is not good for production and Search After is recommended.
But my implementation return null. It is confusing for me what should be in <Project> parameter as shown in Nest API Object Initializer Syntax in docs here.
My code looks like this:
var request = new SearchRequest<ElasticSearchJsonObject._Source>
{
//Sort = new List<ISort>
//{
// new SortField { Field = Field<ElasticSearchJsonObject>(p=>)}
//},
SearchAfter = new List<object> {
},
Size = 20,
Query = query
};
Reality is I don't understand this. Over here ElasticSearchJsonObject._Source is the class to map returned results.
My documents are simple text documents and I only want documents sorted according to score so document Id is not relevant.
There was already a question like this on SO but I can't find it somehow.
Update
After looking at answer I updated my code and though query obtained does work. It return result in kibana but not in NEST.
This is the new updated code:
var request = new SearchRequest<ElasticSearchJsonObject.Rootobject>
{
Sort = new List<ISort>
{
new SortField { Field = "_id", Order = SortOrder.Descending}
},
SearchAfter = new List<object> {
"0fc3ccb625f5d95b973ce1462b9f7"
},
Size = 1,
Query = query
};
Over here I am using size=1 just for test as well as hard code _id value in SearchAfter.
The query generated by NEST is:
{
"size": 1,
"sort": [
{
"_id": {
"order": "desc"
}
}
],
"search_after": [
"0fc3ccb625f5d95b973ce1462b9f7"
],
"query": {
"match": {
"content": {
"query": "lahore",
"fuzziness": "AUTO",
"prefix_length": 3,
"max_expansions": 10
}
}
}
}
The response from the ES does say successful but no results are returned.
Results do return in Kibana
Query status is successful
But...
Total returned is 0 in NEST
Sort value is null in kibana I used TrackScores = true to solve this issue
Here is the debug information:
Valid NEST response built from a successful low level call on POST: /extract/_source/_search?typed_keys=true
# Audit trail of this API call:
- [1] HealthyResponse: Node: http://localhost:9200/ Took: 00:00:00.1002662
# Request:
{"size":1,"sort":[{"_id":{"order":"desc"}}],"search_after":["0fc3ccb625f5d95b973ce1462b9f7"],"query":{"match":{"content":{"query":"lahore","fuzziness":"AUTO","prefix_length":3,"max_expansions":10}}}}
# Response:
{"took":3,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}
So please tell me where I am wrong and what can be the problem and how to solve it.
Update 2:
Code in Controller:
Connection String:
var node = new Uri("http://localhost:9200");
var settings = new ConnectionSettings(node);
settings.DisableDirectStreaming();
settings.DefaultIndex("extract");
var client = new ElasticClient(settings);
Query:
var query = (dynamic)null;
query = new MatchQuery
{
Field = "content",
Query = content,
Fuzziness = Fuzziness.Auto,
PrefixLength = 3,
MaxExpansions = 10
};
Query Builder
var request = new SearchRequest<ElasticSearchJsonObject.Rootobject>
{
Sort = new List<ISort>
{
new SortField { Field = "_id", Order = SortOrder.Descending}
},
SearchAfter = new List<object> {
documentid //sent as parameter
},
Size = 1, //for testing 1 other wise 10
TrackScores = true,
Query = query
};
JSON Query
I use this code to get query I posted above. This query is then passed to kibana with GET <my index name>/_Search and there it works
var stream = new System.IO.MemoryStream();
client.SourceSerializer.Serialize(request, stream);
var jsonQuery = System.Text.Encoding.UTF8.GetString(stream.ToArray());
ES Response
string responseJson = "";
ElasticSearchJsonObject.Rootobject response = new ElasticSearchJsonObject.Rootobject();
var res = client.Search<object>(request);
if (res.ApiCall.ResponseBodyInBytes != null)
{
responseJson = System.Text.Encoding.UTF8.GetString(res.ApiCall.ResponseBodyInBytes);
try
{
response = JsonConvert.DeserializeObject<ElasticSearchJsonObject.Rootobject>(responseJson);
}
catch (Exception)
{
var model1 = new LoginSignUpViewModel();
return PartialView("_NoResultPage", model1);
}
}
This is where things go wrong. Above debug information was captured from response
ElasticSearchJsonObject
Some how I think problem might be here somewhere? The class is generated by taking response from NEST in Search request.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
namespace ESAPI
{
public class ElasticSearchJsonObject
{
public class Rootobject
{
public int took { get; set; }
public bool timed_out { get; set; }
public _Shards _shards { get; set; }
public Hits hits { get; set; }
}
public class _Shards
{
public int total { get; set; }
public int successful { get; set; }
public int skipped { get; set; }
public int failed { get; set; }
}
public class Hits
{
public int total { get; set; }
public float max_score { get; set; }
public Hit[] hits { get; set; }
}
public class Hit
{
public string _index { get; set; }
public string _type { get; set; }
public string _id { get; set; }
public float _score { get; set; }
public _Source _source { get; set; }
}
public class _Source
{
public string content { get; set; }
public Meta meta { get; set; }
public File file { get; set; }
public Path path { get; set; }
}
public class Meta
{
public string title { get; set; }
public Raw raw { get; set; }
}
public class Raw
{
public string XParsedBy { get; set; }
public string Originator { get; set; }
public string dctitle { get; set; }
public string ContentEncoding { get; set; }
public string ContentTypeHint { get; set; }
public string resourceName { get; set; }
public string ProgId { get; set; }
public string title { get; set; }
public string ContentType { get; set; }
public string Generator { get; set; }
}
public class File
{
public string extension { get; set; }
public string content_type { get; set; }
public DateTime last_modified { get; set; }
public DateTime indexing_date { get; set; }
public int filesize { get; set; }
public string filename { get; set; }
public string url { get; set; }
}
public class Path
{
public string root { get; set; }
public string _virtual { get; set; }
public string real { get; set; }
}
}
}
I am sure this can be used to get response.
Please note that in case of simple search this code works:
so for this query below my code is working:
var request = new SearchRequest
{
From = 0,
Size = 20,
Query = query
};

Using from/size is not recommended for deep pagination because of the amount of documents that need to be fetched from all shards for a deep page, only to be discarded when finally returning an overall ordered result set. This operation is inherent to the distributed nature of Elasticsearch, and is common to many distributed systems in relation to deep pagination.
With search_after, you can paginate forward through documents in a stateless fashion and it requires
the documents returned from the first search response are sorted (documents are sorted by _score by default)
passing the values for the sort fields of the last document in the hits from one search request as the values for "search_after": [] for the next request.
In the Search After Usage documentation, a search request is made with sort on NumberOfCommits descending, then by Name descending. The values to use for each of these sort fields are passed in SearchAfter(...) and are the values of Project.First.NumberOfCommits and Project.First.Name properties, respectively. This tells Elasticsearch to return documents that have values for the sort fields that correspond to the sort constraints for each field, and relate to the values supplied in the request. For example, sort descending on NumberOfCommits with a supplied value of 775 means that Elasticsearch should only consider documents with a value less than 775 (and to do this for all sort fields and supplied values).
If you ever need to dig further into any NEST documentation, click the "EDIT" link on the page:
which will take you to the github repository of the documentation, with the original asciidoc markdown for the page:
Within that page will be a link back to the original NEST source code from which the asciidoc was generated. In this case, the original file is SearchAfterUsageTests.cs in the 6.x branch

Visual Studio 2015 is truncating long strings on Text Visualizer

UPDATE: After some research, i've found out that that's an issue with Visual Studio 2015 and Text Visualizer, which you can see here.
To reproduce it, Open QuickWatch (Shift+F9) and in the search box, put new string(' ', 32769). After that click at the magnifier glass and you should see a "..." in the middle of the string...
So, changing my question, is there a way to fix this and make it not truncate, so i can copy without workarounds?
I have this piece of code:
JArray bundlesJson = (JArray)JObject.Parse(System.IO.File.ReadAllText(Server.MapPath("~/itens.json")))["bundles"]; // file "itens.json" can be found at http://pastebin.com/7K15yAVd
System.Text.RegularExpressions.Regex rgx = new System.Text.RegularExpressions.Regex(#"(?<sNome>.*?)\/(?<sClasse>.*?)\/(?<sDescricao>.*?)(?:\/(?<sExtras>.*?)\n|\n|$)");
var text = System.IO.File.ReadAllText(Server.MapPath("~/TextFile1.txt")); // TextFile1.txt is too big to put here, so i've uploaded it here: http://pastebin.com/AtxbYPXc
var matches = rgx.Matches(text);
var lstItens = new List<Item>();
var oJson = new JArray();
foreach (System.Text.RegularExpressions.Match match in matches)
{
var item = new Item()
{
sClasse = match.Groups["sClasse"].Value.Trim(),
sDescricao = match.Groups["sDescricao"].Value.Trim(),
sNome = match.Groups["sNome"].Value.Trim(),
sExtras = match.Groups["sExtras"].Value.Trim(),
};
item.PreencherListaBundles(bundlesJson.ToString());
lstItens.Add(item);
}
var result = JsonConvert.SerializeObject(lstItens, Formatting.Indented);
var backResult = JsonConvert.DeserializeObject<List<Item>>(result);
The lstItens list has all items correctly (501 items), but the result string returns only 48 objects when searched for "Nome":, which is a mandatory field. Why is this happening?
To exemplify the error, look for the item lstItens[166], in the result var, if you search for "Nome": "Fiddlehead Fern" you can see that the item doesn't exists...
What is weird is that backResult.Count will show 501 results, and appears to have every item, but a simple search in the json generated at the result var using a mandatory field "Nome" will result in 48 results, as showed in the image :
Item.cs:
public class Item
{
[JsonProperty(PropertyName = "Nome")]
public string sNome { get; set; }
[JsonProperty(PropertyName = "Classe")]
public string sClasse { get; set; }
[JsonProperty(PropertyName = "Descricao")]
public string sDescricao { get; set; }
[JsonProperty(PropertyName = "Extras")]
public string sExtras { get; set; }
[JsonProperty(PropertyName = "Bundles")]
public List<Bundle> lstBundles { get; set; }
public void PreencherListaBundles(string jsonBundles)
{
List<Bundle> lstBundle = JsonConvert.DeserializeObject<List<Bundle>>(jsonBundles.ToString());
this.lstBundles = new List<Bundle>();
lstBundle.ForEach(x =>
{
if (!lstBundles.Select(y => y.sNome).Contains(x.sNome) && x.lstItens.Select(y => y.sNome).Contains(sNome))
{
lstBundles.Add(new Bundle() { sLocal = x.sLocal, sNome = x.sNome, sRecompensa = x.sRecompensa, lstItens = x.lstItens });
}
});
}
}
Bundle.cs
public class Bundle
{
[JsonProperty(PropertyName = "bundle")]
public string sNome { get; set; }
[JsonProperty(PropertyName = "location")]
public string sLocal { get; set; }
[JsonProperty(PropertyName = "reward")]
public string sRecompensa { get; set; }
[JsonProperty(PropertyName = "items")]
public List<BundleItem> lstItens { get; set; }
public class BundleItem
{
[JsonProperty(PropertyName = "name")]
public string sNome { get; set; }
[JsonProperty(PropertyName = "description")]
public string sDescricao { get; set; }
[JsonProperty(PropertyName = "quantity")]
public int nQuantidade { get; set; }
[JsonProperty(PropertyName = "quality")]
public string sQualidade { get; set; }
}
}
EDIT: Looks like that bug is not happening on some machines, like you can see with the Gerard Sexton's answer, but when i run the same code he ran i still get the 48 results. some more details can be found in this discussion: https://chat.stackoverflow.com/rooms/106307/discussion-between-gerard-sexton-and-gabriel-duarte

Try this:
var result = JsonConvert.SerializeObject(lstItens, Formatting.Indented);
//Testing deserialization
var backResult = JsonConvert.DeserializeObject<List<Item>>(result);
The result variable is a string which contains the properly formatted string.

I was not able to reproduce the bug using VS2015, JSON.NET 8.0.3.
I have included my debugger output.
Using your code and files, unchanged, I got 501 results and sJson contains Fiddlehead Fern and the visualizer shows item 166. Deserialization also works correctly.
Please check character encoding and these kind of things. Your system locale might cause something to act differently. Just a thought.

Creating a Parse class to handle parsing of file being read in

I wrote a parse class trying to handle parsing the data from a string array into it's appropriate value. I am trying to test this program to see if it will print out the value parse.open, and it is not. It is printing up 0's for the moment (which isn't accurate), until i could figure out why it's not showing what I need.
while (!r.EndOfStream)
{
ParseFileRead parse = new ParseFileRead();
string line = r.ReadLine();
//Send this to Parse class
string [] values = line.Split(',');
//parse records
Console.WriteLine(values[6]); //This is printing the accurate value for parse.open
ParseFileRead.Parse(values);
Console.WriteLine(parse.open); //This is not printing the accurate value
}
Console.Read();
vWriteFile.Close();
And here is my ParseFileRead class:
public class ParseFileRead
{
public int open { get; set; }
public int buy { get; set; }
public int sell { get; set; }
public double settleMM { get; set; }
public string account { get; set; }
public string underlying { get; set; }
public string symbol { get; set; }
public static void Parse(string[] arr)
{
ParseFileRead parse = new ParseFileRead();
parse.account = arr[0];
parse.underlying = arr[12];
parse.symbol = arr[1];
parse.open = Convert.ToInt32(arr[6]);
parse.buy = Convert.ToInt32(arr[7]);
parse.sell = Convert.ToInt32(arr[8]);
parse.settleMM = Convert.ToDouble(arr[10]);
}
}

This is actually correct.
The default value for an uninitialized int is 0.
You are creating a new instance of your ParseFileRead class which will have a value of 0 for open. You then check your parsed value to make sure it's reading in correctly using Console.WriteLine(values[6]);.
Next, you try to parse your values using the Parse function of your ParseFileRead class; which is a void function so it has no return value.
Inside your Parse function you have: ParseFileRead parse = new ParseFileRead(); which creates yet another new instance of your class with a value of 0 for open. This particular instance is never used anywhere and is not the same as the values of the properties created with your initial instance of ParseFileRead
If you put a Console.Write in your Parse function, I'm sure that you will see it being parsed correctly.
So you have 2 options:
Set the properties of your ParseFileRead inside the Parse class without creating a new instance of ParseFileRead
Return the newly created ParseFileRead instance out of your Parse function.
Or a 3rd Option, which is probably best as suggested by Plutonix:
/*Parse class*/
public class ParseFileRead
{
public int open { get; set; }
public int buy { get; set; }
public int sell { get; set; }
public double settleMM { get; set; }
public string account { get; set; }
public string underlying { get; set; }
public string symbol { get; set; }
public ParseFileRead(string[] arr)
{
this.account = arr[0];
this.underlying = arr[12];
this.symbol = arr[1];
this.open = Convert.ToInt32(arr[6]);
this.buy = Convert.ToInt32(arr[7]);
this.sell = Convert.ToInt32(arr[8]);
this.settleMM = Convert.ToDouble(arr[10]);
}
}
/*Parsing code*/
while (!r.EndOfStream)
{
string line = r.ReadLine();
//Send this to Parse class
string [] values = line.Split(',');
//parse records
Console.WriteLine(values[6]); //This is printing the accurate value for parse.open
ParseFileRead parse = new ParseFileRead(values);
Console.WriteLine(parse.open); //This is not printing the accurate value
}

Load csv file and save into sql table

I have csv file (file) and structure of file like this:
Amount;P_price;Ean;Number;Name;DPH;certifikate;o1;o2;ZC
0;168,00;8806333394584;E1347;MISSHA Gel;21;106;0002;0001;290
0;156,80;8806336488488;E1357;MISSHA Lotion;21;106;0002;0001;271
0;123,20;8806584752571;E1367;MISSHA Mist;21;106;0002;0001;213
I want to load all rows without first, where i have names of columns...This values i want to save into sql table...I know how i write into sql, but i need to know how i could load value from each row to this variables:
Amount
P_price
Ean
Number
Name
DPH
certifikate
o1
o2
ZC
The filepath i have into: string file;
Have you any ideas?

If you are sure that's the format and there will never be any ; in the actual values, then you can use a quick and dirty method:
foreach(String line in File.ReadAllLines(path).Skip(1))
{
String[] columns = line.Split(';');
String amount = columns[0];
String P_price = columns[1];
//etc
}

please tried with below link and let me know any help need
http://www.mssqltips.com/sqlservertutorial/203/simple-way-to-import-data-into-sql-server/

in sql server
BULK INSERT [dbo].[csv]
FROM 'C:\Users\...' --file path
WITH
(
FIRSTROW = 2,
FIELDTERMINATOR = ',', --CSV field delimiter
ROWTERMINATOR = '\n', --Use to shift the control to next row
ERRORFILE = 'C:\Users\file\...',
TABLOCK
)

DRY - Don't Repeat Yourself
http://www.filehelpers.com/ will help you to work with the csv files
Example:
File with CSV:
10248|VINET|04071996|32.38
10249|TOMSP|05071996|11.61
10250|HANAR|08071996|65.83
10251|VICTE|08071996|41.34
...............
Data transfer object:
[DelimitedRecord("|")]
public class Orders
{
public int OrderID;
public string CustomerID;
[FieldConverter(ConverterKind.Date, "ddMMyyyy")]
public DateTime OrderDate;
public decimal Freight;
}
Reader:
FileHelperEngine<Orders> engine = new FileHelperEngine<Orders>();
// to Read use:
Orders[] res = engine.ReadFile("TestIn.txt");
And than:
For you:
[DelimitedRecord(";")]
[IgnoreFirst(1)]
public class RootObject
{
public string Amount { get; set; }
public string P_price { get; set; }
public object Ean { get; set; }
public string Number { get; set; }
public string Name { get; set; }
public int DPH { get; set; }
public int certifikate { get; set; }
public int o1 { get; set; }
public int o2 { get; set; }
public int ZC { get; set; }
}
FileHelperEngine<Orders> engine = new FileHelperEngine<RootObject>();
// to Read use:
Orders[] res = engine.ReadFile("TestIn.txt");

How to load XML Elements using LINQ from XDocument into a class (not using Descendants)

How to load XML Elements using LINQ from XDocument into a c# class. I don't want to use XDocument.Descendants because the settings are not repeated and occur only once in the XML.
This works but I think there must be another way where I don't have to use IEnumerable or use ToArray() and use the first element [0]. Any suggestions?
CODE
public class BackupItem // class needed so LINQ reads XML into an editable DbGrid
{
public bool Backup { get; set; }
public bool IncludeSubDir { get; set; }
public string BackupLabel { get; set; }
public string BackupPath { get; set; }
}
public class BackupSettings
{
public bool IncludeDateStamp { get; set; }
public bool DatePrefix { get; set; }
public bool DateSuffix { get; set; }
public string DateFormat { get; set; }
public bool ZipCompress { get; set; }
public bool ZipPrefix { get; set; }
public bool ZipSuffix { get; set; }
public string ZipText { get; set; }
public bool BackupToSiblingFolder { get; set; }
public string SiblingFolder { get; set; }
public bool BackupToFolder { get; set; }
public bool IncludeFullPath { get; set; }
public string BackupFolder { get; set; }
}
// use a LINQ query to load xml file into datagridview and make datagrid editable (create class with get/set)
var q = from arg in GvXMLDoc.Descendants("BackupItem")
select new BackupItem()
{
Backup = (bool)arg.Element("IncludeDirectory"),
IncludeSubDir = (bool)arg.Element("IncludeSubDirectories"),
BackupLabel = (string)arg.Element("BackupLabel"),
BackupPath = (string)arg.Element("BackupPath")
};
dataGridView1.DataSource = q.ToList();
// load global variable
IEnumerable<BackupSettings> GvBackupSettings = from arg in GvXMLDoc.Element("Backup").Elements("Settings")
select new BackupSettings()
{
IncludeDateStamp = (bool)arg.Element("IncludeDateStamp"),
DatePrefix = (bool)arg.Element("DatePrefix"),
DateSuffix = (bool)arg.Element("DateSuffix"),
DateFormat = (string)arg.Element("DateFormat"),
ZipCompress = (bool)arg.Element("ZipCompress"),
ZipPrefix = (bool)arg.Element("ZipPrefix"),
ZipSuffix = (bool)arg.Element("ZipSuffix"),
ZipText = (string)arg.Element("ZipText"),
BackupToSiblingFolder = (bool)arg.Element("BackupToSiblingFolder"),
SiblingFolder = (string)arg.Element("SiblingFolder"),
BackupToFolder = (bool)arg.Element("BackupToFolder"),
IncludeFullPath = (bool)arg.Element("IncludeFullPath"),
BackupFolder = (string)arg.Element("BackupFolder")
};
I can access the values but it seems a very 'messy way'.
var s = GvBackupSettings.ToArray()[0].DateSuffix;
var t = GvBackupSettings.ToArray()[0].DateFormat;
XML FILE
<?xml version='1.0'?>
<Backup>
<Settings>
<IncludeDateStamp>true</IncludeDateStamp>
<DatePrefix>false</DatePrefix>
<DateSuffix>true</DateSuffix>
<DateFormat>yyyy-MM-dd h.m.s</DateFormat>
<ZipCompress>false</ZipCompress>
<ZipPrefix>false</ZipPrefix>
<ZipSuffix>false</ZipSuffix>
<ZipText></ZipText>
<BackupToSiblingFolder>true</BackupToSiblingFolder>
<SiblingFolder>backups</SiblingFolder>
<BackupToFolder>false</BackupToFolder>
<IncludeFullPath>true</IncludeFullPath>
<BackupFolder>C:\\backup</BackupFolder>
</Settings>
<BackupItem>
<IncludeDirectory>true</IncludeDirectory>
<IncludeSubDirectories>true</IncludeSubDirectories>
<BackupLabel>Backup1</BackupLabel>
<BackupPath>C:\TestFiles\Xml\Samples</BackupPath>
</BackupItem>
<BackupItem>
<IncludeDirectory>true</IncludeDirectory>
<IncludeSubDirectories>false</IncludeSubDirectories>
<BackupLabel>Backup2</BackupLabel>
<BackupPath>C:\TestFiles\Xml\Samples</BackupPath>
</BackupItem>
</Backup>
EDIT 1
I also am trying to deserialize the XML (thanks for idea) so I don't have to cast each item. This does not work.. I don't want to have to run the XSD tool either and have a huge class file...
XmlRootAttribute rootAttribute = new XmlRootAttribute("Backup");
XmlSerializer deserializer = new XmlSerializer((typeof(BackupSettings)), rootAttribute);
GvBackupSettings = (BackupSettings)deserializer.Deserialize(XmlReader.Create(GvXMLFileName));
EDIT 2
Ended up serializing and deserializing xml as suggested below using stardard XSD.exe tool to generate the c# class. Especially as I had to read and write to same xml file. Note: Make sure you verify/modify the generated xsd file first.

You don't need a query if you just need the first element:
XElement settings = GvXMLDoc.Element("Backup").Element("Settings");
BackupSettings GvBackupSettings = new BackupSettings
{
IncludeDateStamp = (bool)settings.Element("IncludeDateStamp"),
DatePrefix = (bool)settings.Element("DatePrefix"),
...
};
var s = GvBackupSettings.DateSuffix;
var t = GvBackupSettings.DateFormat;