We Keep Coding

Scroll Query is not working Elastic Search

This is my Elastic Search Query for Obtaining the desired output the only reason i am using sroll is that i am having multiple reocrds and i am getting only some records limited by the size i want to get all the records
filters.Add(new TermsQuery
{
Field = new Field("MERCHANTNO"),
Terms = BranchCode,
});
filterClause.Add(new DateRangeQuery
{
Boost = 1.1,
Field = new Field("TRANSACTIONDATE"),
GreaterThanOrEqualTo = DateMath.Anchored(Date).RoundTo(DateMathTimeUnit.Month),
Format = "yyyy-MM",
TimeZone = "+01:00"
});
var SearchRequest = new SearchRequest<Acquirer>("acquiringdata")
{
Size = 2,
Scroll=10, //I have Tried
Scroll="10s" //I have tried this
Scroll=1m //I have tried this
Query = new BoolQuery
{
Must = filters,
Filter = filterClause
}
};
var searchResponse = await _elasticClient.SearchAsync<Acquirer>(SearchRequest);
if (searchResponse.ApiCall.ResponseBodyInBytes != null)
{
var requestJson = System.Text.Encoding.UTF8.GetString(searchResponse.ApiCall.RequestBodyInBytes);
var JsonFormatQuery = JsonConvert.SerializeObject(JsonConvert.DeserializeObject(requestJson), Formatting.Indented);
}
while (searchResponse.Documents.Any())
{
//ProcessResponse(searchResponse);
searchResponse = _elasticClient.Scroll<Acquirer>("10s", searchResponse.ScrollId);
}
It is giving me empty array when i just Removed these While Loop it gives me the desired output the Size of 10 which i have asked for
while (searchResponse.Documents.Any())
{
//ProcessResponse(searchResponse);
searchResponse = _elasticClient.Scroll<Acquirer>("10s", searchResponse.ScrollId);
}
This is my Json i got from the Query and i not getting the point that Scroll is not coming in the Query when i have declared it
{
"query": {
"bool": {
"filter": [
{
"range": {
"TRANSACTIONDATE": {
"format": "yyyy-MM",
"gte": "2019-10||/M",
"time_zone": "+01:00",
"boost": 1.1
}
}
}
]
}
},
"size": 2
}

Casting JArray to Dynamic[] so NEST's IndexMany works

Problem description
I need to receive JSON array of objects (with almost any shape) and store them to ES database using function IndexMany (or some similar bulk indexing function). I found some clumsy solution with one drawback - it doesn't set _id property in ES correctly (according to the id property in JSON object).
And also I would like to know if there is any more elegant way how to achieve my goal without casting every JArray item to string and back to ExpandoObject.
Additional info
Elasticsearch DB 7.5.1
NEST (7.6.1)
Newtonsoft.Json (12.0.3)
TLDR
Is there any elegant solution of following code:
var settings = new ConnectionSettings(new Uri("http://localhost:9200")).DefaultIndex("people");
var client = new ElasticClient(settings);
// this string represents incoming JSON message
string json = #"[
{
""id"": ""1"",
""name"": ""Andrej"",
""surname"": ""Burak"",
""dob"": ""1921-11-10T00:00:00+00:00""
},
{
""id"": ""2"",
""name"": ""Franta"",
""surname"": ""Dobrota"",
""dob"": ""1933-10-05T00:00:00+00:00""
},
{
""id"": ""3"",
""name"": ""Milos"",
""surname"": ""Ovcacek"",
""dob"": ""1988-05-05T00:00:00+00:00""
}
]";
JArray jArray = JArray.Parse(json);
foreach (var jtoken in jArray)
{
var jobj = (JObject)jtoken;
jobj.Add("DbCreated", JToken.FromObject(DateTime.UtcNow));
jobj.Add("DbCreatedBy", JToken.FromObject("authors name"));
}
//working, but seems to me a bit too clumsy to convert every item to string and then back to dynamic object
var converter = new ExpandoObjectConverter();
dynamic[] dlst = jArray.Select(t => (dynamic)JsonConvert.DeserializeObject<ExpandoObject>(t.ToString(), converter)).ToArray();
//not working cast
dynamic[] dlstNW = jArray.ToObject<dynamic[]>();
var indexManyResponse = client.IndexMany(dlst); //working partially (but not using ID as index)
var indexManyResponseNotWorking = client.IndexMany(jArray); //not working
var indexManyResponseNotWorking2 = client.IndexMany(dlstNW); //not working
// expected behavior
dynamic[] jsondyn = new dynamic[]
{
new { Id = "1", Name = "foo" },
new { Id = "2", Name = "bar" },
new { Id = "3", Name = "baz" },
};
var indexManyResponseWithIndex = client.IndexMany(jsondyn); //working perfectly, but don't know how to acieve this

After #gnud pointed me to low-level client I think I have found my answer. Important thing is to build request manually like it was explained here.
var settings = new ConnectionSettings(new Uri("http://localhost:9200")).DefaultIndex("people").DisableDirectStreaming();
var client = new ElasticClient(settings);
// this string represents incoming JSON message
string json = #"[
{
""id"": ""1"",
""name"": ""Andrej"",
""surname"": ""Burak"",
""dob"": ""1921-11-10T00:00:00+00:00""
},
{
""Id"": ""2"",
""name"": ""Franta"",
""surname"": ""Dobrota"",
""dob"": ""1933-10-05T00:00:00+00:00""
},
{
""Id"": ""3"",
""name"": ""Milos"",
""surname"": ""Ovcacek"",
""dob"": ""1988-05-05T00:00:00+00:00""
}
]";
JArray jArray = JArray.Parse(json);
// I have to build my request manually
List<string> esRequest = new List<string>();
foreach (var jtoken in jArray)
{
var jobj = (JObject)jtoken;
jobj.Add("DbCreated", JToken.FromObject(DateTime.UtcNow));
jobj.Add("DbCreatedBy", JToken.FromObject("authors name"));
string id;
// Ensure that Id is set even if we receive it in lowercase
if( jobj["Id"] == null)
{
if( jobj["id"] == null)
{
throw new Exception("missing Id");
}
id = jobj["id"].Value<string>();
}
else
{
id = jobj["Id"].Value<string>();
}
string indexName = "people";
esRequest.Add($"{{\"index\":{{\"_index\":\"{indexName}\",\"_id\":\"{id}\"}}}}");
esRequest.Add(jobj.ToString(Formatting.None));
}
PostData pd = PostData.MultiJson(esRequest);
var llres = client.LowLevel.Bulk<BulkResponse>(pd);
Hope it helps someone.

amchart in listview post back

I have a webform app where I list items using listview.
there are an amchart created in each item, and I'm using stringBuilder write the script from code, then inserting it into a div called "chartArea" to load the script in each amchart item:
Literal lit = new Literal();
lit.Text = sb2.ToString();
chartArea.Controls.Add(lit);
the above works great during the first page load, but then when I trigger a postback event, I get an error on amchart file:
Unhandled exception at line 106, column 73 in http://localhost:33154/Scripts/amcharts/amcharts.js
0x80004005 - JavaScript runtime error: Unspecified error.
What Shall I do?
This is my code to load the chart:
public void buildChart(string ACreg, HtmlGenericControl chartArea)
{
Literal lit = new Literal();
lit.Text = "";
chartArea.Controls.Add(lit);
//-------- create chart data table----------
int Rdays = 90;
DataTable engChartData = new DataTable();
engChartData.Columns.Add("date", typeof(string));
engChartData.Columns.Add("eng1Val", typeof(string));
engChartData.Columns.Add("eng2Val", typeof(string));
DateTime initialDate = DateTime.Now.AddDays(-Rdays);
for (int i = 0; i <= Rdays; i++)
{
string dateMe = initialDate.AddDays(i).ToString("yyyy-MM-dd");
engChartData.Rows.Add(dateMe, "0", "0");
}
//--------- bind chart data-----------------
using (APMEntitiesW APMe = new APMEntitiesW())
{
var countDate = APMe.Points.Where(x => x.point_Del == false && x.point_Act == true && x.point_Reg == ACreg).GroupBy(g => new { g.point_DateTime, g.point_EGT1, g.point_EGT2 }).Select(lg => new { date = EntityFunctions.TruncateTime(lg.Key.point_DateTime), eng1Val = lg.Key.point_EGT1, eng2Val = lg.Key.point_EGT2 }).ToList();
foreach (DataRow dr in engChartData.Rows)
{
foreach (var data in countDate)
{
string datFromQ = data.date.Value.ToString("yyyy-MM-dd").Replace(" 12:00:00 AM", "");
string deteFromDT = dr["date"].ToString().Replace(" 12:00:00 AM", "");
if (datFromQ == deteFromDT)
{
dr["eng1Val"] = data.eng1Val.ToString();
dr["eng2Val"] = data.eng2Val.ToString();
break;
}
}
}
StringBuilder sb = new StringBuilder();
foreach (DataRow item in engChartData.Rows)
{
string date = string.Format("{0:MMM DD,YYYY}", item["date"]);
sb.AppendLine(#"{""date"": """ + date + #""",");
sb.AppendLine(#"""Engine1"": """ + item["eng1Val"] + #""",");
sb.AppendLine(#"""Engine2"": """ + item["eng2Val"] + #"""},");
}
//ChartData = sb.ToString();
StringBuilder sb2 = new StringBuilder();
sb2.AppendLine(#"<script> var chart = AmCharts.makeChart(""chart" + ACreg + #""", {");
sb2.AppendLine(#"""type"": ""serial"",""categoryField"": ""date"", ""startEffect"": ""easeOutSine"",""dataDateFormat"": ""DD-MM-YYYY"", ""startDuration"": 1,");
sb2.AppendLine(#"""categoryAxis"": {""gridPosition"": ""start"", ""autoGridCount"": false, ""balloonDateFormat"": ""DD MMM YYYY""},");
sb2.AppendLine(#" ""chartScrollbar"": {""enabled"": true, ""offset"": 40, ""oppositeAxis"": false, ""scrollbarHeight"": 13},");
sb2.AppendLine(#" ""chartCursor"": { ""enabled"": true}, ""trendLines"": [],");
sb2.AppendLine(#"""graphs"": [{""balloonText"": ""[[title]]: [[value]]"", ""bullet"": ""round"", ""id"": ""AmGraph-1"", ""title"": ""Engine1"", ""type"": ""smoothedLine"", ""valueField"": ""Engine1"", ""bullet"": ""bubble"", ""bulletBorderAlpha"": 1, ""bulletSize"": 1,""bulletBorderColor"": ""#FF0000"",""bulletBorderThickness"": 1, ""bulletColor"": ""#000000"",""fillAlphas"": 0.41, },");
sb2.AppendLine(#"{""balloonText"": ""[[title]]: [[value]]"",""bullet"": ""square"", ""id"": ""AmGraph-2"", ""title"": ""Engine2"", ""type"": ""smoothedLine"", ""valueField"": ""Engine2"", ""bullet"": ""bubble"",""bulletBorderAlpha"": 1, ""bulletSize"": 1, ""bulletBorderColor"": ""#0098FF"",""bulletBorderThickness"": 1, ""bulletColor"": ""#0098FF"", ""lineColor"": ""#0098FF"",}],");
sb2.AppendLine(#"""guides"": [], ""valueAxes"": [ { ""id"": ""ValueAxis-1"", ""title"": ""Consumption Unit""} ], ""allLabels"": [], ""balloon"": {}, ""legend"": { ""enabled"": true, ""markerBorderThickness"": 0, ""useGraphSettings"": true},");
sb2.AppendLine(#"""titles"": [ { ""id"": ""Title-1"", ""size"": 15, ""text"": ""Engine Fuel Consumption"" } ], ""dataProvider"": [" + sb.ToString() + " ] }) </script>");
lit.Text = sb2.ToString();
chartArea.Controls.Add(lit);
}
}

How to Deserialization JSon complex type C#

I'm able to do jSon Deserialised and also Insertion data within Database table, but i want to insert the correct "ParentID & ParentFullPath" as well within table(Database). How to do it with recursive method or without recursion.
I wrote method on Click event of button like below:
protected void btnSave_Click(object sender, EventArgs e)
{
var converter = new ExpandoObjectConverter();
dynamic message = JsonConvert.DeserializeObject<ExpandoObject>(txtJsonData.Text, converter);
RecursiveMethod(message);
}
below is the Recursive Method::
string parentID = string.Empty;
string parentPath = string.Empty;
public void RecursiveMethod(ExpandoObject message)
{
foreach (var item in message)
{
//System.Collections.Generic.Dictionary<string, string> keyValue = new System.Collections.Generic.Dictionary<string, string>();
string K = string.Empty;
string V = string.Empty;
if (item.Value.GetType() == typeof(System.Dynamic.ExpandoObject))
{
parentID = item.Key;
parentPath += item.Key + #"\";
K = item.Key;
jData.Insert(Guid.NewGuid(), string.Empty, parentPath, K, string.Empty);
RecursiveMethod((ExpandoObject)item.Value);
}
else
{
K = item.Key;
V = item.Value.ToString();
jData.Insert(Guid.NewGuid(), parentID, parentPath, K, V);
}
}
}
I have tbJSonData table design like this:
Json data after insertion within data (but ParentID & ParentPath is not accurate as per JsonData give below):
Below is the jSon data which i need to parse & Insert it within tbJSonData table:
{
"interrogation": {
"patient": {
"firstName": "testname",
"lastName": "testfamily",
"dob": "1982-01-01",
"gender": "MALE"
},
"device": {
"manufacturer": "abc",
"manufacturerContact": "John Smith",
"modelName": "model",
"modelNumber": "i-123",
"serialNumber": "EUY1242C",
"type": "CRTD",
"implantDate": "2015-01-01",
"implantingPhysician": "Adam Smith"
},
"leads": [
{
"manufacturer": "BIOTRONIK",
"type": "RA",
"serialNumber": "EUY1242",
"implantDate": "2015-01-01"
}
],
"location": "",
"uploadDate": "",
"shocksAbortedECLClearDate": "",
"shocksAbortedSinceLastReset": "",
"routingLocId": "",
"orderingPhysician": {
"id": "1"
}
},
"patient": {
"id": 1156,
"name": "jmartest",
"family": "jmartest",
"dob": "06-02-2008",
"gender": "MALE"
},
"patientLocation": {
"id": 1159,
"location": {
"id": "1"
},
"mrn": "123"
}
}

I passed extra parameters on Click Event of Button like within RecursiveMethod method like below:
protected void btnSave_Click(object sender, EventArgs e)
{
var converter = new ExpandoObjectConverter();
dynamic message = JsonConvert.DeserializeObject<ExpandoObject>(txtJsonData.Text, converter);
RecursiveMethod(message);
}
Below is the updated Recursive method:
/// <summary>
/// Method is used to INSERT data within JsonData table Recursively
/// </summary>
/// <author>
/// Added :: Author : Irfan Ahmad
/// </author>
/// <param name="message">JSonData Deserialized data as ExpandoObject</param>
/// <param name="parentPath">Parent Path with "\" (Slash) seperated</param>
/// <param name="ParentId">Parent ID</param>
/// <param name="aFieldIndex">Field Index is used if Object is an Array</param>
public void RecursiveMethod(ExpandoObject message, string parentPath, int ParentId, int aFieldIndex)
{
foreach (var item in message)
{
//System.Collections.Generic.Dictionary<string, string> keyValue = new System.Collections.Generic.Dictionary<string, string>();
string K = string.Empty;
string V = string.Empty;
string Path = "";
if (item.Value.GetType() == typeof(System.Dynamic.ExpandoObject) || item.ToString().Contains("System.Collections.Generic.List"))
{
int pID = 0;
K = item.Key;
pID = jData.Insert(MsgID, ParentId, parentPath, K, string.Empty, aFieldIndex);
if (!string.IsNullOrEmpty(parentPath))
{
Path = parentPath + "\\" + item.Key;
}
else
{
Path = item.Key;
}
if (item.ToString().Contains("System.Collections.Generic.List"))
{
IEnumerable temp = (IEnumerable)item.Value;
int fieldIndex = 0;
foreach (var innerItem in temp)
{
if (!string.IsNullOrEmpty(innerItem.ToString()))
{
InsertJsonDataRecursiveMethod((ExpandoObject)innerItem, Path, pID, fieldIndex);
fieldIndex += 1;
}
}
}
else
{
InsertJsonDataRecursiveMethod((ExpandoObject)item.Value, Path, pID, aFieldIndex);
}
}
else
{
K = item.Key;
V = item.Value.ToString();
jData.Insert(MsgID, ParentId, parentPath, K, V, aFieldIndex);
}
}
}
It works like a charm!
Thanks "Eser" for your suggestions.

Parsing through JSON in JSON.NET with unknown property names

I have some JSON Data which looks like this:
{
"response":{
"_token":"StringValue",
"code":"OK",
"user":{
"userid":"2630944",
"firstname":"John",
"lastname":"Doe",
"reference":"999999999",
"guid":"StringValue",
"domainid":"99999",
"username":"jdoe",
"email":"jdoe#jdoe.edu",
"passwordquestion":"",
"flags":"0",
"lastlogindate":"2013-02-05T17:54:06.31Z",
"creationdate":"2011-04-15T14:40:07.22Z",
"version":"3753",
"data":{
"aliasname":{
"$value":"John Doe"
},
"smsaddress":{
"$value":"5555555555#messaging.sprintpcs.com"
},
"blti":{
"hideemail":"false",
"hidefullname":"false"
},
"notify":{
"grades":{
"$value":"0"
},
"messages":{
"$value":"1"
}
},
"beta_component_courseplanexpress_1":{
"$value":"true"
}
}
}
}
I am using C# with JSON.NET to parse through the data. I've been able to sucessfully get data using this algorithm:
User MyUser = new User();
JToken data = JObject.Parse(json);
MyUser.FirstName = (string) data.SelectToken("response.user.firstname");
//The same for all the other properties.
The problem is with the data field. This field is based on user preferences mostly and data is only inserted as it is used. The fields are all custom and developers can put in as many as they want without restrictions. Essentially, it's all free form data. Also as you notice they can be nested really far with data.
I've tried to run:
MyUser.Data = JsonConvert.DeserializeObject<List<JToken>>((string) data.SelectToken("response.user.data");
which doesn't work.
How would you go about converting it to be used in a C# object?

You can access it via the JToken / JArray / JObject methods. For example, this will list all of the keys under the data:
public class StackOverflow_14714085
{
const string JSON = #"{
""response"": {
""_token"": ""StringValue"",
""code"": ""OK"",
""user"": {
""userid"": ""2630944"",
""firstname"": ""John"",
""lastname"": ""Doe"",
""reference"": ""999999999"",
""guid"": ""StringValue"",
""domainid"": ""99999"",
""username"": ""jdoe"",
""email"": ""jdoe#jdoe.edu"",
""passwordquestion"": """",
""flags"": ""0"",
""lastlogindate"": ""2013-02-05T17:54:06.31Z"",
""creationdate"": ""2011-04-15T14:40:07.22Z"",
""version"": ""3753"",
""data"": {
""aliasname"": {
""$value"": ""John Doe""
},
""smsaddress"": {
""$value"": ""5555555555#messaging.sprintpcs.com""
},
""blti"": {
""hideemail"": ""false"",
""hidefullname"": ""false""
},
""notify"": {
""grades"": {
""$value"": ""0""
},
""messages"": {
""$value"": ""1""
}
},
""beta_component_courseplanexpress_1"": {
""$value"": ""true""
}
}
}
}
}";
public static void Test()
{
var jo = JObject.Parse(JSON);
var data = (JObject)jo["response"]["user"]["data"];
foreach (var item in data)
{
Console.WriteLine("{0}: {1}", item.Key, item.Value);
}
}
}

Json.NET can actually parse to a dynamic if that is useful to you.
Which means you can do something like.
dynamic parsedObject = JsonConvert.DeserializeObject("{ test: \"text-value\" }");
parsedObject["test"]; // "text-value"
parsedObject.test; // "text-value"
parsedObject.notHere; // null
Edit: might be more suitable for you to iterate the values if you don't know what you are looking for though.
dynamic parsedObject = JsonConvert.DeserializeObject("{ test: { inner: \"text-value\" } }");
foreach (dynamic entry in parsedObject)
{
string name = entry.Name; // "test"
dynamic value = entry.Value; // { inner: "text-value" }
}