The JSON data is as follows：
{"Sucess":true,
"Code":0,
"Msg":"Sucess",
"Data":{
"UserDayRanking":
{
"UserID":11452112,
"UserCharm":0,
"UserName":"gay",
"UserGender":1,
"UserLevel":36,
"UserPhoto":"http://res.xxx.com/2020/3/16/63719926625601201487545U11452112.jpeg",
"Ranking":0,
"IsNobility":0,
"NobilityType":0,
"NobilityLevel":0,
"UserShowStyle":null,
"LiveLevelUrl":null,
"IsStealth":false},
"DayRankingList":[
{
"UserID":3974854,
"UserCharm":114858,
"UserName":"jack",
"UserGender":1,
"UserLevel":91,
"UserPhoto":"http://res.xxx.com/2020/2/15/63717400601924412312384U3974854.jpeg",
"Ranking":2,
"IsNobility":1,
"NobilityType":1,
"NobilityLevel":3,
"UserShowStyle":
{
"NameColor":100102,
"BorderColor":100403,
"LiangMedal":0,
"DztCountDown":0,
"Mounts":100204,
"LiveLevelCode":0,
"LiveRights":null
},
"LiveLevelUrl":null,
"IsStealth":false
},
{"UserID":6231512,
"UserCharm":22644,
"UserName":"red.girl",
"UserGender":1,
"UserLevel":57,
"UserPhoto":"http://res.xxx.com/2019/11/20/63709843050801519858823U6231512.jpeg",
"Ranking":3,
"IsNobility":0,
"NobilityType":0,
"NobilityLevel":0,
"UserShowStyle":{
"NameColor":0,
"BorderColor":0,
"LiangMedal":0,
"DztCountDown":0,
"Mounts":0,
"LiveLevelCode":0,
"LiveRights":null
},
"LiveLevelUrl":null,
"IsStealth":false}
],
"LiveCharmSwitch":1,
"IsSelf":false
}
}
I want to use c # extraction
"UserID": 3974854,
"UserCharm": 114858,
"UserName": "jack",
"UserID":6231512,
"UserCharm":22644,
"UserName":"red.girl",
That is to extract UserID, UserCharm, UserName，This json has many layers,
What I want after the extraction is，id is sorted in order
id = 1, UserID = 3974854, UserCharm = 114858, UserName = jack
id = 2, UserID = 6231512, UserCharm = 22644, UserName = red.girl
I use the following code, but only extract the first one
string json = #"{"Sucess":true,"Code":0,"Msg":"Sucess","Data":{"UserDayRanking":{"UserID":11452112,"UserCharm":0,"UserName":"gay","UserGender":1,"UserLevel":36,"UserPhoto":"http://res.xxx.com/2020/3/16/63719926625601201487545U11452112.jpeg","Ranking":0,"IsNobility":0,"NobilityType":0,"NobilityLevel":0,"UserShowStyle":null,"LiveLevelUrl":null,"IsStealth":false},"DayRankingList":[{"UserID":3974854,"UserCharm":114858,"UserName":"jack","UserGender":1,"UserLevel":91,"UserPhoto":"http://res.xxx.com/2020/2/15/63717400601924412312384U3974854.jpeg","Ranking":2,"IsNobility":1,"NobilityType":1,"NobilityLevel":3,"UserShowStyle":{"NameColor":100102,"BorderColor":100403,"LiangMedal":0,"DztCountDown":0,"Mounts":100204,"LiveLevelCode":0,"LiveRights":null},"LiveLevelUrl":null,"IsStealth":false},{"UserID":6231512,"UserCharm":22644,"UserName":"red.girl","UserGender":1,"UserLevel":57,"UserPhoto":"http://res.xxx.com/2019/11/20/63709843050801519858823U6231512.jpeg","Ranking":3,"IsNobility":0,"NobilityType":0,"NobilityLevel":0,"UserShowStyle":{"NameColor":0,"BorderColor":0,"LiangMedal":0,"DztCountDown":0,"Mounts":0,"LiveLevelCode":0,"LiveRights":null},"LiveLevelUrl":null,"IsStealth":false}],"LiveCharmSwitch":1,"IsSelf":false}}";
List<Info> jobInfoList = JsonConvert.DeserializeObject<List<Info>>(z);
foreach (Info jobInfo in jobInfoList)
{
//Console.WriteLine("UserName:" + jobInfo.UserName);
}
public class Info
{
public string UserCharm { get; set; }
public string UserName { get; set; }
public data DayRankingList { get; set; }
}
public class data
{
public int UserID { get; set; }
public string UserCharm { get; set; }
public string UserName { get; set; }
public string UserGender { get; set; }
public string UserLevel { get; set; }
}
The above code only shows username = jack，Never show username = red.girl
As it looks to me then you want some details from your JSON has the which is in DayRankingList. As you only want some data then we can use a tool like http://json2csharp.com/ to create our classes and then remove what we don't need. Then we end up with the following classes.
public class DayRankingList
{
public int UserID { get; set; }
public int UserCharm { get; set; }
public string UserName { get; set; }
}
public class Data
{
public List<DayRankingList> DayRankingList { get; set; }
}
public class RootObject
{
public Data Data { get; set; }
}
Which you can deserialise like this
string json = .....
var root = JsonConvert.DeserializeObject<RootObject>(json);
Then if you wish, you can extract the inner data into a new List<> and then just work on that.
List<DayRankingList> rankingLists = root.Data.DayRankingList;
//Do something with this, such as output it
foreach(DayRankingList drl in rankingLists)
{
Console.WriteLine(String.Format("UserId {0} UserCharm {1} UserName {2}",drl.UserId, drl.UserCharm, drl.UserName));
}
You can use Json.Linq to parse your JSON into JObject and enumerate DayRankingList items (since it's an array). Then convert every item into data class and order the result sequence by UserID
var jObject = JObject.Parse(json);
var rankingList = (jObject["Data"] as JObject)?.Property("DayRankingList");
var list = rankingList.Value
.Select(rank => rank.ToObject<data>())
.OrderBy(item => item?.UserID);
foreach (var user in list)
Console.WriteLine($"{user.UserID} {user.UserName}");
Another way is copy your JSON, go to Edit->Paste Special->Paste JSON as classes menu in Visual Studio and generate a proper class hierarchy (I've got 5 classes, they are quite long to post here), then use them during deserialization
The most type-safe way is to define the class structure that you want, like jason.kaisersmith suggested.
To have the final format you need, though, you might want to do an extra Linq Order and Select, to include the id:
var finalList = rankingLists.OrderBy(rl => rl.UserId).Select((value, index) => new
{
id = index,
value.UserId,
value.UserCharm,
value.UserName
});
foreach (var drl in finalList)
{
Console.WriteLine($"Id = {drl.id}, UserId = {drl.UserId}, UserCharm = {drl.UserCharm}, UserName = {drl.UserName}");
}
We are using elastic search just for document search in our application so we don't have any one expert in it. I was able to use TermQuery, SimpleQueryStringQuery and MatchPhraseQuery successfully. But I found out in documentation that using From & Size for pagination is not good for production and Search After is recommended.
But my implementation return null. It is confusing for me what should be in <Project> parameter as shown in Nest API Object Initializer Syntax in docs here.
My code looks like this:
var request = new SearchRequest<ElasticSearchJsonObject._Source>
{
//Sort = new List<ISort>
//{
// new SortField { Field = Field<ElasticSearchJsonObject>(p=>)}
//},
SearchAfter = new List<object> {
},
Size = 20,
Query = query
};
Reality is I don't understand this. Over here ElasticSearchJsonObject._Source is the class to map returned results.
My documents are simple text documents and I only want documents sorted according to score so document Id is not relevant.
There was already a question like this on SO but I can't find it somehow.
Update
After looking at answer I updated my code and though query obtained does work. It return result in kibana but not in NEST.
This is the new updated code:
var request = new SearchRequest<ElasticSearchJsonObject.Rootobject>
{
Sort = new List<ISort>
{
new SortField { Field = "_id", Order = SortOrder.Descending}
},
SearchAfter = new List<object> {
"0fc3ccb625f5d95b973ce1462b9f7"
},
Size = 1,
Query = query
};
Over here I am using size=1 just for test as well as hard code _id value in SearchAfter.
The query generated by NEST is:
{
"size": 1,
"sort": [
{
"_id": {
"order": "desc"
}
}
],
"search_after": [
"0fc3ccb625f5d95b973ce1462b9f7"
],
"query": {
"match": {
"content": {
"query": "lahore",
"fuzziness": "AUTO",
"prefix_length": 3,
"max_expansions": 10
}
}
}
}
The response from the ES does say successful but no results are returned.
Results do return in Kibana
Query status is successful
But...
Total returned is 0 in NEST
Sort value is null in kibana I used TrackScores = true to solve this issue
Here is the debug information:
Valid NEST response built from a successful low level call on POST: /extract/_source/_search?typed_keys=true
# Audit trail of this API call:
- [1] HealthyResponse: Node: http://localhost:9200/ Took: 00:00:00.1002662
# Request:
{"size":1,"sort":[{"_id":{"order":"desc"}}],"search_after":["0fc3ccb625f5d95b973ce1462b9f7"],"query":{"match":{"content":{"query":"lahore","fuzziness":"AUTO","prefix_length":3,"max_expansions":10}}}}
# Response:
{"took":3,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}
So please tell me where I am wrong and what can be the problem and how to solve it.
Update 2:
Code in Controller:
Connection String:
var node = new Uri("http://localhost:9200");
var settings = new ConnectionSettings(node);
settings.DisableDirectStreaming();
settings.DefaultIndex("extract");
var client = new ElasticClient(settings);
Query:
var query = (dynamic)null;
query = new MatchQuery
{
Field = "content",
Query = content,
Fuzziness = Fuzziness.Auto,
PrefixLength = 3,
MaxExpansions = 10
};
Query Builder
var request = new SearchRequest<ElasticSearchJsonObject.Rootobject>
{
Sort = new List<ISort>
{
new SortField { Field = "_id", Order = SortOrder.Descending}
},
SearchAfter = new List<object> {
documentid //sent as parameter
},
Size = 1, //for testing 1 other wise 10
TrackScores = true,
Query = query
};
JSON Query
I use this code to get query I posted above. This query is then passed to kibana with GET <my index name>/_Search and there it works
var stream = new System.IO.MemoryStream();
client.SourceSerializer.Serialize(request, stream);
var jsonQuery = System.Text.Encoding.UTF8.GetString(stream.ToArray());
ES Response
string responseJson = "";
ElasticSearchJsonObject.Rootobject response = new ElasticSearchJsonObject.Rootobject();
var res = client.Search<object>(request);
if (res.ApiCall.ResponseBodyInBytes != null)
{
responseJson = System.Text.Encoding.UTF8.GetString(res.ApiCall.ResponseBodyInBytes);
try
{
response = JsonConvert.DeserializeObject<ElasticSearchJsonObject.Rootobject>(responseJson);
}
catch (Exception)
{
var model1 = new LoginSignUpViewModel();
return PartialView("_NoResultPage", model1);
}
}
This is where things go wrong. Above debug information was captured from response
ElasticSearchJsonObject
Some how I think problem might be here somewhere? The class is generated by taking response from NEST in Search request.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
namespace ESAPI
{
public class ElasticSearchJsonObject
{
public class Rootobject
{
public int took { get; set; }
public bool timed_out { get; set; }
public _Shards _shards { get; set; }
public Hits hits { get; set; }
}
public class _Shards
{
public int total { get; set; }
public int successful { get; set; }
public int skipped { get; set; }
public int failed { get; set; }
}
public class Hits
{
public int total { get; set; }
public float max_score { get; set; }
public Hit[] hits { get; set; }
}
public class Hit
{
public string _index { get; set; }
public string _type { get; set; }
public string _id { get; set; }
public float _score { get; set; }
public _Source _source { get; set; }
}
public class _Source
{
public string content { get; set; }
public Meta meta { get; set; }
public File file { get; set; }
public Path path { get; set; }
}
public class Meta
{
public string title { get; set; }
public Raw raw { get; set; }
}
public class Raw
{
public string XParsedBy { get; set; }
public string Originator { get; set; }
public string dctitle { get; set; }
public string ContentEncoding { get; set; }
public string ContentTypeHint { get; set; }
public string resourceName { get; set; }
public string ProgId { get; set; }
public string title { get; set; }
public string ContentType { get; set; }
public string Generator { get; set; }
}
public class File
{
public string extension { get; set; }
public string content_type { get; set; }
public DateTime last_modified { get; set; }
public DateTime indexing_date { get; set; }
public int filesize { get; set; }
public string filename { get; set; }
public string url { get; set; }
}
public class Path
{
public string root { get; set; }
public string _virtual { get; set; }
public string real { get; set; }
}
}
}
I am sure this can be used to get response.
Please note that in case of simple search this code works:
so for this query below my code is working:
var request = new SearchRequest
{
From = 0,
Size = 20,
Query = query
};
Using from/size is not recommended for deep pagination because of the amount of documents that need to be fetched from all shards for a deep page, only to be discarded when finally returning an overall ordered result set. This operation is inherent to the distributed nature of Elasticsearch, and is common to many distributed systems in relation to deep pagination.
With search_after, you can paginate forward through documents in a stateless fashion and it requires
the documents returned from the first search response are sorted (documents are sorted by _score by default)
passing the values for the sort fields of the last document in the hits from one search request as the values for "search_after": [] for the next request.
In the Search After Usage documentation, a search request is made with sort on NumberOfCommits descending, then by Name descending. The values to use for each of these sort fields are passed in SearchAfter(...) and are the values of Project.First.NumberOfCommits and Project.First.Name properties, respectively. This tells Elasticsearch to return documents that have values for the sort fields that correspond to the sort constraints for each field, and relate to the values supplied in the request. For example, sort descending on NumberOfCommits with a supplied value of 775 means that Elasticsearch should only consider documents with a value less than 775 (and to do this for all sort fields and supplied values).
If you ever need to dig further into any NEST documentation, click the "EDIT" link on the page:
which will take you to the github repository of the documentation, with the original asciidoc markdown for the page:
Within that page will be a link back to the original NEST source code from which the asciidoc was generated. In this case, the original file is SearchAfterUsageTests.cs in the 6.x branch
I have a windows form application and would like to deserialize a JSON string that I'm getting from a web address so that I can get just two values from it, how would I go about doing this?
Below is the code I have to get the JSON string, and if you go to the URL that it's getting, you can also see the JSON string. I want to just get the item name, and current price of it. Which you can see the price under the current key.
private void GrabPrices()
{
using (WebClient webClient = new System.Net.WebClient())
{
WebClient n = new WebClient();
var json = n.DownloadString("http://services.runescape.com/m=itemdb_rs/api/catalogue/detail.json?item=1513");
string valueOriginal = Convert.ToString(json);
Console.WriteLine(json);
}
}
It's also going to be iterating through a SQLite database and getting the same data for multiple items based on the item ID, which I'll be able to do myself.
EDIT I'd like to use JSON.Net if possible, I've been trying to use it and it seems easy enough, but I'm still having trouble.
Okay so first of all you need to know your JSON structure, sample:
[{
name: "Micheal",
age: 20
},
{
name: "Bob",
age: 24
}]
With this information you can derive a C# object
public class Person
{
public string Name {get;set;}
public int Age {get;set;}
}
Now you can use JSON.NET to deserialize your JSON into C#:
var people = JsonConvert.DeserializeObject<List<Person>>(jsonString);
If you look at the original JSON it is an array of objects, to deal with this I have used List<T>.
Key things to remember, you need to have the C# object mirror in properties that of the JSON object. If you don't have a list, then you don't need List<T>.
If your JSON objects have camel casing, and you want this converted to the C# conventions, then use this:
var people = JsonConvert.DeserializeObject<List<Person>>(
jsonString,
new JsonSerializerSettings { ContractResolver = new CamelCasePropertyNamesContractResolver() });
First of all you need to create a class structure for the JSON
public class Wrapper
{
public Item item;
}
public class Item
{
public string icon { get; set; }
public string icon_large { get; set; }
public int id { get; set; }
public string type { get; set; }
public string typeIcon { get; set; }
public string name { get; set; }
public string description { get; set; }
public GrandExchange current { get; set; }
public GrandExchange today { get; set; }
public bool members { get; set; }
public GrandExchange day30 { get; set; }
public GrandExchange day90 { get; set; }
public GrandExchange day180 { get; set; }
}
public class GrandExchange
{
public string trend { get; set; }
public string price { get; set; }
}
Then you need to serialize the current item into a Wrapper class
var wrapper = JsonConvert.DeserializeObject<Wrapper>(json);
Then if you want multiple items in a list, you can do so with this code :
// Items to find
int[] itemIds = {1513, 1514, 1515, 1516, 1517};
// Create blank list
List<Item> items = new List<Item>();
foreach (int id in itemIds)
{
var n = new WebClient();
// Get JSON
var json = n.DownloadString(String.Format("http://services.runescape.com/m=itemdb_rs/api/catalogue/detail.json?item={0}", id));
// Parse to Item object
var wrapper = JsonConvert.DeserializeObject<Wrapper>(json);
// Append to list
items.Add(wrapper.item);
}
// Do something with list
It is also worth noting that Jagex limit how many times this API can be called from a certain IP within a time frame, going over that limit will block your IP for a certain amount of time. (Will try and find a reference for this)
I'm playing around with Web API 4 for the first time and preparing to hook up connection to a MongoDb. I've defined some simple objects to represent the models but when I try to return a collection of them from the GET request of my API only one property is being included in the resulting JSON object.
public class Topic : Entity
{
public string Name { get; set; }
public List<Topic> Parents { get; set; }
public List<Topic> Children { get; set; }
public List<ContentNode> ContentNodes { get; set; }
}
public class ContentNode
{
public enum ContentNodeType { VIDEO, TEXT, AUDIO };
public ContentNodeType ContentType { get; set; }
public string Url { get; set; }
public List<int> Scores { get; set; }
public List<Link> Links { get; set; }
public List<string> Comments { get; set; }
}
public string Get()
{
List<Topic> topics = new List<Topic>();
var programming = new Topic()
{
Id = "1",
Name = "Programming"
};
var inheritanceVideo = new ContentNode()
{
ContentType = ContentNode.ContentNodeType.VIDEO,
Url = "http://youtube.com",
Scores = new List<int>() {
4, 4, 5
},
Comments = new List<string>() {
"Great video about inheritance!"
}
};
var oop = new Topic()
{
Id = "2",
Name = "Object Oriented Programming",
ContentNodes = new List<ContentNode>() {
inheritanceVideo
}
};
programming.Children.Add(oop);
topics.Add(programming);
string test = JsonConvert.SerializeObject(topics);
return test;
}
I'm using the JSON.Net library to serialize the object here but I previously used the default JSON serializer and had the GET return IEnumerable<Topic>. In both cases the JSON being returned is simply:
"[{\"Id\":\"1\"}]"
A browser request for the XML works just fine. According to the documentation for JSON.Net it doesn't seem like there should be a problem serializing these classes to JSON. Any thoughts on why this isn't working? It doesn't seem like I should need to apply explicit attributes for every member.
I'm looking to be able to reuse some of the transform expressions from indexes so I can perform identical transformations in my service layer when the document is already available.
For example, whether it's by a query or by transforming an existing document at the service layer, I want to produce a ViewModel object with this shape:
public class ClientBrief
{
public int Id { get; set; }
public string FullName { get; set; }
public string Email { get; set; }
// ellided
}
From this document model:
public class Client
{
public int Id { get; private set; }
public CompleteName Name { get; private set; }
public Dictionary<EmailAddressKey, EmailAddress> Emails { get; private set; }
// ellided
}
public class CompleteName
{
public string Title { get; set; }
public string GivenName { get; set; }
public string MiddleName { get; set; }
public string Initials { get; set; }
public string Surname { get; set; }
public string Suffix { get; set; }
public string FullName { get; set; }
}
public enum EmailAddressKey
{
EmailAddress1,
EmailAddress2,
EmailAddress3
}
public class EmailAddress
{
public string Address { get; set; }
public string Name { get; set; }
public string RoutingType { get; set; }
}
I have an expression to transform a full Client document to a ClientBrief view model:
static Expression<Func<IClientSideDatabase, Client, ClientBrief>> ClientBrief = (db, client) =>
new ClientBrief
{
Id = client.Id,
FullName = client.Name.FullName,
Email = client.Emails.Select(x => x.Value.Address).FirstOrDefault()
// ellided
};
This expression is then manipulated using an expression visitor so it can be used as the TransformResults property of an index (Client_Search) which, once it has been generated at application startup, has the following definition in Raven Studio:
Map:
docs.Clients.Select(client => new {
Query = new object[] {
client.Name.FullName,
client.Emails.SelectMany(x => x.Value.Address.Split(new char[] {
'#'
})) // ellided
}
})
(The Query field is analysed.)
Transform:
results.Select(result => new {
result = result,
client = Database.Load(result.Id.ToString())
}).Select(this0 => new {
Id = this0.client.__document_id,
FullName = this0.client.Name.FullName,
Email = DynamicEnumerable.FirstOrDefault(this0.client.Emails.Select(x => x.Value.Address))
})
However, the transformation expression used to create the index can then also be used in the service layer locally when I already have a Client document:
var brief = ClientBrief.Compile().Invoke(null, client);
It allows me to only have to have one piece of code that understands the mapping from Client to ClientBrief, whether that code is running in the database or the client app. It all seems to work ok, except the query results all have an Id of 0.
How can I get the Id property (integer) properly populated in the query?
I've read a number of similar questions here but none of the suggested answers seem to work. (Changing the Ids to strings from integers is not an option.)
I have a hard time following your sample fully, Really the best way to dig in to this would be with a failing self-contained unit test.
Nonetheless, let's see if I can pull out the important bits.
In the transform, you have two areas where you are working with the id:
...
client = Database.Load(result.Id.ToString())
...
Id = this0.client.__document_id,
...
The result.Id in the first line and the Id = in the second line are expected to be integers.
The Database.Load() expects a string document key and that is also what you see in __document_id.
The confusion comes from Raven's documentation, code, and examples all use the terms id and key interchangeably, but this is only true when you use string identifiers. When you use non-string identifiers, such as ints or guids, the id may be 123, but the document key is still clients/123.
So try changing your transform so it translates:
...
client = Database.Load("clients/" + result.Id)
...
Id = int.Parse(this0.client.__document_id.Split("/")[1]),
...
... or whatever the c# equivalent linq form would be.