We Keep Coding

Best way to send Customer and Order data from SQL server to MongoDB along with hyperlinks

Problem Statement:
I need to get Customer and all their corresponding Orders stored in the following format in MongoDB.
Current data resides in 2 different tables - Customers and Orders in MSSQL database.
Expected Collection in MongoDb:
{
"_data": [
{
"customer": {
"customerID": "1240014902244000006",
"firstName": "Jerry",
"surname": "Galoski",
"orders": [
{
"_data": {
"orderId": "1240014912244000016",
"suburb": "Delhi",
"postcode": "110051",
"country": "India"
},
"_links": [
{
"rel": "self",
"href": "/customer/1240014902244000006/orders",
"action": "GET"
},
{
"rel": "customer",
"href": "/customer/1240014902244000006",
"action": "GET"
}
]
},
{
"_data": {
"orderId": "1240014912244000024",
"suburb": "Delhi",
"postcode": "110051",
"country": "India"
},
"_links": [
{
"rel": "customer",
"href": "/customer/1240014902244000006/orders",
"action": "GET"
}
]
}
]
},
"_links": [
{
"rel": "self",
"href": "/customer/1240014902244000006",
"action": "GET"
},
{
"rel": "orders",
"href": "/customer/1240014902244000006/orders",
"action": "GET"
}
]
}
]
}
Steps taken to achieve this:
First step is to collate all orders data, which I'm doing using the below SQL query:
SELECT CustomerId, AllOrderIds =
STUFF(
(
SELECT DISTINCT ', ' + CAST(e2.OrderId AS VARCHAR(MAX))
FROM CustomerOrderRelation e2
WHERE e1.CustomerId = e2.CustomerId
FOR XML PATH('')
), 1,1,''
) FROM CustomerOrderRelation cor;
Then I use the above subquery in the customers sql query to get the complete output as shown.
Then in the C# code, I will process the Customer records one by one, to generate the hyperlinks for each.
I need to generate hyperlinks for Order records also.
Once I have the hyperlinks generated in C#, I will call InsertMany command (MongoDB) to insert data. As a result, I will have 2 collections created in MongoDb - one each for Customer and Order.
Finally, I'll be using MongoDb's MapReduce functionality to merge 2 collections to get the data.
db.customers.mapReduce(mapCustomers, reduce, {"out": {"reduce": "customerId"}});
db.orders.mapReduce(mapOrders, reduce, {"out": {"reduce": "customerId"}});
db.customerAndOrders.find().pretty();
I need to do the merge because, I am not finding a clever way to do following operations:
Get customers and their consolidated orders in one single entity
Generate Hyperlinks for customers and all their orders
Push this data into MongoDb with customer details and their order details
Suggestion required:
Is there a better way to accomplish the above?

Is it possible to obtain a parameter's value from JSON response

This is my first post.
I am trying to integrate Shippo to our application and wish to obtain a specific value out of a Json response.
I am getting the following Json response:
{
"count": 3,
"next": null,
"previous": null,
"results": [
{
"object_created": "2019-08-28T12:58:57.064Z",
"object_id": "16b602e0ajdsk87c4313920bc5e3174XYZ",
"object_owner": "some#email.com",
"shipment": "bd62234e151244dab2b2152fdcd15e76",
"attributes": [
"FASTEST"
],
"amount": "31.30",
"currency": "USD",
"amount_local": "31.30",
"currency_local": "USD",
"provider": "USPS",
"provider_image_75": "https://shippo-static.s3.amazonaws.com/providers/75/USPS.png",
"provider_image_200": "https://shippo-static.s3.amazonaws.com/providers/200/USPS.png",
"servicelevel": {
"name": "Priority Mail Express",
"token": "usps_priority_express",
"terms": ""
},
"estimated_days": 1,
"arrives_by": null,
"duration_terms": "Overnight delivery to most U.S. locations.",
"messages": [],
"carrier_account": "4e1506b8b7f7449e90620967e45aa1e9",
"test": false,
"zone": "4"
},
{
"object_created": "2019-08-28T12:58:57.063Z",
"object_id": "ebdee42047aa49a3b7e08b1903ea02ea",
"object_owner": "some#email.com",
"shipment": "bd62234e151244dab2b2152fdcd15e76",
"attributes": [
"BESTVALUE",
"CHEAPEST"
],
"amount": "7.49",
"currency": "USD",
"amount_local": "7.49",
"currency_local": "USD",
"provider": "USPS",
"provider_image_75": "https://shippo-static.s3.amazonaws.com/providers/75/USPS.png",
"provider_image_200": "https://shippo-static.s3.amazonaws.com/providers/200/USPS.png",
"servicelevel": {
"name": "Priority Mail",
"token": "usps_priority",
"terms": ""
},
"estimated_days": 2,
"arrives_by": null,
"duration_terms": "Delivery within 1, 2, or 3 days based on where your package started and where it’s being sent.",
"messages": [],
"carrier_account": "4e1506b8b7f7449e90620967e45aa1e9",
"test": false,
"zone": "4"
},
{
"object_created": "2019-08-28T12:58:57.062Z",
"object_id": "ad410a41c84940ee80eb30c41c507613",
"object_owner": "some#email.com",
"shipment": "bd62234e151244dab2b2152fdcd15e76",
"attributes": [],
"amount": "7.78",
"currency": "USD",
"amount_local": "7.78",
"currency_local": "USD",
"provider": "USPS",
"provider_image_75": "https://shippo-static.s3.amazonaws.com/providers/75/USPS.png",
"provider_image_200": "https://shippo-static.s3.amazonaws.com/providers/200/USPS.png",
"servicelevel": {
"name": "Parcel Select",
"token": "usps_parcel_select",
"terms": ""
},
"estimated_days": 7,
"arrives_by": null,
"duration_terms": "Delivery in 2 to 8 days.",
"messages": [],
"carrier_account": "4e1506b8b7f7449e90620967e45aa1e9",
"test": false,
"zone": "4"
}
]
}
I am using this call:
WebClient webClient = new WebClient();
webClient.Headers.Add("Authorization: ShippoToken " + authToken);
var result = JsonValue.Parse(webClient.DownloadString("https://api.goshippo.com/shipments/"+ theObject.ObjectId + "/rates/USD"));
My question is how could I get just the "amount" value from the response?
Thank you for any help!

Instead of recreating the json layout as a class model you can work with Newtonsofts JToken. Which will leave you with less boilerplate code to create:
var json = "your json data";
// Parse the whole json string
var obj = JObject.Parse(json);
// Extract the results data as JArray
var results = (JArray)obj["results"];
// Iterate over all array entries and get the amount property
foreach(var result in results)
{
var amount = result["amount"].ToString();
// Do something with the amount
}

You could create some classes based on json then deserialize using JsonConvert
e.g https://www.newtonsoft.com/json/help/html/DeserializeObject.htm
tip:
Copy the json and put in json viewer like https://jsonformatter.curiousconcept.com/

Use https://www.newtonsoft.com/json to deserialize the json response. If you didn't want that you could use a series of string.Split()'s to get the amount. But using newtonsoft (via the nuget package manager) would be the easiest.

What is the added value, if any, of a composite entity in the Microsoft Bot framework regarding a single marked entity

In tutorial, Microsoft Bot tutorial, a luis service is started that has the ability to deconstruct a sentence about booking a flight.
The entities that are used within the intent utterances have 2 composite entities named To and From which child to a list entity named Airport.
This produces the following json
"entities": {
"From": [
{
"Airport": [
[
"Berlin"
]
],
"$instance": {
"Airport": [
{
"type": "Airport",
"text": "berlin",
"startIndex": 19,
"length": 6,
"modelTypeId": 5,
"modelType": "List Entity Extractor",
"recognitionSources": [
"model"
]
}
]
}
}
],
"To": [
{
"Airport": [
[
"Paris"
]
],
"$instance": {
"Airport": [
{
"type": "Airport",
"text": "paris",
"startIndex": 29,
"length": 5,
"modelTypeId": 5,
"modelType": "List Entity Extractor",
"recognitionSources": [
"model"
]
}
]
}
}
],
Two things about this seem to be not very efficient but since it is a machine training functionality rather than machine learning I want to know if there is a difference.
Why not make Airport the parent and have 2 child entities named ToCity and FromCity? This would allow Airport to be the city with 2 nested objects of ToCity and FromCity if they are extracted in the utterance?
Why is a composite used here at all? Is there some added benefit? With the above abstraction one could simply make 2 simple entities or list entities of ToCity and FromCity I am not seeing why the organization of the composite is befitting here but I may have a miss-understanding.
Here is an example of what I am speaking about regarding question 1. To me this seems like a better more organized methodology of doing this. But not 100% clear such as is it easier to access or are the score's higher one way or the other. I will tell you imperically in the below test this methodology produced a higher score for each of the 2 entities.
"entities": {
"Color": [
{
"CarColor": [
"blue"
],
"$instance": {
"CarColor": [
{
"type": "CarColor",
"text": "blue",
"startIndex": 6,
"length": 4,
"score": 0.9977741,
"modelTypeId": 1,
"modelType": "Entity Extractor",
"recognitionSources": [
"model"
]
}
]
}
},
{
"InteriorColor": [
"red"
],
"$instance": {
"InteriorColor": [
{
"type": "InteriorColor",
"text": "red",
"startIndex": 20,
"length": 3,
"score": 0.883398235,
"modelTypeId": 1,
"modelType": "Entity Extractor",
"recognitionSources": [
"model"
]
}
]
}
}
],

It's a fair question, however the purpose of the tutorial, and any related example code, is to demonstrate how composite entities can be used. It's true that their usefulness isn't fully realized in this tutorial as there are potentially less complicated ways of achieving the same goal. But, developing an overly complex tutorial would be counter-productive to anyone trying to learn.

Getting distinct values using NEST ElasticSearch client

I'm building a product search engine with Elastic Search in my .NET application, by using the NEST client, and there is one thing i'm having trouble with. Getting a distinct set of values.
I'm search for products, which there are many thousands, but of course i can only return 10 or 20 at a time to the user. And for this paging works fine. But besides this primary result, i want to show my users a list of brands that are found within the complete search, to present these for filtering.
I have read about that i should use Terms Aggregations for this. But, i couldn't get anything better than this. And this still doesn't really give me what i want, because it splits values like "20th Century Fox" into 3 separate values.
var brandResults = client.Search<Product>(s => s
.Query(query)
.Aggregations(a => a.Terms("my_terms_agg", t => t.Field(p => p.BrandName).Size(250))
)
);
var agg = brandResult.Aggs.Terms("my_terms_agg");
Is this even the right approach? Or should is use something totally different? And, how can i get the correct, complete values? (Not split by space .. but i guess that is what you get when you ask for a list of 'Terms'??)
What i'm looking for is what you would get if you would do this in MS SQL
SELECT DISTINCT BrandName FROM [Table To Search] WHERE [Where clause without paging]

You are correct that what you want is a terms aggregation. The problem you're running into is that ES is splitting the field "BrandName" in the results it is returning. This is the expected default behavior of a field in ES.
What I recommend is that you change BrandName into a "Multifield", this will allow you to search on all the various parts, as well as doing a terms aggregation on the "Not Analyzed" (aka full "20th Century Fox") term.
Here is the documentation from ES.
https://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/mapping-multi-field-type.html
[UPDATE]
If you are using ES version 1.4 or newer the syntax for multi-fields is a little different now.
https://www.elasticsearch.org/guide/en/elasticsearch/reference/current/_multi_fields.html#_multi_fields
Here is a full working sample the illustrate the point in ES 1.4.4. Note the mapping specifies a "not_analyzed" version of the field.
PUT hilden1
PUT hilden1/type1/_mapping
{
"properties": {
"brandName": {
"type": "string",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
POST hilden1/type1
{
"brandName": "foo"
}
POST hilden1/type1
{
"brandName": "bar"
}
POST hilden1/type1
{
"brandName": "20th Century Fox"
}
POST hilden1/type1
{
"brandName": "20th Century Fox"
}
POST hilden1/type1
{
"brandName": "foo bar"
}
GET hilden1/type1/_search
{
"size": 0,
"aggs": {
"analyzed_field": {
"terms": {
"field": "brandName",
"size": 10
}
},
"non_analyzed_field": {
"terms": {
"field": "brandName.raw",
"size": 10
}
}
}
}
Results of the last query:
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 5,
"max_score": 0,
"hits": []
},
"aggregations": {
"non_analyzed_field": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "20th Century Fox",
"doc_count": 2
},
{
"key": "bar",
"doc_count": 1
},
{
"key": "foo",
"doc_count": 1
},
{
"key": "foo bar",
"doc_count": 1
}
]
},
"analyzed_field": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "20th",
"doc_count": 2
},
{
"key": "bar",
"doc_count": 2
},
{
"key": "century",
"doc_count": 2
},
{
"key": "foo",
"doc_count": 2
},
{
"key": "fox",
"doc_count": 2
}
]
}
}
}
Notice that not-analyzed fields keep "20th century fox" and "foo bar" together where as the analyzed field breaks them up.

I had a similar issue. I was displaying search results and wanted to show counts on the category and sub category.
You're right to use aggregations. I also had the issue with the strings being tokenised (i.e. 20th century fox being split) - this happens because the fields are analysed. For me, I added the following mappings (i.e. tell ES not to analyse that field):
"category": {
"type": "nested",
"properties": {
"CategoryNameAndSlug": {
"type": "string",
"index": "not_analyzed"
},
"SubCategoryNameAndSlug": {
"type": "string",
"index": "not_analyzed"
}
}
}
As jhilden suggested, if you use this field for more than one reason (e.g. search and aggregation) you can set it up as a multifield. So on one hand it can get analysed and used for searching and on the other hand for not being analysed for aggregation.

How to parse JSONData in C# (Windows Phone) using `this.`

In Windows Phone App.. I am parsing JSON Data By (get,set) method
I got another way of processing JSON data from this URL
In the part of this site (http://www.dfg-team.com/json-feeds-on-windows-phone/?lang=en)
Is used two techniques to parse json data..
in that I want to use this. Key word for processing JSON data ..
But My app have data Have a Lot of Json data which contains lot of JSON Arrays and JSON Objects. But its not a Problem in that I have main problem is My json contains Json array inside json object or json object inside the json array so at here we got problem.
This is My JSon data I want to parse these fields.. using this. method
{
"returnCode": "success",
"SData": {
"results": [
{
"wdetails": [
{
"noffers": [],
"offers_count": 0,
"name": "yahoo.com",
"recentoffers_count": 0,
"sku": "30072826",
"url": "http://www.yahoo.com"
},
{
"noffers": [
{
"id": "3aggEYoyaso",
"price": "179.99",
"firstrecorded_at": 13700,
"lastrecorded_at": 13700,
"seller": "google",
"availability": "In stock, ships same Business Day.",
"currency": "USD"
}
],
"offers_count": 1,
"name": "frys.com",
"recentoffers_count": 1,
"sku": "789",
"url": "http://www.google.com"
},
{
"listprice_currency": "USD",
"noffers": [
{
"id": "2SlCKQKm",
"price": "192.37",
"firstrecorded_at": 111,
"lastrecorded_at": 111,
"seller": "Amazon",
"availability": "13 In Stock",
"currency": "USD"
}
],
"offers_count": 1,
"name": "amazon.com",
"listprice": "288.56",
"recentoffers_count": 1,
"sku": "1084089",
"url": "http://www.amazon.com"
}
],
"model": "GGGG",
"weight": "771107.03",
"price_currency": "USD",
"gtins": [
"00839294"
],
"mpn": "GGGG",
"cat_id": "25552",
"height": "98.81",
"features": {
"Product Type - General": "Tablet",
"Height (in.)": "1.70"
},
"length": "416.56",
"geo": [
"usa"
],
"width": "267.97",
"category": "Tablets",
"price": "179.99",
"updated_at": 138,
"color": "Black",
"manufacturer": "google",
"images_total": 1,
"images": [
"https://evbdn.eventbrite.com/s3-s3/eventlogos/1832816/google.png"
],
"brand": "Dell",
"offers_total": 180
}
],
"total_results_count": 1,
"results_count": 1,
"code": "OK",
"offset": 0
}
}
I want to parse this Using this. in C#.
In Android I done this By hash map method, in Windows Phone this. is similar to is hash map so I am using this..

I am editing my previous answer :-
JObject json = JObject.Parse(yourJson);
if you need results to be parsed
this.ClassOfResult= json ["SData"]["results"];
if you want the 'wdetails' in the Json to get parsed you can try this
this.parentclass.wdetailsClass= json ["SData"]["results"]["wdetails"];
Hope that this will give you some idea.