C# how to insert JSON into Google BigQuery table schema - c#

I have an array of complex JSON objects with embedded arrays of JSON objects. By way of an example, one JSON object might look like...
{
"shopID": "12194",
"name": "Shop Name",
"timeZone": "US\/Eastern",
"timeStamp": "2020-11-30T13:25:13+00:00",
"Contact": {
"contactID": "90",
"Addresses": [
"ContactAddress": {
"address1": "123 Some Street",
"address2": "",
"city": "A city",
"state": "A state",
"zip": "12345",
"country": "United States",
"countryCode": "US"
}
],
"Emails": "",
"Websites": {
"ContactWebsite": {
"url": "www.companyname.com"
}
}
},
"TaxCategory": {
"taxCategoryID": "2",
"isTaxInclusive": "false",
"tax1Name": "Tax region name",
"tax1Rate": "0.07"
}
}
I cannot modify the above format... I am calling a public API that is returning this as a response.
Assuming that the corresponding table schema has been created in Google BigQuery, how do I simply insert this JSON as a row in to the table?
My code looks like this...
bqc = BigQueryClient.Create("a-project-id");
var trShops = new TableReference()
{
ProjectId = "a-project-id",
DatasetId = "a-dataset-name",
TableId = "Shops"
};
# Call to my API data source that returns a JSON array of Shops...
content = await _lsrApiHelper.GetRawDataAsync(uri, currentOffset, _apiLimit);
# After call to API I deserialize the JSON response...
response = JsonSerializer.Deserialize<ShopResponseRoot>(content);
# Then I want loop through Shops in the List<ShopResponse> and insert them into BigQuery...
foreach (ShopResponse shop in response.Shop)
{
var newRow = new BigQueryInsertRow();
newRow.InsertId = "shopID";
newRow.Add(shop); # This is where I am struggling
bqc.InsertRow(trShops, newRow);
}
I am struggling with working out how to use a JSON or C# object to populate the data that I want inserting into BigQuery.
Any and all guidance greatly appreciated.

Related

How to modify the JSON obtained from serializing a DataSet using Json.Net for purposes of ESRI geocoding

How to introduce the "attributes" level into JSON text below? I'm using a C# dataset populated from SQL server with SerializeObject from Newtonsoft.json.
This is for submitting data to ESRI batch geocoder, as described here.
The format their REST service expects looks like this
{
"records": [
{
"attributes": {
"OBJECTID": 1,
"Address": "4550 Cobb Parkway North NW",
"City": "Acworth",
"Region": "GA"
}
},
{
"attributes": {
"OBJECTID": 2,
"Address": "2450 Old Milton Parkway",
"City": "Alpharetta",
"Region": "GA"
}
}
]
}
The format my C# script creates looks like this (missing the "attributes" level.)
{
"records": [
{
"OBJECTID": 1,
"address": "4550 Cobb Parkway North NW",
"city": "Acworth",
"state": "GA",
"zip": 30101.0
},
{
"OBJECTID": 2,
"address": "2450 Old Milton Parkway",
"city": "Alpharetta",
"state": "GA",
"zip": 30009.0
}
]
}
I've read thru json.net documentation and wonder if the JsonConverter class could be helpful. Candidly, I'm at loss for how to resolve this. First time user of Json.net, relative newbie with C#
Here is the C# code used to this point:
SQLStatement = "select OBJECTID, Address, City, Region, Postal from MyAddresses";
SqlDataAdapter geoA = new SqlDataAdapter(SQLStatement, GEOconn);
DataSet GeoDS = new DataSet();
geoA.Fill(GeoDS, "records");
string geoAJSON = JsonConvert.SerializeObject(GeoDS);
Console.WriteLine("{0}", geoAJSON);
You can wrap your rows in another object with an "attributes" property using Json.Net's LINQ-to-JSON API.
In your code, replace this line:
string geoAJSON = JsonConvert.SerializeObject(GeoDS);
with this:
var obj = JObject.FromObject(GeoDS);
obj["records"] = new JArray(
obj["records"].Select(jo => new JObject(new JProperty("attributes", jo)))
);
string geoAJSON = obj.ToString();
Working demo here: https://dotnetfiddle.net/nryw27
Aside: based on your JSON it looks like you are storing postal codes in your database as decimals. Don't do that. They may look like numbers, but you should store them as strings. Postal codes in the US can have leading zeros, which will get dropped when you treat them as numbers. Some international postal codes can contain letters, so a numeric type won't even work in that case.

How to access inner properties of a Json Object?

I am learning C# and I am trying to parse json/xml responses and check each and every key and value pair. For xml I am converting to json so I have only one function/script to work with both cases. My issue is that I am working with a wide range of json responses which are not similar and there may be arrays in some of the json response. I have tried accessing the "Count" of the json object as a way to check for arrays.
Note: The responses will vary. This example is for Products > Product > name, quantity and category. The next response will change and can be like Country > State > Cities and so on. I cannot rely on creating classes since all responses are going to be different. Plus I am working on automating it so it should be able to handle anything thrown at it.
Sample Json I am working with:
{
"products": {
"product": [
{
"name": "Dom quixote de La Mancha",
"quantity": "12",
"category": "Book"
},
{
"name": "Hamlet",
"quantity": "3",
"category": "Book"
},
{
"name": "War and Peace",
"quantity": "7",
"category": "Book"
},
{
"name": "Moby Dick",
"quantity": "14",
"category": "Book"
},
{
"name": "Forrest Gump",
"quantity": "16",
"category": "DVD"
}
]
}
The way I am accessing the count, name and value is as follows:
dynamic dyn = JsonConvert.DeserializeObject<dynamic>(jsonText);
foreach (JProperty property in dyn.Properties())
{
string propname = property.Name;
var propvalue = property.Value;
int count = property.Count;
}
Is there a way to access these without going through the foreach loop like int count = dyn.Count ? All I am getting from this is null instead of actual values.
For the above example my end result will be like:
This responses contains products> product> 5 x (name, quantity, category)
The QuickWatch for the object:
QuickWatch for dyn object
Try to deserialize your JSON into JObject like below:
var jObject = JsonConvert.DeserializeObject<JObject>(jsonText);

How can I use a JSONPath expression as a filter inside another JSONPath expression?

I want to select objects from a JSON string by filtering using a JSONPath expression with another expression embedded in the filter. In other words, I want to filter for a value that is present elsewhere in the JSON data.
For example:
In the following JSON data there is a value in $.Item.State.stepId (currently "QG2.0"). I need to have a JSONPath expression that selects values based on this value, like this:
$..Step[?(#.stepId==$Item.State.stepId)].actionDate
But this will not return any results. If I use the string ("QG2.0") directly like this:
$..Step[?(#.stepId=='QG2.0')].actionDate
it will return the required data.
What's wrong, or is it not even possible? My JSON is below:
{
"Item": {
"Common": {
"folio": "PSH-000016020",
"setName": "123-XZ200-1",
"wfId": "Kat1_002",
"wfIssue": "002",
"wfIdIssue": "Kat1_002.002"
},
"State": {
"status": "IN WORK",
"stepId": "QG2.0",
"stepDescription": "Validation"
},
"Participants": {
"Participant": [
{
"role": "PR",
"roleDescription": "Product Responsible",
"loginName": "marc102",
"email": "mark#abc.de"
}, {
"role": "CR",
"roleDescription": "Chapter Responsible",
"loginName": "uli26819",
"email": "uli#abc.de"
}
]
},
"Steps": {
"Step": [
{
"stepId": "QG1.0",
"stepTitle": "Preparation",
"actionDate": "2016-06-28T10:28:09",
"actionDueDate": "",
"actionBy_Name": "Marc",
"actionBy_Account": "marc102",
"action": "complete",
"Comment": ""
}, {
"stepId": "QG2.0",
"stepTitle": "Check Requirements",
"actionDate": "2016-08-08T14:17:04",
"actionDueDate": "",
"actionBy_Name": "Uli",
"actionBy_Account": "uli26819",
"action": "complete",
"Comment": ""
}
]
}
}
}
I don't think Json.Net's implementation of JSONPath supports this concept.
However, you can still get the information you want if you break the query into two steps:
JObject obj = JObject.Parse(json);
JToken stepId = obj.SelectToken("Item.State.stepId");
JToken actionDate = obj.SelectToken(string.Format("$..Step[?(#.stepId=='{0}')].actionDate", stepId));
Console.WriteLine(actionDate.ToString());
Fiddle: https://dotnetfiddle.net/KunYTf

Need to convert JSON to datatable

I have incoming JSON formatted like this:
{
"users": [
{
"radio_id": "123582",
"callsign": "ABCD",
"name": "First Last",
"city": "Dortmund",
"state": "Nordrhein-Westfalen",
"country": "Germany",
"home_rptr": "W2VL",
"remarks": "None"
},
{
"radio_id": "789456",
"callsign": "EFG",
"name": "Name Here",
"city": "Dortmund",
"state": "Nordrhein-Westfalen",
"country": "Germany",
"home_rptr": "W2VL",
"remarks": "None"
}
]
}
It is coming from a web request that I catch into a string called dataReceived. I then use this line of code to convert to a datatable.
DataTable dtData = (DataTable)JsonConvert.DeserializeObject(dataReceived, (typeof(DataTable)));
I'm getting an error of:
Unexpected JSON token when reading DataTable. Expected StartArray, got StartObject. Path '', line 1, position 1.
I suspect my problem is the data is in an array, but I'm not sure how to solve this. My objective is to have a table with each row one of the "users" objects in the json.
Can anyone push me in the right direction?
var dt = JObject.Parse(json)["users"].ToObject<DataTable>();
That is all.

Printing out the first index of an array in JSON format

What I am trying to do is print out the first index of an array in JSON format.
With the code:
storeInformation = myStoreInfoTable;
strResponseOutput = JsonConvert.SerializeObject(storeInformation);
These are the results:
[
{
"distance": 0,
"descr": "Toronto",
"address": "1300 Castlefield Avenue",
"city": "Toronto"
},
{
"distance": 7.1121883392,
"descr": "Etobicoke - North",
"address": "Resources Road",
"city": "Etobicoke"
}
]
What I tried to do to get the first index is:
storeInformationRow = dtbStoreInformation.Rows[0];
strResponseOutput = JsonConvert.SerializeObject(storeInformationRow);
What I get is:
{
"RowError": "",
"RowState": 2,
"Table": [
{
"distance": 0.0000000000,
"descr": "Toronto",
"address": "1300 Castlefield Avenue",
"city": "Toronto"
},
{
"distance": 7.1121883392,
"descr": "Etobicoke - North",
"address": "Resources Road",
"city": "Etobicoke"
}
]
}
The result I want is just
{
"distance": 0.0000000000,
"descr": "Toronto",
"address": "1300 Castlefield Avenue",
"city": "Toronto"
}
Help anybody?
I seems to me that the JSON serializer is goofing up because it doesn't know how to correctly serialize a DataRow. That's the only difference between the first and second snippet of code.
If you look at the MSDN documentation for the DataRow class you can see that the DataRow has the attributes displayed (RowError, RowState and Table) among many others.
To fix your problem I would suggest one of 2 options:
Create a new DataTable and add the DataRow you want to serialize to it, and call JsonConvert.SerializeObject() on that.
Map the columns (distance, descr, address & city) to another class (Store or something) and try to serialize that object. From what I read in the JsonConvert documentation it's possible (the class attributes are mapped correctly onto the JSON object).
Let me know how it goes!
you can just use :
var row = dtbStoreInformation.Rows[0];
string strResponseOutput = new {
distance = row[0].ToString(),
descr= row[1].ToString(),
address = row[2].ToString(),
city = row[2].ToString()}.ToJson();
don't forget the include of the namespace ServiceStack.Text
You can also create an anonym object
var obj = jsonString = new {
distance = row[0].ToString(),
descr= row[1].ToString(),
address = row[2].ToString(),
city = row[2].ToString()}
strResponseOutput = JsonConvert.SerializeObject(obj);

Categories