We Keep Coding

AWS API Gateway/Lambda using an external API how to retrieve and store data into dynamoDb

Using an open API (im using New York Times book api) create an API Gateway endpoint that does the following:
Retrieves data from an API.
Stores the data in a database.
im able to retrieve the data and store it in a string but im have difficulties extracting that data from the string and storing it into a dynamoDb. Any help is appreciated thank you.
using System;
using System.Collections.Generic;
using System.Dynamic;
using System.Linq;
using System.Net.Http;
using System.Threading.Tasks;
using Newtonsoft.Json;
using Amazon.Lambda.Core;
using Newtonsoft.Json.Linq;
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.DocumentModel;
using Amazon.Lambda.APIGatewayEvents;
using Amazon.DynamoDBv2.Model;
// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]
namespace last
{
public class Result
{
public string url { get; set; }
public string publication_dt { get; set; }
public string byline { get; set; }
public string book_title { get; set; }
public string book_author { get; set; }
public string summary { get; set; }
public List<string> isbn13 { get; set; }
}
public class Root
{
public string status { get; set; }
public string copyright { get; set; }
public int num_results { get; set; }
public List<Result> results { get; set; }
}
public class Function
{
private static AmazonDynamoDBClient client1 = new AmazonDynamoDBClient();
public static readonly HttpClient client = new HttpClient();
public async Task<ExpandoObject> FunctionHandler(string input, ILambdaContext context)
{
string tblName = "last";
string url = "https://api.nytimes.com/svc/books/v3/reviews.json?title=Becoming&api-key=myKey";
string message = await client.GetStringAsync(url);
Result myDeserializedClass = JsonConvert.DeserializeObject<Result>(message);
var request = new PutItemRequest
{
TableName = tblName,
Item = new Dictionary<string, AttributeValue>()
{
{ "bookId", new AttributeValue { S = "202" }},
{ "publication_dt", new AttributeValue { S = myDeserializedClass.publication_dt.ToString() }},
{ "byline", new AttributeValue { S = myDeserializedClass.byline.ToString() }},
{ "book_title", new AttributeValue { S = myDeserializedClass.book_title.ToString()}},
{ "book_author", new AttributeValue { S = myDeserializedClass.book_author.ToString() }},
{ "summary", new AttributeValue { S = myDeserializedClass.summary.ToString() }},
{ "isbn13", new AttributeValue { S = myDeserializedClass.isbn13.ToString()}},
}
};
await client1.PutItemAsync(request);
//Root myDeserializedClass = JsonConvert.DeserializeObject<Root>(message);
//await tblName.PutItemAsync((Document)myDeserializedClass.ToString());
return JsonConvert.DeserializeObject<ExpandoObject>(message);
}
}
}

How can I parse some JSON dynamically without knowing JSON values?

So I am using TDAmeritrade API to receive stock data with a C# Winforms program on Visual Studio. It takes the user input stock symbol and searches for the info. I am using HttpClient and Newtonsoft.Json and have been able to successfully perform the GET request and receive a JSON string back, but I do not know how to get all of the information I need out of it.
Here is the JSON:
https://drive.google.com/file/d/1TpAUwjyqrHArEXGXMof_K1eQe0hFoaw5/view?usp=sharing
Above is the JSON string sent back to me then formatted. My goal is to record information for each price in "callExpDateMap.2021-02-19:11" and "callExpDateMap.2021-03-19:39". The problem is that for each different stock, the dates that show up in "callExpDateMap" are going to be different.
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
var response = await client.GetAsync(url);
var info = await response.Content.ReadAsStringAsync();
dynamic config = JsonConvert.DeserializeObject<dynamic>(info, new ExpandoObjectConverter());
return config;
This is the code I have right now. I know the last for statement is not correct. How can I parse to the specific sections I want (callExpDateMap.expirationdate.StrikePrice) and get the information needed from each without knowing the dates and Strike prices beforehand? Is there a way to innumerate it and search through the JSON as if it were all a bunch of arrays?

The code below is perhaps not the most elegant nor complete, but I think it will get you going. I would start by using the JObject.Parse() from the Newtonsoft.Json.Linq namespace and take it from there.
JObject root = JObject.Parse(info);
string symbol = root["symbol"].ToObject<string>();
foreach (JToken toplevel in root["callExpDateMap"].Children())
{
foreach (JToken nextlevel in toplevel.Children())
{
foreach (JToken bottomlevel in nextlevel.Children())
{
foreach (JToken jToken in bottomlevel.Children())
{
JArray jArray = jToken as JArray;
foreach (var arrayElement in jArray)
{
InfoObject infoObject = arrayElement.ToObject<InfoObject>();
Console.WriteLine(infoObject.putCall);
Console.WriteLine(infoObject.exchangeName);
Console.WriteLine(infoObject.multiplier);
}
}
}
}
}
public class InfoObject
{
public string putCall { get; set; }
public string symbol { get; set; }
public string description { get; set; }
public string exchangeName { get; set; }
// ...
public int multiplier { get; set; }
// ...
}

This is official documentation of Newtonsoft method you are trying to use.
https://www.newtonsoft.com/json/help/html/Overload_Newtonsoft_Json_JsonConvert_DeserializeObject.htm
If an API's method returns different json propeties and you cannot trust it's property names all the times, then you can try using a deserialize method that returns .Net object, for example: JsonConvert.DeserializeObject Method (String)
https://www.newtonsoft.com/json/help/html/M_Newtonsoft_Json_JsonConvert_DeserializeObject.htm
That method's signature is this:
public static Object DeserializeObject(string value)
Parameter is: value of type json string.
Return Value is: Object of type object.
If you do not want an Object, then you can of course use a .Net type you have. Such as this method:
JsonConvert.DeserializeObject Method (String)
Any property that you have in both (the .net type and json object) will get populated. If .net type has properties that do not exist in json object, then those will be ignored. If json object has properties that do not exist in.net, then those will be ignored too.
Here's an example of a .Net type
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using Newtonsoft.Json.Serialization;
using System.Collections.Generic;
using System.IO;
using System.Text;
namespace MyNameSpace
{
public class TDAmeritradeStockData
{
[JsonProperty("symbol")]
public string Symbol { get; set; }
[JsonProperty("status")]
public string Status { get; set; }
[JsonProperty("callExpDateMap")]
public object CallExpDateMap { get; set; }
//...
//...
public CallExpDateMapType[] CallExpDateMapList { get; set; }
}
public class CallExpDateMapType
{
[JsonProperty("expirationdate")]
public string Expirationdate { get; set; }
[JsonProperty("StrikePrice")]
public List<StrikePriceType> StrikePriceList { get; set; }
}
public class StrikePriceType
{
public string StrikePrice { get; set; }
public List<StrikePricePropertiesType> StrikePricePropertiesList { get; set; }
}
public class StrikePricePropertiesType
{
[JsonProperty("putCall")]
public string PutCall { get; set; }
[JsonProperty("symbol")]
public string Symbol { get; set; }
[JsonProperty("description")]
public string Description { get; set; }
[JsonProperty("exchangeName")]
public string ExchangeName { get; set; }
[JsonProperty("bid")]
public double Bid { get; set; }
[JsonProperty("ask")]
public double Ask { get; set; }
//...
//...
}
[TestClass]
public class TestTestTest
{
[TestMethod]
public void JsonTest()
{
var jsondata = ReadFile("data.json");
var model = JsonConvert.DeserializeObject<TDAmeritradeStockData>(jsondata);
JObject jObject = (JObject)model.CallExpDateMap;
var count = ((JObject)model.CallExpDateMap).Count;
model.CallExpDateMapList = new CallExpDateMapType[count];
var jToken = (JToken)jObject.First;
for (var i = 0; i < count; i++)
{
model.CallExpDateMapList[i] = new CallExpDateMapType
{
Expirationdate = jToken.Path,
StrikePriceList = new List<StrikePriceType>()
};
var nextStrikePrice = jToken.First.First;
while (nextStrikePrice != null)
{
var nextStrikePriceProperties = nextStrikePrice;
var srikePriceList = new StrikePriceType
{
StrikePrice = nextStrikePriceProperties.Path,
StrikePricePropertiesList = JsonConvert.DeserializeObject<List<StrikePricePropertiesType>>(nextStrikePrice.First.ToString())
};
model.CallExpDateMapList[i].StrikePriceList.Add(srikePriceList);
nextStrikePrice = nextStrikePrice.Next;
}
jToken = jToken.Next;
}
Assert.IsNotNull(model);
}
private string ReadFile(string fileName)
{
using (var fileStream = new FileStream(fileName, FileMode.Open, FileAccess.Read))
{
var data = new StringBuilder();
using (var streamReader = new StreamReader(fileStream))
{
while (!streamReader.EndOfStream) data.Append(streamReader.ReadLine());
streamReader.Close();
}
fileStream.Close();
return data.ToString();
}
}
}
}

Display JSON DATA in dataGridView

I want to display JSON data in DGview in c#
I tried 2 codes works with Newtonsoft.Json
but both of them shows error invalid input but input is correct as I know
this is an input
https://dastanito.ir/test/ex2/api/storiesmaster/read.php
{"StoriesMasters":[{"id":"4545","story_code":"xv35454","master_code":"mvm,454"},{"id":"2","story_code":"002a","master_code":"he3385_1"},{"id":"60","story_code":"001a","master_code":"he3385_1"},{"id":"3","story_code":"c57675","master_code":"ara3433_2"},{"id":"45","story_code":"hdgw878","master_code":"f998989d"},{"id":"50","story_code":"d8885","master_code":"za76787_3"}]}
this is the first code I used:
using Newtonsoft;
using Newtonsoft.Json;
using System.Net;
using Newtonsoft.Json.Linq;
void get_response()
{
WebClient wp = new WebClient();
string url = "http://dastanito.ir/test/ex2/api/storiesmaster/read.php";
var response=wp.DownloadString(url);
get_data(response);
}
void get_data(string response)
{
dataGridView1.Rows.Clear();
JArray fetch= JArray.Parse(response);
if(fetch.Count()>0)
{
for(int i=0;dataGridView1.Rows.Count>i;i++)
{
int n=dataGridView1.Rows.Add();
dataGridView1.Rows[n].Cells[0].Value=fetch[i]["JsonObjectName1"].ToString();
}
}
}
and the other one is :
using RestSharp;
using Newtonsoft;
using Newtonsoft.Json;
using System.Net;
using Newtonsoft.Json.Linq;
public class JsonResult
{
public int id { get; set; }
public string story_code { get; set; }
public int master_code { get; set; }
}
void getdata()
{
var client = new RestClient("http://dastanito.ir");
var request = new RestRequest("test/ex2/api/storiesmaster/read.php", Method.GET);
// execute the request
IRestResponse response = client.Execute(request);
var content = response.Content;
richTextBox1.Text = response.Content;
var result = JsonConvert.DeserializeObject<List<JsonResult>(content);
dataGridView1.DataSource = result;
}
and the ERROR IS
ERROR

You need to create two classes as following.
public class Data
{
public int Id { get; set; }
public string story_code { get; set; }
public string master_code { get; set; }
}
public class DataList
{
public List<Data> StoriesMasters { get; set; }
}
now you can deserialize the json to an object of DataList class as following.
var result = JsonConvert.DeserializeObject<DataList>(content);
And now you can bind StoriesMasters property to grid view.
dataGridView1.DataSource = result.StoriesMasters;

Your json entry entity is object, not array. Array starts with [ ] and object starts with { } brackets.
Change JArray fetch = JArray.Parse(response); to JObject fetch = JObject.Parse(response);

Unity C# ArrayList of Objects Serialize into XML Utf8 - Primitive Error

I have a ArrayList with some user objects in. I am trying to serialize them into an XML format that I can send to a webserver. The format needs to be UTF 8.
I keep running into this error:
The type of the argument object 'User' is not primitive.
This is effectively two issues, however the main one being that this primitive error will not let me try and other web examples for utf8. I simply to not understand why it does this. I have tried using:
[Serializable]
Currently I have a function which will work but it will not do the xml to a utf8 format. And when I try any other examples on the web I then get this primitive error. Below is my current code:
My User Class:
using System;
using System.Xml;
using System.Xml.Serialization;
[Serializable]
public class User
{
public int UserID { get; set; }
public string Name { get; set; }
public string Password { get; set; }
public DateTime DateCreated { get; set; }
public string DeviceMacAddr { get; set; }
public DateTime LastLoggedIn { get; set; }
public float LastLoggedLat { get; set; }
public float LastLoggedLong { get; set; }
public bool Active { get; set; }
public string SyncStatus { get; set; }
public DateTime LastSyncDate { get; set; }
}
My XML writing Script:
using UnityEngine;
using System;
using System.Collections;
using System.Text;
using System.IO;
using System.Xml;
using System.Xml.Serialization;
using System.Collections.Generic;
public class XmlWriting : MonoBehaviour {
private string formatString = "'yyyy'##'MM'##'dd' 'HH'*'mm'*'ss'";
[XmlAttribute("Users")]
ArrayList Users = new ArrayList();
//List<User> Users = new List<User>();
// Use this for initialization
void Start () {
Users.Add(new User { UserID = 1,
Name = "Test Woman",
Password = "aa",
DateCreated = DateTime.Now,
DeviceMacAddr = "24:70:8c:83:86:BD",
LastLoggedIn = DateTime.Now,
LastLoggedLat = 36.083101f,
LastLoggedLong = -11.263433f,
Active = true,
SyncStatus = "Awaiting Response",
LastSyncDate = DateTime.Now,
}
);
Users.Add(new User { UserID = 2,
Name = "Test Man",
Password = "aa",
DateCreated = DateTime.Now,
DeviceMacAddr = "74:21:0c:93:46:XD",
LastLoggedIn = DateTime.Now,
LastLoggedLat = 83.083101f,
LastLoggedLong = -3.261823f,
Active = true,
SyncStatus = "Complete",
LastSyncDate = DateTime.Now,
}
);
var serializer = new XmlSerializer(typeof(ArrayList));
var memoryStream = new MemoryStream();
var streamWriter = new StreamWriter(memoryStream, System.Text.Encoding.UTF8);
serializer.Serialize(streamWriter, Users);
byte[] utf8EncodedXml = memoryStream.ToArray();
Debug.Log ("SerializeArrayList: " );//+ utf8EncodedXml);
}
// Update is called once per frame
void Update () {
}
private string SerializeArrayList(ArrayList obj)
{
XmlDocument doc = new XmlDocument();
XmlSerializer serializer = new XmlSerializer(typeof(ArrayList), new Type[]{typeof(User)});
using (MemoryStream stream = new System.IO.MemoryStream())
{
try
{
serializer.Serialize(stream, obj);
stream.Position = 0;
doc.Load(stream);
Debug.Log ("stream: " + doc.InnerXml);
return doc.InnerXml;
}
catch (Exception ex)
{
}
}
return string.Empty;
}
public class Utf8StringWriter : StringWriter
{
public override Encoding Encoding
{
get { return Encoding.UTF8; }
}
}
}
Any help is much appreciated.
Thanks

You need to put the [Serializable] attribute above the class definitions for User and every other class you intend to serialize. As for UTF-8, it shouldn't matter, at least not logically - C#'s deserializer on the other side should handle any encoding transparently for you.

There is an error in XML document (1,2) , System.InvalidOperationException: <AuthorizationResult xlms:""> was not expected

XML sent from API
<AuthenticationResult xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
<PAPIErrorCode>0</PAPIErrorCode>
<ErrorMessage/>
<AccessToken>StringAccessToken</AccessToken>
<AccessSecret>StringAccessToken</AccessSecret>
<PolarisUserID>PolarisSampleUser</PolarisUserID>
<BranchID>7</BranchID>
<AuthExpDate>2013-05-27T16:57:46.323</AuthExpDate>
</AuthenticationResult>
Classes for Response
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Xml;
using System.Xml.Schema;
using System.Xml.Serialization;
namespace PAPIAutomatedTestingTool
{
[XmlRoot(ElementName="AuthorizationResult")]
public class AuthorizationResult
{
public int PAPIErrorCode { get; set; }
public string ErrorMessage { get; set; }
public string AccessToken { get; set; }
public string AccessSecret { get; set; }
public int PolarisUserID { get; set; }
public int BranchID { get; set; }
public DateTime AuthExpDate { get; set; }
}
}
Code making request and deserializing
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.IO;
using System.Net;
using System.Web;
using System.Web.Script.Serialization;
using System.Security.Cryptography;
using System.Xml;
using System.Xml.Serialization;
namespace PAPIAutomatedTestingTool
{
public class APICallMethods
{
public string URI { get; set; }
public string accSecret { get; set; }
public string accToken { get; set; }
public string authorizationString { get; set; }
public bool AuthenticateStaffUser()
{
try
{
//Initializing all variables
string authReqMethod = "POST";
string authAccessKey = "Sample Access Key";
string authAccessKeyID = "Sample Access ID";
string authPatronPassword = "";
DateTime authDateTime = DateTime.Now;
string httpAuthDateTime = authDateTime.ToUniversalTime().ToString("r");
string authURI = "Sample URI";
//Composing the papiHash from the given parameters
string papiHash = GetPAPIHash(authAccessKey, authReqMethod, authURI, httpAuthDateTime, authPatronPassword);
//Formating the authorization string
string authorizationString = String.Format("Authorization: PWS {0}:{1}", authAccessKeyID, papiHash);
//Creating and defining the WebRequest
WebRequest req = WebRequest.Create(authURI);
req.Method = "POST";
req.Headers.Add("PolarisDate", httpAuthDateTime);
req.Headers.Add(authorizationString);
req.ContentType = "application/xml";
string requestBody = "<AuthenticationData><Domain>SampleDomain</Domain><Username>SampleUsername</Username><Password>SamplePassword</Password></AuthenticationData>";
byte[] reqBodyBytes = System.Text.Encoding.UTF8.GetBytes(requestBody);
req.ContentLength = reqBodyBytes.Length;
using (Stream requestStream = req.GetRequestStream())
{
requestStream.Write(reqBodyBytes, 0, reqBodyBytes.Length);
}
//Receiving the WebResponse
using (WebResponse resp = req.GetResponse())
{
AuthorizationResult firstResponse = new AuthorizationResult();
Stream respStream = resp.GetResponseStream();
StreamReader sr = new StreamReader(respStream);
XmlSerializer xmlSerializer = new XmlSerializer(typeof(AuthorizationResult));
firstResponse = (AuthorizationResult)xmlSerializer.Deserialize(respStream);
Console.WriteLine("Authorization: PWS" + firstResponse.AccessSecret + ":" + firstResponse.AccessToken);
return true;
}
}
catch (Exception ex)
{
Console.WriteLine(ex);
return false;
}
}
public string GetPAPIHash(string strAccessKey, string strHTTPMethod, string strURI, string strHTTPDate, string strPatronPassword)
{
byte[] secretBytes = UTF8Encoding.UTF8.GetBytes(strAccessKey);
HMACSHA1 hmac = new HMACSHA1(secretBytes);
byte[] dataBytes = null;
if (strPatronPassword.Length > 0)
{
dataBytes = UTF8Encoding.UTF8.GetBytes(strHTTPMethod + strURI + strHTTPDate + strPatronPassword);
}
else
{
dataBytes = UTF8Encoding.UTF8.GetBytes(strHTTPMethod + strURI + strHTTPDate);
}
byte[] computedHash = hmac.ComputeHash(dataBytes);
string computedHashString = Convert.ToBase64String(computedHash);
return computedHashString;
}
}
}
I am making a POST request to the API with a body that contains AuthorizationData. The API is supposed to return the xml to deserialize into firstresponse. I have received the xml(confirmed by printing to console) but i am receiving the There is an error in the XML Document(1,2) and <AuthorizationData xmlns=""> was not expected. Thanks for the help in advance.

It appears to be complaining about an unexpected root element and there is some confusion surrounding that elsewhere in your question.
In the question title you have <AuthorizationResult>
In the example XML response you have <AuthenticationResult>
In the C# you have [XmlRoot(ElementName="AuthorizationResult")]
At the end of your question you mention <AuthorizationData>
That's quite a few subtle variations on a theme.
Assuming the XML sample response is correct, you should change the C# to expect the root element to be <AuthenticationResult>...
[XmlRoot(ElementName="AuthenticationResult")]
public class AuthenticationResult
{
...

If you want to write a single deserialize for both schema and non-schema custom XMLs, you can turn off namespaces to avoid getting the 'There is an error in the XML Document(1,2)' error. But since the XmlReader expects a XML Schema, you should read it with StringReader class then use XmlReader object to remove custom namespaces.
//Namespaces
using System.IO;
using System.Xml.Serialization;
//Field
private static XmlSurrogates.Root _xmlSurrogates = new();
//Method
private static void DeserializeXml(string xml)
{
try
{
XmlSerializer serializer = new XmlSerializer(_xmlSurrogates.GetType());
using (StringReader reader = new StringReader(xml))
{
var xmlTextReader = new System.Xml.XmlTextReader(reader);
xmlTextReader.Namespaces = false;
_xmlSurrogates = (XmlSurrogates.Root)(serializer.Deserialize(xmlTextReader));
}
}
catch (Exception ex)
{
Log.Error($"There is an error while deserialize. Error: {ex.Message}");
}
}
//Root Model Example
public class XmlSurrogates
{
[XmlRoot(ElementName = "XMLROOTELEMENTNAMEHERE FOR THIS ITS root")]
public class Root
{
[XmlElement(ElementName = "Boo")] public Boo Boo { get; set; }
[XmlElement(ElementName = "Control")] public ClassName DeclareName { get; set; }
[XmlAttribute(AttributeName = "MainCode")] public string MainCode { get; set; }
[XmlAttribute(AttributeName = "Caption")] public string Caption { get; set; }
}
[XmlRoot(ElementName = "Boo")]
public class ElementAsClassName
{
[XmlAttribute(AttributeName = "Foo")] public string PropertyName { get; set; }
}
}