I'm trying to parse some JSON data and ultimatley store it in a database.
I'm having issues when storing a collection of strings / values which are not objects themselves.
For example - The "callingCodes" & "altSpellings"
I want to store these in an SQL table which will have reference to the country they belong to.
This is an example of the JSON:
{
"name":"Puerto Rico",
"topLevelDomain":[
".pr"
],
"alpha2Code":"PR",
"alpha3Code":"PRI",
"callingCodes":[
"1787",
"1939"
],
"capital":"San Juan",
"altSpellings":[
"PR",
"Commonwealth of Puerto Rico",
"Estado Libre Asociado de Puerto Rico"
],
"region":"Americas",
"subregion":"Caribbean",
"population":3474182,
"latlng":[
18.25,
-66.5
]
},
I originally created some C# classes to represent the data using http://json2csharp.com/
This sugguested I store the values as a list of strings, which I did:
public List<string> CallingCodes { get; set; }
I now want to store the data in a table, so I created a class "TopLevelDomain" to store / link the data to the parent country:
public class CallingCode
{
public int ID { get; set; }
public int CountryID { get; set; }
public string Code{ get; set; }
}
So I altered the parent to be as follows:
public ICollection<CallingCode> CallingCodes { get; set; }
Is it possible to direct the string values into the "Code" property of my new class?
Or am I trying to crowbar two pieces of logic into one?
Is the correct way to have models for the JSON, and manually restructure these into my new DB / Entity Framework Models?
This is the auto-generated class you get from such JSON. The tricky bit here is List of primitive types.
public class RootObject
{
public string name { get; set; }
public List<string> topLevelDomain { get; set; }
public string alpha2Code { get; set; }
public string alpha3Code { get; set; }
public List<string> callingCodes { get; set; }
public string capital { get; set; }
public List<string> altSpellings { get; set; }
public string region { get; set; }
public string subregion { get; set; }
public int population { get; set; }
public List<double> latlng { get; set; }
}
Certain databases like PostgreSQL supports array as primitive type. If you are using PostgreSQL then you can perhaps make those properties array of primitive type and store them on server as is.
For other databases which does not support array, you cannot store a list of primitive values into single column of database. The easiest way to deal with it is to introduce serialization and create single string which can be stored to server. So looking at above class, for public List<string> topLevelDomain property, you can rewrite it in following way,
[NotMapped]
public List<string> topLevelDomain
{
get => Deserialize(TopLevelDomainString);
set => TopLevelDomainString = Serialize(value);
}
public string TopLevelDomainString { get; set; }
With NotMapped attribute EF will not map topLevelDomain property. But TopLevelDomainString will be persisted to database and it will get values from topLevelDomain. As for Serialize/Deserialize methods, you can use any serialization method. You can use JsonSerializer directly (since you are already using JSON objects. Or you can just combine strings using , as delimiter and split string from server using it.
Starting with EF Core 2.1 version, you can use Value-Conversion feature directly to provide funcs to do conversion (essentially serialization code like above) to EF and EF will do it while reading/saving data from/to server. This will avoid you having to create additional CLR property.
Here is your auto-generated class:
public class RootObject
{
public string name { get; set; }
public List<string> topLevelDomain { get; set; }
public string alpha2Code { get; set; }
public string alpha3Code { get; set; }
public List<string> callingCodes { get; set; }
public string capital { get; set; }
public List<string> altSpellings { get; set; }
public string region { get; set; }
public string subregion { get; set; }
public int population { get; set; }
public List<double> latlng { get; set; }
}
Let's prepare another simple one:
public class MyRootObject
{
public MyRootObject(RootObject root)
{
Name = root.name;
List<CallingCode> callingCodesConverted = new List<CallingCode>();
foreach (string code in root.callingCodes)
{
CallingCode newCode = new CallingCode() { Code = code };
callingCodesConverted.Add(newCode);
}
CallingCodes = callingCodesConverted;
}
public string Name { get; set; }
public List<CallingCode> CallingCodes { get; set; }
}
Now you could first do encoding from json to class RootObject, and then create MyRootObject based on it:
string path = #"D:\test.txt";
var r = new StreamReader(path);
var myJson = r.ReadToEnd();
RootObject root = Json.Decode<RootObject>(myJson);
MyRootObject myroot = new MyRootObject(root);
Sure MyRootObject is only an example.
Is the correct way to have models for the JSON, and manually
restructure these into my new DB / Entity Framework Models?
Well some might use that in their code and some might do even worse than that. But, I personally like to use models/dtos for the things I could and know about the data.
am I trying to crowbar two pieces of logic into one?
Yes. But, it depends. Either strongly type/define objects and all or just Ser/Deser everytime.
Your data is known (no unkown properties or anything, so why serialize and deserialize it everytime?)
Solution 1 : if you use the JSON as is to create a DB Entry
Entites
public class Country //assuming this is country data
{
public int Id { get; set; }
[JsonProperty("name")]
public string Name { get; set; }
[JsonProperty("alpha2Code")]
public string Alpha2Code { get; set; }
[JsonProperty("alpha3Code")]
public string Alpha3Code { get; set; }
[JsonProperty("capital")]
public string Capital { get; set; }
[JsonProperty("region")]
public string Region { get; set; }
[JsonProperty("subregion")]
public string Subregion { get; set; }
[JsonProperty("population")]
public long Population { get; set; }
[JsonProperty("topLevelDomain")]
public virtual List<TopLevelDomain> TopLevelDomains { get; set; }
[JsonProperty("callingCodes")]
public virtual List<CallingCodes> CallingCodes { get; set; }
[JsonProperty("altSpellings")]
public virtual List<AltSpellings> AltSpellings { get; set; }
[JsonProperty("latlng")]
public virtual List<Coordinates> Coordinates { get; set; }
}
public class TopLevelDomain
{
public int Id { get; set; }
[ForeignKey("Country")]
public int CountryId {get; set; }
public virtual Country Country { get; set; }
public string DomainName { get; set; }
}
public class CallingCodes
{
public int Id { get; set; }
[ForeignKey("Country")]
public int CountryId {get; set; }
public virtual Country Country { get; set; }
public string Code { get; set;} // either store it as String
//OR
public long Code { get; set;}
}
public class AltSpellings
{
public int Id { get; set; }
[ForeignKey("Country")]
public int CountryId {get; set; }
public virtual Country Country { get; set; }
public string AltSpelling { get; set; }
}
public class Coordinates
{
public int Id { get; set; }
[ForeignKey("Country")]
public int CountryId {get; set; }
public virtual Country Country { get; set; }
public double Coordinates { get; set; } //again either as string or double, your wish. I would use double
}
Use it like so
//assuming using Newtonsoft
var myJson = ....assuming one Country;
var myJsonList = ...assuming List<Country>;
var country = JsonConvert.DeserializeObject<Country>(myJson);
var countries = JsonConvert.DeserializeObject<List<Country>>(myJson);
Solution 2 : First one can cause too many tables for a little data but First solution is a little more object based and typed, so Here is another one
Entity
public class Country //assuming this is country data
{
public int Id { get; set; }
public string Name { get; set; }
public string Alpha2Code { get; set; }
public string Alpha3Code { get; set; }
public string Capital { get; set; }
public string Region { get; set; }
public string Subregion { get; set; }
public long Population { get; set; }
public string TopLevelDomains { get; set; }
public string CallingCodes { get; set;}
public string AltSpellings { get; set; }
public double Longitude { get; set;}
public double Latitude { get; set; }
}
Model
public class CountryJson
{
public int Id { get; set; }
[JsonProperty("name")]
public string Name { get; set; }
[JsonProperty("alpha2Code")]
public string Alpha2Code { get; set; }
[JsonProperty("alpha3Code")]
public string Alpha3Code { get; set; }
[JsonProperty("capital")]
public string Capital { get; set; }
[JsonProperty("region")]
public string Region { get; set; }
[JsonProperty("subregion")]
public string Subregion { get; set; }
[JsonProperty("population")]
public long Population { get; set; }
[JsonProperty("topLevelDomain")]
public List<string> TopLevelDomains { get; set; }
[JsonProperty("callingCodes")]
public List<string> CallingCodes { get; set;}
[JsonProperty("altSpellings")]
public List<string> AltSpellings { get; set; }
[JsonProperty("latlng")]
public List<string> Latlng { get; set; }
}
Use it like so
//assuming using Newtonsoft
var myJson = ....assuming one Country;
var countryJson = JsonConvert.DeserializeObject<CountryJson>(myJson);
//Write a Mapper Or Manual Map like below
var countryEntity = new Country
{
Name = countryJson.Name,
...
TopLevelDomains = JsonConvert.Serialize(countryJson.TopLevelDomains),
CallingCodes = JsonConvert.Serialize(countryJson.CallingCodes),
...//same for all list (NOTE: YOU NEED TO DESERIALIZE IT WHEN YOU FETCH IT FROM DB
Longitude = countryJson.Latlng.ElementAt(0),//assuming 0 is longi, 1 is lat
Latitude = countryJson.Latlng.ElementAt(1)//you can do it like above as well as string if you want
}
Related
I'm not yet dependent to either Mapster or AutoMapper. For now I'm using handwritten mappings because I couldn't find a mapper who could do this with smaller code.
The problem is how do we map flatten structures to complex objects? I think a lot of people could benefit from a good mapping example for such a complex object. I've got even a mapping condition based on CopyOfficeAddressAsInvoiceAddress whether or not the office address needs to be copied as invoice address. I've looked all over the place but couldn't get it to work.
Maybe I should also use a different naming to make it more clear for the mapping algorithm?!
The biggest question could such a map being resolved by a mapper or is this to complex? Al the demo's I've seen were using dto and model objects that are quite similar to each other. I didn't get the point of mapping an object to another object that 99% similar to each other.
I have a Command (I'm using Mediatr) that looks like as follows:
public class Command : IRequest<IActionResult>
{
public string AccountName { get; set; }
public string ContactFirstName { get; set; }
public string ContactLastName { get; set; }
public string ContactEMail { get; set; }
public string ContactPhoneNumber { get; set; }
public string BankAccount { get; set; }
public string Bank { get; set; }
public string OfficeName { get; set; }
public string OfficeAddressStreet { get; set; }
public int OfficeAddressStreetNumber { get; set; }
public string? OfficeAddressStreetNumberAddition { get; set; }
public string OfficeAddressPostalcode { get; set; }
public string OfficeAddressCity { get; set; }
public string OfficeAddressCountry { get; set; }
public string? OfficeInvoiceAddressStreet { get; set; } = null;
public int? OfficeInvoiceAddressStreetNumber { get; set; } = null;
public string? OfficeInvoiceAddressStreetNumberAddition { get; set; } = null;
public string? OfficeInvoiceAddressPostalcode { get; set; } = null;
public string? OfficeInvoiceAddressCity { get; set; } = null;
public string? OfficeInvoiceAddressCountry { get; set; } = null;
//[Ignore]
public bool? CopyOfficeAddressAsInvoiceAddress { get; set; } = false;
public string? AssociationIdentifier { get; set; } = null;
}
And I want it to be mapped to the following models:
public class Account
{
public int Id { get; set; }
public string AccountName { get; set; }
public IList<Contact> Users { get; set; }
public IList<Office> Offices { get; set; }
public string Bank { get; set; }
public string BankAccount { get; set; }
public string? AssociationIdentifier { get; set; }
}
public class Office
{
public int Id { get; set; }
public string Name { get; set; }
public Address ContactAddress { get; set; }
public Address InvoiceAddress { get; set; }
public bool HeadQuarter { get; set; }
}
public class Address
{
public int Id { get; set; }
public string Street { get; set; }
public string Postalcode { get; set; }
public int StreetNumber { get; set; }
public string StreetNumberAddition { get; set; }
public string City { get; set; }
public string Country { get; set; }
}
public class Contact
{
public int Id { get; set; }
public string FirstName { get; set; }
public string LastName { get; set; }
public string EMail { get; set; }
public string PhoneNumber { get; set; }
}
First of all, my experience is mainly using Automapper, and it is definitely possible to map complex types like this.
But your command does not need to be completely flat. There is nothing inherently wrong with DTOs being similar to your domain models. Using Automapper this is fairly easy as properties with the same name are mapped 1:1.
It could be that you are submitting a form with all the properties flattened in one object. In that case you could define either a seperate map for this object and each domain object.
CreateMap<AccountDto, Account>(); // mapping logic omitted
CreateMap<AccountDto, Office>();
...
Or you could map the one object to a range of objects using Tuples.
CreateMap<AccountDto, (Account, Office, ...)>(); // mapping logic omitted
But if you define seperate DTOs and make mapping profiles for them, it will probably ease your whole mapping experience. For copying the address, you can simply do something like this, in that case.
if (copyAddress)
{
office.InvoiceAddress = _mapper.Map<Address>(addressDto);
}
I am trying to take this JSON:
{
"title":"string",
"description":"string",
"date":"2021-04-19T01:05:38.000Z",
"image":"url",
"images":[
"url1",
"url2"
],
"attributes":{
"phonebrand":"x",
"phonecarrier":"y",
"forsaleby":"z",
"price":12345,
"location":"daLocation",
"type":"OFFERED"
},
"url":"url to listing"
}
And convert it into this C# Object:
public class Listing {
[JsonProperty("title")]
public string Title { get; set; }
[JsonProperty("description")]
public string Description { get; set; }
[JsonProperty("date")]
public DateTime? Date { get; set; }
[JsonProperty("image")]
public string Image { get; set; }
[JsonProperty("images")]
public string[] Images { get; set; }
[JsonProperty("url")]
public string Url { get; set; }
[JsonProperty("price")]
public decimal Price { get; set; }
[JsonProperty("locationId")]
public int LocationId { get; set; }
[JsonProperty("categoryId")]
public int CategoryId { get; set; }
[JsonProperty("sortByName")]
public string SortByName { get; set; }
[JsonProperty("q")]
public string Q { get; set; }
[JsonProperty("location")]
public string Location { get; set; }
[JsonProperty("type")]
public string Type { get; set; }
[JsonProperty("forsaleby")]
public string ForSaleBy { get; set; }
[JsonProperty("fulfillment")]
public string Fulfillment { get; set; }
[JsonProperty("payment")]
public string Payment { get; set; }
[JsonProperty("phonebrand")]
public string? PhoneBrand { get; set; }
[JsonProperty("phonecarrier")]
public string? PhoneCarrier { get; set; }
}
My problem is, I'm trying to deserialize properties like price and phonebrand but those properties are under an object in the JSON. So when I try to deserialize them like this, those properties can't be found and are set as null. How can I deserialize those properties without changing my C# Class to include an Attributes class? I want to do this because I think that it is a cleaner/better design compared the JSON I'm taking in.
I suggest two approaches that are very explicit and easy to follow for the next developer looking at the code.
Two classes
creating a intermediate dto class that is used for deserialisation and then creating the business logic object from that intermediate object.
var withAttributes = Deserialise<ListingDto>();
var flatObject = new Listing(withAttributes);
One class
You could provide accessors at the top level which dip into the subclasses.
public class Listing
{
public AttributesDto Attributes {get; set}
...
public string Url => Attributes.Url; // Maybe '?.Url'
}
I currently have JSON coming in as follows:
{"36879":[{"min_qty":1,"discount_type":"%","csp_price":10}],"57950":[{"min_qty":1,"discount_type":"flat","csp_price":650}]}
This contains a list of the following records
ProductId
MinQty
DiscountType
Price
I need to deserialize this into the following model:
public class CustomerSpecificPricing
{
string productId { get; set; }
public virtual List<CustomerSpecificPricingDetail> CustomerSpecificPricingDetails { get; set; }
}
public class CustomerSpecificPricingDetail
{
public string min_qty { get; set; }
public string discount_type { get; set; }
public string csp_price { get; set; }
}
The problem is that the "productId" of each record is missing the key name.
If I run my JSON through J2C, I get the following:
public class 36879 {
public int min_qty { get; set; }
public string discount_type { get; set; }
public int csp_price { get; set; }
}
public class 57950 {
public int min_qty { get; set; }
public string discount_type { get; set; }
public int csp_price { get; set; }
}
public class Root {
public List<_36879> _36879 { get; set; }
public List<_57950> _57950 { get; set; }
}
Which is obviously incorrect.
How would I deserialize my object correctly?
You would need to deserialize it into a dictionary first and then map it into the format you require after. Something like this should work:
var dict = JsonConvert.DeserializeObject<Dictionary<string, IEnumerable<CustomerSpecificPricingDetail>>>();
var result = dict.Select(kvp => new CustomerSpecificPricing { ProductId = Int32.Parse(kvp.Key), CustomerSpecificPricingDetails = kvp.Value });
Id also recommend you follow the conventional standards of naming. In this case properties in classes should be PascalCase,
e.g. your classes now become:
public class CustomerSpecificPricing
{
[JsonProperty("productId ")]
public string ProductId { get; set; }
public virtual List<CustomerSpecificPricingDetail> CustomerSpecificPricingDetails { get; set; }
}
and
public class CustomerSpecificPricingDetail
{
[JsonProperty("min_qty")]
public string MinQty { get; set; }
[JsonProperty("discount_type ")]
public string DiscountType { get; set; }
[JsonProperty("csp_price ")]
public string CspPrice { get; set; }
}
I'm having problems deserializing the data I'm getting from Shodan. Below are the classes I got from json2csharp and I'm trying to create an array of the matches and loop through them. It seems like I have tried with everything except a working array by now. The data itself is matches as root with objects of them that contain location (with its own data etc). An except below that I cut out a bit.
This is my error:
Cannot deserialize the current JSON object (e.g. {"name":"value"}) into type 'Shodan.Match[]' because the type requ
ires a JSON array (e.g. [1,2,3]) to deserialize correctly.
var data = JsonConvert.DeserializeObject<Match[]>(allData);
{"matches": [{"product": "product", "hash": 0, "ip": 123123, "isp": "Verizon Internet Services"}], "total": 1}
public class Location
{
public string city { get; set; }
public string region_code { get; set; }
public object area_code { get; set; }
public double longitude { get; set; }
public string country_code3 { get; set; }
public double latitude { get; set; }
public string postal_code { get; set; }
public object dma_code { get; set; }
public string country_code { get; set; }
public string country_name { get; set; }
}
public class Options
{
}
public class Shodan
{
public string crawler { get; set; }
public string id { get; set; }
public string module { get; set; }
public Options options { get; set; }
}
public class Match
{
public int hash { get; set; }
public int ip { get; set; }
public string isp { get; set; }
public string transport { get; set; }
public string data { get; set; }
public string asn { get; set; }
public int port { get; set; }
public List<string> hostnames { get; set; }
public Location location { get; set; }
public DateTime timestamp { get; set; }
public List<string> domains { get; set; }
public string org { get; set; }
public object os { get; set; }
public Shodan _shodan { get; set; }
public string ip_str { get; set; }
public string product { get; set; }
}
public class RootObject
{
public List<Match> matches { get; set; }
public int total { get; set; }
}
You could create another class like
var allData =
{"matches": [{"product": "product", "hash": 0, "ip": 123123, "isp": "Verizon Internet Services"}], "total": 1}
public class MyMatches {
public Match[] matches {get; set;}
}
and then use that in the deserializer.
var data = JsonConvert.DeserializeObject<MyMatches>(allData);
This is if the JSON code sample you gave us is correct.
CORRECTION
Just saw the RootObject class.
Just use that.
I have an Api returning JSON data as response. This response is basically answers to a survey taken. We have multiple surveys. Each survey will have multiple questions which can be different from the other surveys. (Hope I make sense here)
I have created a model class from the response I get for one of the survey like follows :-
public string value1 { get; set; }
public string value2 { get; set; }
public string value3 { get; set; }
public string value4 { get; set; }
public int FileSystemObjectType { get; set; }
public int Id { get; set; }
public object someUrl1 { get; set; }
public string SomeUrl2 { get; set; }
public string TypeId { get; set; }
public string DifferentValue1 { get; set; }
public bool DifferentValue2 { get; set; }
public List<string> DifferentValue3 { get; set; }
public List<string> DifferentValue4 { get; set;
The properties DifferentValue1, DifferentValue2, DifferentValue3 .. are the questions that can be different for each survey. Right now this survey has 4 questions, but other surveys can have different number of questions. The rest of the properties remain same throughout the surveys.
Eg. Other survey can have a model
public string value1 { get; set; }
public string value2 { get; set; }
public string value3 { get; set; }
public string value4 { get; set; }
public int FileSystemObjectType { get; set; }
public int Id { get; set; }
public object someUrl1 { get; set; }
public string SomeUrl2 { get; set; }
public string TypeId { get; set; }
public string DifferentValue1 { get; set; }
public bool DifferentValue2 { get; set; }
Is there a way I can make a generic model class where the properties for the question can be different but other properties are same?
Any help is appreciated.
Sounds to me like a typical scenario for inheritance..
Create a base class with properties that you want to have for all surveys and then derive from that class and add properties for the individual questions to that child class.
Example:
public class SurveyBase
{
public int Id { get; set; }
public string SomeUrl { get; set; }
public string TypeId { get; set; }
//...
}
public class Survey1 : SurveyBase
{
public string MyQuestion1 { get; set; }
public string MyQuestion2 { get; set; }
//..
}
public class Survey2 : SurveyBase
{
public string OtherProperty { get; set; }
public int Whatever { get; set; }
//..
}