I'm trying to create a page that is going to read a CSV file and create a JSON file. I need two of the columns in the json. The CSV contains between 15,000-30,000 lines. I need the JSON to be max 10,000 lines and then I am going to check the JSON against an API. The result must get values from the original file, and then write it to a new CSV file with all the columns.
The CSV file looks like this:
number,"surname","forename","emailAddress","taxIdentifier"
101719008,"John","smith","smith#hotmail.com","1997xxxxxxxx"
102358612,"John","doe","doe#gmail.com","1993xxxxxxxx"
I have this code to read the CSV file:
protected void Button2_Click(object sender, EventArgs e)
{
string saved = (#"E:\Temp\Spelpaus\Malmö2022fromNeon.csv");
using (var streamreader = new StreamReader(saved))
using (var csvReader = new CsvReader(streamreader, CultureInfo.InvariantCulture))
{
var records = csvReader.GetRecords<RocketLaunch>().ToList();
}
}
public class RocketLaunch
{
public string number { get; set; }
public string surname { get; set; }
public string forename { get; set; }
public string emailAddress { get; set; }
public string taxIdentifier { get; set; }
}
I need a start in the JSON and then 2 columns (number and taxidentifier) from the CSV and then 2 characters at the end.
After I got the answer from the API I need to find number from original list and get all columns for every number from API.
Any ideas?
You've defined CSVEntry.Parse as static, so the call could look something like:
string line = rd.ReadLine();
CSVEntry entry = CSVEntry.Parse(line);
You could add entry to a list of type List<CSVEntry>.
One issue might be that your sample data is comma separated, but your code is splitting on semicolons.
Split is a good method to get you quickly started, but I notice that some fields are quoted, which means there's the potential for the data within the quotes to include commas. I wrote my own CSV parser many years ago to handle this sort of thing, but I'm sure there must be something on NuGet to do this for you these days (look for something that supports RFC 4180).
Related
I have a XML file which has all information about a voyage and all details inside.
I want to read all records in XML file, after combining records I want to write it to SQL database.
So far I arranged getting header , company and voyage to array but getting details for all records to array I failed.
Here are my task to handle:
Select and read any XML Data to RAM by FileDialog (completed)
Create Arrays and Read XML data to Arrays (partly completed)
Write XML data to DataView (partly completed)
Create T-SQL INSERT Command (Partly completed)
Write Data to Database (Waiting to finish early steps)
While reading from XML to DataView I can get data to memory but could not seperated multi level data as requested.
The exact problem is trying to handle different levels of data in XML in every XML file I recieve.
foreach (var child in childElem.Elements("ManifestData"))
{
foreach(var x in child.Elements())
{
var checkName = x.Name.ToString();
switch (checkName)
{
case "Company":
Globals.Companys.Clear();
foreach (var y in x.Elements())
{
Globals.Companys.Add(y.Name.ToString(), y.Value.ToString());
}
break;
case "Voyage":
Globals.Voyages.Clear();
foreach (var y in x.Elements())
{
Globals.Voyages.Add(y.Name.ToString(), y.Value.ToString());
}
break;
case "BLs":
int recs = 0;
Globals.BL.Clear();
textBox2.Clear();
foreach (var y in x.Elements())
{
foreach (var z in x.Elements("units"))
{
Globals.Units.Add(y.Element("number").Value.ToString(), z.Value.ToString());
}
Globals.BL.Add(y.Element("number").Value.ToString(), y.Value.ToString());
recs = recs + 1;
textBox2.AppendText("\n" + y.ToString());
string output = string.Join("\n", Globals.BL);
MessageBox.Show(output);
}
break;
default:
break;
}
}
}
In my example XML you see that there is 3 BLs and all BL data has different levels.There can be hundreds of BLs with different levels of Goods & Dangerous Goods.
I am having trouble handling multi level XML data here.
I 'll be glad if you help me solve this very basic problem. I am hoping to learn and leave it for the people to figure out to understand making desktop XML reader application for their own DBs.
Here is the XML Data example
You can find all sources here : Project Reading XMLbyC#
The xml processing part can be made simple by deserializing your xml into c# classes which you can then use to do whatever you want.
[XmlRoot(ElementName = "ManifestMessage")]
public class ManifestMessage
{
[XmlElement(ElementName = "Header")]
public Header Header { get; set; }
[XmlElement(ElementName = "ManifestData")]
public ManifestData ManifestData { get; set; }
}
[XmlRoot(ElementName = "Header")]
public class Header
{
[XmlElement(ElementName = "sender")]
public string Sender { get; set; }
[XmlElement(ElementName = "reciever")]
public string Reciever { get; set; }
[XmlElement(ElementName = "timeOfDocument")]
public string TimeOfDocument { get; set; }
[XmlElement(ElementName = "typeOfMessage")]
public string TypeOfMessage { get; set; }
}
// Then when you want to get the xml deserialized into your class hierarchy
var xmlSerializer = new XmlSerializer(typeof(ManifestMessage));
var manifestMessage = xmlSerializer.Deserialize(data) as ManifestMessage;
// now you can use this object to drill down the whole hierarchy
Console.WriteLine(xmlData.Header.Sender);
Console.WriteLine(xmlData.ManifestData.Company.ComanyName);
Console.WriteLine(xmlData.ManifestData.Voyage.CrewNumber);
foreach (var bl in xmlData.ManifestData.BLs.BL)
{
Console.WriteLine(bl.Collect);
Console.WriteLine(bl.Consegnee.Name);
Console.WriteLine(bl.Customer.Name);
}
You can use https://xmltocsharp.azurewebsites.net/ site to generate the whole c# class hierarchy from your xml.
Console.WriteLine is just for demo purpose you can adopt it according to your needs.
Excuse me right off the bat. I am sort of new.
I have an object that contains few properties of which one of the property in that itself is a List. Now, we do not know how big the list of values in the input payload would be like (It could be 1000, it could be 100,000). We are logging this request payload before we process.
We use _logger.Verbose ("Some String...", {object});
When we log, the log file (We use Serilog) saves it as a notepad file with huge values, in JSON format.
Now, when the input is too big, the logger tries to log but fails and retries many times due to big payload.
I am looking for a way to split or do some looping and split and store or something. I dont know how to do in C# code. I tried googling and researched a lot but futile. I found SKIP and TASK methods of Lambda but unsure how to use.
Code below:In this case, imagine, "Model" is like 1000, or 100,1000 it could be anything. I am just looking for a loop logic in C# to divide to a decent number and process.
public class Make
{
public int ID { get; set;}
public string Name { get; set;}
public string Category { get; set;}
public List<Model> Models { get;set;}
}
public class Model
{
public string Name { get; set;}
public string County { get; set;}
public string Submodel { get; set;}
}
public ProcessCars ( Make object)
{
_logger.Verbose ("Some String...", {object});`
// Processing///
//.....//
}
I understand the purpose of yours is to view or debug the values of your list.
If I were you, I would ask myself a few questions
Do I need to write all values? Why can't I filter first before logging?
What's the purpose of writing to a text file, when you can log to database? Serilog support DB logging.
Is it a best practice to log large values to a text file?
I am getting JSON of Form Data from users. Below is the example of JSON Data
{
"Name":"Mike",
"Age":25,
"Gender":"Male",
"Skills":{
".Net":true,
"Mule":""
}
}
I want to save this data in a table (SQL Server). There is no table in the database, I want to define table name before sending this data to sql. Is there any approach to achieve this. Please help.
I suggest using json2csharp to convert the JSON to C# models and alter the names which are not recognized
public class Skills
{
[JsonProperty(PropertyName = ".Net")] //Add JsonProperty to include unclassified names
public bool DotNet { get; set; }
public string Mule { get; set; }
}
public class RootObject10
{
public string Name { get; set; }
public int Age { get; set; }
public string Gender { get; set; }
public Skills Skills { get; set; }
}
then, you can deserialize the json using JsonConvert
using (StreamReader r = new StreamReader(filepath))
{
string json = r.ReadToEnd();
var obj = JsonConvert.DeserializeObject<RootObject10>(json);
}
After this, based on your requirement create single data table or 2 data tables and inject the data
Assuming your question is not about how to translate json into c# object but rather about how to store it in SQL Server while still being able to query it, you can actually work with json data in sql server: https://learn.microsoft.com/en-us/sql/relational-databases/json/json-data-sql-server?view=sql-server-ver15.
I would opt for something like that if I didn’t have schema upfront or I knew it will change often. Then I would create a table called FormData with an Id and Data fields and just stored your JSON in there.
Bear in mind this is likely less performant than defining tables and properly parsing json (which is covered by other answers here) - make sure you make the call after you’ve considered all pros and cons of schema-less storage.
Upd: if you absolutely must create tables at runtime you could potentially run plain SQL DDL statements like ´CREATE TABLE´ using plain ADO.NET
I need to deserialize and/or export to csv a JSON string being sent back to me. The problem is that the JSON string is formatted in a way that results dates being used as the headers.
See example below:
{"2017-01-09":{"chats":129,"missed_chats":0},
"2017-01-10":{"chats":347,"missed_chats":0},
"2017-01-11":{"chats":194,"missed_chats":0},
"2017-01-12":{"chats":229,"missed_chats":0}}
Because of this, I do not believe I can make a class to hold them as each date would need its own class to match the heading of "2017-01-09", and so on. I need to keep the dates as data, so setting my own headers for the csv file programmatically is perfectly fine.
I am not able to use any add ins for visual studio such as JSON.Net, so the built in tool only or some other work around.
Thank you
You can deserialise to a Dictionary, something like this:
public class InnerThing
{
public int Chats { get; set; }
public int Missed_Chats { get; set; }
}
var result = JsonConvert.DeserializeObject<Dictionary<string, InnerThing>>(json);
I am working on a REST API for a project using Visual Studio 2013 with C# and ASP.NET, and I need some guidance.
When the webpage performs a POST, I am passing along a number of fields as a JSON object. By defining a data transfer object in my C# code, I can easily read the values from the JSON, but only if I define all the fields (with the same name).
Here is my current (working) code:
public class AgencyPostDTO
{
public string AgencyName { get; set; }
public string Address1 { get; set; }
public string Address2 { get; set; }
public string City { get; set; }
public string State { get; set; }
public string ZIP { get; set; }
}
// POST: api/Agency
public string Post(AgencyPostDTO Agency)
{
int success;
success = SQLUpdateAgency(Agency);
if (success < 1)
{
return "Failed";
}
else
{
return "Success";
}
}
So far no problems. I need to pass the data over to a second function, where I will perform some data processing (including converting the data into XML) and send the data/XML to MS SQL using a stored procedure:
public int SQLUpdateAgency(AgencyPostDTO Agency)
{
string xml = Agency.SerializeObject();
... code to call SQL stored procedure ommitted here
}
Now to my problem. I would prefer if I did not have to define the parameters of the data transfer object AgencyPostDTO in the code, and instead the code would just read the incoming JSON and pass it along to the next function, where I create the XML containing everything passed along.
As it works now, if the JSON contains for example an email address field, it will be dropped unless I define it in AgencyPostDTO.
So why do I want to do this? For future ease of maintenance. The users may come and say they want to add additional fields to the web form. I can then simply have our SQL expert add that column to the table, give me the name of it and I add an input field to the HTML form and make sure it is included in the JSON sent over. That way we never have to touch the already written, tested and working code. The new field is simply passed though the whole process.
Can this be done? If so, any suggestions on how?
If you used JSON.NET to handle the deserialisation of your objects then that has support for dynamic properties. Once you'd read your JSON string, you could convert it to a JArray or JObject and from there by using the .Children() call to get a list of all properties to convert it to any XML object you needed.
Have a look here:
Deserialize json object into dynamic object using Json.net