We Keep Coding

How to skipped if the column name are same in the datatable

I'm using HTML Agility Pack to web scrape to datatable. However the website have multiple same column name which it was not able to add on for the second table.
The error will be prompt out like this as the "2020" had been added before
My code as below :
public void WebDataScrap()
{
try
{
//Get the content of the URL from the Web
const string url = "https://www.wsj.com/market-data/quotes/MY/XKLS/0146/financials/annual/cash-flow";
var web = new HtmlWeb();
var doc = web.Load(url);
const string classValue = "cr_dataTable"; //cr_datatable
//var nodes = doc.DocumentNode.SelectNodes($"//table[#class='{classValue}']") ?? Enumerable.Empty<HtmlNode>();
var resultDataset = new DataSet();
foreach (HtmlNode table in doc.DocumentNode.SelectNodes($"//table[#class='{classValue}']") ?? Enumerable.Empty<HtmlNode>())
{
var resultTable = new DataTable(table.Id);
foreach (HtmlNode row in table.SelectNodes("//tr"))
{
var headerCells = row.SelectNodes("th");
if (headerCells != null)
{
foreach (HtmlNode cell in headerCells)
{
resultTable.Columns.Add(cell.InnerText);
}
}
var dataCells = row.SelectNodes("td");
if (dataCells != null)
{
var dataRow = resultTable.NewRow();
for (int i = 0; i < dataCells.Count; i++)
{
dataRow[i] = dataCells[i].InnerText;
}
resultTable.Rows.Add(dataRow);
}
}
}
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
}
}
The URL i trying to web scrape : https://www.wsj.com/market-data/quotes/MY/XKLS/0146/financials/annual/cash-flow
I did try to do looping to skip if it was having the same name but it will prompt that the column unable to find when I try to debug.
Is there any solution that can help to solve this? In the end I will need to export the datatable to csv/excel file.
Thanks

I think you want to do this instead:
foreach (HtmlNode table in doc.DocumentNode.SelectNodes($"//table[#class='{classValue}']") ?? Enumerable.Empty<HtmlNode>())
{
var resultTable = new DataTable(table.Id);
// select all the headers and add them to the table
var headerCells = table.SelectNodes("thead/tr/th");
if (headerCells != null)
{
foreach (HtmlNode cell in headerCells)
{
resultTable.Columns.Add(cell.InnerText);
}
}
// select all the rows and add them to the table
foreach (HtmlNode row in table.SelectNodes("tbody/tr"))
{
var dataCells = row.SelectNodes("td");
if (dataCells != null)
{
var dataRow = resultTable.NewRow();
for (int i = 0; i < dataCells.Count; i++)
{
dataRow[i] = dataCells[i].InnerText;
}
resultTable.Rows.Add(dataRow);
}
}
}
The header section and the data section each have their own loop rather than the header section being nested in the data loop. We're also being more explicit about where we want data from: the header should come from thead/tr/th and the data should come from tbody/tr.

How to convert string separated by new line and comma to DataTable in C#

I have a string like this:
"Product,Price,Condition
Cd,13,New
Book,9,Used
"
Which is being passed like this:
"Product,Price,Condition\r\Cd,13,New\r\nBook,9,Used"
How could I convert it to DataTable?
Trying to do it with this helper function:
DataTable dataTable = new DataTable();
bool columnsAdded = false;
foreach (string row in data.Split(new string[] { "\r\n" }, StringSplitOptions.None))
{
DataRow dataRow = dataTable.NewRow();
foreach (string cell in row.Split(','))
{
string[] keyValue = cell.Split('~');
if (!columnsAdded)
{
DataColumn dataColumn = new DataColumn(keyValue[0]);
dataTable.Columns.Add(dataColumn);
}
dataRow[keyValue[0]] = keyValue[1];
}
columnsAdded = true;
dataTable.Rows.Add(dataRow);
}
return dataTable;
However I don't get that "connecting cells with appropriate columns" part - my cells don't have ~ in string[] keyValue = cell.Split('~'); and I obviously get an IndexOutOfRange at DataColumn dataColumn = new DataColumn(keyValue[0]);

Based on your implementation, I have written the code for you, I have not tested it. But you can use the concept.
DataRow dataRow = dataTable.NewRow();
int i = 0;
foreach (string cell in row.Split(','))
{
if (!columnsAdded)
{
DataColumn dataColumn = new DataColumn(cell);
dataTable.Columns.Add(dataColumn);
}
else
{
dataRow[i] = cell;
}
i++;
}
if(columnsAdded)
{
dataTable.Rows.Add(dataRow);
}
columnsAdded = true;

You can do that simply with Linq (and actually there is LinqToCSV on Nuget, maybe you would prefer that):
void Main()
{
string data = #"Product,Price,Condition
Cd,13,New
Book,9,Used
";
var table = ToTable(data);
Form f = new Form();
var dgv = new DataGridView { Dock = DockStyle.Fill, DataSource = table };
f.Controls.Add(dgv);
f.Show();
}
private DataTable ToTable(string CSV)
{
DataTable dataTable = new DataTable();
var lines = CSV.Split(new char[] { '\n' }, StringSplitOptions.RemoveEmptyEntries);
foreach (var colname in lines[0].Split(','))
{
dataTable.Columns.Add(new DataColumn(colname));
}
foreach (var row in lines.Where((r, i) => i > 0))
{
dataTable.Rows.Add(row.Split(','));
}
return dataTable;
}

You can split given string into flattened string array in one call. Then you can iterate through the array and populate list of objects.
That part is optional, since you can immediately populate DataTable but I think it's way easier (more maintainable) to work with strongly-typed objects when dealing with DataTable.
string input = "Product,Price,Condition\r\nCd,13,New\r\nBook,9,Used";
string[] deconstructedInput = input.Split(new string[] { "\r\n", "," }, StringSplitOptions.None);
List<Product> products = new List<Product>();
for (int i = 3; i < deconstructedInput.Length; i += 3)
{
products.Add(new Product
{
Name = deconstructedInput[i],
Price = Decimal.Parse(deconstructedInput[i + 1]),
Condition = deconstructedInput[i + 2]
});
}
public class Product
{
public string Name { get; set; }
public decimal Price { get; set; }
public string Condition { get; set; }
}
So, products collection holds 2 objects which you can easily iterate over and populate your DataTable.
Note: This requires further checks to avoid possible runtime exceptions, also it is not dynamic. That means, if you have differently structured input it won't work.
DataTable dataTable = new DataTable();
dataTable.Columns.Add(new DataColumn(nameof(Product.Name)));
dataTable.Columns.Add(new DataColumn(nameof(Product.Price)));
dataTable.Columns.Add(new DataColumn(nameof(Product.Condition)));
foreach (var product in products)
{
var row = dataTable.NewRow();
row[nameof(Product.Name)] = product.Name;
row[nameof(Product.Price)] = product.Price;
row[nameof(Product.Condition)] = product.Condition;
dataTable.Rows.Add(row);
}

Cannot implicitly convert DateTime to Timespan

Sql Server has the variable TEST_TIME data type as Time(7)
I created the tables in C# and it automatically assigned the Timespan datatype.
Now, i'm trying to upload csv file data to the SQL database and it is giving me an error " Cannot implicitly convert DateTime to Timespan". What would be the best way to fix this?
The user first selects the CSV file:
private void button8_Click(object sender, EventArgs e)
{
try
{
using (OpenFileDialog openfiledialog1 = new OpenFileDialog()
{Filter = "Excel Workbook 97-2003|*.xls|Excel Workbook|*.xlsx|Excel Workbook|*.xlsm|Excel Workbook|*.csv|Excel Workbook|*.txt", ValidateNames = true })
{
--After some IFs--
else if (openfiledialog1.FilterIndex == 4)
{
DataTable oDataTable = null;
int RowCount = 0;
string[] ColumnNames = null;
string[] oStreamDataValues = null;
//using while loop read the stream data till end
while (!oStreamReader.EndOfStream)
{
String oStreamRowData = oStreamReader.ReadLine().Trim();
if (oStreamRowData.Length > 0)
{
oStreamDataValues = oStreamRowData.Split(',');
//Bcoz the first row contains column names, we will populate
//the column name by
//reading the first row and RowCount-0 will be true only once
if (RowCount == 0)
{
RowCount = 1;
ColumnNames = oStreamRowData.Split(',');
oDataTable = new DataTable();
//using foreach looping through all the column names
foreach (string csvcolumn in ColumnNames)
{
DataColumn oDataColumn = new DataColumn(csvcolumn.ToUpper(), typeof(string));
//setting the default value of empty.string to newly created column
oDataColumn.DefaultValue = string.Empty;
//adding the newly created column to the table
oDataTable.Columns.Add(oDataColumn);
}
}
else
{
//creates a new DataRow with the same schema as of the oDataTable
DataRow oDataRow = oDataTable.NewRow();
//using foreach looping through all the column names
for (int i = 0; i < ColumnNames.Length; i++)
{
oDataRow[ColumnNames[i]] = oStreamDataValues[i] == null ? string.Empty : oStreamDataValues[i].ToString();
}
//adding the newly created row with data to the oDataTable
oDataTable.Rows.Add(oDataRow);
}
}
}
result.Tables.Add(oDataTable);
//close the oStreamReader object
oStreamReader.Close();
//release all the resources used by the oStreamReader object
oStreamReader.Dispose();
dataGridView5.DataSource = result.Tables[oDataTable.TableName];
}
Here is the Code:
private void button9_Click(object sender, EventArgs e)
{
try
{
DataClasses1DataContext conn = new DataClasses1DataContext();
else if (textBox3.Text.Contains("GEN_EX"))
{
foreach (DataTable dt in result.Tables)
{
foreach (DataRow dr in dt.Rows)
{
GEN_EX addtable = new GEN_EX()
{
EX_ID = Convert.ToByte(dr[0]),
DOC_ID = Convert.ToByte(dr[1]),
PATIENT_NO = Convert.ToByte(dr[2]),
TEST_DATE = Convert.ToDateTime(dr[3]),
**TEST_TIME = Convert.ToDateTime((dr[4])),**
};
conn.GEN_EXs.InsertOnSubmit(addtable);
}
}
conn.SubmitChanges();
MessageBox.Show("File uploaded successfully");
}
else
{
MessageBox.Show("I guess table is not coded yet");
}
}
EDIT
The TEST_TIME represents HH:MM:SS
The Typed Data Set is defined as:
public virtual int Update(
byte EX_ID,
byte DOC_ID,
byte PATIENT_NO,
System.DateTime TEST_DATE,
System.TimeSpan TEST_TIME)

Based on your input that dr[4] represents time values in hours:minutes:seconds format I recommend following solution.
private TimeSpan GetTimeSpan(string timeString)
{
var timeValues = timeString.Split(new char[] { ':' });
//Assuming that timeValues array will have 3 elements.
var timeSpan = new TimeSpan(Convert.ToInt32(timeValues[0]), Convert.ToInt32(timeValues[1]), Convert.ToInt32(timeValues[2]));
return timeSpan;
}
Use above method as following.
else if (textBox3.Text.Contains("GEN_EX"))
{
foreach (DataTable dt in result.Tables)
{
foreach (DataRow dr in dt.Rows)
{
GEN_EX addtable = new GEN_EX()
{
EX_ID = Convert.ToByte(dr[0]),
DOC_ID = Convert.ToByte(dr[1]),
PATIENT_NO = Convert.ToByte(dr[2]),
TEST_DATE = Convert.ToDateTime(dr[3]),
**TEST_TIME = GetTimeSpan(dr[4].ToString()),**
};
conn.GEN_EXs.InsertOnSubmit(addtable);
}
}
conn.SubmitChanges();
MessageBox.Show("File uploaded successfully");
}
This should give your the value you want. You will face runtime issues if value of dr[4] is not in hours:minutes:seconds format. That I will leave it up to you.

First of all Timespan and DateTime are 2 differents type without implicit conversion available. Since Timespan is a time value between two DateTime, you need to know your referenced time (DateTime) used to start the mesure of your Timespan.
For example it could be from DateTime dtReferential = new DateTime(1900, 01, 01);
In order to give a SQL Timespan value you need to give it a C# Timespan! Change TEST_TIME value to a Timespan. And finally, give it the substracted value of your referenced time.
Using the previous example:
else if (textBox3.Text.Contains("GEN_EX"))
{
foreach (DataTable dt in result.Tables)
{
foreach (DataRow dr in dt.Rows)
{
GEN_EX addtable = new GEN_EX()
{
EX_ID = Convert.ToByte(dr[0]),
DOC_ID = Convert.ToByte(dr[1]),
PATIENT_NO = Convert.ToByte(dr[2]),
TEST_DATE = Convert.ToTimespan(dr[3]),
TEST_TIME = dtReferential.Subtract(Convert.ToDateTime(dr[4]))
};
conn.GEN_EXs.InsertOnSubmit(addtable);
}
}
conn.SubmitChanges();
MessageBox.Show("File uploaded successfully");
}

How to get keys and values of .json file which is generated by parser dynamically in c#

We have following scenario,
We need to parse resumes of candidates and i am using below parser to parse resume in Json format, right.
https://github.com/antonydeepak/ResumeParser
and Json files which i get are in valid format as i checked them in online jsonviewer, but it is cleared that each resume is in different format. so each time parser introduced new pair of Keys ,
Example 1.
Example 2.
Above two formats are of two different resumes and so on...
so now i need to iterate through every key and value that are dynamically generated.
as for as i did to get JObject and JArray at level 0, now i need to iterate through each JObject and JArray to get its values.
I used Json.net to get them
string text = System.IO.File.ReadAllText(#"C:\abc.json");
var d = JsonConvert.DeserializeObject<Dictionary<string, dynamic>>(text);
and it showed me data as
abc.json has JObjects and JArray, so now i need to iterate through each and every line and need to get every key and value from parsed json file and load it to datatable and tried it using google , but it missed some of keys and values.
System.Data.DataTable dt = new System.Data.DataTable();
dt.Columns.Add("Key", typeof(string));
dt.Columns.Add("Value", typeof(string));
string text = System.IO.File.ReadAllText(#"C:\antony_thomas.json");
var d = JsonConvert.DeserializeObject<Dictionary<string, dynamic>>(text);
foreach (var item in d)
{
var key = "";
var val = item.Value;
if (val is JObject)
{
dynamic dynObj = JsonConvert.DeserializeObject(Convert.ToString(val));
foreach (var ite in dynObj)
{
DataRow row = dt.NewRow();
string jsonvalue = Convert.ToString(ite).Replace("[", "").Replace("]", "").Replace("{", "").Replace("}", "").Replace("\"", "");
string jkey = jsonvalue.Split(':')[0];
string jval = jsonvalue.Split(':')[1];
row["Key"] = jkey;
row["Value"] = jval;
dt.Rows.Add(row);
}
// key = item.Key;
}
if (val is JArray)
{
//key = item.Key;
foreach (var it in val)
{
// var newkey=
DataRow row = dt.NewRow();
string jsonvalue = Convert.ToString(it).Replace("[", "").Replace("]", "").Replace("{", "").Replace("}", "").Replace("\"", "");
//Convert.ToString(test);
string jkey = jsonvalue.Split(':')[0];
string jval = jsonvalue.Split(':')[1];
row["Key"] = jkey;
row["Value"] = jval;
dt.Rows.Add(row);
}
}
}
I am using asp.net , C#, Json.net , if anyone have any idea so please guide me how can i get my desired result..

I used this code recently to traverse an arbitrary JSON string.
Each element is dumped with a row of dots at the beginning indicating its level in the hierarchy.
You could modify it to output to a DataTable as you walk through an JArray for example.
public static void JsonFileDump(string path)
{
//Parse the data
string jsonStr = File.ReadAllText(path);
JToken token = JToken.Parse(jsonStr); // get parent token
JsonTokenDump(token);
}
public static void JsonTokenDump(JToken node, int lvl = 0, string nodeName = null)
{
if (nodeName != null)
Console.WriteLine("{0}Node Name={1}, Type={2}", new string('.', lvl), nodeName, node.Type);
else
Console.WriteLine("{0}Node Type={1}", new string('.', lvl), node.Type);
if (node.Type == JTokenType.Object)
{
foreach (JProperty child in node.Children<JProperty>())
{
JsonTokenDump(child.Value, lvl + 1, child.Name);
}
}
else if (node.Type == JTokenType.Array)
{
foreach (JToken child in node.Children())
{
JsonTokenDump(child, lvl + 1);
}
}
}

Reading XML Data and Storing in DataTable

I have an log file like this..
This is the segment 1
============================
<MAINELEMENT><ELEMENT1>10-10-2013 10:10:22.444</ELEMENT1><ELEMENT2>1111</ELEMENT2>
<ELEMENT3>Message 1</ELEMENT3></MAINELEMENT>
<MAINELEMENT><ELEMENT1>10-10-2013 10:10:22.555</ELEMENT1><ELEMENT2>1111</ELEMENT2>
<ELEMENT3>Message 2</ELEMENT3></MAINELEMENT>
This is the segment 2
============================
<MAINELEMENT><ELEMENT1>10-11-2012 10:10:22.444</ELEMENT1><ELEMENT2>2222</ELEMENT2>
<ELEMENT3>Message 1</ELEMENT3></MAINELEMENT>
<MAINELEMENT><ELEMENT1>10-11-2012 10:10:22.555</ELEMENT1><ELEMENT2>2222</ELEMENT2>
<ELEMENT3>Message 2</ELEMENT3></MAINELEMENT>
How can I read this into DataTable excluding the data This is the segment 1 and This is the segment 2 and ====== lines completely.
I would like to have the Datatable as with Columns as "ELEMENT1", "ELEMENT2", "ELEMENT3" and fill the details with the content between those tags in the order of print of line.
It should not change the sequence of the order of records in the table while inserting.

HtmlAgilityPack seems to be a good tool for what you need:
using HtmlAgilityPack;
class Program
{
static void Main(string[] args)
{
var doc = new HtmlDocument();
doc.Load("log.txt");
var dt = new DataTable();
bool hasColumns = false;
foreach (HtmlNode row in doc
.DocumentNode
.SelectNodes("//mainelement"))
{
if (!hasColumns)
{
hasColumns = true;
foreach (var column in row.ChildNodes
.Where(node => node.GetType() == typeof(HtmlNode)))
{
dt.Columns.Add(column.Name);
}
}
dt.Rows.Add(row.ChildNodes
.Where(node => node.GetType() == typeof(HtmlNode))
.Select(node => node.InnerText).ToArray());
}
}
}

could do this, where stringData is the data from the file you have
var array = stringData.Split(new[] { "============================" }, StringSplitOptions.RemoveEmptyEntries);
var document = new XDocument(new XElement("Root"));
foreach (var item in array)
{
if(!item.Contains("<"))
continue;
var subDocument = XDocument.Parse("<Root>" + item.Substring(0, item.LastIndexOf('>') + 1) + "</Root>");
foreach (var element in subDocument.Root.Descendants("MAINELEMENT"))
{
document.Root.Add(element);
}
}
var table = new DataTable();
table.Columns.Add("ELEMENT1");
table.Columns.Add("ELEMENT2");
table.Columns.Add("ELEMENT3");
var rows =
document.Descendants("MAINELEMENT").Select(el =>
{
var row = table.NewRow();
row["ELEMENT1"] = el.Element("ELEMENT1").Value;
row["ELEMENT2"] = el.Element("ELEMENT2").Value;
row["ELEMENT3"] = el.Element("ELEMENT3").Value;
return row;
});
foreach (var row in rows)
{
table.Rows.Add(row);
}
foreach (DataRow dataRow in table.Rows)
{
Console.WriteLine("{0},{1},{2}", dataRow["ELEMENT1"], dataRow["ELEMENT2"], dataRow["ELEMENT3"]);
}

I'm not so sure where you problem is.
You can use XElement for reading the xml and manually creating DataTable.
For Reading the XML See Xml Parsing using XElement
Then you can create dynamically the datatable.
Heres an example of creating a datatable in code
https://sites.google.com/site/bhargavaclub/datatablec
But why do you want to use a DataTable ? There are a lot of downsides...