We Keep Coding

How to skipped if the column name are same in the datatable

I'm using HTML Agility Pack to web scrape to datatable. However the website have multiple same column name which it was not able to add on for the second table.
The error will be prompt out like this as the "2020" had been added before
My code as below :
public void WebDataScrap()
{
try
{
//Get the content of the URL from the Web
const string url = "https://www.wsj.com/market-data/quotes/MY/XKLS/0146/financials/annual/cash-flow";
var web = new HtmlWeb();
var doc = web.Load(url);
const string classValue = "cr_dataTable"; //cr_datatable
//var nodes = doc.DocumentNode.SelectNodes($"//table[#class='{classValue}']") ?? Enumerable.Empty<HtmlNode>();
var resultDataset = new DataSet();
foreach (HtmlNode table in doc.DocumentNode.SelectNodes($"//table[#class='{classValue}']") ?? Enumerable.Empty<HtmlNode>())
{
var resultTable = new DataTable(table.Id);
foreach (HtmlNode row in table.SelectNodes("//tr"))
{
var headerCells = row.SelectNodes("th");
if (headerCells != null)
{
foreach (HtmlNode cell in headerCells)
{
resultTable.Columns.Add(cell.InnerText);
}
}
var dataCells = row.SelectNodes("td");
if (dataCells != null)
{
var dataRow = resultTable.NewRow();
for (int i = 0; i < dataCells.Count; i++)
{
dataRow[i] = dataCells[i].InnerText;
}
resultTable.Rows.Add(dataRow);
}
}
}
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
}
}
The URL i trying to web scrape : https://www.wsj.com/market-data/quotes/MY/XKLS/0146/financials/annual/cash-flow
I did try to do looping to skip if it was having the same name but it will prompt that the column unable to find when I try to debug.
Is there any solution that can help to solve this? In the end I will need to export the datatable to csv/excel file.
Thanks

I think you want to do this instead:
foreach (HtmlNode table in doc.DocumentNode.SelectNodes($"//table[#class='{classValue}']") ?? Enumerable.Empty<HtmlNode>())
{
var resultTable = new DataTable(table.Id);
// select all the headers and add them to the table
var headerCells = table.SelectNodes("thead/tr/th");
if (headerCells != null)
{
foreach (HtmlNode cell in headerCells)
{
resultTable.Columns.Add(cell.InnerText);
}
}
// select all the rows and add them to the table
foreach (HtmlNode row in table.SelectNodes("tbody/tr"))
{
var dataCells = row.SelectNodes("td");
if (dataCells != null)
{
var dataRow = resultTable.NewRow();
for (int i = 0; i < dataCells.Count; i++)
{
dataRow[i] = dataCells[i].InnerText;
}
resultTable.Rows.Add(dataRow);
}
}
}
The header section and the data section each have their own loop rather than the header section being nested in the data loop. We're also being more explicit about where we want data from: the header should come from thead/tr/th and the data should come from tbody/tr.

Get changes after merging datasets

Hello I have two Datasets with the same schemas and i need to get changes between two of them.
Datasets can be created using code below:
DataSet First = new DataSet("DSStore");
DataTable Footer = new DataTable("Footer");
DataColumn Column = new DataColumn("Value", Type.GetType("System.Int32"), "");
DataColumn[] PK = new DataColumn[1];
PK[0] = Column;
DataSet changes;
First.Tables.Add(Footer);
Footer.Columns.Add(Column);
Footer.PrimaryKey = PK;
//Clone to create second one
changes = First.Clone();
now just fill both with data:
for (int i = 0; i < 10; i++)
{
var row2 = changes.Tables["Footer"].NewRow();
row2["Value"] = i;
changes.Tables["Footer"].Rows.Add(row2);
}
var firstRow = First.Tables["Footer"].NewRow();
firstRow["Value"] = 8;
First.Tables["Footer"].Rows.Add(firstRow);
First.AcceptChanges();
changes.AcceptChanges();
Now when we have all data prepared we can get to what I tried:
I tried merging both of them:
First.Merge(changes);
if (First.HasChanges())
Console.WriteLine("has changes");
else
Console.WriteLine("Doesnt");
but unfortunately merge do not change row status so after rows being accepted hasChanges returns false and getchanges is null.
I tried another way:
IEnumerable<DataRow> added = changes.Tables["Footer"].AsEnumerable().Except(First.Tables["Footer"].AsEnumerable(),DataRowComparer.Default);
Console.WriteLine("Added:");
foreach (var row in added)
{
Console.WriteLine(row["Value"]);
}
Now i received some results but it is printing all 9 lines correct. So I tried to insert changes to the first dataset:
foreach(var row in added)
{
changes2.Tables["Footer"].Rows.Add(row);
}
if (changes2.HasChanges())
Console.WriteLine("has changes");
else
Console.WriteLine("Doesnt");
But after trying to add rows I am receving ArgumentException

I needed to change one line:
foreach(var row in added)
{
changes2.Tables["Footer"].Rows.Add(row.ItemArray);
}
now its ok. Adding lines creates changes

Convert IEnumerable string array to datatable

I have a csv file delimited with pipe(|). I am reading it using the following line of code:
IEnumerable<string[]> lineFields = File.ReadAllLines(FilePath).Select(line => line.Split('|'));
Now, I need to bind this to a GridView. So I am creating a dynamic DataTable as follows:
DataTable dt = new DataTable();
int i = 0;
foreach (string[] order in lineFields)
{
if (i == 0)
{
foreach (string column in order)
{
DataColumn _Column = new DataColumn();
_Column.ColumnName = column;
dt.Columns.Add(_Column);
i++;
//Response.Write(column);
//Response.Write("\t");
}
}
else
{
int j = 0;
DataRow row = dt.NewRow();
foreach (string value in order)
{
row[j] = value;
j++;
//Response.Write(column);
//Response.Write("\t");
}
dt.Rows.Add(row);
}
//Response.Write("\n");
}
This works fine. But I want to know if there is a better way to convert IEnumerable<string[]> to a DataTable. I need to read many CSVs like this, so I think the above code might have performance issues.

Starting from .Net 4:
use ReadLines.
DataTable FileToDataTable(string FilePath)
{
var dt = new DataTable();
IEnumerable<string[]> lineFields = File.ReadLines(FilePath).Select(line => line.Split('|'));
dt.Columns.AddRange(lineFields.First().Select(i => new DataColumn(i)).ToArray());
foreach (var order in lineFields.Skip(1))
dt.Rows.Add(order);
return dt;
}
(edit: instead this code, use the code of #Jodrell answer, This prevents double charging of the Enumerator).
Before .Net 4:
use streaming:
DataTable FileToDataTable1(string FilePath)
{
var dt = new DataTable();
using (var st = new StreamReader(FilePath))
{
// first line procces
if (st.Peek() >= 0)
{
var order = st.ReadLine().Split('|');
dt.Columns.AddRange(order.Select(i => new DataColumn(i)).ToArray());
}
while (st.Peek() >= 0)
dt.Rows.Add(st.ReadLine().Split('|'));
}
return dt;
}

since, in your linked example, the file has a header row.
const char Delimiter = '|';
var dt = new DataTable;
using (var m = File.ReadLines(filePath).GetEnumerator())
{
m.MoveNext();
foreach (var name in m.Current.Split(Delimiter))
{
dt.Columns.Add(name);
}
while (m.MoveNext())
{
dt.Rows.Add(m.Current.Split(Delimiter));
}
}
This reads the file in one pass.

How to get SQL Server version only by the server name?

Say, if what I have is only a server name obtained from this enumeration:
//Collect server names
List<string> arrServerNames = new List<string>();
try
{
// Perform the enumeration
DataTable dataTable = null;
try
{
dataTable = System.Data.Sql.SqlDataSourceEnumerator.Instance.GetDataSources();
}
catch
{
dataTable = new DataTable();
dataTable.Locale = System.Globalization.CultureInfo.InvariantCulture;
}
// Create the object array of server names (with instances appended)
for (int i = 0; i < dataTable.Rows.Count; i++)
{
string name = dataTable.Rows[i]["ServerName"].ToString();
string instance = dataTable.Rows[i]["InstanceName"].ToString();
if (instance.Length == 0)
{
arrServerNames.Add(name);
}
else
{
arrServerNames.Add(name + "\\" + instance);
}
}
}
catch
{
//Error
}
How can I know the SQL Server version installed on that server?

Checking the official MSDN documentation for GetDataSources() would have easily revealed that there is a Version column in the result set:
// Create the object array of server names (with instances appended)
for (int i = 0; i < dataTable.Rows.Count; i++)
{
string name = dataTable.Rows[i]["ServerName"].ToString();
string instance = dataTable.Rows[i]["InstanceName"].ToString();
string version = dataTable.Rows[i]["Version"].ToString(); // this gets the version!
..........
}

How to combine lists into single data entity in c#

I have a web method in a c# web service which creates three lists, which are filled from xml input. I want to combine these three lists into one entity (a DataSet would be the best, as the iOS app that is consuming this web service is already programmed to accept and parse DataSets), and return them from the web method.
Here is currently what my code looks like:
[WebMethod]
public DataSet SelectObjects(string ExternalID, string Password)
{
DataSet ds = new DataSet();
MembershipAuthServiceReference.MembershipAuthenticationService objService = new MembershipAuthServiceReference.MembershipAuthenticationService();
MembershipAuthServiceReference.SoapHeaderCredentials objSoapHeader = new MembershipAuthServiceReference.SoapHeaderCredentials();
MembershipAuthServiceReference.MemberUserInfo objMemberInfo = new MembershipAuthServiceReference.MemberUserInfo();
try
{
objSoapHeader.UserName = ExternalID;
objSoapHeader.Password = Password;
objMemberInfo = objService.GetMembershipInfo();
List<Obj1> ListObj1 = new List<Obj1>();
for (int i = 0; i < objMemberInfo.Obj1.Length; i++)
{
Obj1 obj_Obj1 = new Obj1();
obj_Obj1.Stuff = objMemberInfo.Obj1[i].Stuff.ToString();
ListObj1.Add(obj_Obj1);
}
List<Obj2> ListObj2 = new List<Obj2>();
for (int i = 0; i < objMemberInfo.Obj2.Length; i++)
{
Obj2 obj_Obj2 = new Obj2();
obj_Obj2.Stuff = objMemberInfo.Obj2[i].Stuff.ToString();
ListObj2.Add(obj_Obj2);
}
List<Obj3> ListObj3 = new List<Obj3>();
for (int i = 0; i < objMemberInfo.Obj3.Length; i++)
{
Obj3 obj_Obj3 = new Obj3();
obj_Obj3.Stuff = objMemberInfo.Obj3[i].Stuff.ToString();
ListObj3.Add(obj_Obj3);
}
}
catch (Exception ex)
{
string sError;
sError = ex.Message.ToString();
}
return ds;
}
How do I combine these lists into a DataSet? I'm assuming it's possible? If not, is there a viable alternative that does the same thing?

First concatenate your lists as shown below and then use the link to generate the dataset
var combinedList= ListObj1.Concat(ListObj2).Concat(ListObj3);
How do I transform a List<T> into a DataSet?