I have two datatables in my ASP.NET application that are filled from csv files and I am trying to combine the two into one.
Heres what the interface looks like:
When I click the 'Merge Data' button it should merge the test1.csv and test2.csv which kind of works but looks like this:
So my question is how do I align these two datatables so that all the data is on the same row?
Below is the code for the Merge Data Button:
List<string> filepaths = new List<string>();
List<DataTable> allTables = new List<DataTable>();
DataTable mergedTables = new DataTable();
int rowCount = grdFiles.Rows.Count;
for (int i = 0; i < rowCount; i++)
{
string filename = grdFiles.Rows[i].Cells[0].Text;
filepaths.Add(Server.MapPath("~/Uploads/") + filename);
}
foreach(string path in filepaths)
{
DataTable dt = new DataTable();
//converts csv into datatable
dt = GetDataTableFromCsv(path, true);
//add table to list of tables
allTables.Add(dt);
}
foreach(DataTable datatable in allTables)
{
//Merge each table in the list to the mergedTables datatable
mergedTables.Merge(datatable);
}
csvUploadResults.DataSource = mergedTables;
csvUploadResults.DataBind();
Thanks in advance for any help :)
If your objective is just to merge data without considering the relationship between the two data then you can add two more columns into first datatable and through loop get data from second table and assign them to first datatable columns. The way the data is received will be the way data will be saved in first datatable.
public DataTable MergeData(DataTable dtFirst,DataTable dtSecond)
{
dtFirst.Columns.Add("LocalAuthority");
dtFirst.Columns.Add("AverageSpeed");
for (int i = 0; i < dtFirst.Rows.Count; i++)
{
dtFirst.Rows[i]["LocalAuthority"] = dtSecond.Rows[i]["LocalAuthority"];
dtFirst.Rows[i]["AverageSpeed"] = dtSecond.Rows[i]["AverageSpeed"];
}
return dtFirst;
}
Now , you need to pass datatable as parameter in following method.
MergeData(allTables.ElementAt(0), allTables.ElementAt(1));
You're going to need a unique key on both datatables and merge them together. You could add the SchoolName to your second datatable and merge the two tables on the postcode. Or more preferably, add an id to both of the datatables and merge the two datatables on the id.
Related
I have a dataset that looks like the image. I'm trying to filter by table and get all the columns next to it and compare them against other datasets
This dataset has tables named table 1 and table 2 and when they're selected they look like the picture below. It shows the columns and I need to compare those columns against the rows from the matching table in the first dataset
I've looked at dataview but that would be a lot of work and I'm very inexperienced. I'm trying to find a way to implement a foreach loop that'll get the name of the table in the first dataset and then compare the rows in it against the columns inside the datatable in the second dataset that matched the table name from the first dataset.
Without knowing more about these DataSets (like do they have primary keys, the data types of the columns, the number of rows in each table, etc), I can only provide limited help. The following example tries to be as general as possible and avoid some basic problems:
DataSet ds1 = <<fetch dataset1>>;
DataSet ds2 = <<fetch dataset2>>;
foreach (DataTable tbl1 in ds1.Tables)
{
if (ds2.Tables.Contains(tbl1.TableName))
{
DataTable tbl2 = ds2.Tables[tbl1.TableName];
List<string> commonColumnNames = new List<string>(tbl1.Columns.Cast<DataColumn>().Select(c => c.ColumnName).Intersect(tbl2.Columns.Cast<DataColumn>().Select(c => c.ColumnName)));
int maxRows = Math.Min(tbl1.Rows.Count, tbl2.Rows.Count);
for (int r = 0; r <= maxRows; r++)
{
foreach (string colName in commonColumnNames)
{
if (tbl1.Rows[r][colName] != tbl2.Rows[r][colName])
{
// Different value
}
}
}
}
}
Update 1: I've added comments to the following example to explain step-by-step what this code is doing. As I try to say before, since I didn't know much about your data, I had to put in extra code. This extra code is for things like: 'Does the table ABC exist in both DataSets?', 'Do the two tables have the same columns in them?', 'Do the tables have the same number of rows in them?'. Your original question did not have this information, so I made this code a little more robust to handle those unknowns.
DataSet ds1 = <<fetch dataset1>>;
DataSet ds2 = <<fetch dataset2>>;
// Loop through all of the tables in the 1st DataSet
foreach (DataTable tbl1 in ds1.Tables)
{
// If the 2nd DataSet has a table with same name as the one from the 1st DataSet
if (ds2.Tables.Contains(tbl1.TableName))
{
DataTable tbl2 = ds2.Tables[tbl1.TableName];
// Create a list of column names that the two tables have in common.
// We will only compare the values in these two tables, in this set of matching column names.
List<string> commonColumnNames = new List<string>(tbl1.Columns.Cast<DataColumn>().Select(c => c.ColumnName).Intersect(tbl2.Columns.Cast<DataColumn>().Select(c => c.ColumnName)));
// Before we start comparing the rows in the two tables, find out which one has the fewer number of rows in it.
int maxRows = Math.Min(tbl1.Rows.Count, tbl2.Rows.Count);
// If the tables have a different number of rows, then we will only compare the set of rows numbered 0-to-MinRowCount
for (int r = 0; r <= maxRows; r++)
{
// For each row, compare the values of common columns
foreach (string colName in commonColumnNames)
{
if (tbl1.Rows[r][colName] != tbl2.Rows[r][colName])
{
// Different value
}
}
}
}
}
I have a requirement where I need to copy the existing DataTable rows/values to the new DataTable,
New DataTable is of custom(different column names) structure/schema as we need this new DataTable data to be exported to Excel file later.
To copy the required column values to the new DataTable I have created string Array which contains required column names(columns from which we need to copy data to new DataTable) of existing DataTable, given below.
string[] selectedColumns = new[] { "SUPPLIER_NAME", "SUPPLIER_NO", "CONFIRMATION_NO", "RELEASE_NO", "WCO_INVOICE_NO",
"CUSTOMER_BILLED", "BALANCE_TOBILL", "SUPP_INVOICE_NO", "SUPPLIER_PAID", "BALANCE_COST" }; //TODO Add columns "WCO_INVOICE_DATE", "SUPPLIER_INVOICE_DATE" later.
For getting the required DataTable, I have created a method which will be called like below by passing existing DataTable and selected columns of existing DataTable column names as an parameters.
DataTable _dtPrjLedgerExportData = ControllerClass.dtProjectLedgerExport(dtfilter, selectedColumns);
ControllerClass Class dtProjectLedgerExport method will return the newly created DataTable i.e. DataTable with the customize Column names and the values pulled from existing DatTable,
New DataTable additionally contain two new columns (WCO_Invoice_Date, Supplier_Invoice_Date) with default blank values (set from the code).
public static DataTable dtProjectLedgerExport(DataTable dtToExport, string[] selectedColumns)
{
DataTable dt = new DataTable();
dt.Columns.Add("Supplier");
dt.Columns.Add("Supplier_No");
dt.Columns.Add("Confirmation_Number");
dt.Columns.Add("Release_Number");
dt.Columns.Add("WCO_Invoice_No");
dt.Columns.Add("WCO_Invoice_Date");
dt.Columns.Add("Customer_Billed_Amt");
dt.Columns.Add("Balance_Remaining_to_Bill");
dt.Columns.Add("Supplier_Invoice_Number");
dt.Columns.Add("Supplier_Invoice_Date");
dt.Columns.Add("Supplier_Paid_Amt");
dt.Columns.Add("Remaining_Cost_Dollar_Balance");
//temporarily set default value for non-existing rows
dt.Columns["WCO_Invoice_Date"].DefaultValue = string.Empty;
dt.Columns["Supplier_Invoice_Date"].DefaultValue = string.Empty;
//Copy rows to dt
if (dtToExport != null && dtToExport.Rows.Count > 0)
dt = new DataView(dtToExport).ToTable(false, selectedColumns);
return dt;
}
Problem:
The above code is not working as expected it is returning Data of existing DataTable with the same structure.
What changes are required in above code to get the expected result?
I'd use this approach:
public static DataTable DtProjectLedgerExport(DataTable dtToExport, string[] selectedColumns, params string[] additionalColumns)
{
DataTable dt = dtToExport.Copy(); // add columns and data
List<DataColumn> removeColumns = dt.Columns.Cast<DataColumn>()
.Where(c => !selectedColumns.Contains(c.ColumnName, StringComparer.InvariantCultureIgnoreCase))
.ToList();
removeColumns.ForEach(dt.Columns.Remove);
foreach (string colName in additionalColumns)
{
DataColumn newColumn = new DataColumn(colName);
newColumn.DefaultValue = string.Empty;
dt.Columns.Add(newColumn);
}
return dt;
}
After stating in the comments that you have different column names in each table...(Seriously that's a critical piece of information lol)! The only way is to basiaclly map the values yourself for each column like so.
public static DataTable dtProjectLedgerExport(DataTable dtToExport)
{
DataTable dt = new DataTable();
dt.Columns.Add("Supplier");
dt.Columns.Add("Supplier_No");
dt.Columns.Add("Confirmation_Number");
dt.Columns.Add("Release_Number");
dt.Columns.Add("WCO_Invoice_No");
dt.Columns.Add("WCO_Invoice_Date");
dt.Columns.Add("Customer_Billed_Amt");
dt.Columns.Add("Balance_Remaining_to_Bill");
dt.Columns.Add("Supplier_Invoice_Number");
dt.Columns.Add("Supplier_Invoice_Date");
dt.Columns.Add("Supplier_Paid_Amt");
dt.Columns.Add("Remaining_Cost_Dollar_Balance");
foreach (var expRow in dtToExport.Rows)
{
var row = dt.NewRow();
row["Supplier"] = expRow["Supplier_Name"];
//repeat for all columns you want.
dt.Rows.Add(row);
}
return dt;
}
Well, of course, because you finally override dt completely. You should enumerate the rows in dtToExport in a loop and create new rows in dt and assign the values for every field you need.
The problem: inserting rows only if primary key is bigger than existing one when merging source DataTable to actual DataTable (ActualDT.Merge(SourceDT)).
Details of my problem below:
I fill an Actual DataTable with an Int64 primary key by the API from external server after deserializing JSON to Source DataTable. Then I write rows from DataTable to my database and cleanup all rows in DataTable except the biggest primary key. Later I request new data from the API and often the response contains the same rows I already wrote to database and cleanup from my DataTable.
If I won't cleanup the DataTable rows, performance decrease and it's memory pig. So, I leave one row with the biggest primary key after cleaning.
I don't want to compare every PrimaryKey from Source DataTable before merge, comparing can take a lot of time.
What should I do to prevent merging rows that I already wrote to database and removed from Actual DataTable? Maybe I can exclude them even at deserialisation process (I use NewtonSoft JSON.net)? Or any zippy way to prevent merging rows if they primary key < primary key in Actual DataTable?
Thanks for your answers!
UPDATE: merging code
public class MyData
{
DataTable BlackPairs = new DataTable();
DataTable WhiteTable = new DataTable();
public string _Json {
set
{
DataSet TempDS = JsonConvert.DeserializeObject<DataSet>(value);
try
{
foreach (DataTable table in TempDS.Tables)
{
BlackPairs = table.Copy();
WhiteTable.Merge(BlackPairs);
}
}catch{}
}
}
public MyData()
{ //columns initialization
WhiteTable.Columns.AddRange(new DataColumn[]{columns);
WhiteTable.PrimaryKey = new DataColumn[]{tid};
}
I have created custom Merge function based on what we have talked through comments. This function is only if primary column is typeof(int) but it can be easily improved to get all types or just change it to what type you need (string, int, bool...)
public Test()
{
InitializeComponent();
DataTable smallerDatatable = new DataTable();
smallerDatatable.Columns.Add("Col1", typeof(int));
smallerDatatable.Columns.Add("Col2", typeof(string));
DataTable biggerDatatable = new DataTable();
biggerDatatable.Columns.Add("Col1", typeof(int));
biggerDatatable.Columns.Add("Col2", typeof(string));
smallerDatatable.Rows.Add(1, "Row1");
smallerDatatable.Rows.Add(2, "Row2");
smallerDatatable.Rows.Add(3, "Row3");
biggerDatatable.Rows.Add(1, "Row1");
biggerDatatable.Rows.Add(2, "Row2");
biggerDatatable.Rows.Add(3, "Row3");
biggerDatatable.Rows.Add(4, "Row4");
biggerDatatable.Rows.Add(5, "Row5");
DataTable mergedTable = MergeOnUniqueColumn(smallerDatatable, biggerDatatable, "Col1");
dataGridView1.DataSource = mergedTable;
}
private DataTable MergeOnUniqueColumn(DataTable smallTable, DataTable bigTable, string uniqueColumn)
{
DataTable m = smallTable;
for(int i = 0; i < bigTable.Rows.Count; i++)
{
if(!(smallTable.AsEnumerable().Any(row => bigTable.Rows[i][uniqueColumn].Equals(row.Field<object>(uniqueColumn)))))
{
smallTable.Rows.Add(bigTable.Rows[i].ItemArray);
}
}
return m;
}
Function above will fill every missing unique value inside smallTable from bigTable.
If you want to fill smallTable with values from bigTable only after last smallTable row then use this function.
private DataTable MergeOnUniqueColumnAfterLastID(DataTable smallTable, DataTable bigTable, string uniqueColumn)
{
DataTable m = smallTable;
int maxUnique = Convert.ToInt32(m.Compute("max([" + uniqueColumn + "])", string.Empty));
for (int i = 0; i < bigTable.Rows.Count; i++)
{
if (!(smallTable.AsEnumerable().Any(row => (int)bigTable.Rows[i][uniqueColumn] <= maxUnique)))
{
smallTable.Rows.Add(bigTable.Rows[i].ItemArray);
}
}
return m;
}
I have created a datatable which has some records, now what I want is to copy the record of first datatable to another datatable.
I tried like below:
Session["AmountData"] = AmountDatatable; // 1st datatable which has data
DataTable CompanyWiseRecord = new DataTable();
for (int i = 0; i < AmountDatatable.Rows.Count; i++)
{
CompanyWiseRecord.ImportRow(AmountDatatable.Rows[i]); // 2nd datatable which does not have data
}
Kindly let me know where I am going wrong.
You directly use session Like:
DataTable CompanyWiseRecord = (DataTable)Session["AmountData"]; //Retrieving DataTable from Session.
if you are not fetching data from table one based on some condition or assign particular rows or columns then You can do it simply by using
DataTable CompanyWiseRecord=AmountDatatable;
If I have 2 DataTables (dtOne and dtTwo) and I want to merge them and put them in another DataTable (dtAll). How can I do this in C#? I tried the Merge statement on the datatable, but this returns void. Does Merge preserve the data? For example, if I do:
dtOne.Merge(dtTwo);
Does dtOne change or does dtTwo change and if either one changes, do the changes preserve?
I know I can't do this because Merge returns void, but I want to be able to store the Merger of both dtOne and dtTwo in dtAll:
//Will Not work, How do I do this
dtAll = dtOne.Merge(dtTwo);
The Merge method takes the values from the second table and merges them in with the first table, so the first will now hold the values from both.
If you want to preserve both of the original tables, you could copy the original first, then merge:
dtAll = dtOne.Copy();
dtAll.Merge(dtTwo);
Instead of dtAll = dtOne.Copy(); in Jeromy Irvine's answer you can start with an empty DataTable and merge one-by-one iteratively:
dtAll = new DataTable();
...
dtAll.Merge(dtOne);
dtAll.Merge(dtTwo);
dtAll.Merge(dtThree);
...
and so on.
This technique is useful in a loop where you want to iteratively merge data tables:
DataTable dtAllItems = new DataTable();
foreach(var item in items)
{
DataTable dtItem = getDataTable(item); // some function that returns a data table
dtAllItems.Merge(dtItem);
}
dtAll = dtOne.Copy();
dtAll.Merge(dtTwo,true);
The parameter TRUE preserve the changes.
For more details refer to MSDN.
DataTable dtAll = new DataTable();
DataTable dt= new DataTable();
foreach (int id in lst)
{
dt.Merge(GetDataTableByID(id)); // Get Data Methode return DataTable
}
dtAll = dt;
This is what i did for merging two datatables and bind the final result to the gridview
DataTable dtTemp=new DataTable();
for (int k = 0; k < GridView2.Rows.Count; k++)
{
string roomno = GridView2.Rows[k].Cells[1].Text;
DataTable dtx = GetRoomDetails(chk, roomno, out msg);
if (dtx.Rows.Count > 0)
{
dtTemp.Merge(dtx);
dtTemp.AcceptChanges();
}
}