We Keep Coding

How to throuhly join two datatables using linq without column names

This is propably answered somewhere else, but I haven't found working solution yet.
I have two datatables and I want to join them into one datatable containing all data from both of them, or at least from the first of them and some columns from the second datatable.
I don't want to list all columns (totally 180) from the first datatable. I have tried eg. this
var JoinedResult = from t1 in table1.Rows.Cast<DataRow>()
join t2 in table2.Rows.Cast<DataRow>()
on Convert.ToInt32(t1.Field<string>("ProductID")) equals t2.Field<int>("FuelId")
select t1;
but that gives only the columns from table1. How to get colums from table2 too to my result? Finally, I need to add my result to a dataset.
ResultSet.Tables.Add(JoinedResult.CopyToDataTable());
EDIT:
I ended up with this as the solution.
This follows an example given here Create join with Select All (select *) in linq to datasets
DataTable dtProduct = dsProduct.Tables[0];
DataTable dtMoistureLimits = ds.Tables[0];
//clone dt1, copies all the columns to newTable
DataTable dtProductWithMoistureLimits = dtProduct.Clone();
//copies all the columns from dt2 to newTable
foreach (DataColumn c in dtMoistureLimits.Columns)
dtProductWithMoistureLimits.Columns.Add(c.ColumnName, c.DataType);
var ProductsJoinedWithMoistureLimits = dtProduct.Rows.Cast<DataRow>()
.Join(dtMoistureLimits.Rows.Cast<DataRow>(),// join table1 and table2
t1 => new { ProductID = t1.Field<int>("ProductID"), DelivererID = t1.Field<int>("DelivererID") },
t2 => new { ProductID = t2.Field<int>("MoistureLimits_ProductID"), DelivererID = t2.Field<int>("MoistureLimits_DelivererID") },
(t1, t2) => // when they match
{ // make a new object
// containing the matching t1 and t2
DataRow row = dtProductWithMoistureLimits.NewRow();
row.ItemArray = t1.ItemArray.Concat(t2.ItemArray).ToArray();
dtProductWithMoistureLimits.Rows.Add(row);
return row;
});
However, in dtMoistureLimits there is not rows for all "ProductID" and "DelivererID" in dtProduct. Currently my solution returns only matching rows.
How to improve solution to return also those rows where there is not data for "ProductID" and "DelivererID" in dtMoistureLimits?

Solution using method syntax, without having to mention all columns
var result = table1.Rows.Cast<DataRow>()
.Join(table2.Rows.Cast<DataRow>(), // join table1 and table2
t1 => Convert.ToInt32(t1.Field<string>("ProductID")) // from every t1 get the productId
t2 => t2.Field<int>("FuelId") // from every t2 get the fuelId,
(t1, t2) => new // when they match
{ // make a new object
T1 = t1, // containing the matching t1 and t2
T2 = t2,
}

var JoinedResult = (from t1 in table1.Rows.Cast<DataRow>()
join t2 in table2.Rows.Cast<DataRow>()
on Convert.ToInt32(t1.Field<string>("ProductID")) equals t2.Field<int>("FuelId")
select new { T1 = t1,
T2 = t2.column_name // all columns needed can be listed here
}).ToList();
EDIT:
To convert the above result to a DataTable, use the following method:
DataTable dataTable = new DataTable();
//Get all the properties
PropertyInfo[] Props = JoinedResult.Select(y=>y.T1).First().GetType().GetProperties(BindingFlags.Public | BindingFlags.Instance);
foreach (PropertyInfo prop in Props)
{
//Defining type of data column gives proper data table
var type = (prop.PropertyType.IsGenericType && prop.PropertyType.GetGenericTypeDefinition() == typeof(Nullable<>) ? Nullable.GetUnderlyingType(prop.PropertyType) : prop.PropertyType);
//Setting column names as Property names
dataTable.Columns.Add(prop.Name, type);
}
dataTable.Columns.Add(t2_column_name, t2_column_type);
foreach (var item in JoinedResult)
{
var values = new object[Props.Length];
for (int i = 0; i < Props.Length; i++)
{
//inserting property values to datatable rows
values[i] = Props[i].GetValue(item.T1, null);
}
values[Props.Length] = item.T2;
dataTable.Rows.Add(values);
}

Join 2 DataTables on dynamic number of columns

I'm trying to join two DataTables on a dynamic number of columns. I've gotten as far as the code below. The problem is the ON statement of the join. How can I make this dynamic based on how many column names are in the list "joinColumnNames".
I was thinking I will need to build some sort of expression tree, but I can't find any examples of how to do this with multiple join columns and with the DataRow object which doesn't have properties for each column.
private DataTable Join(List<string> joinColumnNames, DataTable pullX, DataTable pullY)
{
DataTable joinedTable = new DataTable();
// Add all the columns from pullX
foreach (string colName in joinColumnNames)
{
joinedTable.Columns.Add(pullX.Columns[colName]);
}
// Add unique columns from PullY
foreach (DataColumn col in pullY.Columns)
{
if (!joinedTable.Columns.Contains((col.ColumnName)))
{
joinedTable.Columns.Add(col);
}
}
var Join = (from PX in pullX.AsEnumerable()
join PY in pullY.AsEnumerable() on
// This must be dynamic and join on every column mentioned in joinColumnNames
new { A = PX[joinColumnNames[0]], B = PX[joinColumnNames[1]] } equals new { A = PY[joinColumnNames[0]], B = PY[joinColumnNames[1]] }
into Outer
from PY in Outer.DefaultIfEmpty<DataRow>(pullY.NewRow())
select new { PX, PY });
foreach (var item in Join)
{
DataRow newRow = joinedTable.NewRow();
foreach (DataColumn col in joinedTable.Columns)
{
var pullXValue = item.PX.Table.Columns.Contains(col.ColumnName) ? item.PX[col.ColumnName] : string.Empty;
var pullYValue = item.PY.Table.Columns.Contains(col.ColumnName) ? item.PY[col.ColumnName] : string.Empty;
newRow[col.ColumnName] = (pullXValue == null || string.IsNullOrEmpty(pullXValue.ToString())) ? pullYValue : pullXValue;
}
joinedTable.Rows.Add(newRow);
}
return joinedTable;
}
Adding a specific example to show input/output using 3 join columns (Country, Company, and DateId):
Pull X:
Country Company DateId Sales
United States Test1 Ltd 20160722 $25
Canada Test3 Ltd 20160723 $30
Italy Test4 Ltd 20160724 $40
India Test2 Ltd 20160725 $35
Pull Y:
Country Company DateId Downloads
United States Test1 Ltd 20160722 500
Mexico Test2 Ltd 20160723 300
Italy Test4 Ltd 20160724 900
Result:
Country Company DateId Sales Downloads
United States Test1 Ltd 20160722 $25 500
Canada Test3 Ltd 20160723 $30
Mexico Test2 Ltd 20160723 300
Italy Test4 Ltd 20160724 $40 900
India Test2 Ltd 20160725 $35

var Join =
from PX in pullX.AsEnumerable()
join PY in pullY.AsEnumerable()
on string.Join("\0", joinColumnNames.Select(c => PX[c]))
equals string.Join("\0", joinColumnNames.Select(c => PY[c]))
into Outer
from PY in Outer.DefaultIfEmpty<DataRow>(pullY.NewRow())
select new { PX, PY };
Another way is to have both DataTable in a DataSet and use DataRelation
How To: Use DataRelation to perform a join on two DataTables in a DataSet?

Since you are using LINQ to Objects, there is no need to use expression trees. You can solve your problem with a custom equality comparer.
Create an equality comparer that can compare equality between two DataRow objects based on the values of specific columns. Here is an example:
public class MyEqualityComparer : IEqualityComparer<DataRow>
{
private readonly string[] columnNames;
public MyEqualityComparer(string[] columnNames)
{
this.columnNames = columnNames;
}
public bool Equals(DataRow x, DataRow y)
{
return columnNames.All(cn => x[cn].Equals(y[cn]));
}
public int GetHashCode(DataRow obj)
{
unchecked
{
int hash = 19;
foreach (var value in columnNames.Select(cn => obj[cn]))
{
hash = hash * 31 + value.GetHashCode();
}
return hash;
}
}
}
Then you can use it to make the join like this:
public class TwoRows
{
public DataRow Row1 { get; set; }
public DataRow Row2 { get; set; }
}
private static List<TwoRows> LeftOuterJoin(
List<string> joinColumnNames,
DataTable leftTable,
DataTable rightTable)
{
return leftTable
.AsEnumerable()
.GroupJoin(
rightTable.AsEnumerable(),
l => l,
r => r,
(l, rlist) => new {LeftValue = l, RightValues = rlist},
new MyEqualityComparer(joinColumnNames.ToArray()))
.SelectMany(
x => x.RightValues.DefaultIfEmpty(rightTable.NewRow()),
(x, y) => new TwoRows {Row1 = x.LeftValue, Row2 = y})
.ToList();
}
Please note that I am using method syntax because I don't think that you can use a custom equality comparer otherwise.
Please note that the method does a left outer join, not a full outer join. Based on the example you provided, you seem to want a full outer join. To do this you need to do two left outer joins (see this answer). Here is how the full method would look like:
private static DataTable FullOuterJoin(
List<string> joinColumnNames,
DataTable pullX,
DataTable pullY)
{
var pullYOtherColumns =
pullY.Columns
.Cast<DataColumn>()
.Where(x => !joinColumnNames.Contains(x.ColumnName))
.ToList();
var allColumns =
pullX.Columns
.Cast<DataColumn>()
.Concat(pullYOtherColumns)
.ToArray();
var allColumnsClone =
allColumns
.Select(x => new DataColumn(x.ColumnName, x.DataType))
.ToArray();
DataTable joinedTable = new DataTable();
joinedTable.Columns.AddRange(allColumnsClone);
var first =
LeftOuterJoin(joinColumnNames, pullX, pullY);
var resultRows = new List<DataRow>();
foreach (var item in first)
{
DataRow newRow = joinedTable.NewRow();
foreach (DataColumn col in joinedTable.Columns)
{
var value = pullX.Columns.Contains(col.ColumnName)
? item.Row1[col.ColumnName]
: item.Row2[col.ColumnName];
newRow[col.ColumnName] = value;
}
resultRows.Add(newRow);
}
var second =
LeftOuterJoin(joinColumnNames, pullY, pullX);
foreach (var item in second)
{
DataRow newRow = joinedTable.NewRow();
foreach (DataColumn col in joinedTable.Columns)
{
var value = pullY.Columns.Contains(col.ColumnName)
? item.Row1[col.ColumnName]
: item.Row2[col.ColumnName];
newRow[col.ColumnName] = value;
}
resultRows.Add(newRow);
}
var uniqueRows =
resultRows
.Distinct(
new MyEqualityComparer(
joinedTable.Columns
.Cast<DataColumn>()
.Select(x => x.ColumnName)
.ToArray()));
foreach (var uniqueRow in uniqueRows)
joinedTable.Rows.Add(uniqueRow);
return joinedTable;
}
Please note also how I clone the columns. You cannot use the same column object in two tables.

How to use Inner Join and then fill a DataSet with result of the join?

Well, this is my question. In short terms; I have two tables, Consequents and Atomic propositions:
AtomicP table
ID Proposition
1 | A |
1 | B |
1 | C |
2 | D |
2 | E |
Consequent Table
ID | Consequent |
1 | A |
2 | B |
And all I just want to do, is to implement a inner join which gives me all the values where the ID for both tables is the same(i.e):
AtomicP Table "A" "B" "C" -> "A" Consequent Table
and withe result given tanks to the inner joins , save that result in a Data Set or in another data structure that could be better.
Best regards.

Assuming the destination table has the values Id, Proposition and Consequent ..
insert into newtable (id,proposition,consequent) select id,atomicP,Consequent from atmicp,consequent where atomicP.id = consequent.id

public class Proposition
{
public int Id;
public string Value;
public Proposition(int id, string value){
Id = id;
Value = value;
}
}
public class Consequent
{
public int Id;
public string Value;
public Consequent(int id, string value){
Id = id;
Value = value;
}
}
var atomicP = new List<Proposition>{
new Proposition(1, "A"),
new Proposition(1, "B"),
new Proposition(1, "C"),
new Proposition(2, "D"),
new Proposition(2, "E"),
}
var consequents = new List<Consequent>{
new Consequent(1, "A"),
new Consequent(2, "B"),
}
var query = from proposition in atomicP
join consequent in consequents on proposition.Id == consequent.Id
select proposition.Value;
return query.ToList();

use this function
private DataTable JoinDataTables(DataTable t1, DataTable t2, params Func<DataRow, DataRow, bool>[] joinOn)
{
DataTable result = new DataTable();
foreach (DataColumn col in t1.Columns)
{
if (result.Columns[col.ColumnName] == null)
result.Columns.Add(col.ColumnName, col.DataType);
}
foreach (DataColumn col in t2.Columns)
{
if (result.Columns[col.ColumnName] == null)
result.Columns.Add(col.ColumnName, col.DataType);
}
foreach (DataRow row1 in t1.Rows)
{
var joinRows = t2.AsEnumerable().Where(row2 =>
{
foreach (var parameter in joinOn)
{
if (!parameter(row1, row2)) return false;
}
return true;
});
foreach (DataRow fromRow in joinRows)
{
DataRow insertRow = result.NewRow();
foreach (DataColumn col1 in t1.Columns)
{
insertRow[col1.ColumnName] = row1[col1.ColumnName];
}
foreach (DataColumn col2 in t2.Columns)
{
insertRow[col2.ColumnName] = fromRow[col2.ColumnName];
}
result.Rows.Add(insertRow);
}
}
return result;
}
An example of how you might use this:
var test = JoinDataTables(Consequents, Atomic,
(row1, row2) =>
row1.Field<int>("ID") == row2.Field<int>("ID"));

I assume you want to join In C# and get DataTable(bit unclear in question).
Code snippets joins two DataTable using Linq and inserts to another Table.
DataTable results = new DataTable();
results.Columns.Add("ID", typeof(int));
results.Columns.Add("Proposition", typeof(string));
results.Columns.Add("Consequent", typeof(string));
var result1 = from arow in AtomicP.AsEnumerable()
join con in Consequent.AsEnumerable()
on arow.Field<int>("ID") equals con.Field<int>("ID")
select results.LoadDataRow(new object[]
{
arow.Field<int>("ID"),
arow.Field<string>("Proposition"),
con.Field<string>("Consequent")
}, false);
Now we can access results by iterating through results.
foreach(DataRow row in results.Rows)
{
foreach(DataColumn column in results.Columns)
{
//Console.WriteLine(row[column]);
}
}
Working Code

C# Linq rows to column

I would like to turn linq result into columns from rows, the field names are user changeable so I need the function to be dynamic.
sample data
ID: 331 FieldName: "BusinessCategory" FieldContents: "Regulatory"
ID: 331 FieldName: "PriorityGroup" FieldContents: "Must Do"
ID: 332 FieldName: "BusinessCategory" FieldContents: "Financial"
ID: 332 FieldName: "PriorityGroup" FieldContents: "Should Do"
Turn it into (sample end output)
ID BusinessCategory PriorityGroup
331 Regulatory Must Do
332 Financial Should DO
Here is the code block to extract to fieldnames and contents from the database.
public static IEnumerable<InitProjectValues1> GetProgramInitiativeAttributesPart1(int id)
{
using (dpm db = new dpm())
{
string partit = (string)HttpContext.Current.Session["sitePart"];
var configrefs = from c in (
from e in db.Metrics
join j in db.ProgramLink on e.ProjectRef equals j.LinkedProject
where (j.ProjectRef == id) && e.PartitNo == partit
select new
{
FieldName = e.FieldName,
FieldContents = e.MetricValue,
ProjectRef = e.ProjectRef,
})
select new InitProjectValues1
{
ProjectRef = c.ProjectRef,
FieldName = c.FieldName,
FieldContents = c.FieldContents,
}; //somewhere here would be the code to cover this into a single row per ProjectRef number.
return configrefs.ToList();
}
}
Here is the data model.
public class InitProjectValues1
{
public int? ProjectRef { get; set; }
public string FieldName { get; set; }
public string FieldContents { get; set; }
}
I really don't know where to go from here, hoping someone can provide guidance / sample code

The kind of operation you need is called a pivot. You are effectively rotating the table around a unique productRef and changing the rows to columns.
You could try this which makes use of a dynamic object which you require for dynamic column generation.
var configrefs = from c in (
from e in db.Metrics
join j in db.ProgramLink on e.ProjectRef equals j.LinkedProject
where (j.ProjectRef == id) && e.PartitNo == partit
select new
{
FieldName = e.FieldName,
FieldContents = e.MetricValue,
ProjectRef = e.ProjectRef,
}).ToArray();
return configrefs.ToPivotArray(
i => i.FieldName,
i => i.ProjectRef,
items => items.Any() ? items.FirstOrDefault().FieldContents : null);
Private method to get dynamic object:
private static dynamic GetAnonymousObject(IEnumerable<string> columns, IEnumerable<object> values)
{
IDictionary<string, object> eo = new ExpandoObject() as IDictionary<string, object>;
int i;
for (i = 0; i < columns.Count(); i++)
{
eo.Add(columns.ElementAt<string>(i), values.ElementAt<object>(i));
}
return eo;
}
And the extension method
public static dynamic[] ToPivotArray<T, TColumn, TRow, TData>(
this IEnumerable<T> source,
Func<T, TColumn> columnSelector,
Expression<Func<T, TRow>> rowSelector,
Func<IEnumerable<T>, TData> dataSelector)
{
var arr = new List<object>();
var cols = new List<string>();
String rowName = ((MemberExpression)rowSelector.Body).Member.Name;
var columns = source.Select(columnSelector).Distinct();
cols =(new []{ rowName}).Concat(columns.Select(x=>x.ToString())).ToList();
var rows = source.GroupBy(rowSelector.Compile())
.Select(rowGroup => new
{
Key = rowGroup.Key,
Values = columns.GroupJoin(
rowGroup,
c => c,
r => columnSelector(r),
(c, columnGroup) => dataSelector(columnGroup))
}).ToArray();
foreach (var row in rows)
{
var items = row.Values.Cast<object>().ToList();
items.Insert(0, row.Key);
var obj = GetAnonymousObject(cols, items);
arr.Add(obj);
}
return arr.ToArray();
}

Modified the ToPivotArray extension to handle multiple column selectors (using an anonymous class as the column selector)
public static dynamic[] ToPivotArrayNew<T, TColumn, TRow, TData>(
this IEnumerable<T> source,
Func<T, TColumn> columnSelector,
Expression<Func<T, TRow>> rowSelector,
Func<IEnumerable<T>, TData> dataSelector)
{
var arr = new List<object>();
var cols = new List<string>();
List<string> rowNames = new List<string>();
bool isObjectSelector = false;
if (rowSelector.Body.GetType() == typeof(MemberExpression))
{
rowNames.Add(((MemberExpression)rowSelector.Body).Member.Name);
}
else if (rowSelector.Body.GetType() == typeof(NewExpression))
{
isObjectSelector = true;
((NewExpression)rowSelector.Body).Members.ToList().ForEach(m => rowNames.Add(m.Name));
}
var columns = source.Select(columnSelector).Distinct();
cols = rowNames.ToArray().Concat(columns.Select(x => x.ToString())).ToList();
var rows = source.GroupBy(rowSelector.Compile())
.Select(rowGroup => new
{
Key = rowGroup.Key,
Values = columns.GroupJoin(
rowGroup,
c => c,
r => columnSelector(r),
(c, columnGroup) => dataSelector(columnGroup))
}).ToArray();
foreach (var row in rows)
{
var items = row.Values.Cast<object>().ToList();
if (isObjectSelector)
{
for (int i = 0; i < rowNames.Count(); i++)
{
items.Insert(i, row.Key.GetType().GetProperty(rowNames[i]).GetValue(row.Key));
}
}
else
{
items.Insert(0, row.Key);
}
var obj = GetAnonymousObject(cols, items);
arr.Add(obj);
}
return arr.ToArray();
}

having trouble with left join LINQ

I have two DataTables t1 and t2. I'm trying to perform a LINQ left join, multiple equijoin, to get the DataRows in t1 that are not in t2.
In SQL, what I'm trying to accomplish is:
select t1.*
from t1
left join t2
on t1.a=t2.a and
t1.b=t2.b and
t1.c=t2.c
where
t2.a is null
So far I have the following:
public DataTable t1_without_t2(DataTable t1, DataTable t2)
{
var query = from t1_row in t1.AsEnumerable()
join t2_row in t2.AsEnumerable()
on
new { t_a = t1_row["a"], t_b = t1_row["b"], t_c = t1_row["c"]}
equals
new { t_a = t2_row["a"], t_b = t2_row["b"], t_c = t2_row["c"]}
into leftJoinT1withoutT2
from join_row in leftJoinT1withoutT2.DefaultIfEmpty()
where t2_row["a"] == null
select new
{
j_a = join_row["a"],
j_b = join_row["b"],
j_c = join_row["c"],
};
DataTable dt = t1.Clone();
foreach (var result in query)
{
dt.LoadDataRow(
new object[]
{
result.j_a,
result.j_b,
result.j_c
},
false);
}
return dt;
}
This is failing on the line j_a = join_row["a"] with this message:
Column 'a' does not belong to table.
I thought that the into leftJoinT1withoutT2 line was supposed to put the results of the join into a var with the column structure of table t1, from which the non-matching entries would be removed using where t2_row["a"] == null . Is that not what's happening here? I'm a little confused.

It should look like this:
var query = from t1_row in t1.AsEnumerable()
join t2_row in t2.AsEnumerable()
on
new { t_a = t1_row["a"], t_b = t1_row["b"], t_c = t1_row["c"] }
equals
new { t_a = t2_row["a"], t_b = t2_row["b"], t_c = t2_row["c"] }
into leftJoinT1withoutT2
from join_row in leftJoinT1withoutT2.DefaultIfEmpty()
.Where(r => r == null)
select new
{
j_a = t1_row["a"],
j_b = t1_row["b"],
j_c = t1_row["c"],
};
Have a look at How to: Perform Left Outer Joins (C# Programming Guide).
The join_row gets null (i.e. default TSource value, see Enumerable.DefaultIfEmpty) when there is no matching element in t2, while t1_row always contains the joined value. So as far as you need only those rows for which join_row is null, I used .Where(r => r == null).