We Keep Coding

Joining datatables dynamic

Im struggling with a following issue:
I have a datatable that contains information retrieved from an SQL Server database. The information retrieved is regarding the price of certain products. So, for example, the datatable has the price of a specific product on an specific price list. One product may be present in several different price list, and today on the datatable, each combination is a different row. For example:
I need to transform the datatable into a datatable that only has one row per product, and the price list information is on columns:
EXAMPLE DATA:
A few notes:
I dont know who many price lists i will have, so i could have the column "cost" (for ex) N amount of times.
The product information (first part) needs to be included.
No more than one row per id_Art.
I been working on the following, but i wanted to step back because i might be going into a rabbit hole and there might be an easier solution.
Currently, i'm creating new datatables filtered by each pricelist. The idea behind this was to join the datatables, but i got stucked.
foreach(var pricelist in pricelists)
{
DataSet dataSetsingle = new DataSet();
dataSetsingle = GetDataSetForMultiTariff(tarifa, dpto, secc, fam, subfam); //This will return a datatable filtered by pricelist
System.Data.DataTable dtnew = dataSetsingle.Tables[0];
var results = from table1 in dtinitial.AsEnumerable()
join table2 in dtnew.AsEnumerable()
on new { A = table1["ID_ART"], B = table1["ID_FORMAT"] } equals new { A = table2["ID_ART"], B = table2["ID_FORMAT"] }
select new
{
table1,table2
};
}
Should i keep moving forward through this approach? I dont get a flattend result the way i'm doing it, and i'm not sure how to solve that.
I have access to the database, so i could potentially change the query.
Pivot tables could work?
Thanks a lot!

If you want to continue with the LINQ join over DataTables, this extension method can help, but I think you would be better off with a pivot. Unfortunately I can't tell from your question what you want to pivot, but I do have a pivot method for a DataTable as well.
public static class DataTableExt {
// ***
// *** T Extensions
// ***
public static IEnumerable<T> AsSingleton<T>(this T first) {
yield return first;
}
// ***
// *** MemberInfo Extensions
// ***
public static Type GetMemberType(this MemberInfo member) {
switch (member) {
case FieldInfo mfi:
return mfi.FieldType;
case PropertyInfo mpi:
return mpi.PropertyType;
case EventInfo mei:
return mei.EventHandlerType;
default:
throw new ArgumentException("MemberInfo must be if type FieldInfo, PropertyInfo or EventInfo", nameof(member));
}
}
public static object GetValue(this MemberInfo member, object srcObject) {
switch (member) {
case FieldInfo mfi:
return mfi.GetValue(srcObject);
case PropertyInfo mpi:
return mpi.GetValue(srcObject);
case MethodInfo mi:
return mi.Invoke(srcObject, null);
default:
throw new ArgumentException("MemberInfo must be of type FieldInfo, PropertyInfo or MethodInfo", nameof(member));
}
}
public static T GetValue<T>(this MemberInfo member, object srcObject) => (T)member.GetValue(srcObject);
// ***
// *** Type Extensions
// ***
public static List<MemberInfo> GetPropertiesOrFields(this Type t, BindingFlags bf = BindingFlags.Public | BindingFlags.Instance) =>
t.GetMembers(bf).Where(mi => mi.MemberType == MemberTypes.Field || mi.MemberType == MemberTypes.Property).ToList();
// ***
// *** DataTable Extensions
// ***
public static IEnumerable<DataColumn> DataColumns(this DataTable aTable) => aTable.Columns.Cast<DataColumn>();
public static IEnumerable<string> ColumnNames(this DataTable aTable) => aTable.DataColumns().Select(dc => dc.ColumnName);
// Create new DataTable from LINQ join results on DataTable
// Expect T to be anonymous object of form new { [DataRow or other] d1, [DataRow or other] d2, ... }
public static DataTable FlattenToDataTable<T>(this IEnumerable<T> src) {
var res = new DataTable();
if (src.Any()) {
var firstRow = src.First();
var memberInfos = typeof(T).GetPropertiesOrFields();
var allDC = memberInfos.SelectMany(mi => (mi.GetMemberType() == typeof(DataRow)) ? mi.GetValue<DataRow>(firstRow).Table.DataColumns() : new DataColumn(mi.Name, mi.GetMemberType()).AsSingleton());
foreach (var dc in allDC) {
var newColumnName = dc.ColumnName;
if (res.ColumnNames().Contains(newColumnName)) {
var suffixNumber = 1;
while (res.ColumnNames().Contains($"{newColumnName}.{suffixNumber}"))
++suffixNumber;
newColumnName = $"{newColumnName}.{suffixNumber}";
}
res.Columns.Add(new DataColumn(newColumnName, dc.DataType));
}
foreach (var objRows in src)
res.Rows.Add(memberInfos.SelectMany(mi => (mi.GetMemberType() == typeof(DataRow)) ? mi.GetValue<DataRow>(objRows).ItemArray : mi.GetValue(objRows).AsSingleton()).ToArray());
}
return res;
}
}
Hmmm... didn't realize how many extension methods that code used :)

How to export data from C# from List<object> or IEnumerable to Excel (MS OpenXml)?

I'm trying to export a List<dynamic> or IEnumerable to an Excel file using the library Open XML of Microsoft.
Library: https://github.com/OfficeDev/Open-XML-SDK
Example of similar request: https://www.codeproject.com/Articles/692121/Csharp-Export-data-to-Excel-using-OpenXML-librarie
The example above works pretty fine but have a lot of code between (As far works neat, but I searching for an optimization)
But the documentation of this library does not extend (or hasn't many examples), and the examples that I saw around about how to export data to Excel it's using a DataTable or Dataset; so, they make the conversation from the List to DataTable to export the DataTable. And that seems pretty complex way to something that other libraries resolve easier.
So, if someone has a sort of example of how to make the export quickly and generic, it will be appreciated.

FYI, here is my implementation, using the code of #mikesknowledgebase (http://www.mikesknowledgebase.com -- The Webpage doesn't work... but to give the credits at least :D)
Post with all information here
So, I used it for .Net Core 2.2; and the intention is to use the method to export a List<dynamic> to Excel.
The final result (In the simplest example):
[HttpGet("[action]")]
public FileResult exportExample()
{
List<dynamic> data = new List<dynamic>();
data.Add(new { a = 1, b = "HELLO WORLD", c = DateTime.Now });
data.Add(new { a = 2, b = "TEST", c = 34 });
// Implementation of Dictionary to limit the columns and the type of the List
// Works with some standard, and flat files (Not with Dynamic Objects that have multiples levels - Indentation of a dynamic into a property)
Dictionary<string, Type> columns = new Dictionary<string, Type>();
columns.Add("a", typeof(int));
columns.Add("b", typeof(string));
columns.Add("c", typeof(object)); // Accepts any (Numbers or DateTime)
string excelContentType;
var excelStream = CreateExcelFile.CreateExcelStream(data, columns, out excelContentType);
return File(excelStream, excelContentType, "report.xlsx");
}
I created some other methods inside the class of Mike...
Method to get the List with the Dictionary<string, type> for the columns... and another param to return the ContentType:
public static byte[] CreateExcelStream<T>(List<T> list, Dictionary<string, Type> columns, out string contentType )
{
DataSet ds = new DataSet();
ds.Tables.Add(ListToDataTable(list, columns));
contentType = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"; // "application/vnd.ms-excel";
return CreateExcelStream(ds).fwToByteArray();
}
Another method to convert the List to a DataTable:
public static DataTable ListToDataTable<dynamic>(List<dynamic> list, Dictionary<string, Type> columns)
{
DataTable dt = new DataTable();
foreach (var column in columns)
dt.Columns.Add(new DataColumn(column.Key, GetNullableType(column.Value)));
foreach (var t in list)
{
DataRow row = dt.NewRow();
((object)t)
.GetType()
.GetProperties()
.ToList()
.ForEach(p =>
{
if (!IsNullableType(p.PropertyType))
row[p.Name] = p.GetValue(t, null);
else
row[p.Name] = (p.GetValue(t, null) ?? DBNull.Value);
});
dt.Rows.Add(row);
}
return dt;
}
I included an extension to convert a stream to a byteArray:
public static byte[] fwToByteArray(this Stream stream)
{
stream.Position = 0;
byte[] buffer = new byte[stream.Length];
for (int totalBytesCopied = 0; totalBytesCopied < stream.Length;)
totalBytesCopied += stream.Read(buffer, totalBytesCopied, Convert.ToInt32(stream.Length) - totalBytesCopied);
return buffer;
}
The rest of the code still the same... here is the result:

More elegant way to group DataTable data in structs in C# (using Linq?)

I have groups of data stored in a database table that I want to transform into a list of group-struct, each containing arrays of data.
I am able to do this using a rather long method. I was wondering if there is a more compact way of achieving this? I suspect Linq is supposed to be perfect for this kind of operation but I really don't know how to start.
The following example illustrates what I am currently doing. My real data is more complex.
The data is to be stored in a struct like this
public struct GroupData
{
private string aString;
private int anInt;
public GroupData(string aString, int anInt)
{
this.aString = aString;
this.anInt = anInt;
}
}
Which again is to be stored in a Group-struct
public struct Group
{
private string groupId;
private GroupData[] groupData;
public Group(string groupId, GroupData[] groupData)
{
this.groupId = groupId;
this.groupData = groupData;
}
}
I am currently doing this
//Create some dummy data
DataTable table = new DataTable();
table.Columns.Add("GROUP_ID", typeof(string));
table.Columns.Add("A_STRING", typeof(string));
table.Columns.Add("AN_INT", typeof(int));
table.Rows.Add("A", "this_is_A2", 7);
table.Rows.Add("A", "this_is_A2", 4);
table.Rows.Add("B", "this_is_B1", 3);
table.Rows.Add("C", "this_is_C1", 1);
table.Rows.Add("D", "this_is_D1", 3);
table.Rows.Add("D", "this_is_D2", 2);
//Create list of groups with arrays of groupdata
string theString, theGroupId;
int theInt;
List<Group> theGroups = new List<Group>();
List<GroupData> groupDataList;
Dictionary<string, List<GroupData>> groupDataDict = new Dictionary<string, List<GroupData>>();
//Read all rows and convert to structs
for (int i = 0; i < table.Rows.Count; i++)
{
theGroupId = (string)table.Rows[i]["GROUP_ID"];
theString = (string)table.Rows[i]["A_STRING"];
theInt = (int)table.Rows[i]["AN_INT"];
//Collect all GroupData into their respective Groups
if (!groupDataDict.TryGetValue(theGroupId, out groupDataList))
{
groupDataList = new List<GroupData>();
groupDataDict.Add(theGroupId, groupDataList);
}
groupDataList.Add(new GroupData(theString, theInt));
}
//Insert each group into the list
foreach (KeyValuePair<string, List<GroupData>> kv in groupDataDict)
theGroups.Add(new Group(kv.Key, kv.Value.ToArray()));
I see that my question is closely related to this post Group by in LINQ and I guess I could first transform my datatable to a list, and then use the prescribed method. But Ideally I would like to bypass the step of first transforming to a list and operate directly on the DataTable.

While I would highly recommend switching from DataTable to using models as most modern applications do (see references below), Microsoft has provided an AsEnumerable() extension method in System.Data.DataTableExtensions that is all you need to use LINQ on a DataTable.
//Create some dummy data
DataTable table = new DataTable();
table.Columns.Add("GROUP_ID", typeof(string));
table.Columns.Add("A_STRING", typeof(string));
table.Columns.Add("AN_INT", typeof(int));
table.Rows.Add("A", "this_is_A2", 7);
table.Rows.Add("A", "this_is_A2", 4);
table.Rows.Add("B", "this_is_B1", 3);
table.Rows.Add("C", "this_is_C1", 1);
table.Rows.Add("D", "this_is_D1", 3);
table.Rows.Add("D", "this_is_D2", 2);
var groups = (from dt in table.AsEnumerable()
group dt by dt.Field<string>("GROUP_ID") into g
select new {
GroupID = g.Key,
GroupData = g.Select(i => i.Field<int>("AN_INT")) }
).ToList();
Reference: LINQ query on a DataTable
A few articles you should read before you decide to use DataTable in the 2020's (keep in mind it is nearly two decades old and is not type safe):
Reasons to Move from DataTables to Generic Collections
What are the advantages of using POCOs over DataTables?

Getting table data as array of rows C# entityframework

My database table 'Table1' has columns 'Column1', 'Column2', 'Column3'.
I can get IEnumerable<Table1> using basic entity framework code.
But, I want my data sent to application layer to be in following format..
var data = [[ 'Column1', 'Column2', 'Column3'],
[Column1-Row1-Value,Column2-Row1-Value, Column3-Row1-Value],
[Column1-Row2-Value,Column2-Row2-Value, Column3-Row2-Value],
[Column1-Row3-Value,Column2-Row3-Value, Column3-Row3-Value]]
Any help is sincerely appreciated.
Thanks

The easist way to do it is just write a function that take in a IEnumerable<T> and outputs the data you want.
public void Example()
{
var myEfModel = GetEFData();
object[][] result = ConvertToArrayFormat(myEfModel, row => new object[] {row.Column1, row.Column2, row.Column3}, new object[] {"Column1", "Column2", "Column3"});
}
public object[][] ConvertToArrayFormat<T>(IEnumerable<T> dataSource, Func<T, object[]> rowSelector, object[] header = null)
{
var result = new List<object[]>();
if(header != null)
result.Add(header);
foreach (var item in dataSource)
{
var row = rowSelector(item);
result.Add(row);
}
return result.ToArray();
}

Is it possible to iterate on List<dynamic>?

In the past few days I've been trying to find a way to iterate on a List<dynamic> without much success.
That's what I'm doing:
while (dr.Read())
{
dynamic e = new ExpandoObject();
var d = e as IDictionary<string, object>;
for (var i = 0; i < dr.FieldCount; i++)
d.Add(dr.GetName(i), DBNull.Value.Equals(dr[i]) ? null : dr[i]);
result.Add(e);
}
the above code is a method that returns an IEnumerable<dynamic> then in my controller I'm getting data back with:
dynamic irionErrorsExport = oracleDbManager.GetStrCtrlNDGWithErrors(tableName, queryParamsList, periodo, "", "", "");
and now I'm stuck since I need to iterate on irionErrorsExport and create a "concrete" object/s to use with EPPlus.
Can anyone please tell me if it is even possible and show a simple example?

Yes, you can iterate over dynamic object:
dynamic source = new List<int>() {1, 2, 3, 4, 5, 6};
foreach (var item in source)
{
Console.Write(item.ToString());
}
Prints 123456 into console.
However, it will cause runtime exception if iteration is not possible:
Consider following code:
dynamic source = 2;
foreach (var item in source)
{
Console.Write(item.ToString());
}
RuntimeBinderException is being thrown:
Cannot implicitly convert type 'int' to 'System.Collections.IEnumerable'
Edit: you should be aware of the differences between foreach on normal variables and dynamic. They are explained in another SO question: C# 4.0 'dynamic' and foreach statement

If you fill a DataTable like here, You can use Json.Net and get a concrete object easily
//Sample DataTable
DataTable dt = new DataTable();
dt.Columns.Add("IntCol");
dt.Columns.Add("StrCol");
dt.Rows.Add(new object[]{1,"1"});
dt.Rows.Add(new object[]{2,"2"});
var jsonstr = JsonConvert.SerializeObject(dt);
var list = JsonConvert.DeserializeObject<List<YourClass>>(jsonstr);
public class YourClass
{
public int IntCol { set; get; }
public string StrCol { set; get; }
}

while (dr.Read())
{
IDictionary<string, object> e = new ExpandoObject();
for (var i = 0; i < dr.FieldCount; i++)
e.Add(dr.GetName(i), DBNull.Value.Equals(dr[i]) ? null : dr[i]);
result.Add(e);
}
From the calling method you "cheat". You know that your dynamic collection is an ExpandoObject, so
foreach (IDictionary<string, object> row in result)
{
foreach (var kv in row)
{
Console.WriteLine("{0}: {1}", kv.Key, kv.Value);
}
}
In the end, it's better if your method simply returns a List<IDictionary<string, object>>, no dynamic necessary.
Reflection on dynamic types is hard. Unless you can use duck typing (duck typing is when you know that the object can Duck(), even if you don't know what exactly it's, so you can do dynamic x = something; x.Duck(); ), then it's only semi-hard. If you don't trust me on this, you can try reading How do I reflect over the members of dynamic object?