We Keep Coding

How to export data from C# from List<object> or IEnumerable to Excel (MS OpenXml)?

I'm trying to export a List<dynamic> or IEnumerable to an Excel file using the library Open XML of Microsoft.
Library: https://github.com/OfficeDev/Open-XML-SDK
Example of similar request: https://www.codeproject.com/Articles/692121/Csharp-Export-data-to-Excel-using-OpenXML-librarie
The example above works pretty fine but have a lot of code between (As far works neat, but I searching for an optimization)
But the documentation of this library does not extend (or hasn't many examples), and the examples that I saw around about how to export data to Excel it's using a DataTable or Dataset; so, they make the conversation from the List to DataTable to export the DataTable. And that seems pretty complex way to something that other libraries resolve easier.
So, if someone has a sort of example of how to make the export quickly and generic, it will be appreciated.

FYI, here is my implementation, using the code of #mikesknowledgebase (http://www.mikesknowledgebase.com -- The Webpage doesn't work... but to give the credits at least :D)
Post with all information here
So, I used it for .Net Core 2.2; and the intention is to use the method to export a List<dynamic> to Excel.
The final result (In the simplest example):
[HttpGet("[action]")]
public FileResult exportExample()
{
List<dynamic> data = new List<dynamic>();
data.Add(new { a = 1, b = "HELLO WORLD", c = DateTime.Now });
data.Add(new { a = 2, b = "TEST", c = 34 });
// Implementation of Dictionary to limit the columns and the type of the List
// Works with some standard, and flat files (Not with Dynamic Objects that have multiples levels - Indentation of a dynamic into a property)
Dictionary<string, Type> columns = new Dictionary<string, Type>();
columns.Add("a", typeof(int));
columns.Add("b", typeof(string));
columns.Add("c", typeof(object)); // Accepts any (Numbers or DateTime)
string excelContentType;
var excelStream = CreateExcelFile.CreateExcelStream(data, columns, out excelContentType);
return File(excelStream, excelContentType, "report.xlsx");
}
I created some other methods inside the class of Mike...
Method to get the List with the Dictionary<string, type> for the columns... and another param to return the ContentType:
public static byte[] CreateExcelStream<T>(List<T> list, Dictionary<string, Type> columns, out string contentType )
{
DataSet ds = new DataSet();
ds.Tables.Add(ListToDataTable(list, columns));
contentType = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"; // "application/vnd.ms-excel";
return CreateExcelStream(ds).fwToByteArray();
}
Another method to convert the List to a DataTable:
public static DataTable ListToDataTable<dynamic>(List<dynamic> list, Dictionary<string, Type> columns)
{
DataTable dt = new DataTable();
foreach (var column in columns)
dt.Columns.Add(new DataColumn(column.Key, GetNullableType(column.Value)));
foreach (var t in list)
{
DataRow row = dt.NewRow();
((object)t)
.GetType()
.GetProperties()
.ToList()
.ForEach(p =>
{
if (!IsNullableType(p.PropertyType))
row[p.Name] = p.GetValue(t, null);
else
row[p.Name] = (p.GetValue(t, null) ?? DBNull.Value);
});
dt.Rows.Add(row);
}
return dt;
}
I included an extension to convert a stream to a byteArray:
public static byte[] fwToByteArray(this Stream stream)
{
stream.Position = 0;
byte[] buffer = new byte[stream.Length];
for (int totalBytesCopied = 0; totalBytesCopied < stream.Length;)
totalBytesCopied += stream.Read(buffer, totalBytesCopied, Convert.ToInt32(stream.Length) - totalBytesCopied);
return buffer;
}
The rest of the code still the same... here is the result:

Mapping Rows Data to MyObject

i have my classes as the following:
class SalesInvlice
{
int id {get;set;}
string number {get;set;}
List<InvoiceItems> {get;set}
}
class InvoiceItems
{
id {get;set}
string item_name {get;set}
int price {get;set;}
}
my application is an agent that can connect to any database specified in the config file, and to execute a certain query. in my case it will execute a query on one client DB as the following select id, number, items.id as items_id, item_name, price from transaction left join transaction_details on transactions.id = transaction_details.transaction_id
let say i got the data using SQLDataReader, but i am open in another solutions as well.
what i am looking for here, is to map the results from SQLDataReader to list of SalesInvoice Object.
the issue i am facing that if the transaction has list of transaction_details, this means the datareader will get them to me in different rows.

You have a choice to make you can return the data in one dataset or two. The first will return the transaction data duplicated see bellow. You will need to manage the dr duplicate transactions.
select id,
number,
items.id as items_id,
item_name, price
from transaction
left join transaction_details
on transactions.id = transaction_details.transaction_id
Or return the data in two sets of data ( transaction and transaction detail). I am assuming that you are interested in data for a particular transaction not many different transactions. Ado.net will allow for multiple groups of data to be returned. You would need to cast each data set to is object type. dimly a case of newing up a transaction and assigning properties.

to solve this issue, i have made my simi general mapper.
I have mapper that do the following:
private List<T> mappingFun<T>( IEnumerable<Dictionary<string, object>> args, object propListObj = null) where T: Entity
{
List<T> listObject = new List<T>();
foreach(var arg in args)
{
T returnedObject = Activator.CreateInstance<T>();
PropertyInfo[] modelProperties = returnedObject.GetType().GetProperties();
var addedobject = listObject.FirstOrDefault(x => x.name == arg[returnedObject.GetType().Name + "_name"].ToString());
if (addedobject != null)
{
returnedObject = addedobject;
}
foreach(var prop in modelProperties)
{
var a = prop.PropertyType;
if (prop.PropertyType == typeof(String))
{
prop.SetValue(returnedObject, arg[returnedObject.GetType().Name + "_" + prop.Name].ToString());
}
else
{
var propModuleObj = GetType().GetMethod("mappingFun", BindingFlags.Instance | BindingFlags.NonPublic, null, new Type[] { typeof(IEnumerable<Dictionary<string, object>>), typeof(object) }, null).MakeGenericMethod(prop.PropertyType.GetGenericArguments().Single()).Invoke(this, new object[] { new List<Dictionary<string, object>>() { arg }, prop.GetValue(returnedObject) });
prop.SetValue(returnedObject, propModuleObj);
}
}
listObject.AddIfNotExist(returnedObject);
if(propListObj != null)
listObject.AddRange((List<T>)propListObj);
}
return listObject;
}
This helps 100%

Convert NEST (Elasticsearch) Search Results to DataSet - C#

I am using NEST to search my Elasticsearch index:
var result = client.Search<MyObject>(s => s
.From(0)
.Size(10)
// Query here
);
This works and returns a Nest.SearchResponse object. The format returned by result.Hits.ToList() is List<Nest.IHit<MyObject>>().
How can I convert the results returned to a DataSet (or DataTable)?
Any help is appreciated.

You need to loop over your results
DataTable dt = new DataTable();
dt.Columns.Add("Field1", typeof(string));
dt.Columns.Add("Field2", typeof(string));
...
foreach (IHit<JObject> x in result.Hits)
{
dt.Rows.Add(
x.Fields.FieldValuesDictionary["Prop1"] as JArray,
x.Fields.FieldValuesDictionary["Prop2"] as JArray
...
);
}
Read
Retrieve data from elasticsearch results
How do you get search results returned in nest 1.x mapped to an object?
as well as
DocumentsWithMetaData
When you do a search with NEST 0.12, you'd get back a QueryResponse
with two ways to loop over your results. .Documents is an
IEnumerable and .DocumentsWithMetaData is and IEnumerable>
depending on your needs one of them might be easier to use.
Starting from NEST 1.0 .DocumentsWithMetaData is now called simply
.Hits.
http://nest.azurewebsites.net/breaking-changes.html

As proposed in this article, you can use an extension method to convert an IEnumerable<T> to a DataTable:
public static class IEnumerableExtensions
{
/*Converts IEnumerable To DataTable*/
public static DataTable ToDataTable<TSource>(this IEnumerable<TSource> data)
{
DataTable dataTable = new DataTable(typeof(TSource).Name);
PropertyInfo[] props = typeof(TSource).GetProperties(BindingFlags.Public | BindingFlags.Instance);
foreach (PropertyInfo prop in props)
{
dataTable.Columns.Add(prop.Name, Nullable.GetUnderlyingType(prop.PropertyType) ??
prop.PropertyType);
}
foreach (TSource item in data)
{
var values = new object[props.Length];
for (int i = 0; i < props.Length; i++)
{
values[i] = props[i].GetValue(item, null);
}
dataTable.Rows.Add(values);
}
return dataTable;
}
}
Then, after the search you could do something like this:
var dataTable = result.Documents.ToDataTable();
Where Documents is an IEnumerable<MyObject> that are the documents inside the hits that are returned. Another way to obtain those documents is:
var documents=result.Hits.Select(h => h.Source);

Is it possible to iterate on List<dynamic>?

In the past few days I've been trying to find a way to iterate on a List<dynamic> without much success.
That's what I'm doing:
while (dr.Read())
{
dynamic e = new ExpandoObject();
var d = e as IDictionary<string, object>;
for (var i = 0; i < dr.FieldCount; i++)
d.Add(dr.GetName(i), DBNull.Value.Equals(dr[i]) ? null : dr[i]);
result.Add(e);
}
the above code is a method that returns an IEnumerable<dynamic> then in my controller I'm getting data back with:
dynamic irionErrorsExport = oracleDbManager.GetStrCtrlNDGWithErrors(tableName, queryParamsList, periodo, "", "", "");
and now I'm stuck since I need to iterate on irionErrorsExport and create a "concrete" object/s to use with EPPlus.
Can anyone please tell me if it is even possible and show a simple example?

Yes, you can iterate over dynamic object:
dynamic source = new List<int>() {1, 2, 3, 4, 5, 6};
foreach (var item in source)
{
Console.Write(item.ToString());
}
Prints 123456 into console.
However, it will cause runtime exception if iteration is not possible:
Consider following code:
dynamic source = 2;
foreach (var item in source)
{
Console.Write(item.ToString());
}
RuntimeBinderException is being thrown:
Cannot implicitly convert type 'int' to 'System.Collections.IEnumerable'
Edit: you should be aware of the differences between foreach on normal variables and dynamic. They are explained in another SO question: C# 4.0 'dynamic' and foreach statement

If you fill a DataTable like here, You can use Json.Net and get a concrete object easily
//Sample DataTable
DataTable dt = new DataTable();
dt.Columns.Add("IntCol");
dt.Columns.Add("StrCol");
dt.Rows.Add(new object[]{1,"1"});
dt.Rows.Add(new object[]{2,"2"});
var jsonstr = JsonConvert.SerializeObject(dt);
var list = JsonConvert.DeserializeObject<List<YourClass>>(jsonstr);
public class YourClass
{
public int IntCol { set; get; }
public string StrCol { set; get; }
}

while (dr.Read())
{
IDictionary<string, object> e = new ExpandoObject();
for (var i = 0; i < dr.FieldCount; i++)
e.Add(dr.GetName(i), DBNull.Value.Equals(dr[i]) ? null : dr[i]);
result.Add(e);
}
From the calling method you "cheat". You know that your dynamic collection is an ExpandoObject, so
foreach (IDictionary<string, object> row in result)
{
foreach (var kv in row)
{
Console.WriteLine("{0}: {1}", kv.Key, kv.Value);
}
}
In the end, it's better if your method simply returns a List<IDictionary<string, object>>, no dynamic necessary.
Reflection on dynamic types is hard. Unless you can use duck typing (duck typing is when you know that the object can Duck(), even if you don't know what exactly it's, so you can do dynamic x = something; x.Duck(); ), then it's only semi-hard. If you don't trust me on this, you can try reading How do I reflect over the members of dynamic object?

Auto-magically add MySQL parameters in C#

At the moment, I'm creating a new method for each mysql query with parameters I want to get performed.
An example:
public DataTable RetreiveAllLinks(int id, int categoryid)
{
const string request =
#"
SELECT *
FROM links
WHERE id=?id,categoryid=?categoryid
ORDER by id DESC
";
using (var query = new MySqlCommand(request))
{
query.Parameters.AddWithValue("?id", id);
query.Parameters.AddWithValue("?categoryid", categoryid);
return _connection.RetreiveData(query);
}
}
This is really getting on my nerves, because I always end up with 10-30 queries, when two queries can do it for me, a simple method for retrieving non-parameter query, and a method for parameters. For example
public DataTable NonParameterCommand(string r)
{
var request = r;
using (var query = new MySqlCommand(request))
{
return _connection.RetreiveData(query);
}
}
What I want to do, is some regex'ing, where i would say e.g.
var pPattern = "^[\\?][\\w]";
and then a method with request and a list as parameters, and the list should be in the same order as the request parameters. So if the request is
`"SELECT * FROM test WHERE id=?id,categoryid=?categoryid"`
then my list would look like
var _dictionary = new Dictionary<string, string>();
_dictionary.Add(_parameter, _value);
and my method
ParameterCommand(string r, Dictionary dic)
but how exactly?

If you are using a dictionary, then the key would already be the parameter name. Why parse the query to extract them?
You could do:
public DataTable CommandWithParams(string sql, Dictionary<string, object> parameters)
{
using (var query = new MySqlCommand(sql))
{
foreach (var param in parameters)
{
query.Parameters.AddWithValue(param.Key, param.Value);
}
return _connection.RetreiveData(query);
}
}
Calling it:
var parameters = new Dictionary<string, object>()
{
{ "?param1", value1 },
{ "?param2", value2 }
};
var result = CommandWithParams(query, parameters);
But maybe you'd like to use a List instead of a Dictionary?
(I wouldn't recommend it, passing the parameters values based on their position would be error-prone)