We Keep Coding

C# Newtonsoft JSON - Deserializing DataTable with nested object

I have the following JSON which needs to be deserialized.
{
"$type":"Lib.TableModel, Lib",
"DataTable":[
{
"Column1":"1",
"Column2":{
"$type":"Lib.RelationCellValue, Lib",
"CellName":"C1",
"FileName":"test1.txt"
},
"Column3":{
"$type":"Lib.RelationCellValue, Lib",
"CellName":"C2",
"FileName":"test2.txt"
}
}
],
"Name":"table1"
}
As the values for Column2 and Column3 are objects, Newtonsoft throws an exception
"Unexpected JSON token when reading DataTable: StartObject"
although the object is annotated with type information.
I'm deserializing like this:
JsonSerializer serializer = new JsonSerializer
{
TypeNameHandling = TypeNameHandling.All,
NullValueHandling = NullValueHandling.Ignore
};
try
{
using (StreamReader sr = new StreamReader(path))
using (JsonReader reader = new JsonTextReader(sr))
{
model = serializer.Deserialize<T>(reader);
}
}
catch (Exception ex)
{
Console.WriteLine(ex.InnerException.ToString());
}
How can I provide a custom type converter for the Newtonsoft DataTable converter (or fix the issue)?
Edit: The TableModel is just a class with a DataTable property.
public class TableModel
{
private DataTable dataTable;
public DataTable DataTable
{
get => dataTable;
set
{
dataTable = value;
RaisePropertyChanged("DataTable");
}
}
}
The items in the DataTable are RelationCellValues:
public class RelationCellValue
{
public string CellName { get; set; }
public string FileName { get; set; }
}
Edit2: The number of columns in the DataTable is dynamic. Test code for generating a sample table:
public void LoadData()
{
dataTable = new DataTable();
DataColumn column;
column = new DataColumn("Column1");
column.DataType = typeof(string);
dataTable.Columns.Add(column);
column = new DataColumn("Column2");
column.DataType = typeof(RelationCellValue);
dataTable.Columns.Add(column);
column = new DataColumn("Column3");
column.DataType = typeof(RelationCellValue);
dataTable.Columns.Add(column);
// Header row
var row = dataTable.NewRow();
dataTable.Rows.Add(row);
//First row
row = dataTable.NewRow();
LoadCellValues(row, 1);
dataTable.Rows.Add(row);
//Second row
row = dataTable.NewRow();
LoadCellValues(row, 2);
dataTable.Rows.Add(row);
//Third row
row = dataTable.NewRow();
LoadCellValues(row, 3);
dataTable.Rows.Add(row);
}
public void LoadCellValues(DataRow dataRow, int ordinalNumber)
{
dataRow[0] = ordinalNumber.ToString();
dataRow[1] = new RelationCellValue() { CellName = "First Cell", FileName = "test.txt" };
dataRow[2] = new RelationCellValue() { CellName = "Second Cell", FileName = "test.txt" }; ;
}

How to convert string separated by new line and comma to DataTable in C#

I have a string like this:
"Product,Price,Condition
Cd,13,New
Book,9,Used
"
Which is being passed like this:
"Product,Price,Condition\r\Cd,13,New\r\nBook,9,Used"
How could I convert it to DataTable?
Trying to do it with this helper function:
DataTable dataTable = new DataTable();
bool columnsAdded = false;
foreach (string row in data.Split(new string[] { "\r\n" }, StringSplitOptions.None))
{
DataRow dataRow = dataTable.NewRow();
foreach (string cell in row.Split(','))
{
string[] keyValue = cell.Split('~');
if (!columnsAdded)
{
DataColumn dataColumn = new DataColumn(keyValue[0]);
dataTable.Columns.Add(dataColumn);
}
dataRow[keyValue[0]] = keyValue[1];
}
columnsAdded = true;
dataTable.Rows.Add(dataRow);
}
return dataTable;
However I don't get that "connecting cells with appropriate columns" part - my cells don't have ~ in string[] keyValue = cell.Split('~'); and I obviously get an IndexOutOfRange at DataColumn dataColumn = new DataColumn(keyValue[0]);

Based on your implementation, I have written the code for you, I have not tested it. But you can use the concept.
DataRow dataRow = dataTable.NewRow();
int i = 0;
foreach (string cell in row.Split(','))
{
if (!columnsAdded)
{
DataColumn dataColumn = new DataColumn(cell);
dataTable.Columns.Add(dataColumn);
}
else
{
dataRow[i] = cell;
}
i++;
}
if(columnsAdded)
{
dataTable.Rows.Add(dataRow);
}
columnsAdded = true;

You can do that simply with Linq (and actually there is LinqToCSV on Nuget, maybe you would prefer that):
void Main()
{
string data = #"Product,Price,Condition
Cd,13,New
Book,9,Used
";
var table = ToTable(data);
Form f = new Form();
var dgv = new DataGridView { Dock = DockStyle.Fill, DataSource = table };
f.Controls.Add(dgv);
f.Show();
}
private DataTable ToTable(string CSV)
{
DataTable dataTable = new DataTable();
var lines = CSV.Split(new char[] { '\n' }, StringSplitOptions.RemoveEmptyEntries);
foreach (var colname in lines[0].Split(','))
{
dataTable.Columns.Add(new DataColumn(colname));
}
foreach (var row in lines.Where((r, i) => i > 0))
{
dataTable.Rows.Add(row.Split(','));
}
return dataTable;
}

You can split given string into flattened string array in one call. Then you can iterate through the array and populate list of objects.
That part is optional, since you can immediately populate DataTable but I think it's way easier (more maintainable) to work with strongly-typed objects when dealing with DataTable.
string input = "Product,Price,Condition\r\nCd,13,New\r\nBook,9,Used";
string[] deconstructedInput = input.Split(new string[] { "\r\n", "," }, StringSplitOptions.None);
List<Product> products = new List<Product>();
for (int i = 3; i < deconstructedInput.Length; i += 3)
{
products.Add(new Product
{
Name = deconstructedInput[i],
Price = Decimal.Parse(deconstructedInput[i + 1]),
Condition = deconstructedInput[i + 2]
});
}
public class Product
{
public string Name { get; set; }
public decimal Price { get; set; }
public string Condition { get; set; }
}
So, products collection holds 2 objects which you can easily iterate over and populate your DataTable.
Note: This requires further checks to avoid possible runtime exceptions, also it is not dynamic. That means, if you have differently structured input it won't work.
DataTable dataTable = new DataTable();
dataTable.Columns.Add(new DataColumn(nameof(Product.Name)));
dataTable.Columns.Add(new DataColumn(nameof(Product.Price)));
dataTable.Columns.Add(new DataColumn(nameof(Product.Condition)));
foreach (var product in products)
{
var row = dataTable.NewRow();
row[nameof(Product.Name)] = product.Name;
row[nameof(Product.Price)] = product.Price;
row[nameof(Product.Condition)] = product.Condition;
dataTable.Rows.Add(row);
}

Cannot implicitly convert DateTime to Timespan

Sql Server has the variable TEST_TIME data type as Time(7)
I created the tables in C# and it automatically assigned the Timespan datatype.
Now, i'm trying to upload csv file data to the SQL database and it is giving me an error " Cannot implicitly convert DateTime to Timespan". What would be the best way to fix this?
The user first selects the CSV file:
private void button8_Click(object sender, EventArgs e)
{
try
{
using (OpenFileDialog openfiledialog1 = new OpenFileDialog()
{Filter = "Excel Workbook 97-2003|*.xls|Excel Workbook|*.xlsx|Excel Workbook|*.xlsm|Excel Workbook|*.csv|Excel Workbook|*.txt", ValidateNames = true })
{
--After some IFs--
else if (openfiledialog1.FilterIndex == 4)
{
DataTable oDataTable = null;
int RowCount = 0;
string[] ColumnNames = null;
string[] oStreamDataValues = null;
//using while loop read the stream data till end
while (!oStreamReader.EndOfStream)
{
String oStreamRowData = oStreamReader.ReadLine().Trim();
if (oStreamRowData.Length > 0)
{
oStreamDataValues = oStreamRowData.Split(',');
//Bcoz the first row contains column names, we will populate
//the column name by
//reading the first row and RowCount-0 will be true only once
if (RowCount == 0)
{
RowCount = 1;
ColumnNames = oStreamRowData.Split(',');
oDataTable = new DataTable();
//using foreach looping through all the column names
foreach (string csvcolumn in ColumnNames)
{
DataColumn oDataColumn = new DataColumn(csvcolumn.ToUpper(), typeof(string));
//setting the default value of empty.string to newly created column
oDataColumn.DefaultValue = string.Empty;
//adding the newly created column to the table
oDataTable.Columns.Add(oDataColumn);
}
}
else
{
//creates a new DataRow with the same schema as of the oDataTable
DataRow oDataRow = oDataTable.NewRow();
//using foreach looping through all the column names
for (int i = 0; i < ColumnNames.Length; i++)
{
oDataRow[ColumnNames[i]] = oStreamDataValues[i] == null ? string.Empty : oStreamDataValues[i].ToString();
}
//adding the newly created row with data to the oDataTable
oDataTable.Rows.Add(oDataRow);
}
}
}
result.Tables.Add(oDataTable);
//close the oStreamReader object
oStreamReader.Close();
//release all the resources used by the oStreamReader object
oStreamReader.Dispose();
dataGridView5.DataSource = result.Tables[oDataTable.TableName];
}
Here is the Code:
private void button9_Click(object sender, EventArgs e)
{
try
{
DataClasses1DataContext conn = new DataClasses1DataContext();
else if (textBox3.Text.Contains("GEN_EX"))
{
foreach (DataTable dt in result.Tables)
{
foreach (DataRow dr in dt.Rows)
{
GEN_EX addtable = new GEN_EX()
{
EX_ID = Convert.ToByte(dr[0]),
DOC_ID = Convert.ToByte(dr[1]),
PATIENT_NO = Convert.ToByte(dr[2]),
TEST_DATE = Convert.ToDateTime(dr[3]),
**TEST_TIME = Convert.ToDateTime((dr[4])),**
};
conn.GEN_EXs.InsertOnSubmit(addtable);
}
}
conn.SubmitChanges();
MessageBox.Show("File uploaded successfully");
}
else
{
MessageBox.Show("I guess table is not coded yet");
}
}
EDIT
The TEST_TIME represents HH:MM:SS
The Typed Data Set is defined as:
public virtual int Update(
byte EX_ID,
byte DOC_ID,
byte PATIENT_NO,
System.DateTime TEST_DATE,
System.TimeSpan TEST_TIME)

Based on your input that dr[4] represents time values in hours:minutes:seconds format I recommend following solution.
private TimeSpan GetTimeSpan(string timeString)
{
var timeValues = timeString.Split(new char[] { ':' });
//Assuming that timeValues array will have 3 elements.
var timeSpan = new TimeSpan(Convert.ToInt32(timeValues[0]), Convert.ToInt32(timeValues[1]), Convert.ToInt32(timeValues[2]));
return timeSpan;
}
Use above method as following.
else if (textBox3.Text.Contains("GEN_EX"))
{
foreach (DataTable dt in result.Tables)
{
foreach (DataRow dr in dt.Rows)
{
GEN_EX addtable = new GEN_EX()
{
EX_ID = Convert.ToByte(dr[0]),
DOC_ID = Convert.ToByte(dr[1]),
PATIENT_NO = Convert.ToByte(dr[2]),
TEST_DATE = Convert.ToDateTime(dr[3]),
**TEST_TIME = GetTimeSpan(dr[4].ToString()),**
};
conn.GEN_EXs.InsertOnSubmit(addtable);
}
}
conn.SubmitChanges();
MessageBox.Show("File uploaded successfully");
}
This should give your the value you want. You will face runtime issues if value of dr[4] is not in hours:minutes:seconds format. That I will leave it up to you.

First of all Timespan and DateTime are 2 differents type without implicit conversion available. Since Timespan is a time value between two DateTime, you need to know your referenced time (DateTime) used to start the mesure of your Timespan.
For example it could be from DateTime dtReferential = new DateTime(1900, 01, 01);
In order to give a SQL Timespan value you need to give it a C# Timespan! Change TEST_TIME value to a Timespan. And finally, give it the substracted value of your referenced time.
Using the previous example:
else if (textBox3.Text.Contains("GEN_EX"))
{
foreach (DataTable dt in result.Tables)
{
foreach (DataRow dr in dt.Rows)
{
GEN_EX addtable = new GEN_EX()
{
EX_ID = Convert.ToByte(dr[0]),
DOC_ID = Convert.ToByte(dr[1]),
PATIENT_NO = Convert.ToByte(dr[2]),
TEST_DATE = Convert.ToTimespan(dr[3]),
TEST_TIME = dtReferential.Subtract(Convert.ToDateTime(dr[4]))
};
conn.GEN_EXs.InsertOnSubmit(addtable);
}
}
conn.SubmitChanges();
MessageBox.Show("File uploaded successfully");
}

C# Reading CSV file with SQL conditions

I am using CsvHelper lib to read CSV file and I can successfully read the file with the lib. However I cannot use SQL condition to filter values. How can I do that without using SQL Server. I am really stuck on it.
It was very easy with Pandas and Pandasql libs in Python but it is being too hard in C#..
My Code:
public static void Main(string[] args)
{
var fileInfo = new FileInfo(#"filePath");
using (TextReader reader = fileInfo.OpenText())
using (var csvReader = new CsvReader(reader))
{
csvReader.Configuration.Delimiter = ",";
csvReader.Configuration.HasHeaderRecord = false;
csvReader.Configuration.IgnoreQuotes = true;
csvReader.Configuration.TrimFields = true;
csvReader.Configuration.WillThrowOnMissingField = false;
while (csvReader.Read())
{
var myStrinVar = csvReader.GetField<string>(0);
Console.Write(myStrinVar); //SELECT * FROM table...
}
}
}

I would suggest using LINQ to filter your results.
https://msdn.microsoft.com/en-us/library/bb397906.aspx
Say you have some class MyClass that you can serialize the lines in your file into.
For example:
public class MyClass
{
public int ID { get; set; }
}
var records = csv.GetRecords<MyClass>().ToList();
var filtered = records.Where(r => r.ID >= 10);
That example is a bit contrived but you can use any boolean expression you like in the where clause.

I know this is too late for OP, but the issue with the accepted answer is that you have to read in the entire result set to memory which may not be tenable for large files. Also, if you can extend this code below to get the top N rows without having to read the entire CSV if you find matches early in the file.
public static void Main(string[] args)
{
var fileInfo = new FileInfo(#"filePath");
var where = ""; //Code to set up where clause part of query goes here
using (TextReader reader = fileInfo.OpenText())
using (var csvReader = new CsvReader(reader))
{
csvReader.Configuration.Delimiter = ",";
csvReader.Configuration.HasHeaderRecord = false;
csvReader.Configuration.IgnoreQuotes = true;
csvReader.Configuration.TrimFields = true;
csvReader.Configuration.WillThrowOnMissingField = false;
DataTable dt = null;
while (csvReader.Read())
{
//Use the first row to initialize the columns.
if (dt == null)
{
dt = new DataTable();
for (var i = 0; i < csvReader.FieldCount; i++)
{
var fieldType = csvReader.GetFieldType(i);
DataColumn dc;
if (fieldType.IsNullableType())
{
dc = new DataColumn(csvReader.GetName(i), Nullable.GetUnderlyingType(fieldType));
dc.AllowDBNull = true;
}
else
dc = new DataColumn(csvReader.GetName(i), data.GetFieldType(i));
dt.Columns.Add(dc);
}
}
//Map DataReader to DataRow
var newRow = dt.Rows.Add();
foreach(DataColumn col in dt.Columns)
{
newRow[col.ColumnName] = csvReader[col.ColumnName];
}
//Create a temporary DataView and filter it with the where clause.
DataView dv = new DataView(dt);
dv.RowFilter = where;
var data = dv.Count > 0 ? dv[0] : null;
if(data != null)
{
//Row in here matches your where clause.
//Code to read this row or do something with it.
}
//Empty the temporary data table.
dt.Rows.Clear();
}
}
}

Convert IEnumerable string array to datatable

I have a csv file delimited with pipe(|). I am reading it using the following line of code:
IEnumerable<string[]> lineFields = File.ReadAllLines(FilePath).Select(line => line.Split('|'));
Now, I need to bind this to a GridView. So I am creating a dynamic DataTable as follows:
DataTable dt = new DataTable();
int i = 0;
foreach (string[] order in lineFields)
{
if (i == 0)
{
foreach (string column in order)
{
DataColumn _Column = new DataColumn();
_Column.ColumnName = column;
dt.Columns.Add(_Column);
i++;
//Response.Write(column);
//Response.Write("\t");
}
}
else
{
int j = 0;
DataRow row = dt.NewRow();
foreach (string value in order)
{
row[j] = value;
j++;
//Response.Write(column);
//Response.Write("\t");
}
dt.Rows.Add(row);
}
//Response.Write("\n");
}
This works fine. But I want to know if there is a better way to convert IEnumerable<string[]> to a DataTable. I need to read many CSVs like this, so I think the above code might have performance issues.

Starting from .Net 4:
use ReadLines.
DataTable FileToDataTable(string FilePath)
{
var dt = new DataTable();
IEnumerable<string[]> lineFields = File.ReadLines(FilePath).Select(line => line.Split('|'));
dt.Columns.AddRange(lineFields.First().Select(i => new DataColumn(i)).ToArray());
foreach (var order in lineFields.Skip(1))
dt.Rows.Add(order);
return dt;
}
(edit: instead this code, use the code of #Jodrell answer, This prevents double charging of the Enumerator).
Before .Net 4:
use streaming:
DataTable FileToDataTable1(string FilePath)
{
var dt = new DataTable();
using (var st = new StreamReader(FilePath))
{
// first line procces
if (st.Peek() >= 0)
{
var order = st.ReadLine().Split('|');
dt.Columns.AddRange(order.Select(i => new DataColumn(i)).ToArray());
}
while (st.Peek() >= 0)
dt.Rows.Add(st.ReadLine().Split('|'));
}
return dt;
}

since, in your linked example, the file has a header row.
const char Delimiter = '|';
var dt = new DataTable;
using (var m = File.ReadLines(filePath).GetEnumerator())
{
m.MoveNext();
foreach (var name in m.Current.Split(Delimiter))
{
dt.Columns.Add(name);
}
while (m.MoveNext())
{
dt.Rows.Add(m.Current.Split(Delimiter));
}
}
This reads the file in one pass.