We Keep Coding

DateTime.Parse date conversion is not consistent

I have an Excel sheet where one column has a date. I have C# code to take that column, convert it to a date and then insert it to a SQL database.
The conversion is done like this:
r.transactionDate = DateTime.Parse(Convert.ToString(xlRange.Cells[i, 1].Value));
The data transfer is working without an issue.
When the date in the Excel sheet is something like 25/05/2022 (25th of May), it is converted properly and the date I get in the SQL database is the 25/05/2022.
However, the problem is when the date is something like 12/05/2022 (12th May); it is converted to 05/12/2022 (05th of December).
How can I fix this issue?

It's an excel sheet downloaded from Paypal that has all the dates of payments. There is a date column with dates.
Paypal does not offer a download in XLS or XLSX format. What you have is, most likely, a CSV file that Excel can open. It appears as an Excel file in your file system because Excel has registered the CSV file extension.
CSV (Comma-Separated Values) is a text file with all values represented as strings, separated by the , character. It has been around for a very long time, but was never formally specified. Or more precisely, formal specifications for CSV have been created several times, and nobody really knows which one is the right one. CSV files that work perfectly with one program can be unreadable to another.
Excel's CSV import is notorious for using US date formats regardless of your computer's regional settings. Each value is separated and examined individually. If the value looks like a date format then Excel attempts to parse it as a US date. If the first set of digits is in the range 1-12 then it is interpreted as the month. If the first set of digits is 13 or more then it tries the regional date strings, and may fall back to day-first if that fails. (This varies apparently.)
For this reason (among others) I strongly recommend that you never open a CSV file via Excel automation. Ever. In fact you should never open a CSV file with dates in it using Excel unless you know that the file was generated with month-first date formats only. Even then it is a needless waste of resources to open a text file with Excel.
You can do better.
There are plenty of libraries out there that will help you with CSV import. I've used a few (CsvHelper is a reasonable starting place), but usually I just write something simple to do the job. Read the file a line at a time (StreamReader.ReadLine), split each line into a collection of values (via my SplitCSV method), then write a ParseCSV method for the type that takes a collection of strings and returns a configured object. That way I have direct control over the way the input data is interpreted without having Excel mess things up for me.
Here's a simple example:
const string InputFile = #"C:\Temp\test.csv";
static void Main()
{
var rows = ReadCSV(InputFile, RowData.ParseCSV);
foreach (var row in rows)
{
Console.WriteLine($"#{row.RowNum}, Date: {row.SomeDate}, Amount: ${row.Amount:#,0.00}");
}
}
class RowData
{
public int RowNum { get; set; }
public DateTime SomeDate { get; set; }
public decimal Amount { get; set; }
public static RowData ParseCSV(string[] values)
{
if (values is null || values.Length < 3)
return null;
if (!int.TryParse(values[0], out var rownum) ||
!DateTime.TryParse(values[1], out var somedate) ||
!decimal.TryParse(values[2], out var amount)
)
return null;
return new RowData
{
RowNum = rownum,
SomeDate = somedate,
Amount = amount
};
}
}
static IEnumerable<T> ReadCSV<T>(string filename, Func<string[], T> parser, bool skipHeaders = true)
where T : class
{
bool first = true;
foreach (var line in Lines(filename))
{
if (first)
{
first = false;
if (skipHeaders)
continue;
}
T curr = null;
try
{
var values = SplitCSV(line);
curr = parser(values);
}
catch
{
// Do something here if you care about bad data.
}
if (curr != null)
yield return curr;
}
}
static string[] SplitCSV(string line)
{
var sb = new StringBuilder();
return internal_split().ToArray();
// The actual split, done as an enumerator.
IEnumerable<string> internal_split()
{
bool inQuote = false;
foreach (char c in line)
{
if (c == ',' && !inQuote)
{
// yield value
yield return sb.ToString();
sb.Clear();
}
else if (c == '"')
inQuote = !inQuote;
else
sb.Append(c);
}
// yield last field
yield return sb.ToString();
}
}
static IEnumerable<string> Lines(string filename)
{
string line;
using var reader = File.OpenText(filename);
while ((line = reader.ReadLine()) != null)
{
yield return line;
}
}
(It's not the best way, but it's a way to do it.)

Getting specific row and column value in .csv file in c#

I have a .csv file that looks like this:
Index,Time,value
1,0,20
2,1,30
What I want to do is to open the .csv file in C# and then read it by getting row = 0, column = 0 which would give me value 1 in the case above. Much like this:
public double GetVal(int row, int column)
{
...
return val;
}
I have looked around for a solution, for example this one: Remove values from rows under specific columns in csv file
But I need to be able to specify both the column and row in the function to get the specific value.

In case CSV file is simple one (no quotations), you can try Linq:
using System.IO;
using System.Linq;
...
public double GetVal(int row, int column) {
return File
.ReadLines(#"c:\MyFile.csv") //TODO: put the right name here
.Where(line => !string.IsNullOrWhiteSpace(line)) // to be on the safe side
.Skip(1) // Skip line with Titles
.Skip(row)
.Select(line => double.Parse(line.Split(',')[column]))
.First();
}
Note, that this code re-reads the file; you may want to read the file once:
string[][] m_Data = File
.ReadLines(#"c:\MyFile.csv")
.Where(line => !string.IsNullOrWhiteSpace(line))
.Skip(1)
.Select(line => line.Split(','))
.ToArray();
...
public double GetVal(int row, int column) => double.Parse(m_Data[row][col]);

Although you can write simple code by yourself, I would suggest you using dedicated CSV library for this like https://www.nuget.org/packages/LumenWorksCsvReader/
There are tons of edge cases like values escaping with double quotes, multiline values in CSV file format.
But if you totally control your files and they are small, you can read all lines at once and parse them. Something like this to get all lines from file and then split every line by ',' character
var lines = File.ReadAllLines('your file');

Quick answer, without considering the performance issue (e.g. the file read should happen once, and other issue like index checking to avoid overflows.
public double GetVal(int row, int column)
{
double output;
using (var reader = new StreamReader("filename.csv"))
{
int m = 1;
while (!reader.EndOfStream)
{
if(m==row)
{
var splits = rd.ReadLine().Split(',');
//You need check the index to avoid overflow
output = double.Parse(splits[column]);
return output;
}
m++;
}
}
return output;
}

Writing Japanese string to excel using OpenXml

I am trying to create Excel file by reading data from the database. One of the columns contains Japanese text. While writing that column to excel cell and saving workbook gives following error (which makes sense as the characters are not valid xml chars ) :'', hexadecimal value 0x0B, is an invalid character.
I am writing the string as following to the excel cell using DocumentFormat.OpenXml package.
var excelCell = new Cell();
var cellValue = dtRow[col.Name].ToString();
var inlineStr = new InlineString(new Text(cellValue));
excelCell.DataType = CellValues.InlineString;
excelCell.InlineString = inlineStr;
What needs to be done to write Japanese characters to the excel using OpenXml in C#

Ok. Found the right way. Putting it as answer so that it can be helpful.
To add text to excel which is not allowed as valid xml, add the text as SharedString to the SharedStringTable
var index = InsertSharedStringItem(text, shareStringPart);
excelCell.CellValue = new CellValue(index.ToString());
excelCell.DataType = new EnumValue<CellValues>(CellValues.SharedString);
private static int InsertSharedStringItem(string text, SharedStringTablePart shareStringPart)
{
// If the part does not contain a SharedStringTable, create one.
if (shareStringPart.SharedStringTable == null)
{
shareStringPart.SharedStringTable = new SharedStringTable();
}
int i = 0;
// Iterate through all the items in the SharedStringTable. If the text already exists, return its index.
foreach (SharedStringItem item in shareStringPart.SharedStringTable.Elements<SharedStringItem>())
{
if (item.InnerText == text)
{
return i;
}
i++;
}
// The text does not exist in the part. Create the SharedStringItem and return its index.
shareStringPart.SharedStringTable.AppendChild(new SharedStringItem(new DocumentFormat.OpenXml.Spreadsheet.Text(text)));
shareStringPart.SharedStringTable.Save();
return i;
}
Full documentation for adding text as shared string to excel using OpenXml
https://msdn.microsoft.com/en-us/library/office/cc861607.aspx

How to validate a .csv file before storage in C#?

I have some .csv files which I am parsing before storing in database.
I would like to make application more robust, and perform validation upon the .csv files before save in the database.
So I am asking you guys if you have some good links, or code examples, patterns, or advice on how to do this?
I will paste an example of my .csv file below. The different data fields in the .csv file are separated by tabs. Each new row of data is on a new line.
I have been thinking a little about the things I should validate against and came up with the list below (I am very open for other suggestions, in case you have anything which you think should be added to the list?)
Correct file encoding.
That file is not empty.
Correct number of lines/columns.
correct number/text/date formats.
correct number ranges.
This is how my .csv file looks like (file with two lines, data on one line is separated by tabs).
4523424 A123456 GT-P1000 mobile phone Samsung XSD1234 135354191325234
345353 A134211 A8181 mobile phome HTC S4112-ad3 111911911932343
The string representation of above looks like:
"4523424\tA123456\tGT-P1000\tmobile phone\tSamsung\tXSD1234\t135354191325234\r
\n345353\tA134211\tA8181\tmobile phome\tHTC\tS4112-ad3\t111911911932343\r\n"
So do you have any good design, links, patterns, code examples, etc. on how to do this in C#?

I do like this:
Create a class to hold each parsed line with expected type
internal sealed class Record {
public int Field1 { get; set; }
public DateTime Field2 { get; set; }
public decimal? PossibleEmptyField3 { get; set; }
...
}
Create a method that parses a line into the record
public Record ParseRecord(string[] fields) {
if (fields.Length < SomeLineLength)
throw new MalformadLineException(...)
var record = new Record();
record.Field1 = int.Parse(fields[0], NumberFormat.None, CultureInvoice.InvariantCulture);
record.Field2 = DateTime.ParseExact(fields[1], "yyyyMMdd", CultureInvoice.InvariantCulture);
if (fields[2] != "")
record.PossibleEmptyField3 = decimal.Parse(fields[2]...)
return record;
}
Create a method parsing the entire file
public List<Record> ParseStream(Stream stream) {
var tfp = new TextFileParser(stream);
...
try {
while (!tfp.EndOfData) {
records.Add(ParseRecord(tfp.ReadFields());
}
}
catch (FormatException ex) {
... // show error
}
catch (MalformadLineException ex) {
... // show error
}
return records;
}
And then I create a number of methods validating the fields
public void ValidateField2(IEnumerable<Record> records) {
foreach (var invalidRecord in records.Where(x => x.Field2 < DateTime.Today))
... // show error
}
I have tried various tools but since the pattern is straight forward they don't help much.
(You should use a tool to split the line into fields)

You can use FileHelpers a free/open source .Net library to deal with CSV and many other file formats.

adrianm and Nipun Ambastha
Thank you for your response to my question.
I solved my problem by writing a solution to validate my .csv file myself.
It's quite possible a more elegant solution could be made by making use of adrianm's code, but I didn't do that, but I am encouraging to give adrianm's code a look.
I am validating the list below.
Empty file
new FileInfo(dto.AbsoluteFileName).Length == 0
Wrong formatting of file lines.
string[] items = line.Split('\t');
if (items.Count() == 20)
Wrong datatype in line fields.
int number;
bool isNumber = int.TryParse(dataRow.ItemArray[0].ToString(), out number);
Missing required line fields.
if (dataRow.ItemArray[4].ToString().Length < 1)
To work through the contents of the .csv file I based my code on this code example:
http://bytes.com/topic/c-sharp/answers/256797-reading-tab-delimited-file

Probably you should take a look to
http://www.codeproject.com/Articles/9258/A-Fast-CSV-Reader
We have been using this in our projects, its quite robust and does what it says.

How to format and read CSV file?

Here is just an example of the data I need to format.
The first column is simple, the problem the second column.
What would be the best approach to format multiple data fields in one column?
How to parse this data?
Important*: The second column needs to contain multiple values, like in an example below
Name Details
Alex Age:25
Height:6
Hair:Brown
Eyes:Hazel

A csv should probably look like this:
Name,Age,Height,Hair,Eyes
Alex,25,6,Brown,Hazel
Each cell should be separated by exactly one comma from its neighbor.
You can reformat it as such by using a simple regex which replaces certain newline and non-newline whitespace with commas (you can easily find each block because it has values in both columns).

A CSV file is normally defined using commas as field separators and CR for a row separator. You are using CR within your second column, this will cause problems. You'll need to reformat your second column to use some other form of separator between multiple values. A common alternate separator is the | (pipe) character.
Your format would then look like:
Alex,Age:25|Height:6|Hair:Brown|Eyes:Hazel
In your parsing, you would first parse the comma separated fields (which would return two values), and then parse the second field as pipe separated.

This is an interesting one - it can be quite difficult to parse specific format files which is why people often write specific classes to deal with them. More conventional file formats like CSV, or other delimited formats are [more] easy to read because they are formatted in a similar way.
A problem like the above can be addressed in the following way:
1) What should the output look like?
In your instance, and this is just a guess, but I believe you are aiming for the following:
Name, Age, Height, Hair, Eyes
Alex, 25, 6, Brown, Hazel
In which case, you have to parse out this information based on the structure above. If it's repeated blocks of text like the above then we can say the following:
a. Every person is in a block starting with Name Details
b. The name value is the first text after Details, with the other columns being delimited in the format Column:Value
However, you might also have sections with addtional attributes, or attributes that are missing if the original input was optional, so tracking the column and ordinal would be useful too.
So one approach might look like the following:
public void ParseFile(){
String currentLine;
bool newSection = false;
//Store the column names and ordinal position here.
List<String> nameOrdinals = new List<String>();
nameOrdinals.Add("Name"); //IndexOf == 0
Dictionary<Int32, List<String>> nameValues = new Dictionary<Int32 ,List<string>>(); //Use this to store each person's details
Int32 rowNumber = 0;
using (TextReader reader = File.OpenText("D:\\temp\\test.txt"))
{
while ((currentLine = reader.ReadLine()) != null) //This will read the file one row at a time until there are no more rows to read
{
string[] lineSegments = currentLine.Split(new[] { " " }, StringSplitOptions.RemoveEmptyEntries);
if (lineSegments.Length == 2 && String.Compare(lineSegments[0], "Name", StringComparison.InvariantCultureIgnoreCase) == 0
&& String.Compare(lineSegments[1], "Details", StringComparison.InvariantCultureIgnoreCase) == 0) //Looking for a Name Details Line - Start of a new section
{
rowNumber++;
newSection = true;
continue;
}
if (newSection && lineSegments.Length > 1) //We can start adding a new person's details - we know that
{
nameValues.Add(rowNumber, new List<String>());
nameValues[rowNumber].Insert(nameOrdinals.IndexOf("Name"), lineSegments[0]);
//Get the first column:value item
ParseColonSeparatedItem(lineSegments[1], nameOrdinals, nameValues, rowNumber);
newSection = false;
continue;
}
if (lineSegments.Length > 0 && lineSegments[0] != String.Empty) //Ignore empty lines
{
ParseColonSeparatedItem(lineSegments[0], nameOrdinals, nameValues, rowNumber);
}
}
}
//At this point we should have collected a big list of items. We can then write out the CSV. We can use a StringBuilder for now, although your requirements will
//be dependent upon how big the source files are.
//Write out the columns
StringBuilder builder = new StringBuilder();
for (int i = 0; i < nameOrdinals.Count; i++)
{
if(i == nameOrdinals.Count - 1)
{
builder.Append(nameOrdinals[i]);
}
else
{
builder.AppendFormat("{0},", nameOrdinals[i]);
}
}
builder.Append(Environment.NewLine);
foreach (int key in nameValues.Keys)
{
List<String> values = nameValues[key];
for (int i = 0; i < values.Count; i++)
{
if (i == values.Count - 1)
{
builder.Append(values[i]);
}
else
{
builder.AppendFormat("{0},", values[i]);
}
}
builder.Append(Environment.NewLine);
}
//At this point you now have a StringBuilder containing the CSV data you can write to a file or similar
}
private void ParseColonSeparatedItem(string textToSeparate, List<String> columns, Dictionary<Int32, List<String>> outputStorage, int outputKey)
{
if (String.IsNullOrWhiteSpace(textToSeparate)) { return; }
string[] colVals = textToSeparate.Split(new[] { ":" }, StringSplitOptions.RemoveEmptyEntries);
List<String> outputValues = outputStorage[outputKey];
if (!columns.Contains(colVals[0]))
{
//Add the column to the list of expected columns. The index of the column determines it's index in the output
columns.Add(colVals[0]);
}
if (outputValues.Count < columns.Count)
{
outputValues.Add(colVals[1]);
}
else
{
outputStorage[outputKey].Insert(columns.IndexOf(colVals[0]), colVals[1]); //We append the value to the list at the place where the column index expects it to be. That way we can miss values in certain sections yet still have the expected output
}
}
After running this against your file, the string builder contains:
"Name,Age,Height,Hair,Eyes\r\nAlex,25,6,Brown,Hazel\r\n"
Which matches the above (\r\n is effectively the Windows new line marker)
This approach demonstrates how a custom parser might work - it's purposefully over verbose as there is plenty of refactoring that could take place here, and is just an example.
Improvements would include:
1) This function assumes there are no spaces in the actual text items themselves. This is a pretty big assumption and, if wrong, would require a different approach to parsing out the line segments. However, this only needs to change in one place - as you read a line at a time, you could apply a reg ex, or just read in characters and assume that everything after the first "column:" section is a value, for example.
2) No exception handling
3) Text output is not quoted. You could test each value to see if it's a date or number - if not, wrap it in quotes as then other programs (like Excel) will attempt to preserve the underlying datatypes more effectively.
4) Assumes no column names are repeated. If they are, then you have to check if a column item has already been added, and then create an ColName2 column in the parsing section.