We Keep Coding

Sylvan CSV Reader C# Check for Missing Column in CSV

#MarkPflug I have a requirement to read 12 columns out of 45 - 85 total columns. This is from multiple csv files (in the hundreds). But here is the problem, a lot of the times a column or two will be missing from some csv data files. How do I check in C# for a missing column in a csv file given I use the nuget package sylvan csv reader. Here is some code:
// Create a reader
CsvDataReader reader = CsvDataReader.Create(file, new CsvDataReaderOptions { ResultSetMode = ResultSetMode.MultiResult });
// Get column by name from csv. This is where the error occurs only in the files that have missing columns. I store these and then use them in a GetString(Ordinal).
reader.GetOrdinal("HomeTeam");
reader.GetOrdinal("AwayTeam");
reader.GetOrdinal("Referee");
reader.GetOrdinal("FTHG");
reader.GetOrdinal("FTAG");
reader.GetOrdinal("Division");
// There is more data here, but anyway you get the point.
// Here I run the reader and for each piece of data I run my database write method.
while (await reader.ReadAsync())
{
await AddEntry(idCounter.ToString(), idCounter.ToString(), attendance, referee, division, date, home_team, away_team, fthg, ftag, hthg, htag, ftr, htr);
}
I tried the following:
// This still causes it to go out of bounds.
if(reader.GetOrdinal("Division") < reader.FieldCount)
// only if the ordinal exists then assign it in a temp variable
else
// skip this column (set the data in add entry method to "")

Looking at the source, it appears that GetOrdinal throws if the column name isn't found or is ambiguous. As such I expect you could do:
int blah1Ord = -1;
try{ blah1Ord = reader.GetOrdinal("blah1"); } catch { }
int blah2Ord = -1;
try{ blah2Ord = reader.GetOrdinal("blah2"); } catch { }
while (await reader.ReadAsync())
{
var x = new Whatever();
if(blah1Ord > -1) x.Blah1 = reader.GetString(blah1Ord);
if(blah2Ord > -1) x.Blah2 = reader.GetString(blah2Ord);
}
And so on, so you effectively sound out whether a column exists - the ordinal remains -1 if it doesn't - and then use that to decide whether to read the column or not
Incidentally, I've been dealing with CSVs with poor/misspelled/partial header names, and I've found myself getting the column schema and searching it for partials, like:
using var cdr = CsvDataReader.Create(sr);
var cs = await cdr.GetColumnSchemaAsync();
var sc = StringComparison.OrdinalIgnoreCase;
var blah1Ord = cs.FirstOrDefault(c => c.ColumnName.Contains("blah1", sc))?.ColumnOrdinal ?? -1;

I started using the Sylvan library and it is really powerful.
Not sure if this could help you but if you use the DataBinder.Create<T> generic method from an entity, you can do the following to get columns in your CSV file that do not map to any of the entity properties:
var dataBinderOptions = new DataBinderOptions()
{
// AllColumns is required to throw UnboundMemberException
BindingMode = DataBindingMode.AllColumns,
};
IDataBinder<TEntity> binder;
try
{
binder = DataBinder.Create<TEntity>(dataReader, dataBinderOptions);
}
catch (UnboundMemberException ex)
{
// Use ex.UnboundColumns to get unmapped columnns
readResult.ValidationProblems.Add($"Unmapped columns: {String.Join(", ", ex.UnboundColumns)}");
return;
}

C# - check which element in a csv is not in an other csv and then write the elements to another csv

My task is to check which of the elements of a column in one csv are not included in the elements of a column in the other csv. There is a country column in both csv and the task is to check which countries are not in the secong csv but are in the first csv.
I guess I have to solve it with Lists after I read the strings from the two csv. But I dont know how to check which items in the first list are not in the other list and then put it to a third list.

There are many way to achieve this, for many real world CSV applications it is helpful to read the CSV input into a typed in-memory store there are standard libraries that can assist with this like CsvHelper as explained in this canonical post: Parsing CSV files in C#, with header
However for this simple requirement we only need to parse the values for Country form the master list, in this case the second csv. We don't need to manage, validate or parse any of the other fields in the CSVs
Build a list of unique Country values from the second csv
Iterate the first csv
Get the Country value
Check against the list of countries from the second csv
Write to the third csv if the country was not found
You can test the following code on .NET Fiddle
NOTE: this code uses StringWriter and StringReader as their interfaces are the same as the file reader and writers in the System.IO namespace. but we can remove the complexity associated with file access for this simple requirement
string inputcsv = #"Id,Field1,Field2,Country,Field3
1,one,two,Australia,three
2,one,two,New Zealand,three
3,one,two,Indonesia,three
4,one,two,China,three
5,one,two,Japan,three";
string masterCsv = #"Field1,Country,Field2
one,Indonesia,...
one,China,...
one,Japan,...";
string errorCsv = "";
// For all in inputCsv where the country value is not listed in the masterCsv
// Write to errorCsv
// Step 1: Build a list of unique Country values
bool csvHasHeader = true;
int countryIndexInMaster = 1;
char delimiter = ',';
List<string> countries = new List<string>();
using (var masterReader = new System.IO.StringReader(masterCsv))
{
string line = null;
if (csvHasHeader)
{
line = masterReader.ReadLine();
// an example of how to find the column index from first principals
if(line != null)
countryIndexInMaster = line.Split(delimiter).ToList().FindIndex(x => x.Trim('"').Equals("Country", StringComparison.OrdinalIgnoreCase));
}
while ((line = masterReader.ReadLine()) != null)
{
string country = line.Split(delimiter)[countryIndexInMaster].Trim('"');
if (!countries.Contains(country))
countries.Add(country);
}
}
// Read the input CSV, if the country is not in the master list "countries", write it to the errorCsv
int countryIndexInInput = 3;
csvHasHeader = true;
var outputStringBuilder = new System.Text.StringBuilder();
using (var outputWriter = new System.IO.StringWriter(outputStringBuilder))
using (var inputReader = new System.IO.StringReader(inputcsv))
{
string line = null;
if (csvHasHeader)
{
line = inputReader.ReadLine();
if (line != null)
{
countryIndexInInput = line.Split(delimiter).ToList().FindIndex(x => x.Trim('"').Equals("Country", StringComparison.OrdinalIgnoreCase));
outputWriter.WriteLine(line);
}
}
while ((line = inputReader.ReadLine()) != null)
{
string country = line.Split(delimiter)[countryIndexInInput].Trim('"');
if(!countries.Contains(country))
{
outputWriter.WriteLine(line);
}
}
outputWriter.Flush();
errorCsv = outputWriter.ToString();
}
// dump output to the console
Console.WriteLine(errorCsv);

Since you write about solving it with lists, I assume you can load those values from the CSV to the lists, so let's start with:
List<string> countriesIn1st = LoadDataFrom1stCsv();
List<string> countriesIn2nd = LoadDataFrom2ndCsv();
Then you can easily solve it with linq:
List<string> countriesNotIn2nd = countriesIn1st.Where(country => !countriesIn2nd.Contains(country)).ToList();
Now you have your third list with countries that are in first, but not in the second list. You can save it.

C# Read and split from .txt to struct array

I'm trying to make a basic login for my console app. I store the user data in a .txt file like this:
ID;Name;IsAdmin. The txt has several lines.
In the app I want to store user data in a struct User array. I can't seem to find the method to read the file, split and put the different data to the right place. This is what I have so far:
Loading user data to struct array
public static void LoadIDs()
{
int entries = FileHandling.CountRows(usersPath);
User[] users = new User[entries]; //Length depends on how many lines are in the .txt
for (int i = 0; i < users.Length; i++)
{
users[i] = new User(1,"a",false); //ID(int), name, isAdmin [This is where I want to put the data from the .txt]
}
}
Reading and spliting the text
public static string ReadFileToArray(string path)
{
String input = File.ReadAllText(path);
foreach (var record in input.Split('\n'))
{
foreach (var data in record.Split(';'))
{
return data;
}
}
return null;
}
I know that this doesn't work at all this way but my knowledge is limited yet and I cannot think of other solutions.

You have a better tool to store your users. Instead of an array (that forces you to know the length of the data loaded) you can use a List where you can add your elements while you read them.
Another point to change is the File.ReadAllText in File.ReadLines. This will allow to read line by line your file directly in the loop
public List<User> BuildUserList(string path)
{
List<User> result = new List<User>();
foreach (var record in File.ReadLines(path)
{
string[] data = record.Split(';'))
User current = new User();
current.ID = Convert.ToInt32(data[0]);
current.Name = data[1];
current.IsAdmin = Convert.ToBoolean(data[2]);
result.Add(current);
}
return result;
}
Now you can use the list like an array if you need
List<User> users = BuildUserList("yourfile.txt");
if(users.Count > 0)
{
Console.WriteLine("Name=" + users[0].Name);
}

If I were to assume your file especially each line having Id;Name;Admin values, I would write something like below to extract it. Please note that there are simple syntax out there but following logic will be helpful for beginners to understand how this could be achieved.
List<User> userList = new List<User>();
// Read the file located at c:\test.txt (this might be different in your case)
System.IO.StreamReader file = new System.IO.StreamReader(#"c:\test.txt");
string line;
while ((line = file.ReadLine()) != null)
{
//following logic will read each line and split by the separator before
// creating a new User instance. Remember to add more defensive logic to
// cover all cases
var extract = line.Split(';');
userList.Add(new User()
{
Id = extract[0],
Name = extract[1],
IsAdmin = extract[2]
});
}
file.Close();
//at this stage you will have List of User and converting it to array using following call
var userArray = userList.ToArray();

And just as another variant, a linq-solution could look like this:
var users = (
from string line in System.IO.File.ReadAllLines(#"..filepath..")
let parts = line.Split(';')
where parts.Length == 3
select new User() {
ID = Convert.ToInt32(parts[0]),
Name = parts[1],
IsAdmin = Convert.ToBoolean(parts[2])}
).ToArray();
This can be elegant and short, error handling may be a bit more difficult.

This will read your file lazily, so it can handle extremely huge files with ease (assuming the rest of your code can):
public IEnumerable<User> ReadUsers(string path)
{
return File.ReadLines(path)
.Select(l=>l.Split(';'))
.Select(l=> new User
{
Id = int.Parse(l[0]),
Name = l[1],
IsAdmin = bool.Parse(l[2])
});
}
or
public IEnumerable<User> ReadUsers(string path)
{
return File.ReadLines(path)
.Select(l=>l.Split(';'))
.Select(l=> new User(int.Parse(l[0]), l[1], bool.Parse(l[2])));
}

C# How to write on files with custom name according to string content?

I'm trying to learn some C# here. My goal is to create and write on multiple custom files which name varies based on a part of the string to be written. Below some examples:
Let's say strings to be written are basically rows of a csv file:
2019-10-28 16:14:14;;15.5;0;;3;false;false;0;111;123;;;10;false;1;2.5;;;;0;
2019-10-28 16:13:11;;18;0;;1;false;false;222;333;123;;;10;false;1;1;;;;0;G
2019-10-29 16:13:11;;18;0;;3;false;false;true;
As you may notice, first field of each string is a date, and that's and that is the key field to choose the name of the file to write to.
First two fields have same date, so both strings will be printed on a single file, the third one in a second file since it has different date.
Expected Result:
First File:
2019-10-28 16:14:14;;15.5;0;;3;false;false;0;111;123;;;10;false;1;2.5;;;;0;
2019-10-28 16:13:11;;18;0;;1;false;false;222;333;123;;;10;false;1;1;;;;0;
Second File:
2019-10-29 16:13:11;;18;0;;3;false;false;true;
Now I have multiple rows like those, and I'd like to print them on different files based on their first value.
I managed to create a class which might represent each row:
class Value {
public DateTime date = DateTime.Now;
public decimal cod = 0;
public decimal quantity = 0;
public decimal price = 0;
//Other irrelevant fields
}
And I also tried to develop a method to write a single Value on given File:
private static void WriteValue(Value content, string folder, string fileName) {
using(StreamWriter writer = new StreamWriter(Path.Combine(folder, fileName), true, Encoding.ASCII)) {
writer.Write(content.dataora.ToString("yyyyMMdd"));
writer.Write("0000");
writer.Write("I");
writer.Write("C");
writer.Write(content.codpro.ToString().PadLeft(14, '0'));
writer.Write(Convert.ToInt64(content.qta * 100).ToString().PadLeft(8, '0'));
writer.WriteLine();
}
}
And a Method to write Values them into files
static void WriteValues(List<Value> fileContent) {
//Once I got all Values of File in a List of Values, I try to write them in files
}
if(fileContent.Count > 0) {
foreach(Value riga in fileContent) {
//Temp Dates, used to compare previous Date in order to know if I have to write Value in new File or it can be written on same File
string dataTemp = riga.dataora.ToString("yyyy-MM-dd");
string lastData = string.Empty;
string FileName = "ordinivoa_999999." + DateTime.Now.ToString("yyMMddHHmmssfff");
//If lastData is Empty we are writing first value
if (string.IsNullOrEmpty(lastData)) {
WriteValue(riga, toLinfaFolder, FileName);
lastData = dataTemp;
}
//Else if lastData is equal as last managed date we write on same file
else if (lastData == dataTemp) {
WriteValue(riga, toLinfaFolder, FileName);
}
else {
//Else current date of Value is new, so we write it in another file
string newFileName = "ordinivoa_999999." + DateTime.Now.AddMilliseconds(1).ToString("yyMMddHHmmssfff");
WriteValue(riga, toLinfaFolder, newFileName);
lastData = dataTemp;
}
}
}
}
My issue is method above has strange behavior, writes first equal dates on a single file, which is good, but writes all other values in a single file, even if we have different dates.
How to make sure each value gets printed on in a single file only if has same date value?

You can group equal dates easily with a LINQ query
private static void WriteValues(List<Value> fileContent)
{
var dateGroups = fileContent
.GroupBy(v => $"ordinivoa_999999.{v.date:yyMMddHHmmssfff}");
foreach (var group in dateGroups) {
string path = Path.Combine(toLinfaFolder, group.Key);
using (var writer = new StreamWriter(path, true, Encoding.ASCII)) {
foreach (Value item in group) {
//TODO: write item to file
writer.WriteLine(...
}
}
}
}
Since a DateTime stores values in units of one ten-millionth of a second, two dates looking equal once formatted, might still be different. So I suggest grouping on the filename to avoid this effect. I used string interpolation to create and format the file name.
Don't open and close the file for each text line.
At the top of your code file you need a
using System.Linq;

You are on the right path declaring a class, but you're also doing a whole bunch of unnecessary stuff. Using LINQ this can be simplified by a great deal.
First I define a class, and since all you want to do is write each record, I would use a DateTime field, and a string field for the entire raw record.
class MyRecordOfSomeType
{
public DateTime Date { get; set; }
public string RawData { get; set; }
}
The DateTime filed is so that it'll come in handy when you're doing LINQ.
Now we iterate through your data, split using ;, then create your class instance list.
var data = new List<string>()
{
"2019-10-28 16:14:14;;15.5;0;;3;false;false;0;111;123;;;10;false;1;2.5;;;;0;",
"2019-10-28 16:13:11;;18;0;;1;false;false;222;333;123;;;10;false;1;1;;;;0;G",
"2019-10-29 16:13:11;;18;0;;3;false;false;true;"
};
var records = new List<MyRecordOfSomeType>();
foreach (var item in data)
{
var parts = item.Split(';');
DateTime.TryParse(parts[0], out DateTime result);
var rec = new MyRecordOfSomeType() { Date = result, RawData = item };
records.Add(rec);
}
Then we group by date. Note that it's important to group by the Date component of the DateTime structure, otherwise it will consider the Time component as well and you'll have more files than you need.
var groups = records.GroupBy(x => x.Date.Date);
Finally, iterate your groups, and write contents of each group to a new file.
foreach (var group in groups)
{
var fileName = string.Format("ordinivoa_999999_{0}.csv", group.Key.ToString("yyMMddHHmmssfff"));
File.WriteAllLines(fileName, group.Select(x => x.RawData));
}

C# Reading From A File

I have an assignment to read text in from a text file. The text is an inventory with department names followed by the quantity of items in the department and then items underneath the separate departments with the item name, quantity, and price. A part of the text file is shown here:
Stationary, 4
Notebook, 20, .99
Pens, 50, .50
Pencils, 25, 0.09
Post It Notes, 30, 4.99
Tools, 6
Band Saw, 3, 299.99
Cresent Wrench, 12, 8.49
Circular Saw, 5, 89.99
Tile Cutter, 2, 149.99
Screwdriver, 70, 2.99
Measuring Tape, 34, 10.99
I'm able to load the text file in just fine. My task is to take in user input for them decide which department they want to shop on. How am I able to display just the departments and then just the items of the desired department from the user? I have a method to output all of the departments and items shown below. This is my first time working with text files with C# so I have no idea what I am doing.
static void ReadDepartments(out List<Dept> s)
{
string line; // detail line read from file
string[] tokens; // break line up into tokens
string deptName; // name of department
int deptQuan; // quan of different items in dept
s = new List<Dept>();
try
{
using (StreamReader sr = new StreamReader(#"..\..\inventory.txt"))
{
while (sr.Peek() >=0)
{
List<Item> myItemList = new List<Item>(); // new instance of tmp List
line = sr.ReadLine();
tokens = line.Split(',');
deptName = tokens[0];
deptQuan = Convert.ToInt32(tokens[1]);
for (int i=0; i< deptQuan; i++)
{
// read each line of dept and build a list of items
line = sr.ReadLine();
tokens = line.Split(',');
Item myItem = new Item(tokens[0], Convert.ToInt32(tokens[1]), Convert.ToDouble(tokens[2]));
myItemList.Add(myItem);
}
s.Add(new Dept(deptName,deptQuan, myItemList));
}
}
}
catch (Exception e)
{
Console.WriteLine("Can't open file because {0}", e.Message);
}
}
static void PrintInventory(List<Dept> s)
{
foreach (Dept d in s)
{
Console.WriteLine("Dept: {0,-20} [{1} items]", d.Name, d.NumItems);
for (int i = 0; i < d.NumItems; i++)
Console.WriteLine(" {0,-15} {1,4} {2,7:$,##0.00}", d.GetItem(i).Name,
d.GetItem(i).Quan, d.GetItem(i).PriceEach);
}
}
I started a method to check if the desired department is a valid department shown below. Is there an easier way to implement the valid[] variable instead of including all of the department names? I will have to error check for valid items and that seems like it would be very tedious.
static string GetDepartment(string prompt)
{
string[] valid = {"BOOKS", "FOOD", "VIDEO", "SPORTS", "STATIONARY", "TOOLS"};
string ans = GetString(prompt, valid, "Inavlid response. Please choose a department.");
return ans;
}
static string GetString(string prompt, string[] valid, string error)
{
string response;
bool OK = false;
do
{
Console.Write(prompt);
response = Console.ReadLine().ToUpper();
foreach (string s in valid) if (response == s) OK = true;
if (!OK) Console.WriteLine(error);
}
while (!OK);
return response;
}

Your method that reads from the text file results in you having a List<Dept>. So you can generate a list of valid department names by going through the list of departments that you have read from the text file.

LINQ is great for searching through data and checking if items exists and what not.
Since you have all of your departments in a List you can query it via some different methods. Either search your raw data directly
using System.Linq;
...
List<Dept> departments;
...
departments.Any(dept => dept.Name == response);
Or if you want to send the names to your GetString method:
GetString(prompt, departments.Select(dept => dept.Name), ...);
...
string GetString(string prompt, IEnumerable<string> valids, string error)
...
valids.Any(valid => valid == response);
If you want to use the Department instead you can use FirstOrDefault instead (which also takes a predicate) and check for null if the item does not exist
Department found = departments.FirstOrDefault(dept => dept.Name == response);
if (found == null) //department name does not exist

If everything is ok on your code then you can add a if statement to check whether if it is your desired department info to print. I didn't check the whole code. You also can solve this problem with Linq (it will be more smart coding then) but your code seems to me as a starter code, so may be it will be a little inefficient but I hope it will solve your problem.
static void PrintInventory(List<Dept> s,string userInputDepartmentName)
{
if(s == null && s.Count <= 0)
return;
foreach (Dept d in s)
{
if(d.Name.Equals(userInputDepartmentName))
{
Console.WriteLine("Dept: {0,-20} [{1} items]", d.Name, d.NumItems);
for (int i = 0; i < d.NumItems; i++)
Console.WriteLine("{0,-15} {1,4} {2,7:$,##0.00}", d.GetItem(i).Name,d.GetItem(i).Quan, d.GetItem(i).PriceEach);
}
}
}