We Keep Coding

CsvHelper delimeter character same as end of line character

I've run into an issue while parsing some csv-like files that I know how to fix, but like to confirm if that's the appropriate way to do.
The file structure
The file I'm trying to parse has a structure similar to .csv in that it's values are separated with a delimeter (in my case it's |), but different to the ones I've previously seen is that it also has a delimeter at the end of the line, e.g:
Column1|Column2|Column3|
Row1Val1|Row1Val2|Row1Val3|
Row2Val1|Row2Val2|Row2Val3|
The issue
The problem arose when I wrote some unit tests to cover my service that wraps over the CsvHelper library. Apparently there is some issue when I provide the following configuration:
var config = new CsvConfiguration(CultureInfo.InvariantCulture)
{
Delimiter = "|",
HasHeaderRecord = true,
NewLine = "|\r\n"
};
With the above configuration, csvReader.GetRecords() returns no results. I believe that's because the order of operations for the parser is to first look for columns, then end of line - and it tries to parse empty column without realizing it's actually part of the delimeter.
(I can paste the code for the getRecords call as well, but it's basically generic code taken from examples - the only difference is I'm using System.IO.Abstractions library for easier unit testing)
The attempts to solve the problem
If I remove the NewLine configuration value, parser works fine when reading the file (even if it has end-of-line delimeter character at the end). Then, however, my "write CSV" tests break, since CsvHelper no longer is adding proper line endings to the file.
The question(s)
Is there any way I can configure CsvHelper to cover both cases with one configuration, or should I basically use two different configurations, depending on whether I'm writing to CSV or reading from it? This seems a little bit counter-intuitive for me, since it's basically the same format I'm trying to follow, but different configurations are expected?

You could manually write the empty column for each line and then you could keep the configuration the same for reading and writing.
void Main()
{
var config = new CsvConfiguration(CultureInfo.InvariantCulture)
{
Delimiter = "|"
};
var records = new List<MyClass>
{
new MyClass {Column1 = "Row1Val1", Column2 = "Row1Val2", Column3 = "Row1Val3"},
new MyClass {Column1 = "Row2Val1", Column2 = "Row2Val2", Column3 = "Row2Val3"}
};
using (var writer = new StreamWriter("file.csv"))
using (var csv = new CsvWriter(writer, config))
{
csv.WriteHeader<MyClass>();
csv.WriteField(string.Empty);
foreach (var record in records)
{
csv.NextRecord();
csv.WriteRecord(record);
csv.WriteField(string.Empty);
}
}
using (var reader = new StreamReader("file.csv"))
using (var csv = new CsvReader(reader, config))
{
var importRecords = csv.GetRecords<MyClass>();
importRecords.Dump();
}
}
public class MyClass
{
public string Column1 { get; set; }
public string Column2 { get; set; }
public string Column3 { get; set; }
}

CSV to Cascading Model using Entity Framework

I need to turn an automatic CSV file into multiple database columns using Entity Framework. It is set up so that each model has children. So that Animal contains a list of Types which contain a list of Classification. In this way Classification is a grandchild of Animal
Right now I have these three models that need to be filled by the CSV file. The file is formatted in the following way:
They are then pulled from the API into a Windows Desktop App as a cascading dropdown box. So far I've tried adding them to separate lists however that did not upload when using Entity Framework. The current way is to try to cascade down the list however I get an error
Sequence contains no events
Here is the portion of the code that I am having a problem with (had to edit due to work rules so classes are different):
var Animal = new List<AnimalModel>();
var lines = await ReadStreamAsync(new StreamReader(uploadModel.File.OpenReadStream()));
foreach(string l in lines)
{
Animal.Add(new AnimalModel
{
AnimalName = cells[0],
});
Animal.Last().Type.Add(new TypeModel
{
TypeName = cells[1],
});
Animal.Last().Type.Last().Classification.Add(new ClassificationModel
{
Type = Type.Last(),
ClassificationName = cells[2],
Color = cells[3],
Age = cells[4]
});
}

I resolved the issue. I needed to initialize the list within the code as I am not doing so within the model. The following worked:
var Animal = new List<AnimalModel>();
var lines = await ReadStreamAsync(new StreamReader(uploadModel.File.OpenReadStream()));
foreach(string l in lines)
{
Animal.Add(new AnimalModel
{
AnimalName = cells[0],
Type = new List<TypeModel>()
{
new TypeModel()
{
TypeName = cells[1]
}
}
});
And so on for the grandchild. I will have to clean this up as it is quite messy but this works for now.

How to trim all column values with CsvEngine.CsvToDataTable()?

I am using FileHelpers 3.3.1 to import CSV data and populate DataTables in my c# app. It works well, and here is how I'm calling it:
DataTable dt = CsvEngine.CsvToDataTable(fullPath, ',');
The problem is that some column values have padding, as in spaces to the left and/or right side of the values, and those spaces are not being trimmed. My CSV files are large and performance of my importer app is important, so I really want to avoid looping through the datatable after the fact and trimming every column value of every row.
Is there a way to invoke a "trim all column values automatically" during the call to CsvToDataTable()?
I know there is a FieldTrim attribute that does this very thing, but I cannot bind rigid classes to my CSV files because I have many different CSV files and they all have different column names and data types. So that's not a practical option for me. It seems like there would be a built-in way to trim using one of the generic CSV parsers like CsvToDataTable().
What is my best option?

The FileHelpers CsvEngine class is quite limited. It is a sealed class so you cannot easily inherit or override from it.
If you don't mind a hacky solution, the following works
// Set the internal TrimChars via reflection
public static class FileBaseExtensions
{
public static void SetTrimCharsViaReflection(this FieldBase field, Char [] value)
{
var prop = typeof(FieldBase).GetProperty("TrimChars", BindingFlags.NonPublic | BindingFlags.Instance);
prop.SetValue(field, value);
}
}
CsvOptions options = new CsvOptions("Records", ',', filename);
var engine = new CsvEngine(options);
foreach (var field in engine.Options.Fields)
{
field.SetTrimCharsViaReflection(new char[] { ' ', '\t' });
field.TrimMode = TrimMode.Both;
}
var dataTable = engine.ReadFileAsDT(filename);
But you would be better off using a standard FileHelperEngine and creating your own version of CsvClassBuilder (source code here) to create the mapping class. You would have to change the AddFields method as follows:
public override DelimitedFieldBuilder AddField(string fieldName, string fieldType)
{
base.AddField(fieldName, fieldType);
if (base.mFields.Count > 1)
{
base.LastField.FieldOptional = true;
base.LastField.FieldQuoted = true;
base.LastField.QuoteMode = QuoteMode.OptionalForBoth;
base.LastField.QuoteMultiline = MultilineMode.AllowForBoth;
// <New>
base.LastField.TrimMode = TrimMode.Both;
base.LastField.TrimChars = " \t"; // trim spaces and tabs
// </New>
}
return base.LastField;
}
If necessary you can lift the code for CsvToDataTable from the source code for CsvEngine which is here.

MVVM get data from text file

I can not get the logic how to search in a text file and then get the data I need using model view view model.
Basically, I have to make a dictionary app and I have word,language and description in the text file. Like:
cat;e English; it is a four leg animal
In the model I have a text box where the client writes a word and two other boxes, where language and description of the word should be shown.
I just can not get how to search in this file. I tried to search online but nothing seemed to meet my exact question.

Unless your file is going to change you can get away with reading the entire file up front when running your application and putting the data into lists of models for your view models.
As this is essentially a CSV file, and assuming each entry is a line, using a Semi-colon as the delimiter we can use the .Net CSV parser to process your file into your models:
Basic Model:
public class DictionaryEntryModel {
public string Word { get; set; }
public string Language { get; set; }
public string Description { get; set; }
}
Example view model with a constructor to fill out your models:
public class DictionaryViewModel {
// This will be a INotify based property in your VM
public List<DictionaryEntryModel> DictionaryEntries { get; set; }
public DictionaryViewModel () {
DictionaryEntries = new List<DictionaryEntryModel>();
// Create a parser with the [;] delimiter
var textFieldParser = new TextFieldParser(new StringReader(File.ReadAllText(filePath)))
{
Delimiters = new string[] { ";" }
};
while (!textFieldParser.EndOfData)
{
var entry = textFieldParser.ReadFields();
DictionaryEntries.Add(new DictionaryEntryModel()
{
Word = entry[0],
Language = entry[1],
Description = entry[2]
});
}
// Don't forget to close!
textFieldParser.Close();
}
}
You can now bind your view using the property DictionaryEntries and as long as your app is open it will preserve your full file as the list of DictionaryEntryModel.
Hope this helps!

I'm not addressing the MVVM part here, but just how to search the text file in order to get resulting data according to a search term, using case insensitive regex.
string dictionaryFileName = #"C:\Test\SampleDictionary.txt"; // replace with your file path
string searchedTerm = "Cat"; // Replace with user input word
string searchRegex = string.Format("^(?<Term>{0});(?<Lang>[^;]*);(?<Desc>.*)$", searchedTerm);
string foundTerm;
string foundLanguage;
string foundDescription;
using (var s = new StreamReader(dictionaryFileName, Encoding.UTF8))
{
string line;
while ((line = s.ReadLine()) != null)
{
var matches = Regex.Match(line, searchRegex, RegexOptions.IgnoreCase);
if (matches.Success)
{
foundTerm = matches.Groups["Term"].Value;
foundLanguage = matches.Groups["Lang"].Value;
foundDescription = matches.Groups["Desc"].Value;
break;
}
}
}
Then you can display the resulting strings to the user.
Note that this will work for typical input words, but it might produce strange results if the user inputs special characters that interfere with the regular expression syntax. Most of this might be corrected by utilizing Regex.Escape(searchedTerm).

Reading files from a txt file and placing contents in a List<T>

I have a file in my project folder containing the data for properties in a Vehicle class. If the class was hardcoded into the XAML it would look like this:
Vehicle v1 = new Car() { Make = "Ford", Model = "Fiesta", Price = 10000, Year = "1999", Colour = "Red", Mileage = 40000, Description = "Lovely red car, 4 wheel, optional steering wheel.", VehicleType = "Car" };
VehicleList.Add(v1);
I don't want to hardcode this class object in, Instead I want to read it from a .txt file, each part separated by a comma ',' and place these into each property and add that vehicle into the Vehicle List<> 'VehicleList' and then display this new read list into a listbox.
Both the hardcode and the .txt follow the same structure with the hardcode containing the variable names and the .txt file containing just the data.
Here is what I have so far, as you can see I tried using System.IO however I am open to alternative methods.
private void Button_Click(object sender, RoutedEventArgs e)
{
string location = #"PATH";
VehicleList = File.ReadAllLines(location).ToList();
var logFile = File.ReadAllLines(location);
foreach (var v[i] in logFile) VehicleList.Add(s);
}
}

public static List<Car> FromTextFile(string fileName)
{
var lines = File.ReadAllLines(fileName);
var cars = new List<Car>();
foreach (var line in lines)
{
var car = FromLine(line);
if (car == null)
continue;
cars.Add(car);
}
return cars;
}
private static Car FromLine(string line)
{
var values = line.Split(',');
if (values.Length != 8) // There is no specification on what to do, if the amount of items is not sufficient.
return null;
// There is also no specification on what order to use, or how to map the ordering.
return new Car
{
Make = values[0],
Model = values[1],
// There is also no specification on how to handle in-correct formats.
// This assumes the Price property to be type of int.
Price = int.Parse(values[2]),
Year = values[3],
Colour = values[4],
// Again; how to handle in-correct formats?
Mileage = int.Parse(values[5]),
Description = values[6],
VehicleType = values[7]
};
}
This code should do what you described. Please be aware of the limitations of this code;
It expects a file that contains some lines, in comma seperated format.
It expects lines to contain 8 non-empty items, otherwise those lines will not be mapped.
It assumes the comma-seperated items to be exactly in this order; Make, Model, Price, Year, Colour, Mileage, Description, VehicleType.
It will not handle in-correct formats for converted types (e.g int). Because you did not ask so.
Keep in mind that this is not a desired code, it is not well written.
Now, as to your code;
Firstly, you've two different values for lines. I don't know why. One of them should be ok.
Second, you try to loop through one of them using foreach, but it doesn't work like that. You don't give or take an index from the foreach loop. The correct what is just foreach (TYPE VARIABLE_NAME in COLLECTION). So your code would be foreach (var v /* not v[i] */ in logFile) // ...
And then you add this to a list which contains those seperated lines again. At the end of the day, you have a list that contains the same lines twice. What you should have done in the adding part is, you should have converted the line to a Car object. Now how you do that, is absolutely up to you. Here are some question that comes to my mind, when the task is something such as mapping a string collection to an object.
What do I do if some members do not exist? (i.e when there is 5 items in the line, even though we have 8 properties)
How do I know which property to map the i'th item? What is the ordering? Do I get it from a "Columns" line? Do I just assume some ordering of my desire? How do I handle, if the user does not follow those rules?
What do I do, when there is something wrong about formatting of some of the members?
Now, this is not really an un-common problem. Everyone somehow needs to transport the data from place A to place B. For your specific instance, it is from disk storage, to memory, in the form of an object.
How do people solve these sorts of problem? They invent generic ways of "serialization" and "deserialization". What are some examples? The most known ones and the most used ones that comes to my mind are XML (eXtensible Markup Language) and JSON (JavaScript Object Notation). What do these actually do? They serialize programming objects to non-binary string formats. Here is an example XML output for your Car class.
<?xml version=\"1.0\" encoding=\"utf-16\"?>
<Car xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\">
<Make>Ford</Make>
<Model>Fiesta</Model>
<Price>10000</Price>
<Year>1999</Year>
<Colour>Red</Colour>
<Mileage>40000</Mileage>
<Description>Lovely red car, 4 wheel, optional steering wheel.</Description>
<VehicleType>Car</VehicleType>
</Car>
Generated with this simple code;
var car = new Car
{
Make = "Ford",
Model = "Fiesta",
Price = 10000,
Year = "1999",
Colour = "Red",
Mileage = 40000,
Description = "Lovely red car, 4 wheel, optional steering wheel.",
VehicleType = "Car"
};
var serializer = new XmlSerializer(typeof(Car));
// https://stackoverflow.com/questions/2434534/serialize-an-object-to-string
using (StringWriter textWriter = new StringWriter())
{
serializer.Serialize(textWriter, car);
var output = textWriter.ToString();
}
And this is, how you would easily read "Car"s from a text file, if it was formatted correctly;
var serializer = new XmlSerializer(typeof(List<Car>));
// If your file should contain one car and one car only, you should change the above line to;
// var serilizer = new XmlSerializer(typeof(Car));
using (var reader = File.OpenRead(#"PATH"))
{
var cars = (List<Car>)serializer.Deserialize(reader);
reader.Close();
}
Similar code can be written for JSON. But I think you get the idea.
In short, most of the times, XML, JSON or a similar generic format is sufficient. When they are not, you can develop your very own format. But comma-seperated line is a very simplistic way of doing serialization. The reasons are exactly my question (see above). And XML, JSON answers those questions easily. Implementations do a very good job as well, most of the time.
Edit
Here is an answer to a very similar question. The answer to that question would high likely also answer your question. How to split csv whose columns may contain ,