We Keep Coding

Is there a way to filter a CSV file for data validation without for loops. (Lumenworks CSVReader)

I want to be able to filter out a CSV file and perform data validation on the filtered data. I imagine for loops, but the file has 2 million cells and it would take a long time. I am using Lumenworks CSVReader for accessing the file using C#.
I found this method csvfile.Where<> but I have no idea what to put in the parameters. Sorry I am still new to coding as well.
[EDIT] This is my code for loading the file. Thanks for all the help!
//Creating C# table from CSV data
var csvTable = new DataTable();
var csvReader = new CsvReader(newStreamReader(System.IO.File.OpenRead(filePath[0])), true);
csvTable.Load(csvReader);
//grabs header from the CSV data table
string[] headers = csvReader.GetFieldHeaders(); //this method gets the headers of the CSV file
string filteredData[] = csvReader.Where // this is where I would want to implement the where method, or some sort of way to filter the data
//I can access the rows and columns with this
csvTable.Rows[0][0]
csvTable.Columns[0][0]
//After filtering (maybe even multiple filters) I want to add up all the filtered data (assuming they are integers)
var dataToValidate = 0;
foreach var data in filteredData{
dataToValidate += data;
}
if (dataToValidate == 123)
//data is validated

I would read some of the documentation for the package you are using:
https://github.com/phatcher/CsvReader
https://www.codeproject.com/Articles/9258/A-Fast-CSV-Reader
To specifically answer the filtering question, so it only contains the data you are searching for consider the following:
var filteredData = new List<List<string>>();
using (CsvReader csv = new CsvReader(new StreamReader(System.IO.File.OpenRead(filePath[0])), true));
{
string searchTerm = "foo";
while (csv.ReadNextRecord())
{
var row = new List<string>();
for (int i = 0; i < csv.FieldCount; i++)
{
if (csv[i].Contains(searchTerm))
{
row.Add(csv[i]);
}
}
filteredData.Add(row);
}
}
This will give you a list of a list of string that you can enumerate over to do your validation
int dataToValidate = 0;
foreach (var row in filteredData)
{
foreach (var data in row)
{
// do the thing
}
}
--- Old Answer ---
Without seeing the code you are using to load the file, it might be a bit difficult to give you a full answer, ~2 Million cells may be slow no matter what what.
Your .Where comes from System.Linq
https://learn.microsoft.com/en-us/dotnet/api/system.linq.enumerable.where?view=net-6.0
A simple example using .Where
//Read the file and return a list of strings that match the where clause
public List<string> ReadCSV()
{
List<string> data = File.ReadLines(#"C:\Users\Public\Documents\test.csv");
.Select(line => line.Split(','))
// token[x] where x is the column number, assumes ID is column 0
.Select(tokens => new CsvFileStructure { Id = tokens[0], Value = tokens[1] })
// Where filters based on whatever you are looking for in the CSV
.Where(csvFileStructure => csvFileStructure.Id == "1")
.ToList();
return data;
}
// Map of your data structure
public class CsvFileStructure
{
public long Id { get; set; }
public string Name { get; set; }
public string Value { get; set; }
}
Modified from this answer:
https://stackoverflow.com/a/10332737/7366061

There is no csvreader.Where method. The "where" is part of Linq in C#. The link below shows an example of computing columns in a csv file using Linq:
https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/linq/how-to-compute-column-values-in-a-csv-text-file-linq

Is it possible to create a property from a string in C#?

I have a list of strings in C# and I want to loop through the list and add each item of the list as a property of my class.
For eg:
public class test
{
List<string> inputList = new List<string> {"add", "subtract", "multiply"};
foreach(var val in inputList)
{
//code to convert string to property
}
}
After I run the above code, when I create a new object of class test, I would like to get:
test.add or test.subtract etc. and I should be able to assign values to these properties.
Is this possible? If yes, could someone please suggest the best way of doing this?
Continuing from above, the aim here is to add API.
The list here is dynamically loaded. I should have started with a more apt list as an example.
public class test
{
List<string> inputList = new List<string> {"name", "age", "dob"};
foreach(var val in inputList)
{
//code to convert string to property of the class test
}
}
After the code is run I should be able to assign values to name, age and dob as user inputs i.e
test.name = "blah"
test.age = "123"
Since the list is dynamically updated, the items (and their number) in the list will vary. All the items in the list have to be added as a property of the class test and the user should be able to assign values to those properties at run time.

You can use dynamic to achieve this. While it's very good to be aware of the concept of dynamic and how to use it, it kills the benefits of using a strongly typed language and compile time checking, and should really only be used when the alternative is more painful:
List<string> inputList = new List<string> {"add", "subtract", "multiply"};
var builder = new System.Dynamic.ExpandoObject() as IDictionary<string, Object>;
foreach(var val in inputList)
{
builder.Add(val, $"{val}: some value here");
}
var output = (dynamic)builder;
Console.WriteLine(output.add);
Console.WriteLine(output.subtract);
Console.WriteLine(output.multiply);

C# - Nested Array/Data structures

Recently, I have been getting into C# (ASP.NET) and moving on from PHP. I want to achieve something like this:
mainArray (
array 1 (
'name' => 'example'
),
array 2 (
'name' => 'example2'
)
);
I know that you can use an Array in C# however, you must indicate the length of the Array before doing so which is where the problem is.
I want to loop through a Database in a Class function which returns an Array of all the columns, ie:
id, username, email.
I have tried:
public Array search_bustype(string match, string forthat)
{
db = new rkdb_07022016Entities2();
var tbl = (from c in db.tblbus_business select c).ToArray();
List<string> List = new List<string>();
int i = 0;
foreach (var toCheck in tbl)
{
if (toCheck.BusType.ToString() == match)
{
if (forthat == "Name")
{
List.Add(toCheck.Name);
}
if (forthat == "Address")
{
}
}
i++;
}
return List.ToArray();
}
But as you can see, I am having to only return the single column because the List is not multidimensional (can't be nested).
What can I use to solve this issue? I have looked at some links:
C# Arrays
StackOverflow post
But these again are an issue for my structure since I don't know how many index's I need in the Array when declaring it - The Database grows everyday.
Thanks in advance.

Try something like this. First, define a class for your business model.
public class Person
{
public string Name {get;set;}
public string Address {get;set;}
}
Then use a generic list instead of a string list.
public Person[] search_bustype(string match, string forthat)
{
var db = new rkdb_07022016Entities2();
List<Person> personList = new List<Person>();
foreach (var toCheck in db.tblbus_business.Where(b => b.BusType.ToString() == match))
{
var model = new Person { Name = toCheck.Name, Address = toCheck.Address };
personList.Add(model);
}
return personList.ToArray();
}
I'm not sure what you are trying to do with the forthat variable.

You can use a list of lists
IList<IList<string>> multiList;

Read all values from CSV into a List using CsvHelper

So I've been reading that I shouldn't write my own CSV reader/writer, so I've been trying to use the CsvHelper library installed via nuget. The CSV file is a grey scale image, with the number of rows being the image height and the number columns the width. I would like to read the values row-wise into a single List<string> or List<byte>.
The code I have so far is:
using CsvHelper;
public static List<string> ReadInCSV(string absolutePath)
{
IEnumerable<string> allValues;
using (TextReader fileReader = File.OpenText(absolutePath))
{
var csv = new CsvReader(fileReader);
csv.Configuration.HasHeaderRecord = false;
allValues = csv.GetRecords<string>
}
return allValues.ToList<string>();
}
But allValues.ToList<string>() is throwing a:
CsvConfigurationException was unhandled by user code
An exception of type 'CsvHelper.Configuration.CsvConfigurationException' occurred in CsvHelper.dll but was not handled in user code
Additional information: Types that inherit IEnumerable cannot be auto mapped. Did you accidentally call GetRecord or WriteRecord which acts on a single record instead of calling GetRecords or WriteRecords which acts on a list of records?
GetRecords is probably expecting my own custom class, but I'm just wanting the values as some primitive type or string. Also, I suspect the entire row is being converted to a single string, instead of each value being a separate string.

According to #Marc L's post you can try this:
public static List<string> ReadInCSV(string absolutePath) {
List<string> result = new List<string>();
string value;
using (TextReader fileReader = File.OpenText(absolutePath)) {
var csv = new CsvReader(fileReader);
csv.Configuration.HasHeaderRecord = false;
while (csv.Read()) {
for(int i=0; csv.TryGetField<string>(i, out value); i++) {
result.Add(value);
}
}
}
return result;
}

If all you need is the string values for each row in an array, you could use the parser directly.
var parser = new CsvParser( textReader );
while( true )
{
string[] row = parser.Read();
if( row == null )
{
break;
}
}
http://joshclose.github.io/CsvHelper/#reading-parsing
Update
Version 3 has support for reading and writing IEnumerable properties.

The whole point here is to read all lines of CSV and deserialize it to a collection of objects. I'm not sure why do you want to read it as a collection of strings. Generic ReadAll() would probably work the best for you in that case as stated before. This library shines when you use it for that purpose:
using System.Linq;
...
using (var reader = new StreamReader(path))
using (var csv = new CsvReader(reader))
{
var yourList = csv.GetRecords<YourClass>().ToList();
}
If you don't use ToList() - it will return a single record at a time (for better performance), please read https://joshclose.github.io/CsvHelper/examples/reading/enumerate-class-records

Please try this. This had worked for me.
TextReader reader = File.OpenText(filePath);
CsvReader csvFile = new CsvReader(reader);
csvFile.Configuration.HasHeaderRecord = true;
csvFile.Read();
var records = csvFile.GetRecords<Server>().ToList();
Server is an entity class. This is how I created.
public class Server
{
private string details_Table0_ProductName;
public string Details_Table0_ProductName
{
get
{
return details_Table0_ProductName;
}
set
{
this.details_Table0_ProductName = value;
}
}
private string details_Table0_Version;
public string Details_Table0_Version
{
get
{
return details_Table0_Version;
}
set
{
this.details_Table0_Version = value;
}
}
}

You are close. It isn't that it's trying to convert the row to a string. CsvHelper tries to map each field in the row to the properties on the type you give it, using names given in a header row. Further, it doesn't understand how to do this with IEnumerable types (which string implements) so it just throws when it's auto-mapping gets to that point in testing the type.
That is a whole lot of complication for what you're doing. If your file format is sufficiently simple, which yours appear to be--well known field format, neither escaped nor quoted delimiters--I see no reason why you need to take on the overhead of importing a library. You should be able to enumerate the values as needed with System.IO.File.ReadLines() and String.Split().
//pseudo-code...you don't need CsvHelper for this
IEnumerable<string> GetFields(string filepath)
{
foreach(string row in File.ReadLines(filepath))
{
foreach(string field in row.Split(',')) yield return field;
}
}

static void WriteCsvFile(string filename, IEnumerable<Person> people)
{
StreamWriter textWriter = File.CreateText(filename);
var csvWriter = new CsvWriter(textWriter, System.Globalization.CultureInfo.CurrentCulture);
csvWriter.WriteRecords(people);
textWriter.Close();
}

dynamicly move records in a dataset into a class is there a way to do it in c#?

assuming we use this on a table using "select * from table " ?
i want to create an array of classes
that use filed names (of the datatable) as class field names initalise them with the datatable
fields
is it possible ?
if so please help :)
soory for the spelling
my writen english is not that good
but i hope i was clear

This is a great time to use anonymous types
var query = (from row in DataTable.Rows.Cast<DataRow>()
select new
{
ID = row.Field<int>("ID"),
Name = row.Field<string>("Caption"),
\\etc etc etc
}).ToArray();
If you already have a class to store the results you could just as easily replace the anonymous type with that class (or use a Tuple in 4.0)
If you do this frequently you could write an extension method that uses reflection to match up data columns with the names of properties in the class array you are trying to generate

that use filed names (of the datatable) as class field names initalise them with the datatable fields
The only way to do that would be to generate a class at runtime, which is (a) not easy and (b) usually not very useful because the code that will use this class won't know the names of the class members.
You could use the dynamic features of C# 4 to build a list of ExpandoObjects:
Func<DataRow, dynamic> projection =
r =>
{
IDictionary<string, object> exp = new ExpandoObject();
foreach (DataColumn col in r.Table.Columns)
exp[col.ColumnName] = r[col];
return exp;
};
dynamic[] objects = dataTable.AsEnumerable().Select(projection).ToArray();
...
dynamic o = objects[0];
int id = o.Id;
string name = o.Name;
But then you loose the benefits of strong typing...
I think a more interesting idea would be to project the data rows to existing classes, based on the member names and data column names:
public T DataRowToObject<T>(DataRow row) where T : new()
{
var properties = typeof(T).GetProperties().Where(p => p.CanWrite);
var columns = row.Table.Columns.Cast<DataColumn>();
// Match properties of T with DataTable columns
var common =
from p in properties
from c in columns
where c.ColumnName == p.Name
&& p.PropertyType.IsAssignableFrom(c.DataType)
select p;
T obj = new T();
foreach (var prop in common)
{
if (!row.IsNull(prop.Name))
{
object value = row[prop.Name];
prop.SetValue(obj, value, null);
}
}
return obj;
}
...
MyClass[] objects = dataTable.AsEnumerable().Select(row => DataRowToObject(row)).ToArray();