We Keep Coding

How to import csv file which contains qoute signs and the delimiter is coma?

So i am having a problem importing a csv file, i want to make an object from the columns but i cant read in properly.
So the header line looks like this: Title,Year,Genre,Rating,Votes,Directors
The data line looks like this: The Last of Us: Mass Effect 2,2010,"Action, Adventure, Drama",9.5,19961,Casey Hudson
The problem is that, i get the exception "Input string is not in correct form"
I am using coma as delimiter, is there a way to make quotes as delimiters too?
Also, what are in the qoutes belongs to the Genre attribute.
I am using this code as the CsvParser right now:
using Games.Models;
using System.Globalization;
using System.Text;
namespace Games.Utils
{
public class CsvParser
{
private readonly string _path;
public char Delimiter { get; set; } = ',';
public bool SkipFirst { get; set; } = true;
public bool Verbose { get; set; } = true;
public NumberFormatInfo NumberFormatInfo { get; private set; } = new NumberFormatInfo();
public Encoding Encoding { get; set; } = Encoding.Default;
public CsvParser(string path) => _path = path;
public IEnumerable<Game> StreamParseGames() => GenerateGames(Enumerables.EnumerateStreamReaderLines(new(_path, Encoding)));
public IEnumerable<Game> TextParseGames() => GenerateGames(File.ReadAllLines(_path, Encoding));
private IEnumerable<Game> GenerateGames(IEnumerable<string> lineProvider)
{
if (SkipFirst) lineProvider = lineProvider.Skip(1);
int lineNum = SkipFirst ? 1 : 0;
foreach (var line in lineProvider)
{
string[] parts = line.Split(Delimiter);
Game game;
try
{
game = new()
{
Title = parts[0],
Year = Convert.ToInt32(parts[1], NumberFormatInfo),
Genre = parts[2],
Rating = Convert.ToDouble(parts[3], NumberFormatInfo),
Votes = Convert.ToDouble(parts[4], NumberFormatInfo),
Directors = parts[5],
};
}
catch (FormatException e)
{
if (Verbose) Console.WriteLine($"Line {lineNum + 1:000000} omitted due: {e.Message}");
continue;
}
catch (IndexOutOfRangeException e)
{
if (Verbose) Console.WriteLine($"Line {lineNum + 1:000000} omitted due: {e.Message}");
continue;
}
finally
{
++lineNum;
}
yield return game;
}
}
}
}

I'd suggest you use CsvHelper which can deal with that instead of rolling your own CSV parser.
using CsvHelper;
using CsvHelper.Configuration;
var config = new CsvConfiguration(CultureInfo.InvariantCulture)
{
Delimiter = ",",
};
using (var reader = new StreamReader("path\\to\\file.csv"))
using (var csv = new CsvReader(reader, config))
{
var records = csv.GetRecords<Foo>();
}

Using CSVhelper to update a single column into a previously written CSV file . I have the code in Java but can't translate it into C#

Essentially I have to read and update the CSVfile (only one column) with the current date after the test finishes executing(ie,there are some values written in at the start of the test execution and then I need to update the same file to input another value). I also have a DateTime error which isn't getting resolved no matter what I try.
Sample of CSV start of test
RunId ProductArea Product Component PageObject Control TimeTakenByLocatorJson
Run_645987 R201 BN2018.5 N778 BC1 C143
CSV one column Needs to get updated after test
( TimeTakenByLocatorJson)
RunId ProductArea Product Component PageObject Control TimeTakenByLocatorJson
Run_645987 R201 BN2018.5 N778 BC1 C143 2021-07-19
I've been trying to update a CSV file using CSVhelper. The code I have is in Java and when I tried translating the same code in C# it doesn't work.
This is the code in Java
public synchronized void writeEndCSV(String runId)
{
CSVWriter csvWriter = null;
try
{
String setupCSVLocation = Reporting.getSetupCSVLocation();
CSVReader csvReader = new CSVReader(new FileReader(setupCSVLocation));
List<String[]> records = csvReader.readAll();
for(int i=0;i<records.size();i++)
{
if(records.get(i)[SETUP_RUNID].equalsIgnoreCase(runId));
{
records.get(i)[SETUP_TimeTakenByLocatorJSON] = Reporting.getExecutionEndDate();
}
}
csvReader.close();
csvWriter = new CSVWriter(new FileWriter(setupCSVLocation));
csvWriter.writeAll(records);
csvWriter.flush();
csvWriter.close();
}
catch (Exception e)
{
e.printStackTrace();
}
}
This is my code in C# (I'm new to .Net so I'm not sure about many parts)
public void writeEnd(string runId)
{
var records = Enumerable.Empty<LocatorTime>();
try
{
var config = new CsvConfiguration(CultureInfo.InvariantCulture)
{
// Don't write the header again.
HasHeaderRecord = false,
};
using (var reader = new StreamReader(#"D:\Reports\" + runId + #"\LocatorTime.csv"))
using (var csv = new CsvReader(reader, config))
{
//csv.Context.RegisterClassMap<LocatorTime>();
records = csv.GetRecords<LocatorTime>().ToList();
foreach (var record in records)
{
if (record.RunID == runId)
{
record.TimeTakenByLocatorJSON = DateTime.Now;
}
// Console.WriteLine("inside loop");
}
}//Endof Stream Reader
using (var stream = File.Open(#"D:\Reports\" + runId + #"\LocatorTime.csv", FileMode.Append)) //not sure what the file mode should be
using (var writer = new StreamWriter(stream))
using (var csv = new CsvWriter(writer, CultureInfo.InvariantCulture))
{
csv.WriteRecords(records);
}
}
catch (Exception e)
{
Console.WriteLine(e);
}
}//end func writeEnd
This is the class used for the csv file & are also the column names in the csvfile
public class LocatorTime
{
public string RunID { get; set; }
public string ProductArea { get; set; }
public string Product { get; set; }
public string Component { get; set; }
public string PageObject { get; set; }
public string Control { get; set; }
public DateTime TimeTakenByLocatorJSON //only this value needs to be written for update at end of exec
{
get;
set;
}/*error because of DateTime datatype how to resolve?*/ }//LocatorTimeClass
/*public void SetExeDate() //tried this for removing DateTime error, didn't work
{
DateTime today = DateTime.Today; // As DateTime
string s_today = today.ToString("yyyy-MM-dd"); // As String
//TimeTakenByLocatorJSON = s_today.Trim();
TimeTakenByLocatorJSON = Convert.ToDateTime(s_today);}
*/
public sealed class LocatorTimeMap : ClassMap<LocatorTime> //is mapping helpful for updating? currently commented out
{
public LocatorTimeMap()
{
Map(m => m.RunID).Index(0);
Map(m => m.ProductArea).Index(1);
Map(m => m.Product).Index(2);
Map(m => m.Component).Index(3);
Map(m => m.PageObject).Index(4);
Map(m => m.Control).Index(5);
Map(m => m.TimeTakenByLocatorJSON).Index(6); //error
}
}
I had used the below link as reference for trying to update the CSV file hence the use of "HasHeaderRecord = false"
https://joshclose.github.io/CsvHelper/examples/writing/appending-to-an-existing-file/

I noticed 3 things. 1st - In your sample data, you have 7 column headers, but only 6 columns of data. 2nd - Your class names for "RunId" and "TimeTakenByLocatorJson" don't exactly match the columns in your sample data. 3rd - Your config says "Don't write the header again.", but you are using it for reading.
For the 1st issue, I'm going to assume this was a misprint and I'll add another column of data.
For the 2nd issue, there are at least 3 ways to handle it. You already handled it one way by mapping to indexes in LocatorTimeMap. I'll give you a 2nd way by casting the header to lower case. A 3rd way is to use the name attribute
For the 3rd issue, the header is there for reading and I assume you want the header when you write it, so you can leave HasHeaderRecord = false out.
void Main()
{
writeEnd("Run_645987");
}
// You can define other methods, fields, classes and namespaces here
public void writeEnd(string runId)
{
var records = Enumerable.Empty<LocatorTime>();
try
{
var config = new CsvConfiguration(CultureInfo.InvariantCulture)
{
// This will convert both the header coming in and your class property to lower case.
PrepareHeaderForMatch = args => args.Header.ToLower()
};
var input = new StringBuilder();
input.AppendLine("RunId,ProductArea,Product,Component,PageObject,Control,TimeTakenByLocatorJson");
input.AppendLine("Run_645987,R201,BN2018.5,N778,BC1,control1,2021-07-19");
using (var reader = new StringReader(input.ToString()))
//using (var reader = new StreamReader(#"D:\Reports\" + runId + #"\LocatorTime.csv"))
using (var csv = new CsvReader(reader, config))
{
records = csv.GetRecords<LocatorTime>().ToList();
foreach (var record in records)
{
if (record.RunID == runId)
{
record.TimeTakenByLocatorJSON = DateTime.Now;
}
}
}//Endof Stream Reader
//using (var stream = File.Open(#"D:\Reports\" + runId + #"\LocatorTime.csv", FileMode.Append)) //not sure what the file mode should be
//using (var writer = new StreamWriter(stream))
using (var csv = new CsvWriter(Console.Out, CultureInfo.InvariantCulture))
{
csv.WriteRecords(records);
}
}
catch (Exception e)
{
Console.WriteLine(e);
}
}//end func writeEnd
public class LocatorTime
{
public string RunID { get; set; }
public string ProductArea { get; set; }
public string Product { get; set; }
public string Component { get; set; }
public string PageObject { get; set; }
public string Control { get; set; }
public DateTime TimeTakenByLocatorJSON //only this value needs to be written for update at end of exec
{
get;
set;
}
}//LocatorTimeClass

System.InvalidOperationException: 'There is an error in the XML document (1, 1).'

This error is showing up in my code, there is a second one that is as follows:
XmlException: The existing data at the root level is invalid. Line 1, position 1
I checked this second one saying there is a error with the file when there isn't any since I have 5 files inside my XMLFiles directory.
public static void Main()
{
XmlSerializer serializer = new XmlSerializer(typeof(ImportSession));
MemoryStream stream = new MemoryStream();
using (StreamWriter sw = new StreamWriter(stream))
{
sw.Write(stream);
sw.Flush();
stream.Position = 0;
}
Console.ReadKey();
foreach (string filename in Directory.EnumerateFiles(#"C:\XMLFiles", "*.xml"))
{
ProcessFile(filename, stream, serializer);
}
void ProcessFile(string Filename, MemoryStream stream, XmlSerializer serializer)
{
bool temErro = false;
Console.WriteLine("A processar xml: " + Filename);
XmlDocument xml = new XmlDocument();
xml.Load(Filename);
ImportSession session = (ImportSession)serializer.Deserialize(stream);
foreach (Batch batch in session.Batches)
{
foreach (Document doc in batch.Documents)
{
foreach (Page page in doc.Pages)
{
if (!string.IsNullOrEmpty(batch.Processed.ToString()))
{
if (!string.IsNullOrEmpty(page.HasError.ToString()))
{
string Import = page.ImportFileName;
Console.WriteLine("Página com erro:" + Import);
temErro = true;
}
}
}
}
}
if (temErro)
Console.WriteLine("Ficheiro com erro: " + Filename);
else
Console.WriteLine("Ficheiro processado: " + Filename);
Console.WriteLine(Filename);
}
}
public class ImportSession
{
public Batch[] Batches { get; set; }
}
public class Batch
{
[XmlAttribute]
public string Name { get; set; }
[XmlAttribute]
public string Description { get; set; }
[XmlAttribute]
public string BatchClassName { get; set; }
[XmlAttribute]
public bool Processed { get; set; }
public Document[] Documents { get; set; }
}
public class Document
{
[XmlAttribute]
public string FormTypeName { get; set; }
public IndexField[] IndexFields { get; set; }
public Page[] Pages { get; set; }
}
public class IndexField
{
[XmlAttribute]
public string Name { get; set; }
[XmlAttribute]
public string Value { get; set; }
}
public class Page
{
[XmlAttribute]
public string ImportFileName { get; set; }
[XmlAttribute]
public string ErrorCode { get; set; }
[XmlAttribute]
public string ErrorMessage { get; set; }
[XmlIgnore]
public bool HasError => !string.IsNullOrWhiteSpace(ErrorMessage);
}
This app right now is only trying to read all the files and point out some parts that need to show up in the console and it was doing it but I was adviced on here to change into this object oriented and memory stream.

This:
MemoryStream stream = new MemoryStream();
using (StreamWriter sw = new StreamWriter(stream))
{
sw.Write(stream);
sw.Flush();
stream.Position = 0;
is basically meaningless. Whatever the contents of stream are meant to be: it isn't this. Ask yourself:
What is stream meant to contain?
At the moment it contains... itself, sort of, but not really?
If you intend the stream to be the file contents: just use File.OpenRead

I think this is based on a misunderstanding from answers to previous questions on the topic.
This should make it work. BUT keep in mind, that it is in no way production-ready.
public static void Main()
{
XmlSerializer serializer = new XmlSerializer(typeof(ImportSession));
foreach (string filename in Directory.EnumerateFiles(#"C:\XMLFiles", "*.xml"))
{
ProcessFile(filename, serializer);
}
Console.ReadKey();
}
private static void ProcessFile(string Filename, XmlSerializer serializer)
{
bool temErro = false;
Console.WriteLine("A processar xml: " + Filename);
using (var file = File.OpenRead(Filename)) {
var session = (ImportSession)serializer.Deserialize(file);
// from here on the rest of your code ...
To minimize the code that keeps the file opened:
ImportSession session;
using (var file = File.OpenRead(Filename))
{
session = (ImportSession)serializer.Deserialize(file);
}
// file will be closed by disposal of FileStream using this notation
// rest of code
Addendum
if (!string.IsNullOrEmpty(batch.Processed.ToString()))
{ // Will ALWAYS be entered!
if (!string.IsNullOrEmpty(page.HasError.ToString()))
{ // Will ALWAYS be entered!
string Import = page.ImportFileName;
Console.WriteLine("Página com erro:" + Import);
temErro = true;
}
}
Let's look at it:
!string.IsNullOrEmpty(page.HasError.ToString()) is always true. Why?
page.HasError is of type bool. So, page.HasError.ToString() "Converts the value of this instance to its equivalent string representation (either "True" or "False")."
So, it will never be null or empty. So, string.IsNullOrEmpty will always be false, and !string.IsNullOrEmpty therefore always be true.
If you want to check the boolean value, you simply do if( page.HasError ) => "Page has an error"

How to iterate through LiteDB.LiteCollection for specific key values from a large litedb database file

I am creating a standalone database using LiteDB with C# code for storing every files metadata like filesize, absolute path ,createtime, extension etc. I am having problem in querying and iterating for the exact key value from the db file, please help out Thankyou.
This is my Insert query which is taking the absolute paths of every files from directories and passing the path as variable to my class for setting the file metadata values and returning the object with values to store in the LiteDB.
/INSERT QUERY/
public void Search(/* Drive paths */)
{
foreach(string file in Directory.GetFiles(path, "."))
{
try
{
using (var db = new LiteDatabase(/path of my DB file/))
{
var FileDBptr = db.GetCollection("FileDB");
var data = new FileInfoDB(file);
FileDBptr.Insert(data);
Console.WriteLine("SUCCESS INSERT");
}
}
catch (Exception e)
{
Console.WriteLine("COuld Not Create DB!!!!" + e);
}
}
foreach (string subDir in Directory.GetDirectories(path))
{
try
{
Search(subDir);
}
catch
{
throw;
}
}
}
/* CLASS MODEL */
[Serializable]
public class FileInfoDB
{
[BsonId]
public ObjectId FileId { get; set; }
public string FileName { get; set; }
public string FilePath { get; set; }
public string FileExtension { get; set; }
public long FileSize { get; set; }
public DateTime FileCreateTime { get; set; }
public DateTime FileLastAccess { get; set; }
public DateTime FileLastWrite { get; set; }
public FileInfoDB(string path)
{
if (File.Exists(path))
{
try
{
FileInfo f = new FileInfo(path);
FileId= ObjectId.NewObjectId();
FileName = f.Name;
FilePath = f.FullName;
FileCreateTime = f.CreationTime;
FileSize = f.Length; //Size in bytes
FileExtension = f.Extension;
FileLastAccess = f.LastAccessTime;
FileLastWrite = f.LastWriteTime;
}
catch (Exception)
{
throw;
}
}
}
}
//QUERY CODE FOR RETRIEVAL
string fileName = "$ICN03Z0.txt";
try
{
using (var db = new LiteDatabase(_jsonpath))
{
var FileDBptr = db.GetCollection<FileInfoDB>("FileDB");
FileDBptr.EnsureIndex(x=>x.FileName);
var data2 = FileDBptr.Find(Query.EQ("FileName", fileName));
int c = FileDBptr.Count();
Console.WriteLine(c); //Correct output
if (data2 !=null)
{
foreach (var a in data2) //Throwing an Exception
{
Console.WriteLine(a.FileName);
}
}
}
}
This is the data format which is stored in lite database file
{
"_id": {"$oid":"5c4ebee0f2e2d05814dcf865"},
"FileName": "$ICN03Z0.txt",
"FilePath": "C:\$Recycle.Bin\S-1-5-21-3439349906-2439027251-2956315770-1001\$ICN03Z0.txt",
"FileExtension":".txt",
"FileSize": {"$numberLong":"114"},
"FileCreateTime": {"$date":"2019-01-16T09:04:16.0810000Z"},
"FileLastAccess": {"$date":"2019-01-16T09:04:16.0810000Z"},
"FileLastWrite": {"$date":"2019-01-16T09:04:16.0810000Z"}
}
I expect to search the value according to filename first and then extract all the other key value pairs data of same file.
I tried with different query using LINQ as mentioned in GitHub but always throwing exception. Also have verified the data stored inside Lite database using LiteDB shell its inserting as the format above, but retrieval is giving problem.
Thanks in Advance!

Csv Helper. First Value is shifting to the right

I have been trying to create a simple csv file using the csv helper. However, the result, I am getting is not what I expected.
For some reason which I cannot find it, the first value is shifting towards the right and appear as the header.
Could someone point me what I am doing wrong here?
public class Record
{
public string Vrm { get; set; }
public string Version { get; set; }
public DateTime Started { get; set; }
public DateTime? Completed { get; set; }
public string Status { get; set; }
public string Comments { get; set; }
}
static void Main(string[] args)
{
var source = new List<Record> {
new Record {
Status = "Success",
Version = "enhance",
Started = DateTime.Parse("2017-11-15 13:27:56.9933333"),
Completed = DateTime.Parse("2017-11-15 13:27:57.7300000"),
Vrm = "16aux",
Comments = "Completed Successfully"
}
};
var month = DateTime.UtcNow.Month;
var year = DateTime.UtcNow.Year;
var fileName = $"TestFile_{month}{year}.csv";
using (var sw = new StreamWriter(fileName))
{
var writer = new CsvWriter(sw);
try
{
writer.WriteHeader<Record>();
foreach (var record in source)
{
writer.WriteField(record.Vrm);
writer.WriteField(record.Version);
writer.WriteField(record.Started);
writer.WriteField(record.Completed);
writer.WriteField(record.Status);
writer.WriteField(record.Comments);
writer.NextRecord();
}
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
var i = sw;
}
}
The result is something like this:

Read up on http://joshclose.github.io/CsvHelper/writing#writing-all-records
You need to advance the writer one line by calling writer.NextRecord(); after writer.WriteHeader<Record>();.
You could also simply write all data at once, using csv.WriteRecords( records ); instead of foreaching over them