The CSVHelper .NET library seems fantastic so far, but the documentation is a little lacking for a pseudo-beginner like myself.
I need to read a csv file and write the results to our SQL Server database. For the table I'm writing to, I need to map to its columns from the CSV columns, including some concatenation of multiple fields to one.
This is what I have for reading the csv file:
public static void Main(string[] args)
{
using (var reader = new StreamReader(#"C:\Users\me\Documents\file.csv"))
using (var csv = new CsvReader(reader))
{
csv.Configuration.PrepareHeaderForMatch = (string header, int index) =>
header.Replace(" ", "_").Replace("(", "").Replace(")", "").Replace(".", "");
var records = csv.GetRecords<EntityCsv>().ToList();
}
}
My EntityCsv class contains property names for all columns of the csv file.
Then, I also have a class called TaskEntity which contains the property names and types for the destination database table (although I'm unclear as to whether I need this).
Finally, per advice from a colleague, I have a method set up to make use of SQLBulkCopy as thus:
public void AddBulk(List<TaskEntity> entities)
{
using (var con = GetConnection())
{
SqlBulkCopy bulk = new SqlBulkCopy(con);
bulk.BatchSize = 2000;
bulk.BulkCopyTimeout = 0;
bulk.DestinationTableName = "dbo.CsvExports";
bulk.WriteToServer(entities.AsDataTable());
bulk.Close();
}
}
I borrowed that code block from him and would theoretically run that method as the final step.
But I know I'm missing a step in between, and that is mapping the fields from the csv to the SQL server field. I'm scratching my head at how to implement this step.
So let's say for simplicity's sake I have 3 columns in the csv file, and I want to map them to 2 columns of the SQL table as follows:
CsvColumn1 -> SQLtableColumn1
CsvColumn2 + CsvColumn3 -> SQLtableColumn2
How would I go about accomplishing this with CsvReader and C#? I have explored the Mapping section of the CSVReader documentation but everything I'm seeing in there seems to refer to mapping column names from an input file to names in an output file. I don't see anything there (nor anywhere on the Google) that speaks specifically to taking the input file and exporting its rows to a SQL database.
You can use a ClassMap to map the csv columns to the sql table columns and skip the CsvEntity class.
public static void Main(string[] args)
{
using (var reader = new StreamReader(#"C:\Users\me\Documents\file.csv"))
using (var csv = new CsvReader(reader))
{
csv.Configuration.PrepareHeaderForMatch = (string header, int index) =>
header.Replace(" ", "_").Replace("(", "").Replace(")", "").Replace(".", "");
csv.Configuration.RegisterClassMap<TaskEntityMap>();
var records = csv.GetRecords<TaskEntity>().ToList();
}
}
public class TaskEntity
{
public int Id { get; set; }
public string SqlTableColumn1 { get; set; }
public string SqlTableColumn2 { get; set; }
}
public sealed class TaskEntityMap : ClassMap<TaskEntity>
{
public TaskEntityMap()
{
Map(m => m.SqlTableColumn1).Name("CsvColumn1");
Map(m => m.SqlTableColumn2).ConvertUsing(row => row.GetField<string>("CsvColumn2") + " " + row.GetField<string>("CsvColumn3"));
}
}
I have used the SqlBulkCopy along with the csvhelper to dump data in to the sql server.
SqlBulkCopy is an awesome utility that writes data to the sql server from nearly any data source that can be loaded into a DataTable instance.
var lines = File.ReadAllLines(file);
if (lines.Count() == 0)
return;
var tableName = GetTableName(file);
var columns = lines[0].Split(',').ToList();
var table = new DataTable();
sqlBulk.ColumnMappings.Clear();
foreach (var c in columns)
{
table.Columns.Add(c);
sqlBulk.ColumnMappings.Add(c, c);
}
for (int i = 1; i < lines.Count() - 1; i++)
{
var line = lines[i];
// Explicitly mark empty values as null for SQL import to work
var row = line.Split(',')
.Select(a => string.IsNullOrEmpty(a) ? null : a).ToArray();
table.Rows.Add(row);
}
sqlBulk.DestinationTableName = tableName;
sqlBulk.WriteToServer(table);
Related
This project is an ASP.Net Api project with Angular. What I'm trying to do is export data from a database table and into an excel file. So far, I've managed to export all the table data into an excel file, but struggle to select 2 or 3 fields in the table to export.
[HttpGet("download")]
public IActionResult DownloadExcel(string field)
{
string dbFileName = "DbTableName.xlsx";
FileInfo file = new FileInfo(dbFileName);
byte[] fileContents;
var stream = new MemoryStream();
using (ExcelPackage package = new ExcelPackage(file))
{
IList<UserTable> userList = _context.UserTable.ToList();
ExcelWorksheet worksheet = package.Workbook.Worksheets.Add("DbTableName");
int totalUserRows = userList.Count();
}
return File(fileContents, fileType, dbFileName);
}
There's no need to write so many if ... else if ... else if ... else if ... to get the related field names.
A nicer way is to
Use a field list (IList<string>)as a parameter.
And then generate a required field list by intersect.
Finally, we could use reflection to retrieve all the related values.
Implementation
public IActionResult DownloadExcel(IList<string> fields)
{
// get the required field list
var userType = typeof(UserTable);
fields = userType.GetProperties().Select(p => p.Name).Intersect(fields).ToList();
if(fields.Count == 0){ return BadRequest(); }
using (ExcelPackage package = new ExcelPackage())
{
IList<UserTable> userList = _context.UserTable.ToList();
ExcelWorksheet worksheet = package.Workbook.Worksheets.Add("DbTableName");
// generate header line
for(var i= 0; i< fields.Count; i++ ){
var fieldName = fields[i];
var pi= userType.GetProperty(fieldName);
var displayName = pi.GetCustomAttribute<DisplayNameAttribute>()?.DisplayName;
worksheet.Cells[1,i+1].Value = string.IsNullOrEmpty(displayName ) ? fieldName : displayName ;
}
// generate row lines
int totalUserRows = userList.Count();
for(var r=0; r< userList.Count(); r++){
var row = userList[r];
for(var c=0 ; c< fields.Count;c++){
var fieldName = fields[c];
var pi = userType.GetProperty(fieldName);
// because the first row is header
worksheet.Cells[r+2, c+1].Value = pi.GetValue(row);
}
}
var stream = new MemoryStream(package.GetAsByteArray());
return new FileStreamResult(stream,"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
}
}
You could configure the display name using the DsiplayNameAttribute:
public class UserTable
{
public int Id{get;set;}
[DisplayName("First Name")]
public string fName { get; set; }
[DisplayName("Last Name")]
public string lName { get; set; }
[DisplayName("Gender")]
public string gender { get; set; }
}
It's possible to add any properties as you like without hard-coding in your DownloadExcel method.
Demo :
passing a field list fields[0]=fName&fields[1]=lName&fields[2]=Non-Exist will generate an excel as below:
[Update]
To export all the fields, we could assume the client will not pass a fields parameter. That means when the fields is null or if the fields.Count==0, we'll export all the fields:
[HttpGet("download")]
public IActionResult DownloadExcel(IList<string> fields)
{
// get the required field list
var userType = typeof(UserTable);
var pis= userType.GetProperties().Select(p => p.Name);
if(fields?.Count >0){
fields = pis.Intersect(fields).ToList();
} else{
fields = pis.ToList();
}
using (ExcelPackage package = new ExcelPackage()){
....
}
}
if you want to use the datatable then we can define which you need to select from the datatable in this way
string[] selectedColumns = new[] { "Column1","Column2"};
DataTable dt= new DataView(fromDataTable).ToTable(false, selectedColumns);
or else if you wanna you list then you can use linq for selection of particular columns
var xyz = from a in prod.Categories
where a.CatName.EndsWith("A")
select new { CatName=a.CatName, CatID=a.CatID, CatQty = a.CatQty};
I have a working solution for uploading a CSV file. Currently, I use the IFormCollection for a user to upload multiple CSV files from a view.
The CSV files are saved as a temp file as follows:
List<string> fileLocations = new List<string>();
foreach (var formFile in files)
{
filePath = Path.GetTempFileName();
if (formFile.Length > 0)
{
using (var stream = new FileStream(filePath, FileMode.Create))
{
await formFile.CopyToAsync(stream);
}
}
fileLocations.Add(filePath);
}
I send the list of file locations to another method (just below). I loop through the file locations and stream the data from the temp files, I then use a data table and SqlBulkCopyto insert the data. I currently upload between 50 and 200 files at a time and each file is around 330KB. To insert a hundred, it takes around 6 minutes, which is around 30-35MB.
public void SplitCsvData(string fileLocation, Guid uid)
{
MetaDataModel MetaDatas;
List<RawDataModel> RawDatas;
var reader = new StreamReader(File.OpenRead(fileLocation));
List<string> listRows = new List<string>();
while (!reader.EndOfStream)
{
listRows.Add(reader.ReadLine());
}
var metaData = new List<string>();
var rawData = new List<string>();
foreach (var row in listRows)
{
var rowName = row.Split(',')[0];
bool parsed = int.TryParse(rowName, out int result);
if (parsed == false)
{
metaData.Add(row);
}
else
{
rawData.Add(row);
}
}
//Assigns the vertical header name and value to the object by splitting string
RawDatas = GetRawData.SplitRawData(rawData);
SaveRawData(RawDatas);
MetaDatas = GetMetaData.SplitRawData(rawData);
SaveRawData(RawDatas);
}
This code then passes the object to the to create the datatable and insert the data.
private DataTable CreateRawDataTable
{
get
{
var dt = new DataTable();
dt.Columns.Add("Id", typeof(int));
dt.Columns.Add("SerialNumber", typeof(string));
dt.Columns.Add("ReadingNumber", typeof(int));
dt.Columns.Add("ReadingDate", typeof(string));
dt.Columns.Add("ReadingTime", typeof(string));
dt.Columns.Add("RunTime", typeof(string));
dt.Columns.Add("Temperature", typeof(double));
dt.Columns.Add("ProjectGuid", typeof(Guid));
dt.Columns.Add("CombineDateTime", typeof(string));
return dt;
}
}
public void SaveRawData(List<RawDataModel> data)
{
DataTable dt = CreateRawDataTable;
var count = data.Count;
for (var i = 1; i < count; i++)
{
DataRow row = dt.NewRow();
row["Id"] = data[i].Id;
row["ProjectGuid"] = data[i].ProjectGuid;
row["SerialNumber"] = data[i].SerialNumber;
row["ReadingNumber"] = data[i].ReadingNumber;
row["ReadingDate"] = data[i].ReadingDate;
row["ReadingTime"] = data[i].ReadingTime;
row["CombineDateTime"] = data[i].CombineDateTime;
row["RunTime"] = data[i].RunTime;
row["Temperature"] = data[i].Temperature;
dt.Rows.Add(row);
}
using (var conn = new SqlConnection(connectionString))
{
conn.Open();
using (SqlTransaction tr = conn.BeginTransaction())
{
using (var sqlBulk = new SqlBulkCopy(conn, SqlBulkCopyOptions.Default, tr))
{
sqlBulk.BatchSize = 1000;
sqlBulk.DestinationTableName = "RawData";
sqlBulk.WriteToServer(dt);
}
tr.Commit();
}
}
}
Is there another way to do this or a better way to improve performance so that the time to upload is reduced as it can take a long time and I am seeing an ever increasing use of memory to around 500MB.
TIA
You can improve performance by removing the DataTable and reading from the input stream directly.
SqlBulkCopy has a WriteToServer overload that accepts an IDataReader instead of an entire DataTable.
CsvHelper can CSV files using a StreamReader as an input. It provides CsvDataReader as an IDataReader implementation on top of the CSV data. This allows reading directly from the input stream and writing to SqlBulkCopy.
The following method will read from an IFormFile, parse the stream using CsvHelper and use the CSV's fields to configure a SqlBulkCopy instance :
public async Task ToTable(IFormFile file, string table)
{
using (var stream = file.OpenReadStream())
using (var tx = new StreamReader(stream))
using (var reader = new CsvReader(tx))
using (var rd = new CsvDataReader(reader))
{
var headers = reader.Context.HeaderRecord;
var bcp = new SqlBulkCopy(_connection)
{
DestinationTableName = table
};
//Assume the file headers and table fields have the same names
foreach(var header in headers)
{
bcp.ColumnMappings.Add(header, header);
}
await bcp.WriteToServerAsync(rd);
}
}
This way nothing is ever written to a temp table or cached in memory. The uploaded files are parsed and written to the database directly.
In addition to #Panagiotis's answer, why don't you interleave your file processing with the file upload? Wrap up your file processing logic in an async method and change the loop to a Parallel.Foreach and process each file as it arrives instead of waiting for all of them?
private static readonly object listLock = new Object(); // only once at class level
List<string> fileLocations = new List<string>();
Parallel.ForEach(files, (formFile) =>
{
filePath = Path.GetTempFileName();
if (formFile.Length > 0)
{
using (var stream = new FileStream(filePath, FileMode.Create))
{
await formFile.CopyToAsync(stream);
}
await ProcessFileInToDbAsync(filePath);
}
// Added lock for thread safety of the List
lock (listLock)
{
fileLocations.Add(filePath);
}
});
Thanks to #Panagiotis Kanavos, I was able to work out what to do. Firstly, the way I was calling the methods, was leaving them in memory. The CSV file I have is in two parts, vertical metadata and then the usual horizontal information. So I needed to split them into two. Saving them as tmp files was also causing an overhead. It has gone from taking 5-6 minutes to now taking a minute, which for a 100 files containing 8,500 rows isn't bad I suppose.
Calling the method:
public async Task<IActionResult> UploadCsvFiles(ICollection<IFormFile> files, IFormCollection fc)
{
foreach (var f in files)
{
var getData = new GetData(_configuration);
await getData.SplitCsvData(f, uid);
}
return whatever;
}
This is the method doing the splitting:
public async Task SplitCsvData(IFormFile file, string uid)
{
var data = string.Empty;
var m = new List<string>();
var r = new List<string>();
var records = new List<string>();
using (var stream = file.OpenReadStream())
using (var reader = new StreamReader(stream))
{
while (!reader.EndOfStream)
{
var line = reader.ReadLine();
var header = line.Split(',')[0].ToString();
bool parsed = int.TryParse(header, out int result);
if (!parsed)
{
m.Add(line);
}
else
{
r.Add(line);
}
}
}
//TODO: Validation
//This splits the list into the Meta data model. This is just a single object, with static fields.
var metaData = SplitCsvMetaData.SplitMetaData(m, uid);
DataTable dtm = CreateMetaData(metaData);
var serialNumber = metaData.LoggerId;
await SaveMetaData("MetaData", dtm);
//
var lrd = new List<RawDataModel>();
foreach (string row in r)
{
lrd.Add(new RawDataModel
{
Id = 0,
SerialNumber = serialNumber,
ReadingNumber = Convert.ToInt32(row.Split(',')[0]),
ReadingDate = Convert.ToDateTime(row.Split(',')[1]).ToString("yyyy-MM-dd"),
ReadingTime = Convert.ToDateTime(row.Split(',')[2]).ToString("HH:mm:ss"),
RunTime = row.Split(',')[3].ToString(),
Temperature = Convert.ToDouble(row.Split(',')[4]),
ProjectGuid = uid.ToString(),
CombineDateTime = Convert.ToDateTime(row.Split(',')[1] + " " + row.Split(',')[2]).ToString("yyyy-MM-dd HH:mm:ss")
});
}
await SaveRawData("RawData", lrd);
}
I then use a data table for the metadata (which takes 20 seconds for a 100 files) as I map the field names to the columns.
public async Task SaveMetaData(string table, DataTable dt)
{
using (SqlBulkCopy sqlBulk = new SqlBulkCopy(_configuration.GetConnectionString("DefaultConnection"), SqlBulkCopyOptions.Default))
{
sqlBulk.DestinationTableName = table;
await sqlBulk.WriteToServerAsync(dt);
}
}
I then use FastMember for the large data parts for the raw data, which is more like a traditional CSV.
public async Task SaveRawData(string table, IEnumerable<LogTagRawDataModel> lrd)
{
using (SqlBulkCopy sqlBulk = new SqlBulkCopy(_configuration.GetConnectionString("DefaultConnection"), SqlBulkCopyOptions.Default))
using (var reader = ObjectReader.Create(lrd, "Id","SerialNumber", "ReadingNumber", "ReadingDate", "ReadingTime", "RunTime", "Temperature", "ProjectGuid", "CombineDateTime"))
{
sqlBulk.DestinationTableName = table;
await sqlBulk.WriteToServerAsync(reader);
}
}
I am sure this can be improved on, but for now, this works really well.
This is my first time posting so I apologize for any ignorance or failed use of examples.
I have a console app project to create where I have been given a fair few CSV files and I need to create some kind of Parent/Child/Grandchild relationship out of them (XML? maybe? - then I can use that to do the uploads and writes to the DMS with minimal calls - I don't want to be querying if a folder exists over and over)
I am a little out of my depth on this one
I need to know the best way to do this without 3rd party library dependencies, pure C#, using the OLEDB JET provider is most likely required as it will handle the parsing required, there is no order to the CSV files in regards to date, previous years could appear down the list and vice versa.
Here's an example of the CSV output
"DESCRIPTION1","8, 5/8\" X 6.4MM","STRING","filename001.pdf","2016-09-19","1"
"DESCRIPTION2","12, 3/4\" X 6.4MM","STRING","filename001.pdf","2016-09-19","1"
"DESCRIPTION3","12, 3/4\" X 6.4MM","STRING","filename001.pdf","2016-09-19","1"
"another description 20# gw","1","388015","Scan123.pdf","2015-10-24","1"
"another description 20# gw","3","385902","Scan456.pdf","2015-04-14","1"
"STRINGVAL1","273.10 X 9.27 X 6000","45032-01","KHJDWNEJWKFD9101529.pdf","2012-02-03","1"
"STRINGVAL2","273.10 X 21.44 X 6000","7-09372","DJSWH68767681540.pdf","2017-02-03","1"
The end output will be (YEAR/MONTH/FILENAME + (Attributes for each file - these are for eventually updating columns inside a DMS))
Year and Month retrieved from the column with the date
If the YEAR alread exists then it will not be created again
If the month under that year exists it will not be created again
If the filename already exists under that YEAR/MONTH it will not be created again BUT the additional ATTRIBUTES for that FileName will be added to the attributes - "line seperated?"
Required Output:
I have attempted a Linq query to begin to output the possible required XML for me to progress but it outputs every row and does no grouping, I am not familiar with Linq at the moment.
I also ran into issues with the basic escaping on the .Split(',') doing it this way (see original CSV examples above compared to me using TAB separation in my test file and example below) which is why I want the Oledb provider to handle it.
string[] source = File.ReadAllLines(#"C:\Processing\In\mockCsv.csv");
XElement item = new XElement("Root",
from str in source
let fields = str.Split('\t')
select new XElement("Year", fields[4].Substring(0, 4),
new XElement("Month", fields[4].Substring(5, 2),
new XElement("FileName", fields[3]),
new XElement("Description",fields[0]),
new XElement("Length", fields[1]),
new XElement("Type", fields[2]),
new XElement("FileName", fields[3]),
new XElement("Date", fields[4]),
new XElement("Authorised", fields[5]))
)
);
I also need to log every step of the process so I have setup a Logger class
private class Logger
{
private static string LogFile = null;
internal enum MsgType
{
Info,
Debug,
Error
}
static Logger()
{
var processingDetails = ConfigurationManager.GetSection(SECTION_PROCESSINGDETAILS) as NameValueCollection;
LogFile = Path.Combine(processingDetails[KEY_WORKINGFOLDER],
String.Format("Log_{0}.txt", StartTime.ToString("MMMyyyy")));
if (File.Exists(LogFile))
File.Delete(LogFile);
}
internal static void Write(string msg, MsgType msgType, bool isNewLine, bool closeLine)
{
if (isNewLine)
msg = String.Format("{0} - {1} : {2}", DateTime.Now.ToString("dd/MM/yyyy HH:mm:ss"), msgType, msg);
if (closeLine)
Console.WriteLine(msg);
else
Console.Write(msg);
if (String.IsNullOrEmpty(LogFile))
return;
try
{
using (StreamWriter sw = new StreamWriter(LogFile, true))
{
if (closeLine)
sw.WriteLine(msg);
else
sw.Write(msg);
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
}
Used as such
Logger.Write(String.Format("Reading records from csv file ({0})... ",
csvFile), Logger.MsgType.Info, true, false);
Try following. If you are reading from a file use StreamReader instead of StringReader :
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Xml;
using System.Xml.Linq;
using System.IO;
using System.Text.RegularExpressions;
namespace ConsoleApplication74
{
class Program
{
static void Main(string[] args)
{
string input =
"\"DESCRIPTION1\",\"8, 5/8 X 6.4MM\",\"STRING\",\"filename001.pdf\",\"2016-09-19\",\"1\"\n" +
"\"DESCRIPTION2\",\"12, 3/4 X 6.4MM\",\"STRING\",\"filename001.pdf\",\"2016-09-19\",\"1\"\n" +
"\"DESCRIPTION3\",\"12, 3/4 X 6.4MM\",\"STRING\",\"filename001.pdf\",\"2016-09-19\",\"1\"\n" +
"\"another description 20# gw\",\"1\",\"388015\",\"Scan123.pdf\",\"2015-10-24\",\"1\"\n" +
"\"another description 20# gw\",\"3\",\"385902\",\"Scan456.pdf\",\"2015-04-14\",\"1\"\n" +
"\"STRINGVAL1\",\"273.10 X 9.27 X 6000\",\"45032-01\",\"KHJDWNEJWKFD9101529.pdf\",\"2012-02-03\",\"1\"\n" +
"\"STRINGVAL2\",\"273.10 X 21.44 X 6000\",\"7-09372\",\"DJSWH68767681540.pdf\",\"2017-02-03\",\"1\"\n";
string pattern = "\\\"\\s*,\\s*\\\"";
string inputline = "";
StringReader reader = new StringReader(input);
XElement root = new XElement("Root");
while ((inputline = reader.ReadLine()) != null)
{
string[] splitLine = Regex.Split(inputline,pattern);
Item newItem = new Item() {
description = splitLine[0].Replace("\"",""),
length = splitLine[1],
type = splitLine[2],
filename = splitLine[3],
date = DateTime.Parse(splitLine[4]),
authorized = splitLine[5].Replace("\"", "") == "1" ? true : false
};
Item.items.Add(newItem);
}
foreach(var year in Item.items.GroupBy(x => x.date.Year).OrderBy(x => x.Key))
{
XElement newYear = new XElement("_" + year.Key.ToString());
root.Add(newYear);
foreach(var month in year.GroupBy(x => x.date.Month).OrderBy(x => x.Key))
{
XElement newMonth = new XElement("_" + month.Key.ToString());
newYear.Add(newMonth);
newMonth.Add(
month.OrderBy(x => x.date).Select(x => new XElement(
x.filename,
string.Join("\r\n", new object[] {
x.description,
x.length,
x.type,
x.date.ToString(),
x.authorized.ToString()
}).ToList()
)));
}
}
}
}
public class Item
{
public static List<Item> items = new List<Item>();
public string description { get; set; }
public string length { get; set; }
public string type { get; set; }
public string filename { get; set; }
public DateTime date { get; set; }
public Boolean authorized { get; set; }
}
}
I am getting some SQL statement that user inputs on UI:
select * from [table] where [col1] is not null and col2 <> 0
I need to verify that all column names (col1 and col2) are in brackets (like col1). And show some popup message if any column name is not in brackets (in this case, col2).
Is there some way to do such SQL verification in C#?
Regarding parsing SQL, you can use the TSqlParser library that ships with Visual Studio. This parser accepts a visitor class that inherits from TSqlFragmentVisitor and overrides the visit method for ColumnReferenceExpression.
using System;
using System.Collections.Generic;
using System.IO;
using System.Text;
using Microsoft.SqlServer.TransactSql.ScriptDom;
namespace ConsoleApplication8
{
public class QueryParser
{
public IEnumerable<string> Parse(string sqlSelect)
{
TSql100Parser parser = new TSql100Parser(false);
TextReader rd = new StringReader(sqlSelect);
IList<ParseError> errors;
var columns = new List<string>();
var fragments = parser.Parse(rd, out errors);
var columnVisitor = new SQLVisitor();
fragments.Accept(columnVisitor);
columns = new List<string>(columnVisitor.Columns);
return columns;
}
}
internal class SQLVisitor : TSqlFragmentVisitor
{
private List<string> columns = new List<string>();
private string GetNodeTokenText(TSqlFragment fragment)
{
StringBuilder tokenText = new StringBuilder();
for (int counter = fragment.FirstTokenIndex; counter <= fragment.LastTokenIndex; counter++)
{
tokenText.Append(fragment.ScriptTokenStream[counter].Text);
}
return tokenText.ToString();
}
public override void ExplicitVisit(ColumnReferenceExpression node)
{
columns.Add(GetNodeTokenText(node));
}
public IEnumerable<string> Columns {
get { return columns; }
}
}
public class Program
{
private static void Main(string[] args)
{
QueryParser queryParser = new QueryParser();
var columns = queryParser.Parse("SELECT A,[B],C,[D],E FROM T WHERE isnumeric(col3) = 1 Order by Id desc");
foreach (var column in columns)
{
Console.WriteLine(column);
}
}
}
}
However, I am not sure that parsing SQL that an user inputs in UI is a good idea. Even if you run the SQL statements in a sand boxed environment where the user only has read permissions (I do hope you are doing this), it is not very user friendly. As a user I would prefer to be able to select the columns I want using check boxes or by dragging and dropping them rather than having to write a SQL query.
I'm trying to read a text file and print out into a table.
I want the output to be this
But now I having different output
var column1 = new List<string>();
var column2 = new List<string>();
var column3 = new List<string>();
using (var rd = new StreamReader(#"C:\test.txt"))
{
while (!rd.EndOfStream)
{
var splits = rd.ReadLine().Split(';');
column1.Add(splits[0]);
column2.Add(splits[1]);
column3.Add(splits[2]);
}
}
Console.WriteLine("Date/Time \t Movie \t Seat");
foreach (var element in column1) Console.WriteLine(element);
foreach (var element in column2) Console.WriteLine(element);
foreach (var element in column3) Console.WriteLine(element);
You can use Linq to construct a convenient structure (e.g. List<String[]>) and then print out all the data wanted:
List<String[]> data = File
.ReadLines(#"C:\test.txt")
//.Skip(1) // <- uncomment this to skip caption if the csv has it
.Select(line => line.Split(';').Take(3).ToArray()) // 3 items only
.ToList();
// Table output (wanted one):
String report = String.Join(Environment.NewLine,
data.Select(items => String.Join("\t", items)));
Console.WriteLine(report);
// Column after column output (actual one)
Console.WriteLine(String.Join(Environment.NewLine, data.Select(item => item[0])));
Console.WriteLine(String.Join(Environment.NewLine, data.Select(item => item[1])));
Console.WriteLine(String.Join(Environment.NewLine, data.Select(item => item[2])));
EDIT: if you want to choose the movie, buy the ticket etc. elaborate the structure:
// Create a custom class where implement your logic
public class MovieRecord {
private Date m_Start;
private String m_Name;
private int m_Seats;
...
public MovieRecord(DateTime start, String name, int seats) {
...
m_Seats = seats;
...
}
...
public String ToString() {
return String.Join("\t", m_Start, m_Name, m_Seats);
}
public void Buy() {...}
...
}
And then convert to conventinal structure:
List<MovieRecord> data = File
.ReadLines(#"C:\test.txt")
//.Skip(1) // <- uncomment this to skip caption if the csv has it
.Select(line => {
String items[] = line.Split(';');
return new MovieRecord(
DateTime.ParseExact(items[0], "PutActualFormat", CultureInfo.InvariantCulture),
items[1],
int.Parse(items[2]));
}
.ToList();
And the table output will be
Console.Write(String.Join(Envrironment.NewLine, data));
Don't use Console.WriteLine if you want to add a "column". You should also use a single List<string[]> instead of multiple List<string>.
List<string[]> allLineFields = new List<string[]>();
using (var rd = new StreamReader(#"C:\test.txt"))
{
while (!rd.EndOfStream)
{
var splits = rd.ReadLine().Split(';');
allLineFields.Add(splits);
}
}
Console.WriteLine("Date/Time \t Movie \t Seat");
foreach(string[] line in allLineFields)
Console.WriteLine(String.Join("\t", line));
In general you should use a real csv parser if you want to parse a csv-file, not string methods or regex.
You could use the TextFieldParser which is the only one available in the framework directly:
var allLineFields = new List<string[]>();
using (var parser = new Microsoft.VisualBasic.FileIO.TextFieldParser(#"C:\test.txt"))
{
parser.Delimiters = new string[] { ";" };
parser.HasFieldsEnclosedInQuotes = false; // very useful
string[] lineFields;
while ((lineFields = parser.ReadFields()) != null)
{
allLineFields.Add(lineFields);
}
}
You need to add a reference to the Microsoft.VisualBasic dll to your project.
There are other available: Parsing CSV files in C#, with header
You could attempt to solve this in a more Object-Orientated manner, which might make it a bit easier for you to work with:
You can declare a simple class to represent a movie seat:
class MovieSeat
{
public readonly string Date, Name, Number;
public MovieSeat(string source)
{
string[] data = source.Split(';');
Date = data[0];
Name = data[1];
Number = data[2];
}
}
And then you can read in and print out the data in a few lines of code:
// Read in the text file and create a new MovieSeat object for each line in the file.
// Iterate over all MovieSeat objets and print them to console.
foreach(var seat in File.ReadAllLines(#"C:\test.txt").Select(x => new MovieSeat(x)))
Console.WriteLine(string.Join("\t", seat.Date, seat.Name, seat.Number));