I've been trying to read through some csv files that are stored in blob storage and append and populate two columns. I'm using csvhelper to do this but finding it difficult when trying to modify the actual rows.
I would like this process to be dynamic as I'll be looping through a containers paths, and I do not have the structure of all the csv files at hand.
using (StreamReader reader = new StreamReader(blob.OpenRead()))
using (CsvReader csv = new CsvReader(reader))
{
csv.Read();
csv.ReadHeader();
List<string> headers = csv.Context.HeaderRecord.ToList();
headers.Add("ColA");
headers.Add("ColB");
newHeaderContent = string.Join(",", headers);
// Not sure how to read through the csv and populate the two columns I just appended
}
using (StreamWriter writer = new StreamWriter(processedblob.OpenWrite()))
using (CsvWriter csvwriter = new CsvWriter(writer) )
{
writer.WriteLine(String.Concat(newHeaderContent));
//writer code for rows
}
As per this issue, CsvHelper does not support this kind of edit feature. And he also mentions that you need 2 files, read from the old one(and then make some changes), then write the updates to the new one.
The code below works and following the above instructions:
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Auth;
using Microsoft.WindowsAzure.Storage.Blob;
using System;
using System.IO;
using System.Text;
namespace AzureBlobConsole
{
class Program
{
static void Main(string[] args)
{
CloudStorageAccount storageAccount = new CloudStorageAccount(new StorageCredentials("your_storage_account", "storage_account_key"), true);
CloudBlobClient client = storageAccount.CreateCloudBlobClient();
CloudBlobContainer blobContainer = client.GetContainerReference("f11");
CloudBlockBlob blob = blobContainer.GetBlockBlobReference("t2.csv");
//download the .csv file into a text
string s1 = blob.DownloadText();
//split the text into an array
string[] temp = s1.Split('\r');
// variable s used to store the updated text from .csv
string s = "";
//because the last element of the array is redundant data("\n"), so we use temp.Length-1 to abandon it.
for (int i=0;i<temp.Length-1;i++)
{
if (i == 0)
{
temp[i] += ",ColA,ColB"+"\r\n";
}
else
{
temp[i] += ",cc,dd"+"\r\n";
}
s += temp[i];
}
//upload the updated .csv file into azure
var stream = new MemoryStream(Encoding.UTF8.GetBytes(s));
blob.UploadFromStream(stream);
Console.WriteLine("completed.");
Console.ReadLine();
}
}
}
Test result:
Before update:
After update:
Related
I need to get all the files from a storage with url: "https://example.blob.core.windows.net/file/subfile1/subfile2"
I only need to read data files( only .txt for example, so if I have .jpg or something else I should skip those) and only from subfile2, not the whole storage.
With c# btw
I tried in my system able to download the all txt files in container subfolder(test is container and test1 is folder in container) ,Try with this code .
using Microsoft.Azure.Storage.Blob;
using System;
using System.Collections.Generic;
using System.IO;
namespace listtxtblobs
{
class Program
{
static void Main(string[] args)
{
Console.WriteLine("Hello World!");
CloudStorageAccount storageAccount = CloudStorageAccount.Parse("Connection string");
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference("test//container name");
List<string> blobnames = new List<string>();
var allblobs = container.ListBlobs(useFlatBlobListing: true);
foreach (var b in allblobs)
{
string name = ((CloudBlockBlob)b).Name;
Console.WriteLine(name);
string[] names = name.Split('/');
blobnames.Add(names[names.Length - 1]);
}
foreach(string name in blobnames)
{
Console.WriteLine(name);
if (name.Contains(".txt"))
{
string filePath = "local file path\\" + name;
CloudBlockBlob blockBlob = container.GetBlockBlobReference("test1\\"+ name);
blockBlob.DownloadToFile(filePath, FileMode.OpenOrCreate);
}
}
}
}
}
Folder structure in azure storage
OUTPUT
Downloaded the all txt files in local folder
This question already has answers here:
Reading CSV files using C#
(12 answers)
Closed 3 years ago.
The code below reads a CSV file and looks for a line containing the serial number which is the first column of the file. Then copies that line to another file. The code works fine.
I need to read the text in the second and third fields of the row (there are 12 fields) and assign them to string variables (for other uses).
Can you help me, Please.
I am a novice.
List<string> found = new List<string>();
string line;
using(StreamReader file = new StreamReader(input_filename))
{
while((line=file.ReadLine())!=null)
{
if(line.Contains("XA2345")) // Serial Number
{
found.Add(line);
using(StreamWriter w = File.AppendText(output_filename))
{
// Console.WriteLine(line);
w.WriteLine(line);
w.Flush();
}
}
}
}
I'd start with some best practices for parsing CSV files with the post Parsing CSV files in C#, with header.
I've also noticed you've got that "found" variable. Are you trying to avoid duplicate lines in your output file but the code is incomplete? I've written the following code under that assumption.
Here are the using statements:
using Microsoft.VisualBasic.FileIO;
using System.Collections.Generic;
using System.IO;
using System.Linq;
Here's the main code:
List<string> foundLines = new List<string>();
using (TextFieldParser parser = new TextFieldParser(inputFilename))
{
// Set up the parser for CSV files
parser.TextFieldType = FieldType.Delimited;
parser.SetDelimiters(",");
using (StreamWriter writer = new StreamWriter(outputFilename, false))
{
while (!parser.EndOfData)
{
string[] values = parser.ReadFields();
string serialNumber = values[0];
if (string.Equals(serialNumber, "XA2345"))
{
string line = string.Join(",", values.Select(Escape));
if (foundLines.Contains(line))
continue; // Skip writing this line more than once
else
foundLines.Add(line); // Remember this line for later
writer.WriteLine(line);
// Do what you need to with the individual column values
string secondValue = values[1];
string thirdValue = values[2];
// ... Etc. ...
}
}
}
}
And here's a CSV helping method for escaping values as needed at Good CSV writer for C#:
static private string Escape(string s)
{
const string QUOTE = "\"";
const string ESCAPED_QUOTE = "\"\"";
char[] CHARACTERS_THAT_MUST_BE_QUOTED = { ',', '"', '\n' };
if (s.Contains(QUOTE))
s = s.Replace(QUOTE, ESCAPED_QUOTE);
if (s.IndexOfAny(CHARACTERS_THAT_MUST_BE_QUOTED) > -1)
s = QUOTE + s + QUOTE;
return s;
}
I have a text file (filename.txt) and I want to delete everything in this file from a special row number to the end of the text file.
How can I do this?
Is it even possible?
P.S.:
The number of the line is not constant.
It depends on the value of another variable in my code.
You can do the following:
Read the count of lines which you need.
Delete the file.
Write your lines to a newly created file with the same path and name.
Below is an example of the code:
using System.IO;
using System.Collections.Generic;
namespace ConsoleApp3
{
public class Program
{
static void Main(string[] args)
{
int lineNumber = 4;
string filePath = #"C:\Users\Admin\Desktop\test - Copy.txt";
List<string> lines = new List<string>();
using (TextReader tr = new StreamReader(filePath))
{
int i = 0;
while(i!=lineNumber)
{
lines.Add(tr.ReadLine());
i++;
}
}
File.Delete(filePath);
File.WriteAllLines(filePath,lines);
}
}
}
I am trying to read only headers from a csv file using CSVHELPER but i am unable to get GetFieldHeaders() method of csvhelper.
I have taken code from this link :Source
public static String[] GetHeaders(string filePath)
{
using (CsvReader csv = new CsvReader(new StreamReader("data.csv")))
{
int fieldCount = csv.FieldCount;
string[] headers = csv.GetFieldHeaders();//Error:doesnt contains definition
}
}
But GetFieldHeaders is not working.
Note: I only want to read headers from csv file
Update : Headers in my csv files are like below :
Id,Address,Name,Rank,Degree,Fahrenheit,Celcius,Location,Type,Stats
So can anybody tell me what i am missing??
Please try below code ... hope this will help you.
var csv = new CsvReader(new StreamReader("YOUR FILE PATH"));
csv.ReadHeader();
var headers = csv.Parser.RawRecord;
Note: headers will return all headers together.. you will need to make substring(s) for each comma to get each header separately.
I did not try to use this library. But quick overview of documentation brought this possible solution:
public static String[] GetHeaders(string filePath)
{
using (CsvReader csv = new CsvReader(new StreamReader("data.csv")))
{
csv.Configuration.HasHeaderRecord = true;
int fieldCount = csv.FieldCount;
string[] headers = csv.GetFieldHeaders();
}
}
* See documentation and search for header.
You can try this instead:
public IEnumerable<string> ReadHeaders(string path)
{
using (var reader = new StreamReader(path))
{
var csv = new CsvReader(reader);
if (csv.Read())
return csv.FieldHeaders;
}
return null;
}
All of the methods in the other answers seem to have been removed in the latest version?
This is what worked for me:
using (var fileReader = File.OpenText("data.csv"))
{
var csv = new CsvReader(fileReader);
csv.Configuration.HasHeaderRecord = true;
csv.Read();
csv.ReadHeader();
string[] headers = ((CsvFieldReader)((CsvParser)csv.Parser).FieldReader).Context.HeaderRecord;
}
I'm trying to convert XML file to CSV using XSLT. It works. I was able to convert XML and get the output I want using XSLT.
The challenge I'm facing right now is that I have many many XML files in a single location. I want to get ALL the data from all XMLs and put them into a single CSV file. I have a for loop that goes through the folder and gets the XML files and then exports it to CSV. However, every time it converts new XML it overrides the data in the current CSV file. So the end result is I only get one row in the CSV file instead of 500 (if there's 500 xml files).
Here's the C# code:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Xml;
using System.Xml.Xsl;
using System.Xml.XPath;
using System.IO;
using System.Diagnostics;
namespace XSL
{
class Program
{
public static void Main(string[] args)
{
try
{
//declaring xmlFile
string xmlfile ;
//Loading the XSLT template
String xsltfile = "C:\\Win\\XMLReader\\XSL\\csv.xsl";
//get folder location
string d = DateTime.Now.ToString("yyyyMMdd");
//Console.WriteLine(d.ToString());
//first part of the location path
String firstPath = #"\\tripx-exportm\\output\\Skill 53115;1_";
//full path
string fullPath = firstPath + d.ToString() + "_000000" + #"\\IDX";
//Get files from a folder
string[] filePath = Directory.GetFiles(fullPath, "*.xml");
//get each file in the folder
foreach (string file in filePath)
{
Console.WriteLine(file);
xmlfile = file;
Transform(xmlfile, xsltfile);
}
//Get the count of XML files in the current folder
DirectoryInfo dir = new DirectoryInfo(fullPath);
int count = dir.GetFiles().Length;
Console.WriteLine("Count of XML files: " + count);
//Transform(xmlfile, xsltfile);
Console.WriteLine("press any key");
Console.ReadKey();
}
catch(Exception e){
Console.WriteLine(e.ToString());
}
}
public static void Transform(string xml, string xslt)
{
try
{
//load the xml doc
XPathDocument myDoc = new XPathDocument(xml);
XslCompiledTransform myXL = new XslCompiledTransform();
//load the xslt doc
myXL.Load(xslt);
//create the output
XmlTextWriter myWriter = new XmlTextWriter("result.csv", null);
myXL.Transform(myDoc, null, myWriter);
myWriter.Close();
}
catch (Exception e)
{
Console.WriteLine("exception : {0}", e.ToString());
};
}
}
}
XSLT file
Any suggestions on how I can put the data from multiple XMLs into a single CSV?
Thank you
You need to use the right constructor overload for the XmTextWriter to take a stream, rather than a file name.
The help on MSDN says this about the file name overload:
The filename to write to. If the file exists, it truncates it and
overwrites it with the new content.
So you should to do this to Transform to make it work:
public static void Transform(Stream stream, string xml, string xslt)
{
var myDoc = new XPathDocument(xml);
var myXL = new XslCompiledTransform();
myXL.Load(xslt);
using(var myWriter = new XmlTextWriter(stream, Encoding.Default))
{
myXL.Transform(myDoc, null, myWriter);
myWriter.Flush();
myWriter.Close();
}
}
Then the calling code should look like this:
using (var fs = new FileStream("result.csv", FileMode.Create))
{
foreach (string file in filePath)
{
Transform(fs, file, xsltfile);
}
fs.Flush();
fs.Close();
}
I've taken out the (bad) exception handling, and comments, etc.
I have come up with a different approach. It may not be the best possible solution, but it works for me.
In my code, I added a counter to count the number of XMLs that were processed. It increments.
In my Transform procedure instead of hard coding the name "result.csv", I generate the name:
string Result = "result" + count.ToString() + ".csv";
And I use that name in the XMLWriter
XmlTextWriter myWriter = new XmlTextWriter(Result, null);
This way it generates a single CSV for every XML and it will not override the existing one.
Then I wrote another procedure that combines all the CVSs into one:
private static void JoinCsvFiles(string[] csvFileNames, string outputDestinationPath)
{
StringBuilder sb = new StringBuilder();
bool columnHeadersRead = false;
foreach (string csvFileName in csvFileNames)
{
TextReader tr = new StreamReader(csvFileName);
string columnHeaders = tr.ReadLine();
// Skip appending column headers if already appended
if (!columnHeadersRead)
{
sb.AppendLine(columnHeaders);
columnHeadersRead = true;
}
sb.AppendLine(tr.ReadToEnd());
}
File.WriteAllText(outputDestinationPath, sb.ToString());
}
And in the main, I just call
string[] csvFileNames = Directory.GetFiles(".", "result*.csv");
JoinCsvFiles(csvFileNames, "CsvOutput.csv");
Hope this helps somebody.