We Keep Coding

Convert .XYZ to .csv using c#

Hi i am using this method to replace " " to "," but is failing when i try to use it on data that have 32 millions lines. Is anyone knows how to modify it to make it running?
List<String> lines = new List<String>();
//loop through each line of file and replace " " sight to ","
using (StreamReader sr = new StreamReader(inputfile))
{
int id = 1;
int i = File.ReadAllLines(inputfile).Count();
while (sr.Peek() >= 0)
{
//Out of memory issuee
string fileLine = sr.ReadLine();
//do something with line
string ttt = fileLine.Replace(" ", ", ");
//Debug.WriteLine(ttt);
lines.Add(ttt);
//lines.Add(id++, 'ID');
}
using (StreamWriter writer = new StreamWriter(outputfile, false))
{
foreach (String line in lines)
{
writer.WriteLine(line+","+id);
id++;
}
}
}
//change extension to .csv
FileInfo f = new FileInfo(outputfile);
f.MoveTo(Path.ChangeExtension(outputfile, ".csv"));
I general i am trying to convert big .XYZ file to .csv format and add incremental field at the end. I am using c# for first time in my life to be honest :) Can you help me?

See my comment above - you could modify your reading / writing as follows :
using (StreamReader sr = new StreamReader(inputfile))
{
using (StreamWriter writer = new StreamWriter(outputfile, false))
{
int id = 1;
while (sr.Peek() >= 0)
{
string fileLine = sr.ReadLine();
//do something with line
string ttt = fileLine.Replace(" ", ", ");
writer.WriteLine(ttt + "," + id);
id++;
}
}
}

SSIS 2015 Script task convert text file to UTF8 in C# or VB

I want to convert my resulting txt file into a UTF8 formatted file so I can load it into my Azure SQL DW via Polybase. It is required the source file be in UTF8.
MSDN has an "IO Streaming example" HERE works perfectly for a single job. I am trying to architect an SSIS solution for around 30 tables though. I believe using this method would cause a race condition where the PS script will be locked by 1 SSIS package when another SSIS package needs it.
I am a sql dev, not a .NET dev so please forgive me. How would one convert the above to an SSIS C# Script task assuming I know how to pass parameters into the Script task?
PowerShell Code from MSDN
#Static variables
$ascii = [System.Text.Encoding]::ASCII
$utf16le = [System.Text.Encoding]::Unicode
$utf8 = [System.Text.Encoding]::UTF8
$ansi = [System.Text.Encoding]::Default
$append = $False
#Set source file path and file name
$src = [System.IO.Path]::Combine("<MySrcFolder>","<MyUtf8stage>.txt")
#Set source file encoding (using list above)
$src_enc = $ascii
#Set target file path and file name
$tgt = [System.IO.Path]::Combine("<MyDestFolder>","<MyFinalstage>.txt")
#Set target file encoding (using list above)
$tgt_enc = $utf8
$read = New-Object System.IO.StreamReader($src,$src_enc)
$write = New-Object System.IO.StreamWriter($tgt,$append,$tgt_enc)
while ($read.Peek() -ne -1)
{
$line = $read.ReadLine();
$write.WriteLine($line);
}
$read.Close()
$read.Dispose()
$write.Close()
$write.Dispose()
Update
I found a similar post which I was able to tweak to my needs, I swear I searched high and low before posting. Anyway here is what IS working for me. If you see anyway to improve it please share:
public void Main()
{
//$Package::SourceSQLObject = tablename
//$Package::StageFile_DestinationFolderPath = rootpath eg "C:\temp\"
string path = (string)Dts.Variables["$Package::StageFile_DestinationFolderPath"].Value;
string name = (string)Dts.Variables["$Package::SourceSQLObject"].Value;
string from = Path.Combine(path, name) + ".csv";
string to = Path.ChangeExtension(from, "txt");
Dts.Log("Starting " + to.ToUpper(), 0, null);
using (StreamReader reader = new StreamReader(from, Encoding.ASCII, false, 10))
using (StreamWriter writer = new StreamWriter(to, false, Encoding.UTF8, 10))
{
while (reader.Peek() >= 0)
{
writer.WriteLine(reader.ReadLine());
}
}
Dts.TaskResult = (int)ScriptResults.Success;

Your code indicates that your are trying to convert an ASCII file to UTF-8 however that article also states the following:
As UTF-8 uses the same character encoding as ASCII PolyBase will also
support loading data that is ASCII encoded.
So my advice to you is to try the file first with Polybase, check for any conversion issues before you spend any time trying to convert the files.

var mySrcFolder = ""; // something from user variables?
var myUtf8stage = ""; // something from user variables?
var myFinalstage = ""; // something from user variables?
// Static variables
var ascii = System.Text.Encoding.ASCII;
var utf16le = System.Text.Encoding.Unicode;
var utf8 = System.Text.Encoding.UTF8;
var ansi = System.Text.Encoding.Default;
var append = false;
// Set source file path and file name
var src = System.IO.Path.Combine(
mySrcFolder,
String.Format("{0}.txt", myUtf8stage));
// Set source file encoding (using list above)
var src_enc = ascii;
// Set target file path and file name
var tgt = System.IO.Path.Combine(
mySrcFolder,
String.Format("{0}.txt", myFinalstage));
// Set target file encoding (using list above)
var tgt_enc = utf8;
using (var read = new System.IO.StreamReader(src, src_enc))
using (var write = new System.IO.StreamWriter(tgt, append, tgt_enc))
{
while (read.Peek() != -1)
{
var line = read.ReadLine();
write.WriteLine(line);
}
}

public void Main()
{
//$Package::SourceSQLObject = tablename
//$Package::StageFile_DestinationFolderPath = rootpath eg "C:\temp\"
string path = (string)Dts.Variables["$Package::StageFile_DestinationFolderPath"].Value;
string name = (string)Dts.Variables["$Package::SourceSQLObject"].Value;
string from = Path.Combine(path, name) + ".csv";
string to = Path.ChangeExtension(from, "txt");
Dts.Log("Starting " + to.ToUpper(), 0, null);
using (StreamReader reader = new StreamReader(from, Encoding.ASCII, false, 10))
using (StreamWriter writer = new StreamWriter(to, false, Encoding.UTF8, 10))
{
while (reader.Peek() >= 0)
{
writer.WriteLine(reader.ReadLine());
}
}
Dts.TaskResult = (int)ScriptResults.Success;

C# unique file naming with String from Streamreader

ok i have solved my problem of finding a unique word within the file that is then used as the newly created .txt file name.
for example: current.txt files have 200 lines of words/data per file but one of the words is unique("92222225") with every current.txt file.
so the newly created output files from streamwriter becomes 92222225.txt, 933333334.txt and so on.
the whole time i though what i needed was within streamreader or streamwriter.
but what i need to add to the two was "Regex.Match".
here is the code i figured out to use for pulling strings out of a .txt file to use as a name for the output files. also to add other words to the new output file.
string mydocpath = Environment.GetFolderPath(Environment.SpecialFolder.Desktop) + #"\reporting";
foreach (string txtName in Directory.EnumerateFiles(mydocpath, "*.txt"))
{
StringBuilder sb = new StringBuilder();
StreamReader sr = new StreamReader(txtName);
string content = sr.ReadToEnd();
sb.AppendLine(txtName.ToString());
sb.AppendLine("= = = = = =");
sb.Append(content);
if (content.Contains("helloworld"))
{
sb.AppendLine();
sb.AppendLine("byeworld");
}
sb.AppendLine();
sb.AppendLine();
//string fileName = content.Contains("helloworld").ToString();
string FindMatch = content;
Match match = Regex.Match(FindMatch, #"9(([A-Za-z0-9\-])\d+)");
if (match.Success)
{
//this is what adds unique word as 922225.txt file name.
string capture = match.Groups[1].Captures[0].Value;
using (StreamWriter outfile = new StreamWriter(mydocpath + #"\" + capture + ".txt"))
{
outfile.Write(sb.ToString());
}
}
}
i updated this whole post so if anyone else may need this.
never even used Regex.Match before, nor knew about it or maybe i forgot about it.

If you want to name your output file use:
using (StreamWriter outfile = new StreamWriter(mydocpath + #"\901232lOi.txt"))

how to change a line in a txt-document with C#

I'm having a problem here. I have a .txt-file where one line contains "message", this is the line I wanna change. But I can't get this code to work, anyone that can help me?
I have this code here that is working for only replacing a string, but I don't know how to do it so it changes the whole line.
public void t()
{
string filename = #"F:\test\test.txt";
StringBuilder result = new StringBuilder();
if (System.IO.File.Exists(filename))
{
using (StreamReader streamReader = new StreamReader(filename))
{
String line;
while ((line = streamReader.ReadLine()) != null)
{
string newLine = String.Concat(line, Environment.NewLine);
newLine = newLine.Replace("message", "HEJHEJ ");
result.Append(newLine);
}
}
}
using (FileStream fileStream = new FileStream(filename, FileMode.Open, FileAccess.ReadWrite))
{
StreamWriter streamWriter = new StreamWriter(fileStream);
streamWriter.Write(result);
streamWriter.Close();
fileStream.Close();
}
}
This code is changeing "" to "HEJHEJ", but I want to whole line in the txt-document to change to "HEJHEJ", not just only the "message" part

How about changing this:
string newLine = String.Concat(line, Environment.NewLine);
newLine = newLine.Replace("message", "HEJHEJ ");
result.Append(newLine);
to this:
string newLine;
if (line.Contains("message")) {
newLine = String.Concat("HEJHEJ ", Environment.NewLine);
}
else {
newLine = String.Concat(line, Environment.NewLine);
}
result.Append(newLine);
There's a lot cleaner ways to do this, of course.

I have my txt-document with the line:
message helloworld
with your code this line changes into:
HEJHEJ
elloworld
//I would like it to be:
HEJHEJ
So in the end, i want it to change/replace to whole line of text that contains "message"

Rewriting a text file after reading it

i got a file that is store in my appliction directory, and he got some site list.
i dont have any problem reading it, but when i want to write to it, i get
System.ArgumentException: Stream is not writeable
this is how i accsess the file:
FileStream theTextFileStream = new FileStream(Environment.CurrentDirectory + "/fourmlinks.txt",FileMode.OpenOrCreate);
and this is the function that throw me the expection:
public static void WriteNewTextToFile(string text, FileStream theFile)
{
string fileText = GetAllTextFromFile(theFile);
ArrayList fileLIst = populateListFromText(fileText);
using (StreamWriter fileWriter = new StreamWriter(theFile))
{
fileWriter.Write(String.Empty);
for (int i = 0; i < fileLIst.Count; i++)
{
fileWriter.WriteLine(fileLIst[i].ToString());
}
}
}
the function read the old and new text and add it to an arry. then i clean the file from every thing, and rewriting it with the old and new data from the arry i made.
i dont know if that will help but here is the file proprites:
Build Action: None
Copy To Out Put Directory: Copy always
why i cant rewrite the file?
this is the function i use to read the file content:
public static string GetAllTextFromFile(FileStream theFile)
{
string fileText = "";
using (theFile)
{
using (StreamReader stream = new StreamReader(theFile))
{
string currentLine = "";
while ((currentLine = stream.ReadLine()) != null)
{
fileText += currentLine + "\n";
}
}
}
return fileText;
}

You have to use Read/Write file access as third parameter -
FileStream theTextFileStream = new FileStream(Environment.CurrentDirectory + "/fourmlinks.txt",FileMode.OpenOrCreate, FileAccess.ReadWrite
);
Important - Remove using(theFile) statement:
public static string GetAllTextFromFile(FileStream theFile)
{
string fileText = "";
using (StreamReader stream = new StreamReader(theFile))
{
string currentLine = "";
while ((currentLine = stream.ReadLine()) != null)
{
fileText += currentLine + "\n";
}
}
return fileText;
}
Do not use using construct in your case as it will close the underlying stream as in your case you have to manually open and close stream objects.
This will allow you to write in the file as well.
For more information refer following links -
FileStream Constructor
FileAccess Enumeration