Given a SQL table Document with data:
ID Content ByteLength Sequence
1 Part1... 945669 1
1 Part2... 945669 2
1 Part3... 945669 3
...
2 Part1... 45234 1
2 Part2... 45234 2
Where:
Document.Content = Up to 32KB of data
Document.ByteLength = Total data in bytes for the Document ID
Document.Sequence = Order of content in the Document ID
How can I read all of Document.Content into a single byte array byte[] Content?
using (var command = new SqlCommand("SELECT Content FROM Document WHERE ID=1 ORDER BY Sequence", connection))
using (var reader = command.ExecuteReader())
{
while(reader.Read())
{
// What goes here?
// if we just want one row:
//var fileBytes = (byte[])reader.GetValue(0); // read the file data from the selected row (first column in above query)
}
}
Given the fact that this Content field contains text data, you can simply use a StringBuilder to add data while you read the content field
using (var command = new SqlCommand("SELECT Content FROM Document WHERE ID=1 ORDER BY Sequence", connection))
using (var reader = command.ExecuteReader())
{
// Set a large enough initial capacity
StringBuilder sb = new StringBuilder(32767);
while(reader.Read())
{
sb.Append(reader.GetString(0));
}
}
Now, at the loop exit, all the content is in the StringBuilder buffer and you can get it back in a byte array with
byte[] buffer = System.Text.Encoding.UTF8.GetBytes(sb.ToString());
Related
I have the following code where I look at columns headers before some condition statement:
var csvConfig = new CsvHelper.Configuration.CsvConfiguration(CultureInfo.InvariantCulture)
{
// csvconfig.Delimiter = "\t";
HasHeaderRecord = true
};
using (StreamReader reader = new StreamReader(filePath, Encoding.UTF8))
using (CsvReader csv1 = new CsvReader(reader, csvConfig))
{
CsvReader csv2 = csv1;
csv2.Read();
csv2.ReadHeader();
string[] headers = csv2.HeaderRecord;
if(headers[0].Replace("\0", "").ToUpper() == "TEST")
{
using (var dr1 = new CsvDataReader(csv1))
{
var dt1 = new DataTable();
dt1.Load(dr1);
}
}
}
I thought that by having 2 variables for the CsvReader (csv1 and csv2) it would be feasible but it seems that they both use the same object in memory.
Therefore when I want to use csv2 to fill my datatable, the header row has been already read in csv1 and is not loaded in my datatable.
How can I make sure that csv2 contains the whole csv and is distinct from csv1? Is there a method to go back to the beginning or do I need to read the whole CSV again using CsvReader?
Thank you
In C#, datas types are categorized based on how they store their value in the memory : by value or by reference (by pointer exists too).
For example, a string is usually stored by value but complex objects like your reader are almost always stored by reference.
When you do csv2 = csv1 both now refer to the same memory area. This means that they are 2 names for the same thing. When you do an action on csv1, csv2 also receives it since they are 2 aliases for the same information.
Try if CSVreader implements Clone() :
CsvReader csv2 = csv1.Clone();
If it is, the function will create a new object with the same informations which do not share the same memory area.
You need a separate CsvReader for csv1 and csv2. As both you #UserNam3 advise, you don't want csv2 to be referencing the same object as csv1 in memory. They will both be using the same Stream, so after reading the header with csv2 you will need to reset the Stream back to the beginning.
using (StreamReader reader = new StreamReader(filePath, Encoding.UTF8))
using (CsvReader csv1 = new CsvReader(reader, csvConfig))
using (CsvReader csv2 = new CsvReader(reader, csvConfig))
{
csv2.Read();
csv2.ReadHeader();
string[] headers = csv2.HeaderRecord;
reader.BaseStream.Position = 0;
if(headers[0].Replace("\0", "").ToUpper() == "TEST")
{
using (var dr1 = new CsvDataReader(csv1))
{
var dt1 = new DataTable();
dt1.Load(dr1);
}
}
}
I wrote a method which creates a list of strings. The string's values are accountance data.
When I click on a button, a new .csv-file will be created.
It looks like this:
As you can see, there is no newline carriage return feed at the end of the line.
I would like to combine all of these .csv files to 1, each dataset for 1 row.
I tried that manually with this simple cmd copy command copy *.csv allcsv.csv but they are all appended in the first row instead of added to the next row:
What do I need to add/change in my code to include the newline character at the end of each row?
How could I include the cmd copy command in my method the easiest way possible?
private void BuchungssatzBilden(object obj)
{
//Lieferschein-Buchungswerte in Liste speichern
List<string> bs = new List<string>();
bs.Add(SelItem.Umsatz.ToString());
bs.Add(SelItem.Gegenkonto);
bs.Add(SelItem.Beleg);
bs.Add(SelItem.Buchungsdatum);
bs.Add(SelItem.Konto);
bs.Add(SelItem.Kost1);
bs.Add(SelItem.Kost2);
bs.Add(SelItem.Text);
using (var stream = new MemoryStream())
using (var reader = new StreamReader(stream))
using (var sr = new StreamWriter(#"C:\" + SelItem.Beleg + SelItem.Text + SelItem.Hv + ".csv", true, Encoding.UTF8))
{
using (var csv = new CsvWriter(sr, System.Globalization.CultureInfo.CurrentCulture))
{
//csv.Configuration.Delimiter = ";";
//csv.Configuration.HasHeaderRecord = true;
foreach (var s in bs)
{
csv.WriteField(s);
}
csv.Flush();
stream.Position = 0;
reader.ReadToEnd();
}
}
MessageBox.Show("CSV erfolgreich erstellt!");
}
Hello I've got a data(in txt file) which contains 1,4 million rows ++ and i need to separate the data based on ID. So if in the data has 10 differents ID, then the console application will create 10 different files which each files contains data who has same ID.
My problem is , the data that have been given to me , not all columns to be insert, so if the data has has 10 columns, i only need to take 8 columns
Here's the code that i use to write and separate data into files with differents id
string appPath = AppDomain.CurrentDomain.BaseDirectory;
string dirTxt = appPath + "VA_" + tglskrg;
string dirZip = appPath + "VA_" + tglskrg + "\\ZIP";
var writers = new Dictionary<string, TextWriter>();
string header = "COMPANY CODE;CUSTOMER NUMBER;CUSTOMER NAME;INSERT DATE;TRANSACTION ID;TRANSACTION AMOUNT;ADMIN FEE;TRANSACTION REF;FLAG STATUS;TRANSACTION STATUS"; //its still 10 columns because my code still write all the columns in the old data .
string inputFile = appPath + "va_txn_success_daily_"+tglkemarin+".txt";
string outputPath = dirTxt;
string outputPathh = dirZip;
TextWriter writer;
using (var reader = File.OpenText(inputFile))
{
//skip header
reader.ReadLine();
try
{
while (!reader.EndOfStream)
{
//read one line and separate key and value
var line = reader.ReadLine();
var separatorIndex = line.IndexOf(';');
var id = line.Substring(0, separatorIndex);
var value = line.Substring(separatorIndex - 5);
//get a writer or create one
if (!writers.TryGetValue(id, out writer))
{
writer = File.CreateText(dirTxt + "\\" + "va_txn_success_" + id + "_" + tglskrg + ".txt");
writer.WriteLine(header);
writers.Add(id, writer);
}
//write the line to the correct file
writer.WriteLine(value);
}
}
finally
{
reader.Close();
//dispose all the writers
foreach (var writerr in writers.Values)
{
writerr.Dispose();
}
}
I cant show the data , because the data its restricted
if i can give example so will be
COMPANY CODE;CUSTOMER NUMBER;CUSTOMER NAME;INSERT DATE;TRANSACTION ID;TRANSACTION AMOUNT;ADMIN FEE;TRANSACTION REF;FLAG STATUS;TRANSACTION STATUS;
A;01;Ricky;15-Jan;ABC01;1000;0;BCD123;Success;Trans success
B;02;John;15-Feb;ABC02;1500;1000;BCC122;Success;Trans success
A;02;Ricky;1-Jan;Abc03;2000;0;BCC;123;Success;Trans success
So it will be create 2 file , for A company Code and B company Code
And i want to take the company code, cust number, cust name, insert date, trans amount, trans ref, flag status ,and trans status only
Thankyou
Read the items of the line into an array. Easiest way: String.Split
string[] items = String.Split(";");
Then you are able to acces the columns;
string company = item[0};
string customer = item[1];
....
I'm trying to set up code to import .CSV files into .NET.
I've tried both Microsoft.Jet.OLEDB.4.0 and Microsoft.ACE.OLEDB.12.0 providers, including modifying the Extended Properties and even modifying corresponding registry keys for each. I have yet to come up with a solution for what I am attempting to do:
I would like to import each field as text, but leave fields longer than 255 characters un-truncated.
What I've found so far is that I can have one or the other, but not both.
If I set the ImportMixedTypes registry value to Majority Type, it leaves 255+ character text fields un-truncated, but converts other fields to unwanted types.
If I set the ImportMixedTypes registry value to Text, it truncates 255+ character text fields, but leaves the other field types as text.
How do I accomplish this using OleDb?
Additional info:
I have a "notes" column, which can contain very lengthy text. I also have a "zip code" column, which contains mixed zip-code formats (5-digit and 9-digit with a dash). Typically, the 5-digit zip-code format is more popular, so the importer thinks that the column should be integer type, leaving the 9-digit zip-codes as null values after import.
Have you considered using something as versatile as the FileHelpers library (http://filehelpers.sourceforge.net/) instead?
Or alternatively if your requirements are no more than you state (read csv file, get string fields), use something really simple such as:
public static class SimpleCsvImport
{
public static IEnumerable<List<string>> Import(string csvFileName)
{
using (var reader = File.OpenText(csvFileName))
{
while (!reader.EndOfStream)
{
var fields = reader.ReadLine().Split(new[] { ',' }, StringSplitOptions.None).Select(f => f.Trim()).ToList();
if (fields.Count > 0)
yield return fields;
}
}
}
}
i have implemented this code to read memo field (Microsoft Access):
private string GetMemoField(string TableName, string FieldName, string IdentityFieldName, string IdentityFieldValue, OleDbConnection conn)
{
string ret = "";
OleDbCommand cmd1 = new OleDbCommand("SELECT " + FieldName + " FROM “ + TableName + “ WHERE " + IdentityFieldName + "=" + IdentityFieldValue, conn);
var reader = cmd1.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); // Create the DataReader that will get the memo field one buffer at a time
if (reader.Read())
{
long numberOfChars = reader.GetChars(/*Field pos*/ 0, 0, null, 0, 0); // Total number of memo field's chars
if (numberOfChars > 0)
{
int bufferSize = 1024;
char[] totalBuffer = new char[64*bufferSize]; // Array to hold memo field content
long dataIndex = 0;
do
{
char[] buffer = new char[bufferSize]; // Buffer to hold single read
long numberOfCharsReaded = reader.GetChars(0, dataIndex, buffer, 0, bufferSize);
if (numberOfCharsReaded == 0)
{
ret = new string(totalBuffer,0, (int)numberOfChars);
break;
}
Array.Copy(buffer, 0, totalBuffer, dataIndex, numberOfCharsReaded); // Add temporary buffer to main buffer
dataIndex += numberOfCharsReaded;
} while (true);
}
}
return ret;
}
Hi all i have my database structure as follows
Field Type
FileHeader longblob
BatchHeader longblob
Entry longblob
BtchEntry longblob
FileControl longblob
I will have the data to be inserted is as follows
101 111111111 1111111111104021031A094101
52201 1 1 PPD1 110402110402 1111000020000001
6221110000251 00000000011 1 1 0111000020000001
822000000100111000020000000000000000000000011 111000020000001
52251 1 1 CCD1 110402110402 1111000020000002
6281110000251 00000000011 1 1 0111000020000002
822500000100111000020000000000010000000000001 111000020000002
9000006000001000000060066600012000000000003000000000003
as you can observe there are multiple lines that starts with 5,6 and 8. I would like to save those individually to the corresponding columns of my table. Is it possible to do if so can any mention the best method to do it. If unclear please specify
The code i written is
using (StreamReader srRead = new StreamReader(filePath))
{
while (srRead.Peek() >= 0)
{
strLine = srRead.ReadLine();
if (strLine.StartsWith("1"))
{
strFileHeader = strLine;
}
if (strLine.StartsWith("5"))
{
strBatchHeader = strLine;
}
if (strLine.StartsWith("6"))
{
strEntry = strLine;
}
if (strLine.StartsWith("8"))
{
strBtchcntrl = strLine;
}
if (strLine.StartsWith("9"))
{
strFileCntrl = strLine;
}
}
string strQuery = "insert into tblfiles(FName, FData,FileHeader,BatchHeader,Entry,BtchEntry,FileControl) values (#_FName,#_FData,#_FileHeader,#_BtchHeader,#_EntryDets,#_BtchCntrl,#_FileCntrl)";
MySqlCommand cmd = new MySqlCommand(strQuery);
cmd.Parameters.Add("#_FName", MySqlDbType.VarChar).Value = filename;
cmd.Parameters.Add("#_FData", MySqlDbType.LongBlob).Value = bytes;
cmd.Parameters.Add("#_FileHeader", MySqlDbType.LongBlob).Value = strFileHeader;
cmd.Parameters.Add("#_BtchHeader", MySqlDbType.LongBlob).Value = strBatchHeader;
cmd.Parameters.Add("#_EntryDets", MySqlDbType.LongBlob).Value = strEntry;
cmd.Parameters.Add("#_BtchCntrl", MySqlDbType.LongBlob).Value = strBtchcntrl;
cmd.Parameters.Add("#_FileCntrl", MySqlDbType.LongBlob).Value = strFileCntrl;
InsertUpdateData(cmd);
But this will insert the latest to the DB but i would like to save each and every line as per i stated
No - a column can only store one value per row. You could combine all your batch headers into one blob and store that as a single value, but you would have to be able to split them apart again when your read the data.
Instead - it looks as though:
each file starts with a '1' record and ends with a '9' record
each file contains zero or more batches
each batch starts with a '5' record and ends with an '8' record
each batch contains zero or more entries ('6' records)
If that is all correct, then you need 3 tables that would look something like:
File table:
Field Type
----------- --------
FileID integer # unique file ID - see AUTO_INCREMENT in the MySQL reference
FName varchar
FData longblob
FileHeader longblob # '1' record
FileControl longblob # '9' record
Batch table:
Field Type
----------- --------
FileID integer # references a row in the File table
BatchID integer # unique batch ID
BatchHeader longblob # '5' record
BatchControl longblob # '8' record
BatchEntry table:
Field Type
----------- --------
BatchID integer # references a row in the Batch table
EntryId integer # unique file ID
Entry longblob # '6' record
That should get you started. Good luck.
Why don't you use Stringbuilder and append the required lines to that string builder and write them to the DB instead of using strings. Seperating each column will be a tough one to retrieve the data if you need. So declare a string builder and append the lines to each and every one you required and after all write to the DB
string strFileHeader = string.Empty;
StringBuilder strBatchHeader=new StringBuilder();
StringBuilder strEntry=new StringBuilder();
StringBuilder strBtchcntrl=new StringBuilder();
string strFileCntrl = string.Empty;
using (StreamReader srRead = new StreamReader(filePath))
{
while (srRead.Peek() >= 0)
{
strLine = srRead.ReadLine();
if (strLine.StartsWith("1"))
{
strFileHeader = strLine;
}
if (strLine.StartsWith("5"))
{
strBatchHeader.AppendLine(strLine);
}
if (strLine.StartsWith("6"))
{
strEntry.AppendLine(strLine);
}
if (strLine.StartsWith("8"))
{
strBtchcntrl.AppendLine(strLine);
}
if (strLine.StartsWith("9"))
{
strFileCntrl = strLine;
}
}
string strQuery = "insert into tblfiles(FName, FData,FileHeader,BatchHeader,Entry,BtchEntry,FileControl) values (#_FName,#_FData,#_FileHeader,#_BtchHeader,#_EntryDets,#_BtchCntrl,#_FileCntrl)";
MySqlCommand cmd = new MySqlCommand(strQuery);
cmd.Parameters.Add("#_FName", MySqlDbType.VarChar).Value = filename;
cmd.Parameters.Add("#_FData", MySqlDbType.LongBlob).Value = bytes;
cmd.Parameters.Add("#_FileHeader", MySqlDbType.LongBlob).Value = strFileHeader;
cmd.Parameters.Add("#_BtchHeader", MySqlDbType.LongBlob).Value = strBatchHeader.ToString();
cmd.Parameters.Add("#_EntryDets", MySqlDbType.LongBlob).Value = strEntry.ToString();
cmd.Parameters.Add("#_BtchCntrl", MySqlDbType.LongBlob).Value = strBtchcntrl.ToString();
cmd.Parameters.Add("#_FileCntrl", MySqlDbType.LongBlob).Value = strFileCntrl;
InsertUpdateData(cmd);