I am trying to insert data from access database to .csv file in C#. I have 54744 records in Access database. Inserting the records in CSV file is taking more than 2 hours. Below is my code:
public void CreateCSVFile()
{
DataSet dsRecLoad = new DataSet();
DataTable dt = new DataTable("RecsCDDt");
StringBuilder sb = new StringBuilder();
using (OleDbConnection connection = new OleDbConnection(_configuration.GetConnectionString("AccessConnection")))
{
using (OleDbCommand cmd = new OleDbCommand("select * from RecsCD", connection))
{
connection.Open();
OleDbDataReader reader = cmd.ExecuteReader(CommandBehavior.Default);
dt.Load(reader);
}
}
foreach (DataRow row in dt.Rows)
{
foreach (DataColumn dc in dt.Columns)
{
if (!String.IsNullOrEmpty(sb.ToString()))
sb.Append(",");
sb.Append(row[dc.ColumnName].ToString());
}
sb.Append("\n");
}
string CSVfilePath = _configuration.GetValue<string>("MySettings:CSVFile");
if (!File.Exists(CSVfilePath))
{
File.WriteAllText(CSVfilePath, sb.ToString()) ;
}
}
Can anyone suggest me a more efficient way to insert the records in .CSV file.
Thank You.
Here's a small example of how you might achieve the same thing by streaming the data using a reader and writing to the file as a stream (untested):
await using var textWriter = new StreamWriter("path\\to\\file.csv");
await using var writer = new CsvHelper.CsvWriter(textWriter, CultureInfo.InvariantCulture);
await using var connection = new SqlConnection("connection_string");
await using var cmd = new SqlCommand("select * from RecsCD", connection);
await connection.OpenAsync();
await using var reader = await cmd.ExecuteReaderAsync();
while (await reader.ReadAsync()) {
foreach (int i in Enumerable.Range(0, reader.FieldCount)) {
writer.WriteField(await reader.GetFieldValueAsync<object>(i));
}
await writer.NextRecordAsync();
}
await writer.FlushAsync();
Note that the code depends on the CsvHelper library.
Related
I have a following code with List of guids and I need to get all corresponding items from database:
var guids = new List<string>()
{
"1HFAoQchL1NB9Tc5wJLYHm",
"1OLd2nLP92c87kmkHaMAtO",
"2FwmMnAX98sg537nfo$S39",
"38pd0XqUvEvA8nrnF$3vvZ",
"0iqYqWKBvC_x62wk0jcKW3",
"0njEfI2BzDEeQpiOo5S18q",
"0chMA4Bgf9nxnq2i1gG9Fb",
"3TrnV9L79FfBz0ni6_bJ9N"
};
foreach (var gd in guids)
{
MemoryStream stream = null;
using (var conn = new NpgsqlConnection(connectionString))
{
string sQL = $"SELECT geometricaldata FROM entities WHERE guid= '{gd}'";
using (var command = new NpgsqlCommand(sQL, conn))
{
conn.Open();
using (var reader = command.ExecuteReader())
{
while (reader.Read())
{
stream = new MemoryStream((byte[])reader[0]);
}
reader.Close();
}
}
}
}
Do I really need to do this in a loop one by one or it would be better to batch SELECT statements? Or maybe there is another way how to get multiple rows with one query that I don't know of?
I am using oledb to read excel file and then i am trying to apply some conditions and then save in db, but i just want to read excel row one by one using loop.
var cmd = conn.CreateCommand();
cmd.CommandText = "select * from [الحيازات$]";// want to get this rows using loop so
//that only specific row is selected perform opertion blow then i will save in db,but
//its selecting all rows now
using (var rdr = cmd.ExecuteReader())
{
//LINQ query - when executed will create anonymous objects for each row
var query = (from DbDataRecord row in rdr
select row)
.Select(x =>
{
//dynamic item = new ExpandoObject();
Dictionary<string, string> item = new Dictionary<string, string>();
for (int i = 0; i < x.FieldCount; i++)
{
//code
}
});
}
Please check this
private void ReadExcel()
{
try
{
string connectionString = #"Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + filePath + #";Extended Properties=""Excel 12.0;HDR=YES;IMEX=1;""";
using (OleDbConnection connection = new OleDbConnection())
{
connection.ConnectionString = connectionString;
using (OleDbCommand command = connection.CreateCommand())
{
command.CommandText = "SELECT * FROM [Sheet1$]";
connection.Open();
OleDbDataAdapter dataAdapter = new OleDbDataAdapter(command);
DataSet dataSet = new DataSet();
dataAdapter.Fill(dataSet);
if (dataSet != null && dataSet.Tables[0] != null)
{
foreach (DataRow row in dataSet.Tables[0].Rows)
{
string value = row["EmailColumnName"].ToString();
}
}
}
}
}
catch (Exception ex)
{
//Handle Exception
}
}
In my database table, I have a column name 'SectionDatatable' which are in XML datatype. In my C# code, after I have connection to database to my database and I make a query to get SectionDatatablewhich is XML format in my database, UserDefinedSectionData. I need to convert the 'SectionDatatable' in XML Datatype and convert it into DataSet, how can I do it. I have stuck for 1 day, the following is my code.
SqlConnectionStringBuilder csb = new SqlConnectionStringBuilder();
csb.DataSource = #"CPX-XSYPQKSA91D\SQLEXPRESS";
csb.InitialCatalog = "DNN_Database";
csb.IntegratedSecurity = true;
string connString = csb.ToString();
string queryString = "select * FROM UserDefinedSectionData WHERE SectionDataTabId = #tabId";
using (SqlConnection connection = new SqlConnection(connString))
using (SqlCommand command = connection.CreateCommand())
{
command.CommandText = queryString;
command.Parameters.Add(new SqlParameter("tabId", tabId));
connection.Open();
using (SqlDataReader reader = command.ExecuteReader())
{
while (reader.Read())
{
string sectionData= reader["SectionDatatable"].ToString();
int moduleId = Int32.Parse(reader["SectionDataModuleId"].ToString());
}
}
}
This is the simple example of converting XML string to DataSet. This sample also demonstrates processing all tables in DataSet.
You need to replace XML string in this sample with your XML output from database. You can change code as per you need to access data.
string RESULT_OF_SectionDatatable = "<note><to>Tove</to><from>Jani</from><heading>Reminder</heading><body>Don't forget me this weekend!</body></note>";
var xmlReader = XmlReader.Create(new StringReader(RESULT_OF_SectionDatatable));
DataSet ds = new DataSet();
ds.ReadXml(xmlReader);
foreach (DataTable table in ds.Tables)
{
Console.WriteLine(table);
Console.WriteLine();
foreach (var row in table.AsEnumerable())
{
for (int i = 0; i < table.Columns.Count; ++i)
{
Console.WriteLine(table.Columns[i].ColumnName +"\t" + row[i]);
}
Console.WriteLine();
}
}
I have a xslx file with following data
www.url.com
www.url.com
www.url.com
www.url.com
www.url.com
www.url.com
www.url.com
www.url.com
...
Like you can see I have only 1 column used and a lot of rows.
I need to read that column from the xslx file somehow and convert it to List<string>.
Any help?
Thanks!
You can use EPPlus, it's simple, something like this :
var ep = new ExcelPackage(new FileInfo(excelFile));
var ws = ep.Workbook.Worksheets["Sheet1"];
var domains = new List<string>();
for (int rw = 1; rw <= ws.Dimension.End.Row; rw++)
{
if (ws.Cells[rw, 1].Value != null)
domains.Add(ws.Cells[rw, 1].Value.ToString());
}
the easiest way is to use OleDb, you can do something like this:
List<string> values = new List<string>();
string constr = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=C:\\your\\path\\file.xlsx;Extended Properties=\"Excel 12.0 Xml;HDR=NO;\"";
using (OleDbConnection conn = new OleDbConnection(constr))
{
conn.Open();
OleDbCommand command = new OleDbCommand("Select * from [SheetName$]", conn);
OleDbDataReader reader = command.ExecuteReader();
if (reader.HasRows)
{
while (reader.Read())
{
// this assumes just one column, and the value is text
string value = reader[0].ToString();
values.Add(value);
}
}
}
foreach (string value in values)
Console.WriteLine(value);
Have you tried using ClosedXML? Super simple to do.
var wb = new XLWorkbook(FileName);
var ws2 = wb.Worksheet(1);
List<string> myData = new List<string>();
foreach (var r in ws2.RangeUsed().RowsUsed())
{
myData.Add(r.Cell(1).GetString());
}
You can use OOXML to read the file and this library simplify your work http://simpleooxml.codeplex.com.
The list will grow and shrink depending on how many items I have in my database.
I need to populate a list not a listbox. I understand I will need to open a connection.
using (var conn = new SqlConnection(Properties.Settings.Default.DBConnectionString))
{
using (var cmd = conn.CreateCommand())
{
conn.Open();
List<string> TagList = new List<string>();
for (int i = 0; i < TagList.Count; i++)
TagList[i].Add("Data from database");
cmd.ExecuteNonQuery();
}
}
I'm really not sure how to do this and I'm sure my method up here looks very wrong so I really need help.
Could someone show me what I'm doing wrong?
public IEnumerable<string> GetTagList()
{
using (var connection = new SqlConnection(Properties.Settings.Default.DBConnectionString))
using (var cmd = connection.CreateCommand())
{
connection.Open();
cmd.CommandText = "select Tag from TagsTable"; // update select command accordingly
using (var reader = cmd.ExecuteReader())
{
while (reader.Read())
{
yield return reader.GetString(reader.GetOrdinal("Tag"));
}
}
}
}
then you can call it as below
List<string> tags = GetTagList().ToList();
I would like to share my solution, hope helps someone in the future:
public List<string> getFromDataBase()
{
List<string> result = new List<string>();
using(SqlConnection con = new SqlConnection("connectionString"))
{
con.Open();
DataTable tap = new DataTable();
new SqlDataAdapter(query, con).Fill(tap);
result = tap.Rows.OfType<DataRow>().Select(dr => dr.Field<string>("columnName")).ToList();
}
return result;
}
This would do as it is (if I didn't do any typos...)
private void LoadList()
{
List<string> tagsList = new List<string>();
using (IDbConnection connection = new SqlConnection(Properties.Settings.Default.DBConnectionString))
{
connection.Open();
using (IDbCommand command = connection.CreateCommand())
{
command.CommandText = "SELECT TAGCOLUMN FROM TAGSTABLE";
using (IDataReader reader = command.ExecuteReader())
{
while (reader.Read())
{
if (!reader.IsDBNull(0))
tagsList.Add(reader.GetString(0));
}
reader.Close();
}
}
connection.Close();
}
}
EDIT:
Of course you have to change the select statement to the correct one from your database.
I just used a pseudo one to show you what to put there.