We Keep Coding

How to Search Data From Huge CSV Files (20Gb) C# ASP.NET

I want to create a program using .Net to read or search data in a 20Gb CSV file
Is there any way to do it ?
My Code For Search
string search = txtBoxSearch.Text;
string pathOnly = Path.GetDirectoryName(csvPath);
string fileName = Path.GetFileName(csvPath);
string sql = #"SELECT F1 AS StringID, F2 AS StringContent FROM [" + fileName + "] WHERE F2 LIKE '%" + search + "%'";
using (OleDbConnection connection = new OleDbConnection(
#"Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + pathOnly +
";Extended Properties=\"Text;HDR=No\""))
using (OleDbCommand command = new OleDbCommand(sql, connection))
using (OleDbDataAdapter adapter = new OleDbDataAdapter(command))
{
DataTable dataTable = new DataTable();
adapter.Fill(dataTable);
dataTable.Columns.Add("MatchTimes", typeof(System.Int32));
foreach (DataRow row in dataTable.Rows)
{
row["MatchTimes"] = Regex.Matches(row["StringContent"].ToString(), search).Count;
}
GridViewResult.DataSource = dataTable;
GridViewResult.DataBind();
My Code for generate the CSV File
int records = 100000;
File.AppendAllLines(csvPath,
(from r in Enumerable.Range(0, records)
let guid = Guid.NewGuid()
let stringContent = GenerateRandomString(256000)
select $"{guid},{stringContent}"));

This really depends on exactly how you're searching. If you're just doing a single search, you could simply read this one line at a time and do a string comparison or something. If you do this, do not load the whole thing into memory - load it one at a time.
If you have access to the "full" edition of SQL Server, you could do a BULK INSERT. If you don't, though (e.g. you're using one of the express editions), you might run into the maximum table size. In this case, I've never tried this, but you could try SQLite. In theory at least, the database can handle multiple terabytes. Be sure to insert a large number of records in each transaction, though; if you do a commit after each insert your performance will be absolutely wretched. Also, be sure that you're not creating an in-memory database, or you'll just run out of memory again.

How to bulk copy data from CSV file to SQL?

I tried like this:
public Datatable GetDatatable(){
string[] csv=#"1,ABC,Arun,12/12/2017\n
2,BCD,Sam,10/12/2017\n
3,XYZ,Ammy,11/12/2017\n
4,PQR,Varun,9/12/2017\n";
DataTable table = new DataTable();
table.Columns.Add("Dosage", typeof(int));
table.Columns.Add("Drug", typeof(string));
table.Columns.Add("Patient", typeof(string));
table.Columns.Add("Date", typeof(DateTime));
foreach (var line in csv)
{
string[] l=line.split(',');
table.Rows.Add(l[0]);
table.Rows.Add(l[1]);
table.Rows.Add(l[2]);
table.Rows.Add(l[3]);
}
return table;
}
csv like
1,ABC,Arun,12/12/2017
2,BCD,Sam,10/12/2017
3,XYZ,Ammy,11/12/2017
4,PQR,Varun,9/12/2017
It is done by using foreach. I want same result without using for loop
or foreach because CSV contain above 3 million lines. Please give some ideas or suggestions.

You could use OLEDB to read the CSV file using SQL (kinda) queries:
Check this answer for an example: Most efficient way to process a large csv in .NET
You could also use an existing library such as FileHelpers: http://www.filehelpers.net/ it allows to parse large CSV files easily and very fast.

In a way or another a loop will be always executed to read the file, split the line and insert it in the DataTable.
There are many free libraries to work with CSV file and if you execute a simple search you will find them really easily.
However your code could be improved (and fixed) with splitting the line just one time and not 4 times in that loop and adding the row with just one single line
(Actually your code adds 4 rows for each line and this seems pretty wrong)
foreach (var line in csv)
{
string[] parts = line.Split(',');
table.Rows.Add(parts);
}
return table;
If this will be enough for your requirements it is up to you to evaluate. With so many rows to read I would do an extensive comparison of the features and performances of different approaches. (Using libraries or self made code)

This is working no need to go for third party Use SQLBulk Copy Instead
public static void BulInsert()
{
var destination = #"Data Source = (localdb)\MSSQLLocalDB; Initial Catalog = SampleDatabase; Integrated Security = True; Connect Timeout = 30; Encrypt = False; TrustServerCertificate = False; "; // your comnnection string
var filename = #"d:\db.csv";//your source file
var connString = string.Format(#"Provider=Microsoft.Jet.OleDb.4.0; Data Source={0};Extended Properties=""Text;HDR=No;FMT=Delimited""", Path.GetDirectoryName(filename));
string query = string.Format("Select * from [{0}]", Path.GetFileName(filename));
using (var conn = new OleDbConnection(connString))
{
conn.Open();
OleDbCommand command = new OleDbCommand(query, conn);
var reader = command.ExecuteReader();
using (SqlConnection destConnection = new SqlConnection(destination))
{
try
{
destConnection.Open();
using (SqlBulkCopy bulkCopy =
new SqlBulkCopy(destConnection))
{
// what ever mapping with ordinal
bulkCopy.ColumnMappings.Add(new SqlBulkCopyColumnMapping(0, "Id"));
bulkCopy.ColumnMappings.Add(new SqlBulkCopyColumnMapping(1, "name"));
bulkCopy.ColumnMappings.Add(new SqlBulkCopyColumnMapping(2, "visit"));
bulkCopy.DestinationTableName =
"dbo.Patients";
try
{
bulkCopy.WriteToServer(reader);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
reader.Close();
}
}
}
catch (Exception)
{
}
}
}
}

You can use Cinchoo ETL library, if you want to convert CSV to Datatable or bulk import CSV to sqlserver easily.
To convert CSV to datatable, the code below shows how to do it
using (var p = new ChoCSVReader("emp.csv").WithFirstLineHeader())
{
DataTable dt = p.AsDataTable();
}
If you want to bulk import CSV to sqlserver, you can refer this codeproject article on how to do it.
Cinchoo ETL - Bulk Insert CSV File into SQLServer
Hope this helps.
Disclaimer: I'm author of this library.

SQL Bulk Insert in C# not inserting values

I'm completely new to C#, so I'm sure I'm going to get a lot of comments about how my code is formatted - I welcome them. Please feel free to throw any advice or constructive criticisms you might have along the way.
I'm building a very simple Windows Form App that is eventually supposed to take data from an Excel file of varying size, potentially several times per day, and insert it into a table in SQL Server 2005. Thereafter, a stored procedure within the database takes over to perform various update and insert tasks depending on the values inserted into this table.
For this reason, I've decided to use the SQL Bulk Insert method, since I can't know if the user will only insert 10 rows - or 10,000 - at any given execution.
The function I'm using looks like this:
public void BulkImportFromExcel(string excelFilePath)
{
excelApp = new Excel.Application();
excelBook = excelApp.Workbooks.Open(excelFilePath);
excelSheet = excelBook.Worksheets.get_Item(sheetName);
excelRange = excelSheet.UsedRange;
excelBook.Close(0);
try
{
using (SqlConnection sqlConn = new SqlConnection())
{
sqlConn.ConnectionString =
"Data Source=" + serverName + ";" +
"Initial Catalog=" + dbName + ";" +
"User id=" + dbUserName + ";" +
"Password=" + dbPassword + ";";
using (OleDbConnection excelConn = new OleDbConnection())
{
excelQuery = "SELECT InvLakNo FROM [" + sheetName + "$]";
excelConn.ConnectionString = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + excelFilePath + ";Extended Properties='Excel 8.0;HDR=Yes'";
excelConn.Open();
using (OleDbCommand oleDBCmd = new OleDbCommand(excelQuery, excelConn))
{
OleDbDataReader dataReader = oleDBCmd.ExecuteReader();
using (SqlBulkCopy bulkImport = new SqlBulkCopy(sqlConn.ConnectionString))
{
bulkImport.DestinationTableName = sqlTable;
SqlBulkCopyColumnMapping InvLakNo = new SqlBulkCopyColumnMapping("InvLakNo", "InvLakNo");
bulkImport.ColumnMappings.Add(InvLakNo);
sqlQuery = "IF OBJECT_ID('ImportFromExcel') IS NOT NULL BEGIN SELECT * INTO [" + DateTime.Now.ToString().Replace(" ", "_") + "_ImportFromExcel] FROM ImportFromExcel; DROP TABLE ImportFromExcel; END CREATE TABLE ImportFromExcel (InvLakNo INT);";
using (SqlCommand sqlCmd = new SqlCommand(sqlQuery, sqlConn))
{
sqlConn.Open();
sqlCmd.ExecuteNonQuery();
while (dataReader.Read())
{
bulkImport.WriteToServer(dataReader);
}
}
}
}
}
}
}
catch(Exception ex)
{
MessageBox.Show(ex.ToString());
}
finally
{
excelApp.Quit();
}
}
The function runs without errors or warnings, and if I replace the WriteToServer with manual SQL commands, the rows are inserted; but the bulkImport isn't inserting anything.
NOTE: There is only one field in this example, and in the actual function I'm currently running to test; but in the end there will be dozens and dozens of fields being inserted, and I'll be doing a ColumnMapping for all of them.
Also, as stated, I am aware that my code is probably horrible - please feel free to give me any pointers you deem helpful. I'm ready and willing to learn.
Thanks!

I think it would be a very long and messy answer if I commented on your code and also gave pointer sample codes in the same message, so I decided to divide then into two messages. Comments first:
You are using automation to get what? You already have the sheet name as I see it and worse you are doing app.Quit() at the end. Completely remove that automation code.
If you needed some information from excel (like sheet names, column names) then you could use OleDbConnecton's GetOleDbSchemaTable method.
You might do the mapping basically in 2 ways:
Excel column ordinal to SQL table column name
Excel column name to SQL table column name
both would do. In a generic code, assuming you have column names same in both sources, but their ordinal and count may differ, you could get the column names from OleDbConnection schema table and do the mapping in a loop.
You are dropping and creating a table named "ImportFromExcel" for the purpose of temp data insertion, then why not simply create a temp SQL server table by using a # prefix in table name? OTOH that code piece is a little weird, it would do an import from "ImportFromExcel" if it is there, then drop and create a new one and attempt to do bulk import into that new one. In first run, SqlBulkCopy (SBC) would fill ImportFromExcel and on next run it would be copied to a table named (DateTime.Now ...) and then emptied via drop and create again. BTW, naming:
DateTime.Now.ToString().Replace(" ", "_") + "_ImportFromExcel"
doesn't feel right. While it looks tempting, it is not sortable, probably you would want something like this instead:
DateTime.Now.ToString("yyyyMMddHHmmss") + "_ImportFromExcel"
Or better yet:
"ImportFromExcel_" +DateTime.Now.ToString("yyyyMMddHHmmss")
so you would have something that is sorted and selectable for all the imports as a wildcard or looping for some reason.
Then you are writing to server inside a reader.Read() loop. That is not the way WriteToServer works. You wouldn't do reader.Read() but simply:
sbc.WriteToServer(reader);
In my next message e I will give simple schema reading and a simple SBC sample from excel into a temp table, as well as a suggestion how you should do that instead.

Here is the sample for reading schema information from Excel (here we read the tablenames - sheet names with tables in them):
private IEnumerable<string> GetTablesFromExcel(string dataSource)
{
IEnumerable<string> tables;
using (OleDbConnection con = new OleDbConnection("Provider=Microsoft.ACE.OLEDB.12.0;" +
string.Format("Data Source={0};", dataSource) +
"Extended Properties=\"Excel 12.0;HDR=Yes\""))
{
con.Open();
var schemaTable = con.GetOleDbSchemaTable(OleDbSchemaGuid.Tables, null);
tables = schemaTable.AsEnumerable().Select(t => t.Field<string>("TABLE_NAME"));
con.Close();
}
return tables;
}
And here is a sample that does SBC from excel into a temp table:
void Main()
{
string sqlConnectionString = #"server=.\SQLExpress;Trusted_Connection=yes;Database=Test";
string path = #"C:\Users\Cetin\Documents\ExcelFill.xlsx"; // sample excel sheet
string sheetName = "Sheet1$";
using (OleDbConnection cn = new OleDbConnection(
"Provider=Microsoft.ACE.OLEDB.12.0;Data Source="+path+
";Extended Properties=\"Excel 8.0;HDR=Yes\""))
using (SqlConnection scn = new SqlConnection( sqlConnectionString ))
{
scn.Open();
// create temp SQL server table
new SqlCommand(#"create table #ExcelData
(
[Id] int,
[Barkod] varchar(20)
)", scn).ExecuteNonQuery();
// get data from Excel and write to server via SBC
OleDbCommand cmd = new OleDbCommand(String.Format("select * from [{0}]",sheetName), cn);
SqlBulkCopy sbc = new SqlBulkCopy(scn);
// Mapping sample using column ordinals
sbc.ColumnMappings.Add(0,"[Id]");
sbc.ColumnMappings.Add(1,"[Barkod]");
cn.Open();
OleDbDataReader rdr = cmd.ExecuteReader();
// SqlBulkCopy properties
sbc.DestinationTableName = "#ExcelData";
// write to server via reader
sbc.WriteToServer(rdr);
if (!rdr.IsClosed) { rdr.Close(); }
cn.Close();
// Excel data is now in SQL server temp table
// It might be used to do any internal insert/update
// i.e.: Select into myTable+DateTime.Now
new SqlCommand(string.Format(#"select * into [{0}]
from [#ExcelData]",
"ImportFromExcel_" +DateTime.Now.ToString("yyyyMMddHHmmss")),scn)
.ExecuteNonQuery();
scn.Close();
}
}
While this would work, thinking in the long run, you need column names, and maybe their types differ, it might be an overkill to do this stuff using SBC and you might instead directly do it from MS SQL server's OpenQuery:
SELECT * into ... from OpenQuery(...)

the WriteToServer(IDataReader) is intended to do internally the IDataReader.Read()operation.
using (SqlCommand sqlCmd = new SqlCommand(sqlQuery, sqlConn))
{
sqlConn.Open();
sqlCmd.ExecuteNonQuery();
bulkImport.WriteToServer(dataReader);
}
You can check the MSDN doc on that function, has a working example: https://msdn.microsoft.com/en-us/library/434atets(v=vs.110).aspx

local database won't update and doesn't show errors

Hey i'm new to this and from what i managed to pick up this should be working but it doesn't update my local database.
I have a TelemarketingDatabaseDataSet that was auto generated when i created my local database, which then i dragged the table onto the dataset and i guess they're linked.
Now i have this code :
SqlCeConnection connection = new SqlCeConnection();
connection.ConnectionString = TelemarketingTracker.Properties.Settings.Default.TelemarketingDatabaseConnectionString;
TelemarketingDatabaseDataSet ds = new TelemarketingDatabaseDataSet();
// DataTable tbl = new DataTable();
SqlCeDataAdapter adapter = new SqlCeDataAdapter("select * from Calls", connection);
//adapter.InsertCommand = new SqlCeCommand("InsertQuery", connection);
adapter.Fill(ds,"Calls");
DataTable tbl = ds.Tables["Calls"];
//tbl.Columns.Add("caller");
//tbl.Columns.Add("called");
//tbl.Columns.Add("duration");
//tbl.Columns.Add("time");
var row = tbl.NewRow();
row[1] = Convert.ToString(caller);
row[2] = Convert.ToString(called);
row[3] = Convert.ToString(duration);
row[4] = Convert.ToDateTime(time);
tbl.Rows.Add(row);
adapter.Update(ds, "Calls");
connection.Close();
MessageBox.Show("Database should be updated!");
And please, i'm not intrested in using an SqlCommand as i prefer using DataSet.
Could the problem be related to datatypes of my table? it doesn't show errors to suggest that but i guess this could be the problem. my Table consists of :
ID - int,key
caller - varchar
called - varchar
duration - varchar
time - datetime
EDIT:
Now if i uncomment the insertQuery row i get an unhandled error occured in Syste.Data dll.
Now even if i try to use a regular insert command i get no errors but the database won't update.
if this makes any diffrence after i close the debugging window i see an X next to the local database but it doesn't show any errors.
This is the command i've tried :
using (SqlCeCommand com = new SqlCeCommand("INSERT INTO Calls (caller, called, duration, time) Values(#Caller,#Called,#Duration,#Time)", connection))
{
com.Parameters.AddWithValue("#Caller", row[1]);
com.Parameters.AddWithValue("#Called", row[2]);
com.Parameters.AddWithValue("#Duration", row[3]);
com.Parameters.AddWithValue("#Time", row[4]);
com.ExecuteNonQuery();
}

The Fill() method "Adds or refreshes rows in the DataSet to match those in the data source." The key part of this sentence being "to match those in the data source". The row you're adding gets wiped out when you call Fill() because it's not already in the source.
I'm not positive, but I don't think that you need to even call Fill() if you're only adding new records and not worried about modifying/removing existing ones. If you do need to call it though, it would obviously need to be moved before any new record insertions you make.

Try something similar to this..
string dbfile = new System.IO.FileInfo(System.Reflection.Assembly.GetExecutingAssembly().Location).DirectoryName + "\\TelemarketingDatabase.sdf";
SqlCeConnection connection = new SqlCeConnection("datasource=" + dbfile);
TelemarketingDatabaseDataSet ds = new TelemarketingDatabaseDataSet();
SqlCeDataAdapter adapter = new SqlCeDataAdapter();
string qry = #"select * from Calls";
da.SelectCommand = new SqlCommand(qry, connection);
SqlCommandBuilder cb = new SqlCommandBuilder(adapter);
adapter.Fill(ds,"Calls");
DataTable tbl = ds.Tables["Calls"];
var row = tbl.NewRow();
row[0] = caller;
row[1] = called;
row[2] = duration;
row[3] = Convert.ToDateTime(time);
tbl.Rows.Add(row);
adapter.Update(ds,"Calls");
See the example here http://www.java2s.com/Code/CSharp/Database-ADO.net/UseDataTabletoupdatetableinDatabase.htm

Well in the end i didn't manage to solve this, instead i used a remote database and regular sql commands.
Thanks for those who helped!

just want to share this even the if the question is old
using System;
using System.IO; //needed for path.getdirectoryname() and directory.getcurrentdirectory()
string path = Path.GetDirectoryName(Path.GetDirectoryName(Directory.GetCurrentDirectory()));
AppDomain.CurrentDomain.SetData("DataDirectory", path);
Directory.GetcurrentDirectory() will output "C:/..projectname/bin/debug" which is where the temporary database.mdf is located
by using Path.GetDirectoryName(Directory.GetcurrentDirectory()) it will give the directory of the current directory thus moving one location back
"C:/..projectname/bin"
then use it again
Path.GetDirectoryName(Path.GetDirectoryName(Directory.GetCurrentDirectory())) will give you the location of the root database in your project folder
"C:/..projectname"
then just use AppDomain.CurrentDomain.SetData()

Reading excel file in c#.net cell by cell

I'm new to c#.net
I have excel sheet and I want to import into database.
I want to read it cell by cell and want to insert value in database.
this.openFileDialog1.FileName = "*.xls";
DialogResult dr = this.openFileDialog1.ShowDialog();
if (dr == System.Windows.Forms.DialogResult.OK)
{
string path = openFileDialog1.FileName;
string connectionString = String.Format(#"Provider=Microsoft.Jet.OLEDB.4.0;Data Source={0};Extended Properties=""Excel 8.0;HDR=no;IMEX=1;""", openFileDialog1.FileName);
string query = String.Format("select * from [{0}$]", "Sheet3");
OleDbDataAdapter dataAdapter = new OleDbDataAdapter(query, connectionString);
DataSet dataSet = new DataSet();
dataAdapter.Fill(dataSet);
dataGridView1.DataSource = dataSet.Tables[0];

I assume that after you execute the code in your question, you can see the values within dataGridView1.
The actual reading from the excel sheet is done when calling dataAdapter.Fill. So, in your case, reading the cells comes down to indexing columns and rows in dataSet.Tables[0].
For example:
for (int row = 0; row < dataSet.Tables[0].Rows.Count; row++)
{
DataRow r = dataSet.Tables[0].Rows[row];
}
Accessing the cells in row r is trivial (like the sample above, just for cell).
EDIT
I forgot to describe the "insert the values into a database" part. I presume that the database is SQL Server (may be Express edition, too).
First: create a database connection. Instead of manually composing the connection string, use the SqlConnectionStringBuilder:
SqlConnectionStringBuilder csb = new SqlConnectionStringBuilder();
csb.DataSource = <your server instance, e.g. "localhost\sqlexpress">;
csb.InitialCatalog = <name of your database>;
csb.IntegratedSecurity = <true if you use integrated security, false otherwise>;
if (!csb.IntegratedSecurity)
{
csb.UserId = <User name>;
csb.Password = <Password>;
}
Then, create and open a new SqlConnection with the connection string:
using (SqlConnection conn = new SqlConnection(csb.ConnectionString))
{
conn.Open();
Iterate over all the values you want to insert and execute a respective insert command:
for (...)
{
SqlCommand cmd = new SqlCommand("INSERT INTO ... VALUES (#param1, ..., #paramn)", conn);
cmd.Parameters.AddWithValue("#param1", value1);
...
cmd.Parameters.AddWithValue("#paramn", valuen);
cmd.ExecuteNonQuery();
}
This closes the connection, as the using block ends:
}
And there you go. Alternatively, you could use a data adapter with a special insert-command. Then, inserting the values would come down to a one-liner, however, your database table must have the same structure as the Excel-sheet (respectively: as the data table you obtained in the code you posted.

Check out NPOI
http://npoi.codeplex.com/
It's the .NET version of Apache's POI Excel implementation. It'll easily do what you need it to do, and will help avoid some of the problems ( i.e. local copy of Excel, or worse, copy of Excel on the server ) that you'll face when using the Jet provider.