We Keep Coding

How to Search Data From Huge CSV Files (20Gb) C# ASP.NET

I want to create a program using .Net to read or search data in a 20Gb CSV file
Is there any way to do it ?
My Code For Search
string search = txtBoxSearch.Text;
string pathOnly = Path.GetDirectoryName(csvPath);
string fileName = Path.GetFileName(csvPath);
string sql = #"SELECT F1 AS StringID, F2 AS StringContent FROM [" + fileName + "] WHERE F2 LIKE '%" + search + "%'";
using (OleDbConnection connection = new OleDbConnection(
#"Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + pathOnly +
";Extended Properties=\"Text;HDR=No\""))
using (OleDbCommand command = new OleDbCommand(sql, connection))
using (OleDbDataAdapter adapter = new OleDbDataAdapter(command))
{
DataTable dataTable = new DataTable();
adapter.Fill(dataTable);
dataTable.Columns.Add("MatchTimes", typeof(System.Int32));
foreach (DataRow row in dataTable.Rows)
{
row["MatchTimes"] = Regex.Matches(row["StringContent"].ToString(), search).Count;
}
GridViewResult.DataSource = dataTable;
GridViewResult.DataBind();
My Code for generate the CSV File
int records = 100000;
File.AppendAllLines(csvPath,
(from r in Enumerable.Range(0, records)
let guid = Guid.NewGuid()
let stringContent = GenerateRandomString(256000)
select $"{guid},{stringContent}"));

This really depends on exactly how you're searching. If you're just doing a single search, you could simply read this one line at a time and do a string comparison or something. If you do this, do not load the whole thing into memory - load it one at a time.
If you have access to the "full" edition of SQL Server, you could do a BULK INSERT. If you don't, though (e.g. you're using one of the express editions), you might run into the maximum table size. In this case, I've never tried this, but you could try SQLite. In theory at least, the database can handle multiple terabytes. Be sure to insert a large number of records in each transaction, though; if you do a commit after each insert your performance will be absolutely wretched. Also, be sure that you're not creating an in-memory database, or you'll just run out of memory again.

Copying data from one database to another - Out of Memory Exception

I have a task where i want to copy all data from one database to another database & skipping 2 tables. There are more than 200 tables.
I have table structure ready for my 2nd databas.
So as a solution i created a page & on a button click i have below code :-
DataSet ds = new DataSet();
string connectionString = "Data Source=COMP112\\MSSQLSERVER2014;Initial Catalog=HCMBL;Integrated Security=True;Persist Security Info=True";
SqlConnection con = new SqlConnection(connectionString);
//render table name from database
string sqlTable = "SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE='BASE TABLE' and TABLE_Schema='" + Session["SchemaName"].ToString() + "' and TABLE_NAME!='ENTRY' and TABLE_NAME!='OT' and TABLE_NAME!='BL_ENTRY' and TABLE_NAME!='BL_OT'";
con.Open();
SqlDataAdapter da = new SqlDataAdapter();
SqlCommand cmd = new SqlCommand(sqlTable, con);
cmd.CommandType = CommandType.Text;
da.SelectCommand = cmd;
da.Fill(ds);
con.Close();
//render connection string from WebConfig file
string strcon = ConfigurationManager.ConnectionStrings["SPSchema"].ConnectionString;
for (int i = 0; i < ds.Tables[0].Rows.Count; i++)
{
if (!(ds.Tables[0].Rows[i]["TABLE_NAME"].ToString().Contains("Asp")))
{
string deleteQuery = "Truncate table " + Session["SchemaName"].ToString() + "." + ds.Tables[0].Rows[i]["TABLE_NAME"];
con.Open();
SqlCommand cmdDelete = new SqlCommand(deleteQuery, con);
cmdDelete.ExecuteNonQuery();
con.Close();
DataSet dataSet = new DataSet();
SqlConnection conn = new SqlConnection(strcon);
conn.Open();
string selectData = "select * from " + Session["SchemaName"].ToString() + "." + ds.Tables[0].Rows[i]["TABLE_NAME"];
SqlCommand command = new SqlCommand(selectData, conn);
DataTable dataTable = new DataTable();
SqlDataAdapter dataAdapter = new SqlDataAdapter(selectData, conn);
dataAdapter.FillSchema(dataSet, SchemaType.Mapped);
dataAdapter.Fill(dataSet);
dataTable = dataSet.Tables[0];
conn.Close();
if (dataSet.Tables[0].Rows.Count > 0)
{
//Connect to second Database and Insert row/rows.
SqlConnection conn2 = new SqlConnection(connectionString);
conn2.Open();
SqlBulkCopy bulkCopy = new SqlBulkCopy(conn2);
bulkCopy.DestinationTableName = Session["SchemaName"].ToString() + "." + ds.Tables[0].Rows[i]["TABLE_NAME"].ToString();
bulkCopy.WriteToServer(dataTable);
conn2.Close();
}
}
}
As i run the above code after inserting data in less than 10 tables, it gives out of memory exception & program crashes.
How to handle this? I tried increasing the memory capacity of SQL Server but still same error.
Is there any other way to achieve the task?

What you are doing is very far from the best solution. You are using an ASP.NET MVC process to get all data of your entire database into memory, and then outputting it to another database. If your database is anything more than small and trivial, that will most definitely fill your process's alotted memory.
This type of task should never be done through the memory of a process, but rather using some form of Backup/Restore pattern.
You should look into SSIS projects and create an extract, transfer, and load (ETL) solution, which can be triggered from your ASP.NET MVC solution asynchronously.
An SSIS solution can be triggered from C# code in this way:
var app = new Application();
var package = app.LoadPackage("compiled-package.dtsx", null);
var results = package.Execute();
See this question for a little more information (not specifically about duplicating databases, but has information about triggering SSIS packages from code): How to execute an SSIS package from .NET?
Alternatively
You also have the option of running a query against both databases at once, however this requires some additional plumbing to be done. The user account of your ASP.NET MVC solution needs to have access to both databases. If your databases are hosted on different servers, you also need to link one server to the other: Create linked servers
To perform an insert directly from the output of a select, consider this:
string source = "NAME_OF_SOURCE_DATABASE";
string target = "NAME_OF_TARGET_DATABASE";
string schema = Session["SchemaName"].ToString();
string table = ds.Tables[0].Rows[i]["TABLE_NAME"];
// Uncomment this if you need to deal with autoincrement columns
/*string idInsQuery = $"SET IDENTITY_INSERT {target}.{schema}.{table} ON";
var idInsCommand = new SqlCommand(idInsQuery, conn);
idInsCommand.ExecuteNonQuery();*/
string insQuery = $"INSERT INTO {target}.{schema}.{table} SELECT * FROM {source}.{schema}.{table}";
var insCommand = new SqlCommand(insQuery, conn);
insCommand.ExecuteNonQuery();
// Uncomment this if you need to deal with autoincrement columns
/*string idInsQuery2 = $"SET IDENTITY_INSERT {target}.{schema}.{table} OFF";
var idInsCommand2 = new SqlCommand(idInsQuery2, conn);
idInsCommand2.ExecuteNonQuery();*/
This will only work if the table structures are identical. There might be problems with autoincrement ids or columns with default values, too.

This will copy data from a table in database 1 to a table in database 2
Insert into db2.dbo.table2 (col1,col2)
Select col1,col2 from db1.dbo.table1
Run this sql statement and the data will be copied without a round trip to your app.

Let me know if you find my approach is useful.
First of all, why you want to write down one whole application to do this job while SQL Server have inherited property to do it.
My approach would be configure an Linked Server and configure it which tables you want to copy and which one not.
https://learn.microsoft.com/en-us/sql/relational-databases/linked-servers/create-linked-servers-sql-server-database-engine
Secondly, You can just write down simple stored procedure and schedule that in your sql server to push into another server database as per your schedule. In this way you can control it in N number of ways. I mean about controlling any dependencies(Table level or Business level).
To do this in t-sql, you can use the following system stored procedures to schedule a daily job. This example schedules daily at 1:00 AM. See Microsoft help for details on syntax of the individual stored procedures and valid range of parameters.
DECLARE #job_name NVARCHAR(128), #description NVARCHAR(512), #owner_login_name NVARCHAR(128), #database_name NVARCHAR(128);
SET #job_name = N'Some Title';
SET #description = N'Periodically do something';
SET #owner_login_name = N'login';
SET #database_name = N'Database_Name';
-- Delete job if it already exists:
IF EXISTS(SELECT job_id FROM msdb.dbo.sysjobs WHERE (name = #job_name))
BEGIN
EXEC msdb.dbo.sp_delete_job
#job_name = #job_name;
END
-- Create the job:
EXEC msdb.dbo.sp_add_job
#job_name=#job_name,
#enabled=1,
#notify_level_eventlog=0,
#notify_level_email=2,
#notify_level_netsend=2,
#notify_level_page=2,
#delete_level=0,
#description=#description,
#category_name=N'[Uncategorized (Local)]',
#owner_login_name=#owner_login_name;
-- Add server:
EXEC msdb.dbo.sp_add_jobserver #job_name=#job_name;
-- Add step to execute SQL:
EXEC msdb.dbo.sp_add_jobstep
#job_name=#job_name,
#step_name=N'Execute SQL',
#step_id=1,
#cmdexec_success_code=0,
#on_success_action=1,
#on_fail_action=2,
#retry_attempts=0,
#retry_interval=0,
#os_run_priority=0,
#subsystem=N'TSQL',
#command=N'EXEC my_stored_procedure; -- OR ANY SQL STATEMENT',
#database_name=#database_name,
#flags=0;
-- Update job to set start step:
EXEC msdb.dbo.sp_update_job
#job_name=#job_name,
#enabled=1,
#start_step_id=1,
#notify_level_eventlog=0,
#notify_level_email=2,
#notify_level_netsend=2,
#notify_level_page=2,
#delete_level=0,
#description=#description,
#category_name=N'[Uncategorized (Local)]',
#owner_login_name=#owner_login_name,
#notify_email_operator_name=N'',
#notify_netsend_operator_name=N'',
#notify_page_operator_name=N'';
-- Schedule job:
EXEC msdb.dbo.sp_add_jobschedule
#job_name=#job_name,
#name=N'Daily',
#enabled=1,
#freq_type=4,
#freq_interval=1,
#freq_subday_type=1,
#freq_subday_interval=0,
#freq_relative_interval=0,
#freq_recurrence_factor=1,
#active_start_date=20170101, --YYYYMMDD
#active_end_date=99991231, --YYYYMMDD (this represents no end date)
#active_start_time=010000, --HHMMSS
#active_end_time=235959; --HHMMSS
Let me know in case you need more details on this.
Thanks,
Ayan

How to update a large table using ADO.NET

Ok, so here's the problem I have to solve. I need to write a method in C# that will modify a table in SQL Server 2008. The table could potentially contain millions of records. The modifications include altering the table by adding a new column and then calculating and setting the value of the new field for every row in the table.
Adding the column is not a problem. It's setting the values efficiently that is the issue. I don't want to read in the whole table into a DataTable and then update and commit for obvious reasons. I'm thinking that I would like to use a cursor to iterate over the rows in the table and update them one by one. I haven't done a whole lot of ADO.NET development, but it is my understanding that only read-only server side (firehose) cursors are supported.
So what is the correct way to go about doing something like this (preferably with some sample code in C#)? Stored procedures or other such modifications to the DB are not allowed.

jpgoody,
Here is an example to chew on using the NerdDinner database and some SQLConnection, SQLCommand, and SQLDataReader objects. It adds one day to each of the Event Dates in the Dinners table.
using System;
using System.Data.SqlClient;
namespace NerdDinner
{
public class Class1
{
public void Execute()
{
SqlConnection readerConnection = new SqlConnection(Properties.Settings.Default.ConnectionString);
readerConnection.Open();
SqlCommand cmd = new SqlCommand("SELECT DinnerID, EventDate FROM Dinners", readerConnection);
SqlDataReader reader = cmd.ExecuteReader();
SqlConnection writerConnection = new SqlConnection(Properties.Settings.Default.ConnectionString);
writerConnection.Open();
SqlCommand writerCommand = new SqlCommand("", writerConnection);
while (reader.Read())
{
int DinnerID = reader.GetInt32(0);
DateTime EventDate = reader.GetDateTime(1);
writerCommand.CommandText = "UPDATE Dinners SET EventDate = '" + EventDate.AddDays(1).ToString() + "' WHERE DinnerID = " + DinnerID.ToString();
writerCommand.ExecuteNonQuery();
}
}
}
}

Your problem looks like something that you should be solving using T-SQL and not C#, unless there is some business rule that you are picking up dynamically and calculating the column values T-SQL should be the way to go. Just write a stored procedure or just open up Management studio and write the code to make your changes.
If this does not help then please elaborate on what exactly you want to do to the table, then we can help you figure out if this can be done via T-SQL or not.
[EDIT] you can do something like this
string sql = " USE " + paramDbName;
sql+= " ALTER TABLE XYZ ADD COLUMN " + param1 + " datatype etc, then put semicolon to separate the commands as well"
sql+= " UPDATE XYZ SET Columnx = " + some logic here
cmd.CommandText = sql;
cmd.ExecuteNonQuery();
get this executed on the required instance of Sql Server 2008.
If you have too many lines of text then use StringBuilder.

Here's a suggestion:
You can read data using a DataReader , create a update command for current row and add it to a list of commands.Then run update commands in a transaction.
something like this:
var commands=new List<SqlCommand>();
while(dr.Read())
{
var cmd=new SqlCommand();
cmd.CommandText="Add your command text here";
commands.Add(cmd);
}
using(var cnn=new SqlConnection("Connection String"))
{
IDbTransaction transaction;
try
{
cnn.Open();
transaction=cnn.BeginTransaction();
foreach(var cmd in commands)
{
cmd.Transaction=transaction;
cmd.ExecuteNonQuery();
cmd.Dispose();
}
transaction.Commit();
}
catch(SqlException)
{
if(transaction!=null)
transaction.Rollback();
throw;
}
}

C# Dynamically Add Columns to table in Database

I am still learning quite a bit about programming so this may be an obvious question but I am in the process of writing a program where a user can import an excel file with no column constraints (meaning each excel file could have a different number of columns and so on). What I want to do is be able to convert the imported excel file into a database table using SQLite. As far as I have read the syntax for SQLite is the same as ADO syntax but I am unsure of how to dynamically create the tables in the database. I am looking for any form of suggestions, if someone felt like showing an example in code that would be cool.
Thanks!
Nathan

Here is a piece of code for creating a table in sqlite:
SQLiteConnection mDBcon = new SQLiteConnection();
mDBcon.ConnectionString = "Data Source=" + DataSourcePath;
mDBcon.Open();
SQLiteCommand cmd = new SQLiteCommand(mDBcon);
cmd.CommandText = "CREATE TABLE IF NOT EXISTS tags (ISBN VARCHAR(15), Tag VARCHAR(15));";
cmd.ExecuteNonQuery();
note that in SQLite you can only add coloumn to existing tables, and only to the end of the table:
cmd.CommandText = "ALTER TABLE books ADD COLUMN PublishDate DateTime;";
cmd.ExecuteNonQuery();
Addition
Assuming you imported you data from the excel to a DataSet,
you can now iterate through the DataSet's DataTables, create the corresponding tables and fill them:
(disclaimer, haven't tested it)
foreach(DataTable table in dataSet.Tables)
{
SQLiteCommand cmd = new SQLiteCommand(mDBcon);
cmd.CommandText = "CREATE TABLE IF NOT EXISTS " + table.Name + "(";
bool first = true;
foreach (DataColumn column in table.Columns)
{
cmd.CommandText += "#"+column.Name;
if (!first) cmd.CommandText += ",";
else first = false;
cmd.Parameters.Add(new SQLiteParameter("#"+column.Name, column.Name));
}
cmd.CommandText += ");";
cmd.ExecuteNonQuery();
// Fill the new table:
SQLiteDataAdapter da = new SQLiteDataAdapter("select * from " + table.Name, mDBcon);
da.Fill(table);
}

You can just use ExecuteNonQuery to create your table, using the standard SQLite Query Language for CREATE TABLE.

Passing data from one database to another database table

I want to take a backup of my Access database Pragmatically.
And After taking all data in backup i want to delete data from source database.
( So that it will not take much time while querying and filtering through application.)
The source database name is Data.mdb
The destination database name is Backup.mdb
Both are protected by same password.
For these purpose i am writing a query in C# like this.
string conString = "Provider=Microsoft.Jet.OLEDB.4.0 ;Data Source=Backup.mdb;Jet
OLEDB:Database Password=12345";
OleDbConnection dbconn = new OleDbConnection();
OleDbDataAdapter dAdapter = new OleDbDataAdapter();
OleDbCommand dbcommand = new OleDbCommand();
try
{
if (dbconn.State == ConnectionState.Closed)
dbconn.Open();
string selQuery = "INSERT INTO [Bill_Master] SELECT * FROM [MS Access;DATABASE="+
"\\Data.mdb" + "; Jet OLEDB:Database Password=12345;].[Bill_Master]";
dbcommand.CommandText = selQuery;
dbcommand.CommandType = CommandType.Text;
dbcommand.Connection = dbconn;
int result = dbcommand.ExecuteNonQuery();
}
catch(Exception ex) {}
Everything goes fine if i try with without password database file.
I think error in passing password on query statement.
I am trying to execute through access query but it is saying "Invalid argument".
Please is there any other programing logic for doing that.
Thanks
prashant
YuvaDeveloper

Are Data.mdb and Backup.mdb identically in strcuture? If so, I wouldn't bother copying data via SQL but just copy the whole file.

Try remove the space between the ; and Jet …
So the format would be:
INSERT INTO [Bill_Master] SELECT * FROM [MS Access;DATABASE="+
"\\Data.mdb" + ";Jet OLEDB:Database Password=12345;].[Bill_Master]

You can copy and rename Data.mdb, and then truncate all the tables in Data.mdb. Far easier than trying to copy a table at a time..

Don't delete data. This becomes a lot mroe difficult in the future to do analysis or inquiries. If it's taking a long time then review indexing or upszing to SQL Server. The Express edition is free and can handle databases up to 4 Gb.