Dealing with huge amount of data when inserting into sql database - c#

in my code the user can upload an excel document wish contains it's phone contact list.Me as a developer should read that excel file turn it into a dataTable and insert it into the database .
The Problem is that some clients have a huge amount of contacts like saying 5000 and more contacts and when i am trying to insert this amount of data into the database it's crashing and giving me a timeout exception.
What would be the best way to avoid this kind of exception and is their any code that can reduce the time of the insert statement so the user don't wait too long ?
the code
public SqlConnection connection = new SqlConnection(System.Configuration.ConfigurationManager.ConnectionStrings["ConnectionString"].ConnectionString);
public void Insert(string InsertQuery)
{
SqlDataAdapter adp = new SqlDataAdapter();
adp.InsertCommand = new SqlCommand(InsertQuery, connection);
if (connection.State == System.Data.ConnectionState.Closed)
{
connection.Open();
}
adp.InsertCommand.ExecuteNonQuery();
connection.Close();
}
protected void submit_Click(object sender, EventArgs e)
{
string UploadFolder = "Savedfiles/";
if (Upload.HasFile) {
string fileName = Upload.PostedFile.FileName;
string path=Server.MapPath(UploadFolder+fileName);
Upload.SaveAs(path);
Msg.Text = "successfully uploaded";
DataTable ValuesDt = new DataTable();
ValuesDt = ConvertExcelFileToDataTable(path);
Session["valuesdt"] = ValuesDt;
Excel_grd.DataSource = ValuesDt;
Excel_grd.DataBind();
}
}
protected void SendToServer_Click(object sender, EventArgs e)
{
DataTable Values = Session["valuesdt"] as DataTable ;
if(Values.Rows.Count>0)
{
DataTable dv = Values.DefaultView.ToTable(true, "Mobile1", "Mobile2", "Tel", "Category");
double Mobile1,Mobile2,Tel;string Category="";
for (int i = 0; i < Values.Rows.Count; i++)
{
Mobile1 =Values.Rows[i]["Mobile1"].ToString()==""?0: double.Parse(Values.Rows[i]["Mobile1"].ToString());
Mobile2 = Values.Rows[i]["Mobile2"].ToString() == "" ? 0 : double.Parse(Values.Rows[i]["Mobile2"].ToString());
Tel = Values.Rows[i]["Tel"].ToString() == "" ? 0 : double.Parse(Values.Rows[i]["Tel"].ToString());
Category = Values.Rows[i]["Category"].ToString();
Insert("INSERT INTO client(Mobile1,Mobile2,Tel,Category) VALUES(" + Mobile1 + "," + Mobile2 + "," + Tel + ",'" + Category + "')");
Msg.Text = "Submitied successfully to the server ";
}
}
}

You can try SqlBulkCopy to insert Datatable to Database Table
Something like this,
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(sqlConnection, SqlBulkCopyOptions.KeepIdentity))
{
bulkCopy.DestinationTableName = DestTableName;
string[] DtColumnName = YourDataTableColumns;
foreach (string dbcol in DbColumnName)//To map Column of Datatable to that of DataBase tabele
{
foreach (string dtcol in DtColumnName)
{
if (dbcol.ToLower() == dtcol.ToLower())
{
SqlBulkCopyColumnMapping mapID = new SqlBulkCopyColumnMapping(dtcol, dbcol);
bulkCopy.ColumnMappings.Add(mapID);
break;
}
}
}
bulkCopy.WriteToServer(YourDataTableName.CreateDataReader());
bulkCopy.Close();
}
For more Read http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlbulkcopy.aspx

You are inserting 1 row at a time, which is very expensive for this amount of data
In those cases you should use bulk insert, so the round trip to DB will be only once, if you need to roll back - all is the same transaction

You can use SqlBulkCopy which is more work, or you can use the batch update feature of the SqlAdpater. Instead of creating your own insert statement, then building a sqladapter, and then manually executing it, create a dataset, fill it, create one sqldataadpater, set the number of inserts in a batch, then execute the adapter once.
I could repeat the code, but this article shows exactly how to do it: http://msdn.microsoft.com/en-us/library/kbbwt18a%28v=vs.80%29.aspx
protected void SendToServer_Click(object sender, EventArgs e)
{
DataTable Values = Session["valuesdt"] as DataTable ;
if(Values.Rows.Count>0)
{
DataTable dv = Values.DefaultView.ToTable(true, "Mobile1", "Mobile2", "Tel", "Category");
//Fix up default values
for (int i = 0; i < Values.Rows.Count; i++)
{
Values.Rows[i]["Mobile1"] =Values.Rows[i]["Mobile1"].ToString()==""?0: double.Parse(Values.Rows[i]["Mobile1"].ToString());
Values.Rows[i]["Mobile2"] = Values.Rows[i]["Mobile2"].ToString() == "" ? 0 : double.Parse(Values.Rows[i]["Mobile2"].ToString());
Values.Rows[i]["Tel"] = Values.Rows[i]["Tel"].ToString() == "" ? 0 : double.Parse(Values.Rows[i]["Tel"].ToString());
Values.Rows[i]["Category"] = Values.Rows[i]["Category"].ToString();
}
BatchUpdate(dv,1000);
}
}
public static void BatchUpdate(DataTable dataTable,Int32 batchSize)
{
// Assumes GetConnectionString() returns a valid connection string.
string connectionString = GetConnectionString();
// Connect to the database.
using (SqlConnection connection = new SqlConnection(connectionString))
{
// Create a SqlDataAdapter.
SqlDataAdapter adapter = new SqlDataAdapter();
// Set the INSERT command and parameter.
adapter.InsertCommand = new SqlCommand(
"INSERT INTO client(Mobile1,Mobile2,Tel,Category) VALUES(#Mobile1,#Mobile2,#Tel,#Category);", connection);
adapter.InsertCommand.Parameters.Add("#Mobile1",
SqlDbType.Float);
adapter.InsertCommand.Parameters.Add("#Mobile2",
SqlDbType.Float);
adapter.InsertCommand.Parameters.Add("#Tel",
SqlDbType.Float);
adapter.InsertCommand.Parameters.Add("#Category",
SqlDbType.NVarchar, 50);
adapter.InsertCommand.UpdatedRowSource = UpdateRowSource.None;
// Set the batch size.
adapter.UpdateBatchSize = batchSize;
// Execute the update.
adapter.Update(dataTable);
}
}

I know this is a super old post, but you should not need to use the bulk operations explained in the existing answers for 5000 inserts. Your performance is suffering so much because you close and reopen the connection for each row insert. Here is some code I have used in the past that keeps one connection open and executes as many commands as needed to push all the data to the DB:
public static class DataWorker
{
public static Func<IEnumerable<T>, Task> GetStoredProcedureWorker<T>(Func<SqlConnection> connectionSource, string storedProcedureName, Func<T, IEnumerable<(string paramName, object paramValue)>> parameterizer)
{
if (connectionSource is null) throw new ArgumentNullException(nameof(connectionSource));
SqlConnection openConnection()
{
var conn = connectionSource() ?? throw new ArgumentNullException(nameof(connectionSource), $"Connection from {nameof(connectionSource)} cannot be null");
var connState = conn.State;
if (connState != ConnectionState.Open)
{
conn.Open();
}
return conn;
}
async Task DoStoredProcedureWork(IEnumerable<T> workData)
{
using (var connection = openConnection())
using (var command = connection.CreateCommand())
{
command.CommandType = CommandType.StoredProcedure;
command.CommandText = storedProcedureName;
command.Prepare();
foreach (var thing in workData)
{
command.Parameters.Clear();
foreach (var (paramName, paramValue) in parameterizer(thing))
{
command.Parameters.AddWithValue(paramName, paramValue ?? DBNull.Value);
}
await command.ExecuteNonQueryAsync().ConfigureAwait(false);
}
}
}
return DoStoredProcedureWork;
}
}
This was actually from a project where I was gathering emails for a restriction list, so kind of relevant example of what a parameterizer argument might look like and how to use the above code:
IEnumerable<(string,object)> RestrictionToParameter(EmailRestriction emailRestriction)
{
yield return ("#emailAddress", emailRestriction.Email);
yield return ("#reason", emailRestriction.Reason);
yield return ("#restrictionType", emailRestriction.RestrictionType);
yield return ("#dateTime", emailRestriction.Date);
}
var worker = DataWorker.GetStoredProcedureWorker<EmailRestriction>(ConnectionFactory, #"[emaildata].[AddRestrictedEmail]", RestrictionToParameter);
await worker(emailRestrictions).ConfigureAwait(false);

Related

How to execute DataAdapter.Fill() simultaneously

I have been working on an ASP.Net application for a long time and there are above 10 clients using the application. But now I found a problem in the application, that is, I have a stored procedure call which takes about 30 seconds to execute. It is not a problem, because the SQL code highly complicated and looping many times. The problem is :
Whenever that stored procedure call is executing, I am not able to using any other functions or stored procedure call.
When I tried debugging, the problem is that 'DataAdapter.Fill()' function is waiting for the first stored procedure call to finish.
My code that executes stored procedure call and returning data is :
public static DataSet ExecuteQuery_SP(string ProcedureName, object[,] ParamArray)
{
SqlDataAdapter DataAdapter = new SqlDataAdapter();
DataSet DS = new DataSet();
try
{
if (CON.State != ConnectionState.Open)
OpenConnection();
SqlCommand cmd = new SqlCommand();
cmd.CommandTimeout = 0;
cmd.CommandText = ProcedureName;
cmd.CommandType = CommandType.StoredProcedure;
cmd.Connection = CON;
cmd.Transaction = SqlTrans;
string ParamName;
object ParamValue;
for (int i = 0; i < ParamArray.Length / 2; i++)
{
ParamName = ParamArray[i, 0].ToString();
ParamValue = ParamArray[i, 1];
cmd.Parameters.AddWithValue(ParamName, ParamValue);
}
DataAdapter = new SqlDataAdapter(cmd);
DataAdapter.Fill(DS);
cmd.CommandText = "";
}
catch (Exception ea)
{
}
return DS;
}
All stored procedure calls are working through this function. Hence when my first stored procedure call 'A' is running, stored procedure call 'B' will not execute until 'A' is finished.
This reduces overall performance of the application and causes problem in data retrieval.
I surfed google and found that 'Threading' can be helpful but I am not able to execute threading properly. I am not so familiar with these kind of things. It will helpful if you can rectify the problem.
My first stored procedure call is:
ds = DB.ExecuteQuery_SP("SelectOutstandingReportDetailed", parArray);
Where ds is the DataSet object.
Second stored procedure call is :
ds = DB.ExecuteQuery_SP("[SelectAccLedgersDetailsByID]", ParamArray);
My current DB connection open function is :
public static bool OpenConnection()
{
try
{
Server = (String)HttpContext.GetGlobalResourceObject("Resource", "Server");
DBName = (String)HttpContext.GetGlobalResourceObject("Resource", "DBName");
UserName = (String)HttpContext.GetGlobalResourceObject("Resource", "UserName");
PassWord = (String)HttpContext.GetGlobalResourceObject("Resource", "PassWord");
string ConnectionString;
ConnectionString = "server=" + Server + "; database=" + DBName + "; uid=" + UserName + "; pwd=" + PassWord + "; Pooling='true';Max Pool Size=100;MultipleActiveResultSets=true;Asynchronous Processing=true";
CON.ConnectionString = ConnectionString;
if (CON.State != ConnectionState.Open)
{
CON.Close();
CON.Open();
}
}
catch (Exception ea)
{
}
return false;
}
Where 'CON' is a public SqlConnection variable
static SqlConnection CON = new SqlConnection();
I found the problem, that is, all stored procedure calls are performed through this 'CON' object. If there is seperate SqlConnection object for each stored procedure call, then no problem is there.
So is it possible to make separate SqlConnection for every ExecuteQuery_SP calls.
If any doubt in question, kindly comment.
Thank you
SQL Server will allow thousands of connections simultaneously, by default. This is NOT the source of your problem. You have forced every call to a stored procedure to be funneled through a single method. Factor out your calls to the stored procedures - in other words, lose the ExecuteQuery_SP method which is a bottleneck. Then test again.
Here's is an introduction to data layers.
Heres the simplest version I can create for you.
Important: to understand you should read about async-await.
You can start at the Microsoft C# Async-Await Docs
// TODO set up your connection string
private string connectionString = "<your connection string>";
// Gets data assyncronously
public static async Task<DataTable> GetDataAsync(string procedureName, object[,] ParamArray)
{
try
{
var asyncConnectionString = new SqlConnectionStringBuilder(connectionString)
{
AsynchronousProcessing = true
}.ToString();
using (var conn = new SqlConnection(asyncConnectionString))
{
using (var SqlCommand = new SqlCommand())
{
SqlCommand.Connection = conn;
SqlCommand.CommandText = procedureName;
SqlCommand.CommandType = CommandType.StoredProcedure;
string ParamName;
object ParamValue;
for (int i = 0; i < ParamArray.Length / 2; i++)
{
ParamName = ParamArray[i, 0].ToString();
ParamValue = ParamArray[i, 1];
SqlCommand.Parameters.AddWithValue(ParamName, ParamValue);
}
conn.Open();
var data = new DataTable();
data.BeginLoadData();
using (var reader = await SqlCommand.ExecuteReaderAsync().ConfigureAwait(true))
{
if (reader.HasRows)
data.Load(reader);
}
data.EndLoadData();
return data;
}
}
}
catch (Exception Ex)
{
// Log error or something else
throw;
}
}
public static async Task<DataTable> GetData(object General, object Type, string FromDate, string ToDate)
{
object[,] parArray = new object[,]{
{"#BranchID",General.BranchID},
{"#FinancialYearID",General.FinancialYearID},
{"#Type",Type},
{"#FromDate",DateTime.ParseExact(FromDate, "dd/MM/yyyy", System.Globalization.CultureInfo.InvariantCulture)},
{"#ToDate",DateTime.ParseExact(ToDate, "dd/MM/yyyy", System.Globalization.CultureInfo.InvariantCulture)}
};
return await DataBaseHelper.GetDataAsync("SelectOutstandingReportDetailed", parArray);
}
// Calls database assyncronously
private async Task ConsumeData()
{
DataTable dt = null;
try
{
// TODO configure your parameters here
object general = null;
object type = null;
string fromDate = "";
string toDate = "";
dt = await GetData(general, type, fromDate, toDate);
}
catch (Exception Ex)
{
// do something if an error occurs
System.Diagnostics.Debug.WriteLine("Error occurred: " + Ex.ToString());
return;
}
foreach (DataRow dr in dt.Rows)
{
System.Diagnostics.Debug.WriteLine(dr.ToString());
}
}
// Fired when some button is clicked. Get and use the data assyncronously, i.e. without blocking the UI.
private async void button1_Click(object sender, EventArgs e)
{
await ConsumeData();
}

Sqlite DataAdapter isn't updating when called

I have to write identical tables to two different Sqlite database. The tables rewrite existing data and ids, so I currently just delete the entire table and rewrite. Depending on the order that I write to the db, the C# SQLiteCommandBuilder does not update the second db called.
If I call Db_A before Db_B, then A gets written and B gets deleted and vice versa. Can anyone tell me why the second table enters the code to get deleted, but the Sqlite adapter never updates the second table? It doesn't throw an error either.
public static bool WriteDt_Name(DataTable dt)
{
using (connA = GetDbAConn())
{
SaveDataTable(connA, dt);
}
using (connB = GetDbBConn())
{
SaveDataTable(connB, dt);
}
return true;
}
public static void SaveDataTable(SQLiteConnection conn, DataTable dt)
{
string table = dt.TableName;
var cmd = conn.CreateCommand();
cmd.CommandText = string.Format("DELETE FROM {0}", table);
int val = cmd.ExecuteNonQuery();
cmd.CommandText = string.Format("SELECT * FROM {0}", table);
using (var adapter = new SQLiteDataAdapter(cmd))
{
using (SQLiteCommandBuilder builder = new SQLiteCommandBuilder(adapter))
{
adapter.Update(dt);
conn.Close();
}
}
}
public static SQLiteConnection GetDbAConn()
{
string base_dir = System.AppDomain.CurrentDomain.BaseDirectory;
string path = Directory.GetParent(Directory.GetParent(Directory.GetParent(Directory.GetParent(Directory.GetParent(base_dir).ToString()).ToString()).ToString()).ToString()).ToString();
path = path + "\\db\\DbA.sqlite;";
SQLiteConnection conn = new SQLiteConnection("Data Source=" + path + "Version=3;");
return conn;
}
I have tried splitting SaveDataTable into a SaveDt_A and SaveDt_B and calling it that way. I still get the same result.

C# Mysql multiple queries

Im trying to build up a little status-tool. I need to get results of multiple queries (about 4-5). The general connection-setup and 'how-to-read-data' is already done but I cant figure out how the another query executed.
Everything I found while searching for it is for the SqlClient. Im totally overcharged with this.
Here is my code so far (be patient, im a newbie to this):
private void button1_Click(object sender, EventArgs e)
{
if(listView1.Items.Count > 1)
{
listView1.Items.Clear();
}
var listMember = new List<string>{};
var listOnline = new List<string>{};
// SQL PART //
string connString = "Server=10*****;Port=3306;Database=e***;Uid=e***;password=********************;";
MySqlConnection conn = new MySqlConnection(connString);
MySqlCommand command = conn.CreateCommand();
command.CommandText = "SELECT fullname,online FROM member WHERE active = '1' ORDER BY online DESC";
try
{
conn.Open();
}
catch (Exception ex)
{
listView1.Items.Add("Error: " + ex);
}
MySqlDataReader reader = command.ExecuteReader();
while(reader.Read())
{
listMember.Add(reader["fullname"].ToString());
listOnline.Add(reader["online"].ToString());
}
conn.Close();
// SQL ENDING //
// SET ENTRIES TO LISTVIEW //
int counter = 0;
foreach(string member in listMember)
{
ListViewItem item = new ListViewItem(new[] { member, listOnline.ElementAt(counter) });
item.ForeColor = Color.Green;
listView1.Items.Add(item);
counter++;
}
}
Im not really sure how the design/layout will look like in the end, so I would like to just append the results to lists in the sql-part to process the data later out of the lists.
Do I really have to setup a complete new connection after conn.Close()? Or is there any other way? I can just imagine: 5 queries with their own connection,try,catch and 2 loops... this will get about 100-200 lines just for getting the results out of 5 queries. Isnt that a bit too much for such an easy thing?
Hope for some help.
Greetings.
According to the new comments my latest code:
Top:
public partial class Form1 : Form
{
public static string connString = "Server=10****;Port=3306;Database=e****;Uid=e****;password=****;";
public Form1()
{
InitializeComponent();
MySqlConnection conn = new MySqlConnection(connString); // Error gone!
}
Body part:
public void QueryTwoFields(string s, List<string> S1, List<string> S2)
{
try
{
MySqlCommand cmd = conn.CreateCommand(); // ERROR: conn does not exist in the current context.
cmd.CommandType = CommandType.Text;
string command = s;
cmd.CommandText = command;
MySqlDataReader sqlreader = cmd.ExecuteReader();
while (sqlreader.Read())
{
S1.Add(sqlreader[0].ToString());
S2.Add(sqlreader[1].ToString());
}
sqlreader.Close();
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
}
}
private void button1_Click(object sender, EventArgs e)
{
if(listView1.Items.Count > 1)
{
listView1.Items.Clear();
}
var listMember = new List<string>{};
var listOnline = new List<string>{};
using (conn) // ERROR: conn does not exist in the current context.
{
conn.Open();
///...1st Query
QueryTwoFields("SELECT fullname,online FROM member WHERE active = '1' ORDER BY online DESC",listMember,listOnline);
//...2nd query
//QueryTwoFields("your new Select Statement", otherList, otherList);
}
}
You don't have to close connection every time you execute one query rarher than close the sqlreader assigned to that connection. Finally when all of your queries have been executed you close the connection. Consider also the use of using:
You cal also define a method for execution your Query in order for your code not to be repetive:
public void QueryTwoFields(string s, List<string> S1, List<string> S2)
///Select into List S1 and List S2 from Database (2 fields)
{
try
{
MySqlCommand cmd = conn.CreateCommand();
cmd.CommandType = CommandType.Text;
string command = s;
cmd.CommandText = command;
MySqlDataReader sqlreader = cmd.ExecuteReader();
while (sqlreader.Read())
{
S1.Add(sqlreader[0].ToString());
S2.Add(sqlreader[1].ToString());
}
sqlreader.Close();
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
}
}
private void button1_Click(object sender, EventArgs e)
{
if(listView1.Items.Count > 1)
{
listView1.Items.Clear();
}
var listMember = new List<string>{};
var listOnline = new List<string>{};
// SQL PART //
using (conn)
{
conn.Open();
///...1st Query
QueryTwoFields("SELECT fullname,online FROM member WHERE active = '1' ORDER BY online DESC",listmember,listonline)
//...2nd query
QueryTwoFields("your new Select Statement",myOtherList1,myOtherlist2)
....
}
}
EDIT :
Take in mind you cant define QueryTwoFields method inside button handler. You must define it outside (see code above).
Also Define your connection data in the start of the programm:
namespace MyProject
{
/// <summary>
/// Defiine your connectionstring and connection
/// </summary>
///
public partial class Form1 : Form
{ public static string connString = "Server=10*****;Port=3306;Database=e***;Uid=e***;password=********************;";
MySqlConnection conn = new MySqlConnection(connString);
.........
Datatables are fantastic
Using a data table is a nice way to do both read and write. And it comes with the luxury of eveything you can do with a datatable - like asssigning it directly to a datagrid control, sorting, selecting and deleting while disconnected.
The sample below assumes a MySqlConnection conection property managed by calls to your own OpenConnection() and CloseConnection() methods not shown.
Simple datatable read demo:
public DataTable Select(string query = "")
{
//Typical sql: "SELECT * FROM motorparameter"
DataTable dt = new DataTable();
//Open connection
if (this.OpenConnection() == true)
{
//Create Command
MySqlCommand cmd = new MySqlCommand(query, connection);
//Create a data reader and Execute the command
MySqlDataReader dataReader = cmd.ExecuteReader();
dt.Load(dataReader);
//close Data Reader
dataReader.Close();
//close Connection
this.CloseConnection();
//return data table
return dt;
}
else
{
return dt;
}
}
In case of writing back the datatable to the database - supply the SQL you used in the read (or would have used to read to the data table):
public void Save(DataTable dt, string DataTableSqlSelect)
{
//Typically "SELECT * FROM motorparameter"
string query = DataTableSqlSelect;
//Open connection
if (this.OpenConnection() == true)
{
//Create Command
MySqlCommand mySqlCmd = new MySqlCommand(query, connection);
MySqlDataAdapter adapter = new MySqlDataAdapter(mySqlCmd);
MySqlCommandBuilder myCB = new MySqlCommandBuilder(adapter);
adapter.UpdateCommand = myCB.GetUpdateCommand();
adapter.Update(dt);
//close Connection
this.CloseConnection();
}
else
{
}
}
The neat thing the datatable is extremely flexible. You can run your own selects against the table once it contains data and before writing back you can set or reset what rows needs updating and by default the datatable keeps track of what rows you update in the table. Do not forget primary key column(s) for all tables in the db.
For multiple queries consider if possible using a join between the database tables or same table if data related or use a UNION sql syntax if column count and type of data is the same. You can allways "create" your extra column in the select to differ what data comes from what part of the UNION.
Also consider using CASE WHEN sql syntax to conditionally select data from different sources.

How to Insert Data from DataTable to Oracle Database Table :

I've data in DataTable with 2 rows and 3 columns. I want to insert that data into Oracle table.
How can I insert? please give me with some example.
And also
How can I pass datatable to storedprocedure in ORACLE...
I pass datatable in below mensioned manner, but datatable type problem is comming. how can I solve this?
cmd.Parameters.Add("#Details",dtSupplier);
(OR)
cmd.Parameters.Add("Details", DbType.Single).Value = dtSupplier.ToString();
want to insert dataset or a datatable into ORACLE,
create an ORACLE data adapter.
create a command object for insertion,
set the CommandType to StoredProcedure.
Update command of the data adapter,
pass the dataset or datatable as parameter.
like this:
OracleDataAdapter da = new OracleDataAdapter();
OracleCommand cmdOra = new OracleCommand(StoredProcedureName, Connection);
cmdOra.CommandType = CommandType.StoredProcedure;
da.InsertCommand = cmdOra;
da.Update(dsDataSet);
OR
if above dont work than pass datatable as xml prameter than than process it
For details check : ADO.NET DataTable as XML parameter to an Oracle/SQL Server Database Stored Procedure
OR
Check this thread on Oracle site : Thread: Pass data table to Oracle stored procedure
Check existing answer : How to Pass datatable as input to procedure in C#?
I'm very late for this answer, but I elaborated a bit to have some more readable (I hope) code, and to avoid all those .ToString() for the values so nulls and other less common values can be handled; here it is:
public void Copy(String tableName, DataTable dataTable)
{
var insert = $"insert into {tableName} ({GetColumnNames(dataTable)}) values ({GetParamPlaceholders(dataTable)})";
using (var connection = /*a method to get a new open connection*/)
{
for (var row = 0; row < dataTable.Rows.Count; row++)
{
InsertRow(dataTable, insert, connection, row);
}
}
}
private static void InsertRow(DataTable dataTable, String insert, OracleConnection connection, Int32 row)
{
using (var command = new OracleCommand(insert, connection))
{
AssembleParameters(dataTable, command, row);
command.ExecuteNonQuery();
}
}
private static void AssembleParameters(DataTable dataTable, OracleCommand command, Int32 row)
{
for (var col = 0; col < dataTable.Columns.Count; col++)
{
command.Parameters.Add(ParameterFor(dataTable, row, col));
}
}
private static OracleParameter ParameterFor(DataTable dataTable, Int32 row, Int32 col)
{
return new OracleParameter(GetParamName(dataTable.Columns[col]), dataTable.Rows[row].ItemArray.GetValue(col));
}
private static String GetColumnNames(DataTable data) => (from DataColumn column in data.Columns select column.ColumnName).StringJoin(", ");
private static String GetParamPlaceholders(DataTable data) => (from DataColumn column in data.Columns select GetParamName(column)).StringJoin(", ");
private static String GetParamName(DataColumn column) => $":{column.ColumnName}_param";
Hope this can be still useful to somebody
The best idea would be follow the step mentioned below
Create a transaction
Begin the transaction
Loop through you data table
call your procedure
If no error occurred commit transaction
else roll back transaction
Regarding this part of your question:
cmd.Parameters.Add("#Details",dtSupplier);
(OR)
cmd.Parameters.Add("Details", DbType.Single).Value = dtSupplier.ToString();
What is the type of the "Details" parameter? Is it a Single? Then you would have to pick one (1) value from your DataTable and pass it to your parameter, something like dtSupplier.Rows[0]["col"].
If you use dtSupplier.ToString() you are just making a string of the entire DataTable (which i guess will always be the type name of DataTable).
First of all, you need to add Oracle.DataAccess.dll as reference in Visual Studio. In most cases, you can find this dll in the directory C:\ProgramData\Oracle11g\product\11.2.0\client_1\ODP.NET\bin\2.x\Oracle.DataAccess.dll
If just you need to insert the records from DataTable to Oracle table, then you can call the below function. Consider that your DataTable name is dt.
string error = "";
int noOfInserts = DataTableToTable(dt,out error);
1. Without using Oracle Parameters(special character non-safe)
The definition of the function is given below. Here, we are just making the query dynamic for passing this as a sql statement to the InsertWithQuery function.
public int DataTableToTable(DataTable dt,out string error)
{
error = "";
for (int i = 0; i < dt.Rows.Count; i++)
{
finalSql = "INSERT INTO TABLENAME SELECT ";
for (int j = 0; j < dt.Columns.Count; j++)
{
colValue += "'" + dt.Rows[i][j].ToString() + "',";
}
colValue = colValue.Remove(colValue.Length - 1, 1);
finalSql += colValue + " FROM DUAL";
InsertWithQuery(finalSql, out error);
if (error != "")
return error;
inserts++;
colValue = "";
}
}
The code for InsertWithQuery function is given below. Here, in the connection string you have to place you database details like Host,Username,Password etc.
public int InsertWithQuery(string query, out string error)
{
error = "";
int rowsInserted = 0;
if (error == "")
{
OracleConnection con = new OracleConnection("Data Source=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=)(PORT=)))(CONNECT_DATA=(SERVER=DEDICATED)(SID=)));User Id=;Password=");
OracleTransaction trans = con.BeginTransaction();
try
{
error = "";
OracleCommand cmd = new OracleCommand();
cmd.Transaction = trans;
cmd.Connection = con;
cmd.CommandText = query;
rowsInserted = cmd.ExecuteNonQuery();
trans.Commit();
con.Dispose();
return rowsInserted;
}
catch (Exception ex)
{
trans.Rollback();
error = ex.Message;
rowsInserted = 0;
}
finally
{
con.Dispose();
}
}
return rowsInserted;
}
2. With using Oracle Parameters(special character safe)
This can handle special characters like single quotes like scenarios in the column values.
public int DataTableToTable(DataTable dt,out string error)
{
error = "";
string finalSql = "";
List<string> colValue = new List<string>();
List<string> cols = new List<string>() {"COLUMN1","COLUMN2","COLUMN3"};
for (int i = 0; i < dt.Rows.Count; i++)
{
finalSql = "INSERT INTO TABLENAME(COLUMN1,COLUMN2,COLUMN3) VALUES(:COLUMN1,:COLUMN2,:COLUMN3) ";
for (int j = 0; j < dt.Columns.Count; j++)
{
colValue.Add(dt.Rows[i][j].ToString());
}
objDAL.InsertWithParams(finalSql,colValue,cols, out error);
if (error != "")
return error;
inserts++;
colValue.Clear();
}
}
And the InsertWithParams is given below
public string InsertWithParams(string sql, List<string> colValue, List<string> cols, out string error)
{
error = "";
try
{
OracleConnection con = new OracleConnection("Data Source=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=)(PORT=)))(CONNECT_DATA=(SERVER=DEDICATED)(SID=)));User Id=;Password=");
OracleCommand command = new OracleCommand(sql, con);
for (int i = 0; i < colValue.Count; i++)
{
command.Parameters.Add(new OracleParameter(cols[i], colValue[i]));
}
command.ExecuteNonQuery();
command.Connection.Close();
}
catch (Exception ex)
{
error = ex.Message;
}
return null;
}
try {
//Suppose you have DataTable dt
string connectionString = #"Provider=Microsoft.ACE.OLEDB.12.0;" +
#"Data Source='Give path of your access database file here';Persist Security Info=False";
OleDbConnection dbConn = new OleDbConnection(connectionString);
dbConn.Open();
using (dbConn)
{
int j = 0;
for (int i = 0; i < 2; i++)
{
OleDbCommand cmd = new OleDbCommand(
"INSERT INTO Participant_Profile ([column1], [column2] , [column3] ) VALUES (#c1 , #c2 , #c3 )", dbConn);
cmd.Parameters.AddWithValue("#c1", dt.rows[i][j].ToString());
cmd.Parameters.AddWithValue("#c2", dt.rows[i][j].ToString());
cmd.Parameters.AddWithValue("#c3", dt.rows[i][j].ToString());
cmd.ExecuteNonQuery();
j++;
}
}
}
catch (OleDbException exception)
{
Console.WriteLine("SQL Error occured: " + exception);
}
I know it's been a big WHILE upon the matter, but the same need: "to insert data from a datatable to an Oracle table" has happened to me. I found this thread. I also tried the answers and came to the conclusion that executing a
...
cmd.ExecuteNonQuery();
...
in a loop, is bad. Reeaaally bad. The first thing that is bad is performance, the second is unnecessary complexity, the third is unnecessary Oracle Objects (stored proc). The time it takes to complete, lets say 200 rows, is almost 1 minute and that's me rounding it down. So in the hope that someone else will find this helpful here's my experience.
I got stubborn and searched some more, so I found out this, true it's from 2018. But I'm in 2021 myself...
So the base code is:
using Oracle.ManagedDataAccess.Client; // you don't need other dll, just install this from nuget gallery
using System.Data;
public static void Datatable2Oracle(string tableName, DataTable dataTable)
{
string connString = "connection string";
OracleBulkCopy copy= new(connString, OracleBulkCopyOptions.UseInternalTransaction /*I don't know what this option does*/);
copy.DestinationTableName = tableName;
copy.WriteToServer(dataTable);
copy.Dispose();
}
This should match a raw oracle DDL performance:
create table table_name as select * from other_table_name

SQL Syntax Error (INSERT command)

I have a form with a text box and button, such that when the user clicks the button, the specified name in the text box is added to a table in my sql database. The code for the button is as follows:
private void btnAddDiaryItem_Click(object sender, EventArgs e)
{
try
{
string strNewDiaryItem = txtAddDiaryItem.Text;
if (strNewDiaryItem.Length == 0)
{
MessageBox.Show("You have not specified the name of a new Diary Item");
return;
}
string sqlText = "INSERT INTO tblDiaryTypes (DiaryType) VALUES = ('" + strNewDiaryItem + "');";
cSqlQuery cS = new cSqlQuery(sqlText, "non query");
PopulateInitialDiaryItems();
MessageBox.Show("New Diary Item added succesfully");
}
catch (Exception ex)
{
MessageBox.Show("Unhandled Error: " + ex.Message);
}
}
The class cSqlQuery is a simple class that executes various T-SQL actions for me and its code is as follows:
class cSqlQuery
{
public string cSqlStat;
public DataTable cQueryResults;
public int cScalarResult;
public cSqlQuery()
{
this.cSqlStat = "empty";
}
public cSqlQuery(string paramSqlStat, string paramMode)
{
this.cSqlStat = paramSqlStat;
string strConnection = BuildConnectionString();
SqlConnection linkToDB = new SqlConnection(strConnection);
if (paramMode == "non query")
{
linkToDB.Open();
SqlCommand sqlCom = new SqlCommand(paramSqlStat, linkToDB);
sqlCom.ExecuteNonQuery();
linkToDB.Close();
}
if (paramMode == "table")
{
using (linkToDB)
using (var adapter = new SqlDataAdapter(cSqlStat, linkToDB))
{
DataTable table = new DataTable();
adapter.Fill(table);
this.cQueryResults = table;
}
}
if (paramMode == "scalar")
{
linkToDB.Open();
SqlCommand sqlCom = new SqlCommand(paramSqlStat, linkToDB);
this.cScalarResult = (Int32)sqlCom.ExecuteScalar();
linkToDB.Close();
}
}
public cSqlQuery(SqlCommand paramSqlCom, string paramMode)
{
string strConnection = BuildConnectionString();
SqlConnection linkToDB = new SqlConnection(strConnection);
paramSqlCom.Connection = linkToDB;
if (paramMode == "table")
{
using (linkToDB)
using (var adapter = new SqlDataAdapter(paramSqlCom))
{
DataTable table = new DataTable();
adapter.Fill(table);
this.cQueryResults = table;
}
}
if (paramMode == "scalar")
{
linkToDB.Open();
paramSqlCom.Connection = linkToDB;
this.cScalarResult = (Int32)paramSqlCom.ExecuteScalar();
linkToDB.Close();
}
}
public string BuildConnectionString()
{
cConnectionString cCS = new cConnectionString();
return cCS.strConnect;
}
}
The class works well throughout my application so I don't think the error is in the class, but then I can't be sure.
When I click the button I get the following error message:
Incorrect syntax near =
Which is really annoying me, because when I run the exact same command in SQL Management Studio it works fine.
I'm sure I'm missing something rather simple, but after reading my code through many times, I'm struggling to see where I have gone wrong.
you have to remove = after values.
string sqlText = "INSERT INTO tblDiaryTypes (DiaryType) VALUES ('" + strNewDiaryItem + "');"
and try to use Parameterized queries to avoid Sql injection. use your code like this. Sql Parameters
string sqlText = "INSERT INTO tblDiaryTypes (DiaryType) VALUES (#DairyItem);"
YourCOmmandObj.Parameters.AddwithValue("#DairyItem",strNewDiaryIItem)
Remove the = after VALUES.
You do not need the =
A valid insert would look like
INSERT INTO table_name (column1, column2, column3,...)
VALUES (value1, value2, value3,...)
Source: http://www.w3schools.com/sql/sql_insert.asp
Please use following:
insert into <table name> Values (value);
Remove "=", and also i would recommend you to use string.format() instead of string concatenation.
sqlText = string.format(INSERT INTO tblDiaryTypes (DiaryType) VALUES ('{0}'), strNewDiaryItem);"

Categories