I am running a SQLQuery that takes roughly 45 seconds to run and display results. I am using Task<DataSet> to populate two drop downs on my page. Well the 1st drop down populates fine (the query is completed in about 2 seconds), the second it seems that the adapter.Fill(dataSet) is not waiting on the query to complete before it begins to fill the drop down with a null dataset. What should I alter so that the code execution halts until the query executes completely?
Task.Factory.ContinueWhenAll(new[]
{
One("Data Source=server;Initial Catalog=db;Integrated Security=True;MultipleActiveResultSets=True"),
Two("Data Source=server;Initial Catalog=db;Integrated Security=True;MultipleActiveResultSets=True"),
}, tasks =>
{
try
{
this.ddl1.DataSource = tasks[0].Result.Tables[0];
this.ddl1.DataTextField = "One";
this.ddl1.DataValueField = "ID";
this.ddl1.DataBind();
int indexOfLastItem = this.ddl1.Items.Count - 1;
this.ddl1.SelectedIndex = indexOfLastItem;
ddl2.DataSource = tasks[1].Result.Tables[0];
this.ddl2.DataTextField = "Two";
this.ddl2.DataValueField = "ID";
this.ddl2.DataBind();
this.ddl2.Items.Insert(0, new ListItem(Constants.All, Constants.All));
}
catch (Exception exception) { throw exception; }
}, CancellationToken.None, TaskContinuationOptions.None, TaskScheduler.FromCurrentSynchronizationContext());
public System.Threading.Tasks.Task<DataSet> One(string databaseConnection)
{
return FillDS("Select * from activeemployees", databaseConnection);
}
public System.Threading.Tasks.Task<DataSet> Two(string databaseConnection)
{
return FillDS("Select * from mastersalesdatabase", databaseConnection);
}
public System.Threading.Tasks.Task<DataSet> FillDS(string sqlQuery, string connectionString)
{
try
{
var dataSet = new DataSet();
using (var adapter = new SqlDataAdapter(sqlQuery, connectionString))
{
adapter.Fill(dataSet);
return dataSet;
}
}
catch (Exception exception) { throw exception; }
}
My query Select * from activeemployees completes in about 2 seconds and populates fine, my query Select * from mastersalesdatabase takes roughly 45 seconds and it seems the code just moves on w/o a delay to let the query execute to completion.
If you're going to do async to retrieve data into a datatable, it should look more like this:
public static async Task<DataTable> GetDataTableAsync(string connectionString, SqlCommand command)
{
using (var connection = new SqlConnection(connectionString))
{
command.Connection = connection;
await connection.OpenAsync();
using (var dataReader = await command.ExecuteReaderAsync())
{
var dataTable = new DataTable();
dataTable.Load(dataReader);
return dataTable;
}
}
}
Notice there's no need for a dataset.
Then in WebForms, we have to handle async code differently.
protected void Page_Load(object sender, EventArgs e)
{
RegisterAsyncTask(new PageAsyncTask(DoWorkAsync));
}
private async Task DoWorkAsync()
{
ActiveEmployeesDropDownList.DataSource = GetDataTableAsync(databaseConnection, new SqlCommand("select * from activeemployees"));
ActiveEmployeesDropDownList.DataBind();
}
Notice I renamed the control from ddl1 to ActiveEmployeesDropDownList because ddl1 is a horrible name. Your names should have semantic meaning.
You'll need to add the async=true attribute to your page according to MSDN.
And you should also fix your query to not take 45 seconds, but that's a separate question entirely.
Related
I have been working on an ASP.Net application for a long time and there are above 10 clients using the application. But now I found a problem in the application, that is, I have a stored procedure call which takes about 30 seconds to execute. It is not a problem, because the SQL code highly complicated and looping many times. The problem is :
Whenever that stored procedure call is executing, I am not able to using any other functions or stored procedure call.
When I tried debugging, the problem is that 'DataAdapter.Fill()' function is waiting for the first stored procedure call to finish.
My code that executes stored procedure call and returning data is :
public static DataSet ExecuteQuery_SP(string ProcedureName, object[,] ParamArray)
{
SqlDataAdapter DataAdapter = new SqlDataAdapter();
DataSet DS = new DataSet();
try
{
if (CON.State != ConnectionState.Open)
OpenConnection();
SqlCommand cmd = new SqlCommand();
cmd.CommandTimeout = 0;
cmd.CommandText = ProcedureName;
cmd.CommandType = CommandType.StoredProcedure;
cmd.Connection = CON;
cmd.Transaction = SqlTrans;
string ParamName;
object ParamValue;
for (int i = 0; i < ParamArray.Length / 2; i++)
{
ParamName = ParamArray[i, 0].ToString();
ParamValue = ParamArray[i, 1];
cmd.Parameters.AddWithValue(ParamName, ParamValue);
}
DataAdapter = new SqlDataAdapter(cmd);
DataAdapter.Fill(DS);
cmd.CommandText = "";
}
catch (Exception ea)
{
}
return DS;
}
All stored procedure calls are working through this function. Hence when my first stored procedure call 'A' is running, stored procedure call 'B' will not execute until 'A' is finished.
This reduces overall performance of the application and causes problem in data retrieval.
I surfed google and found that 'Threading' can be helpful but I am not able to execute threading properly. I am not so familiar with these kind of things. It will helpful if you can rectify the problem.
My first stored procedure call is:
ds = DB.ExecuteQuery_SP("SelectOutstandingReportDetailed", parArray);
Where ds is the DataSet object.
Second stored procedure call is :
ds = DB.ExecuteQuery_SP("[SelectAccLedgersDetailsByID]", ParamArray);
My current DB connection open function is :
public static bool OpenConnection()
{
try
{
Server = (String)HttpContext.GetGlobalResourceObject("Resource", "Server");
DBName = (String)HttpContext.GetGlobalResourceObject("Resource", "DBName");
UserName = (String)HttpContext.GetGlobalResourceObject("Resource", "UserName");
PassWord = (String)HttpContext.GetGlobalResourceObject("Resource", "PassWord");
string ConnectionString;
ConnectionString = "server=" + Server + "; database=" + DBName + "; uid=" + UserName + "; pwd=" + PassWord + "; Pooling='true';Max Pool Size=100;MultipleActiveResultSets=true;Asynchronous Processing=true";
CON.ConnectionString = ConnectionString;
if (CON.State != ConnectionState.Open)
{
CON.Close();
CON.Open();
}
}
catch (Exception ea)
{
}
return false;
}
Where 'CON' is a public SqlConnection variable
static SqlConnection CON = new SqlConnection();
I found the problem, that is, all stored procedure calls are performed through this 'CON' object. If there is seperate SqlConnection object for each stored procedure call, then no problem is there.
So is it possible to make separate SqlConnection for every ExecuteQuery_SP calls.
If any doubt in question, kindly comment.
Thank you
SQL Server will allow thousands of connections simultaneously, by default. This is NOT the source of your problem. You have forced every call to a stored procedure to be funneled through a single method. Factor out your calls to the stored procedures - in other words, lose the ExecuteQuery_SP method which is a bottleneck. Then test again.
Here's is an introduction to data layers.
Heres the simplest version I can create for you.
Important: to understand you should read about async-await.
You can start at the Microsoft C# Async-Await Docs
// TODO set up your connection string
private string connectionString = "<your connection string>";
// Gets data assyncronously
public static async Task<DataTable> GetDataAsync(string procedureName, object[,] ParamArray)
{
try
{
var asyncConnectionString = new SqlConnectionStringBuilder(connectionString)
{
AsynchronousProcessing = true
}.ToString();
using (var conn = new SqlConnection(asyncConnectionString))
{
using (var SqlCommand = new SqlCommand())
{
SqlCommand.Connection = conn;
SqlCommand.CommandText = procedureName;
SqlCommand.CommandType = CommandType.StoredProcedure;
string ParamName;
object ParamValue;
for (int i = 0; i < ParamArray.Length / 2; i++)
{
ParamName = ParamArray[i, 0].ToString();
ParamValue = ParamArray[i, 1];
SqlCommand.Parameters.AddWithValue(ParamName, ParamValue);
}
conn.Open();
var data = new DataTable();
data.BeginLoadData();
using (var reader = await SqlCommand.ExecuteReaderAsync().ConfigureAwait(true))
{
if (reader.HasRows)
data.Load(reader);
}
data.EndLoadData();
return data;
}
}
}
catch (Exception Ex)
{
// Log error or something else
throw;
}
}
public static async Task<DataTable> GetData(object General, object Type, string FromDate, string ToDate)
{
object[,] parArray = new object[,]{
{"#BranchID",General.BranchID},
{"#FinancialYearID",General.FinancialYearID},
{"#Type",Type},
{"#FromDate",DateTime.ParseExact(FromDate, "dd/MM/yyyy", System.Globalization.CultureInfo.InvariantCulture)},
{"#ToDate",DateTime.ParseExact(ToDate, "dd/MM/yyyy", System.Globalization.CultureInfo.InvariantCulture)}
};
return await DataBaseHelper.GetDataAsync("SelectOutstandingReportDetailed", parArray);
}
// Calls database assyncronously
private async Task ConsumeData()
{
DataTable dt = null;
try
{
// TODO configure your parameters here
object general = null;
object type = null;
string fromDate = "";
string toDate = "";
dt = await GetData(general, type, fromDate, toDate);
}
catch (Exception Ex)
{
// do something if an error occurs
System.Diagnostics.Debug.WriteLine("Error occurred: " + Ex.ToString());
return;
}
foreach (DataRow dr in dt.Rows)
{
System.Diagnostics.Debug.WriteLine(dr.ToString());
}
}
// Fired when some button is clicked. Get and use the data assyncronously, i.e. without blocking the UI.
private async void button1_Click(object sender, EventArgs e)
{
await ConsumeData();
}
Hi all: I have a program that is running 4 threads that talk to an Oracle database. I have the Oracle connections local to each thread, and I'm employing the USING statement as well as manually closing the recordset and closing the connection. As I understand it, the ORA-01000 error arises when there are more open recordsets than configured cursors on the database. I do not understand why my recordsets are staying open or why I'm getting this error. Here's the code:
static void CheckPaths()
{
int pathcount = paths.Count; //paths is a typed list
Parallel.ForEach(paths, new ParallelOptions { MaxDegreeOfParallelism = 4 }, (p) =>
{
try
{
CheckSinglePathAllHours(p);
}
catch (Exception ex)
{
//there is logging here, this is where the exception hits
}
});
}
static void CheckSinglePathAllHours(Path p)
{
string sqlBase = #"Select * from table ";//this is actually a big SQL statement
using (DBManager localdbm = new DBManager())
{
string sql = sqlBase;
OracleDataReader reader = localdbm.GetData(sql);
while (reader.Read())
{
//process the path, query always returns 24 or less rows
}
reader.Close();
reader = null; //is this even necessary?
localdbm.Close(); //is this necessary in conjunction with the USING statement?
}
}
class DBManager : IDisposable
{
OracleConnection conn;
OracleCommand cmd;
public DBManager()
{
string connStr = "blah blah blah";
conn = new OracleConnection(connStr);
conn.Open();
cmd = conn.CreateCommand();
}
public OracleDataReader GetData(string sql)
{
cmd.CommandText = sql;
cmd.CommandTimeout = 900;
return cmd.ExecuteReader();
}
public void RunSQL(string sql)
{
cmd.CommandText = sql;
cmd.CommandTimeout = 900;
cmd.ExecuteNonQuery();
}
public void Close()
{
conn.Close();
}
public void Dispose()
{
try
{
conn.Close();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
}
The code will usually run for about a minute or two before the exception. The exception message is two-fold: ORA-00604: error occured at recursive SQL level 1; and ORA-01000: maximum open cursors exceeded. Any ideas what I'm doing wrong?
Changed the code to call .Dispose() on OracleDataReader and OracleConnection as suggested by Paul Abbott. Also increased the number of cursors per session from 50 to 150 on the database.
at time my query times out and cause a failure. What is the best way to retry to execute a query?
I validate that the connection is open before executing the query. However, due to the server load at any given time, it may take <1 min to 5+ minutes. I thought about extending the CommandTimeout but I do not believe that is really the way to go.
Here is my sql query code. Thanks for all the assistance.
private static void ExecuteQuery(string connectionString, string query)
{
SqlConnection connection = new SqlConnection(connectionString);
DataTable output = new DataTable();
try
{
//create new SqlAdataAdapter
SqlDataAdapter command = new SqlDataAdapter {SelectCommand = new SqlCommand(query, connection)};
//connect to Sqldb
connection.Open();
//validate connection to database before executing query
if (connection.State != ConnectionState.Open) return;
Console.WriteLine("Connection successful\nExecuting query...");
//set connection timeout
command.SelectCommand.CommandTimeout = 200;
//create new dataSet in order to input output of query to
DataSet dataSet = new DataSet();
//fill the dataSet
command.Fill(dataSet, "capacity");
DataTable dtTable1 = dataSet.Tables["capacity"];
Console.WriteLine("There are " + dtTable1.Rows.Count + " clusters within the capacity anlaysis.");
output = dtTable1;
}
catch (Exception e)
{
Console.WriteLine("Unable to execute capacity (all records) query due to {0}", e.Message);
}
finally
{
connection.Close();
Declarations.NumOfClusters = output.Rows.Count;
Declarations.finalIssues = Issues(output, 2m, 20, true);
Console.WriteLine("\n---------------Successfully Created Capacity DataSet---------------\n");
}
}
Use Palmer library: https://github.com/mitchdenny/palmer
Retry.On<Exception>().For(TimeSpan.FromSeconds(15)).With(context =>
{
// Code that might periodically fail due to some issues.
ExecuteQuery(string connectionString, string query)
if (contect.DidExceptionLastTime)
Thread.Sleep(200); // what ever you wish
});
Refer to the API on the github page. You can for example check the context for exceptions and decide to sleep for a while if an exception did happen.
You can Retry on more specific exception.
You can try forever, etc.
Re-structure you code in manner so that it would allow you to call the query recursively till you get a desired result.
Eg.
private static void ExecuteQuery(string connectionString, string query)
{
SqlConnection connection = new SqlConnection(connectionString);
DataTable output = null;
while output is null
{
output = getDataFromDB(query);
}
if(output is DataTable && output.Rows.Count > 0)
{
Console.WriteLine("There are " + output.Rows.Count + " clusters within the capacity analysis.");
}
}
private DataTable getDataFromDB(string query)
{
DataTable oDTResult = null;
try
{
//create new SqlAdataAdapter
SqlDataAdapter command = new SqlDataAdapter {SelectCommand = new SqlCommand(query, connection)};
//connect to Sqldb
connection.Open();
//validate connection to database before executing query
if (connection.State != ConnectionState.Open) return;
Console.WriteLine("Connection successful\nExecuting query...");
//set connection timeout
command.SelectCommand.CommandTimeout = 200;
//create new dataSet in order to input output of query to
DataSet dataSet = new DataSet();
//fill the dataSet
command.Fill(dataSet, "capacity");
DataTable dtTable1 = dataSet.Tables["capacity"];
oDTResult = dtTable1;
}
catch (Exception e)
{
Console.WriteLine("Unable to execute capacity (all records) query due to {0}", e.Message);
}
finally
{
connection.Close();
Declarations.NumOfClusters = output.Rows.Count;
Declarations.finalIssues = Issues(output, 2m, 20, true);
Console.WriteLine("\n---------------Successfully Created Capacity DataSet---------------\n");
}
return oDTResult;
}
You would want in-cooperate the ability to number of retries the program should attempt to executing this to give it some flexibility.
Further if your query is taking such a long time you should look into ways how you optimise the SQL Server Database to cut down on the execution time with the aid of Views/ Indexes etc.
A generic method for retry the action
public static class Retry
{
public static void Do(
Action action,
TimeSpan retryInterval,
int retryCount = 3)
{
Do<object>(() =>
{
action();
return null;
}, retryInterval, retryCount);
}
public static T Do<T>(
Func<T> action,
TimeSpan retryInterval,
int retryCount = 3)
{
var exceptions = new List<Exception>();
for (int retry = 0; retry < retryCount; retry++)
{
try
{
if (retry > 0)
Thread.Sleep(retryInterval);
return action();
}
catch (Exception ex)
{
exceptions.Add(ex);
}
}
throw new AggregateException(exceptions);
}
}
You can now use this utility method to perform retry logic:
Retry.Do(() => SomeFunctionThatCanFail(), TimeSpan.FromSeconds(1));
I write a sample code in asp.net web form which runs for 10 times .
static int noOfTries = 0;
protected void Page_Load(object sender, EventArgs e)
{
function();
}
private void function()
{
try
{
if (noOfTries == 10) goto XX;
noOfTries++;
int a = 0;
int b = 1 / a;
}
catch (Exception ew)
{
Response.Write(ew.Message + "<br/>");
function();
}
XX:
int c = 0;
}
Note:
Its is not thread safe as use of static variable
static int noOfTries=0
Multi-thread execution may not work as you except because static variable will be shared in multi-thread.
Solve use
Session["noOfTries"]
if multi-thread execution environment .
I have just taken over a project using webforms and C# with my asp.net project. My page load times are very slow and I assume it is due to the multiple database calls that are being made, however I am not sure how to do it any differently. For example, I have one page that holds 3 different dropdownlists each dropdownlist is populated in the Page_Load() event handler, but all 3 have there own database call.
Below is pseducode to show the approach being used. What is the proper way to accomplish something like this?
namespace CEDS
{
public partial class BBLL : System.Web.UI.UserControl
{
private DataSet DS = new DataSet();
private C2 _C2 = new C2();
protected void Page_Load(object sender, EventArgs e)
{
if (!IsPostBack)
{
GetDataForDropDown1();
GetDataForDropDown2();
GetDataForDropDown3();
}
}
private void GetDataForDropDown1()
{
DS = _C2.GetDataForDropDown1();
this.gv1.DataSource = DS;
this.gv1.DataBind();
this.gv1.Visible = true;
}
private void GetDataForDropDown2()
{
DS = _C2.GetDataForDropDown2();
this.gv2.DataSource = DS;
this.gv2.DataBind();
this.gv2.Visible = true;
}
private void GetDataForDropDown3()
{
DS = _C2.GetDataForDropDown3();
this.gv3.DataSource = DS;
this.gv3.DataBind();
this.gv3.Visible = true;
}
}
public class C2
{
private DataSet DS = new DataSet();
private DatabaseAccessLayer DAL = new DatabaseAccessLayer();
public DataSet GetDataForDropDown1()
{
DS = new DataSet();
DAL.SqlQueryBuilder = new StringBuilder();
DAL.SqlQueryBuilder.Append("exec dbo.RunStoredProcedure1 ");
DS = DAL.ExecuteSqlQuery(databaseConnection, DAL.SqlQueryBuilder.ToString());
return DS;
}
public DataSet GetDataForDropDown2()
{
DS = new DataSet();
DAL.SqlQueryBuilder = new StringBuilder();
DAL.SqlQueryBuilder.Append("exec dbo.RunStoredProcedure2 ");
DS = DAL.ExecuteSqlQuery(databaseConnection, DAL.SqlQueryBuilder.ToString());
return DS;
}
public DataSet GetDataForDropDown3()
{
DS = new DataSet();
DAL.SqlQueryBuilder = new StringBuilder();
DAL.SqlQueryBuilder.Append("exec dbo.RunStoredProcedure3 ");
DS = DAL.ExecuteSqlQuery(databaseConnection, DAL.SqlQueryBuilder.ToString());
return DS;
}
}
public class DatabaseAccessLayer
{
public DataSet ExecuteSqlQuery(string connectionString, string sqlQuery)
{
try
{
_connectionString = System.Configuration.ConfigurationManager.AppSettings[connectionString].ToString();
_sqlDatabaseConnection = new SqlConnection(_connectionString);
_sqlCommand = new SqlCommand(sqlQuery, _sqlDatabaseConnection);
_sqlDatabaseConnection.Open();
_sqlCommand.CommandTimeout = 0;
_dataSet = new DataSet();
_sqlDataAdapter = new SqlDataAdapter(_sqlCommand);
_sqlDataAdapter.Fill(_dataSet, "Data");
return _dataSet;
}
catch (Exception exception) { throw exception; }
finally
{
_sqlDatabaseConnection.Close();
_sqlCommand.Dispose();
_sqlDataAdapter.Dispose();
}
}
}
}
Page_Load()
{
var t1 = GetDataForDropDown1();
var t2 = GetDataForDropDown2();
var t3 = GetDataForDropDown3();
await Task.WhenAll(t1, t2, t3);
PopulateDD1();
PopulateDD2();
PopulateDD3();
}
async Task GetDataForDropDown1()
{
SqlQuery
Call To Database Access Layer
await Execute Stored Procedure
Store Returned Result In Dataset
}
async Task GetDataForDropDown2()
{
SqlQuery
Call To Database Access Layer
await Execute Stored Procedure
Store Returned Result In Dataset
}
async Task GetDataForDropDown3()
{
SqlQuery
Call To Database Access Layer
await Execute Stored Procedure
Store Returned Result In Dataset
}
You should be able to do
Parallel.Invoke(GetDataForDropDown1, GetDataForDropDown2, GetDataForDropDown3);
So at least you aren't waiting for the first to complete until you start waiting for the second and third.
It might be more effective to have a single stored procedure that returns all three recordsets, so your database connection and retrieval roundtrip is only made once. But that will probably mean you have to change your data layer code.
The better approach might be this, try it.
Page_Load()
{
if(!Page.IsPostBack)
{
LoadData();
}
}
LoadData()
{
// PopulateDropDown1 Code
// PopulateDropDown2 Code
// PopulateDropDown3 Code
}
if(!Page.IsPostBack) prevents LoadData() to call on every postback.
in my code the user can upload an excel document wish contains it's phone contact list.Me as a developer should read that excel file turn it into a dataTable and insert it into the database .
The Problem is that some clients have a huge amount of contacts like saying 5000 and more contacts and when i am trying to insert this amount of data into the database it's crashing and giving me a timeout exception.
What would be the best way to avoid this kind of exception and is their any code that can reduce the time of the insert statement so the user don't wait too long ?
the code
public SqlConnection connection = new SqlConnection(System.Configuration.ConfigurationManager.ConnectionStrings["ConnectionString"].ConnectionString);
public void Insert(string InsertQuery)
{
SqlDataAdapter adp = new SqlDataAdapter();
adp.InsertCommand = new SqlCommand(InsertQuery, connection);
if (connection.State == System.Data.ConnectionState.Closed)
{
connection.Open();
}
adp.InsertCommand.ExecuteNonQuery();
connection.Close();
}
protected void submit_Click(object sender, EventArgs e)
{
string UploadFolder = "Savedfiles/";
if (Upload.HasFile) {
string fileName = Upload.PostedFile.FileName;
string path=Server.MapPath(UploadFolder+fileName);
Upload.SaveAs(path);
Msg.Text = "successfully uploaded";
DataTable ValuesDt = new DataTable();
ValuesDt = ConvertExcelFileToDataTable(path);
Session["valuesdt"] = ValuesDt;
Excel_grd.DataSource = ValuesDt;
Excel_grd.DataBind();
}
}
protected void SendToServer_Click(object sender, EventArgs e)
{
DataTable Values = Session["valuesdt"] as DataTable ;
if(Values.Rows.Count>0)
{
DataTable dv = Values.DefaultView.ToTable(true, "Mobile1", "Mobile2", "Tel", "Category");
double Mobile1,Mobile2,Tel;string Category="";
for (int i = 0; i < Values.Rows.Count; i++)
{
Mobile1 =Values.Rows[i]["Mobile1"].ToString()==""?0: double.Parse(Values.Rows[i]["Mobile1"].ToString());
Mobile2 = Values.Rows[i]["Mobile2"].ToString() == "" ? 0 : double.Parse(Values.Rows[i]["Mobile2"].ToString());
Tel = Values.Rows[i]["Tel"].ToString() == "" ? 0 : double.Parse(Values.Rows[i]["Tel"].ToString());
Category = Values.Rows[i]["Category"].ToString();
Insert("INSERT INTO client(Mobile1,Mobile2,Tel,Category) VALUES(" + Mobile1 + "," + Mobile2 + "," + Tel + ",'" + Category + "')");
Msg.Text = "Submitied successfully to the server ";
}
}
}
You can try SqlBulkCopy to insert Datatable to Database Table
Something like this,
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(sqlConnection, SqlBulkCopyOptions.KeepIdentity))
{
bulkCopy.DestinationTableName = DestTableName;
string[] DtColumnName = YourDataTableColumns;
foreach (string dbcol in DbColumnName)//To map Column of Datatable to that of DataBase tabele
{
foreach (string dtcol in DtColumnName)
{
if (dbcol.ToLower() == dtcol.ToLower())
{
SqlBulkCopyColumnMapping mapID = new SqlBulkCopyColumnMapping(dtcol, dbcol);
bulkCopy.ColumnMappings.Add(mapID);
break;
}
}
}
bulkCopy.WriteToServer(YourDataTableName.CreateDataReader());
bulkCopy.Close();
}
For more Read http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlbulkcopy.aspx
You are inserting 1 row at a time, which is very expensive for this amount of data
In those cases you should use bulk insert, so the round trip to DB will be only once, if you need to roll back - all is the same transaction
You can use SqlBulkCopy which is more work, or you can use the batch update feature of the SqlAdpater. Instead of creating your own insert statement, then building a sqladapter, and then manually executing it, create a dataset, fill it, create one sqldataadpater, set the number of inserts in a batch, then execute the adapter once.
I could repeat the code, but this article shows exactly how to do it: http://msdn.microsoft.com/en-us/library/kbbwt18a%28v=vs.80%29.aspx
protected void SendToServer_Click(object sender, EventArgs e)
{
DataTable Values = Session["valuesdt"] as DataTable ;
if(Values.Rows.Count>0)
{
DataTable dv = Values.DefaultView.ToTable(true, "Mobile1", "Mobile2", "Tel", "Category");
//Fix up default values
for (int i = 0; i < Values.Rows.Count; i++)
{
Values.Rows[i]["Mobile1"] =Values.Rows[i]["Mobile1"].ToString()==""?0: double.Parse(Values.Rows[i]["Mobile1"].ToString());
Values.Rows[i]["Mobile2"] = Values.Rows[i]["Mobile2"].ToString() == "" ? 0 : double.Parse(Values.Rows[i]["Mobile2"].ToString());
Values.Rows[i]["Tel"] = Values.Rows[i]["Tel"].ToString() == "" ? 0 : double.Parse(Values.Rows[i]["Tel"].ToString());
Values.Rows[i]["Category"] = Values.Rows[i]["Category"].ToString();
}
BatchUpdate(dv,1000);
}
}
public static void BatchUpdate(DataTable dataTable,Int32 batchSize)
{
// Assumes GetConnectionString() returns a valid connection string.
string connectionString = GetConnectionString();
// Connect to the database.
using (SqlConnection connection = new SqlConnection(connectionString))
{
// Create a SqlDataAdapter.
SqlDataAdapter adapter = new SqlDataAdapter();
// Set the INSERT command and parameter.
adapter.InsertCommand = new SqlCommand(
"INSERT INTO client(Mobile1,Mobile2,Tel,Category) VALUES(#Mobile1,#Mobile2,#Tel,#Category);", connection);
adapter.InsertCommand.Parameters.Add("#Mobile1",
SqlDbType.Float);
adapter.InsertCommand.Parameters.Add("#Mobile2",
SqlDbType.Float);
adapter.InsertCommand.Parameters.Add("#Tel",
SqlDbType.Float);
adapter.InsertCommand.Parameters.Add("#Category",
SqlDbType.NVarchar, 50);
adapter.InsertCommand.UpdatedRowSource = UpdateRowSource.None;
// Set the batch size.
adapter.UpdateBatchSize = batchSize;
// Execute the update.
adapter.Update(dataTable);
}
}
I know this is a super old post, but you should not need to use the bulk operations explained in the existing answers for 5000 inserts. Your performance is suffering so much because you close and reopen the connection for each row insert. Here is some code I have used in the past that keeps one connection open and executes as many commands as needed to push all the data to the DB:
public static class DataWorker
{
public static Func<IEnumerable<T>, Task> GetStoredProcedureWorker<T>(Func<SqlConnection> connectionSource, string storedProcedureName, Func<T, IEnumerable<(string paramName, object paramValue)>> parameterizer)
{
if (connectionSource is null) throw new ArgumentNullException(nameof(connectionSource));
SqlConnection openConnection()
{
var conn = connectionSource() ?? throw new ArgumentNullException(nameof(connectionSource), $"Connection from {nameof(connectionSource)} cannot be null");
var connState = conn.State;
if (connState != ConnectionState.Open)
{
conn.Open();
}
return conn;
}
async Task DoStoredProcedureWork(IEnumerable<T> workData)
{
using (var connection = openConnection())
using (var command = connection.CreateCommand())
{
command.CommandType = CommandType.StoredProcedure;
command.CommandText = storedProcedureName;
command.Prepare();
foreach (var thing in workData)
{
command.Parameters.Clear();
foreach (var (paramName, paramValue) in parameterizer(thing))
{
command.Parameters.AddWithValue(paramName, paramValue ?? DBNull.Value);
}
await command.ExecuteNonQueryAsync().ConfigureAwait(false);
}
}
}
return DoStoredProcedureWork;
}
}
This was actually from a project where I was gathering emails for a restriction list, so kind of relevant example of what a parameterizer argument might look like and how to use the above code:
IEnumerable<(string,object)> RestrictionToParameter(EmailRestriction emailRestriction)
{
yield return ("#emailAddress", emailRestriction.Email);
yield return ("#reason", emailRestriction.Reason);
yield return ("#restrictionType", emailRestriction.RestrictionType);
yield return ("#dateTime", emailRestriction.Date);
}
var worker = DataWorker.GetStoredProcedureWorker<EmailRestriction>(ConnectionFactory, #"[emaildata].[AddRestrictedEmail]", RestrictionToParameter);
await worker(emailRestrictions).ConfigureAwait(false);