Setting a specific value in a DataTable + writing on DB - C# - c#

The steps of my program are:
Read a CSV file
Read, from a DB, the ID from a specific file name (from step 1)
Create a DataTable with the CSV file
Add a column with the ID from step 2
Write the result on a DB
My problem:
The ID is not being written on the DB
The code has several parts. I will paste the main ones:
Calls:
string id_file = Get_id_file(filenameout, id_costumer);
//MessageBox.Show("the id of the file will be: " + id_file);
DataTable csvFileData = GetDataTabletFromCSVFile(filenameout, id_file);
InsertDataIntoSQLServerUsingSQLBulkCopy(csvFileData, destinytablename);
Reading from the DB:
public static string Get_id_file(string filenameout,string id_id_costumer)
{
var fileName = Path.GetFileNameWithoutExtension(filenameout);
string return_id_file = "";
using (SqlConnection dbConnection = new SqlConnection("XXXXX"))
{
try
{
dbConnection.Open();
SqlDataReader myReader = null;
// Using the DB to find the value
SqlCommand myCommand = new SqlCommand("select ID from TB_FILE_NAMES where path_name='" + fileName + "' and id_program='" + id_id_costumer + "'",
dbConnection);
myReader = myCommand.ExecuteReader();
while (myReader.Read())
{
MessageBox.Show("the ID selected for the file is: " + myReader["ID"].ToString());
return_id_file = (myReader["ID"].ToString());
}
//return_id_file = "9999";
return return_id_file;
}
catch (Exception e)
{
MessageBox.Show(e.ToString());
return ("Error in return_id_file");
}
} // end of using (SqlConnection dbConnection)
}
Part of "GetDataTableFromCSVFile"
On this part, I just wanna set the ID (found in the previous step) to all rows for that specific CSV
// Add a column with the ID of the file
//DataColumn Col =
csvData.Columns.Add("ID_TB_FILE_NAMES", System.Type.GetType("System.String"));
// Set a value for the entire column - ID of the File
//var column2 = csvData.Columns["ID_TB_FILE_NAMES"];
foreach (DataRow row in csvData.Rows)
{
//row.SetField(id_file_name, column2);
row.SetField("ID_TB_FILE_NAMES", id_file_name);
}
return csvData;
} // end of GetDataTabletFromCSVFile
Writing, on the DB, the csvData:
static void InsertDataIntoSQLServerUsingSQLBulkCopy(DataTable csvFileData,string destinytablename)
{
using (SqlConnection dbConnection = new SqlConnection("XXXXX"))
{
dbConnection.Open();
// Clean the destiny table
//string query = "TRUNCATE TABLE " + destinytablename;
//SqlCommand cmd = new SqlCommand(query, dbConnection);
//cmd.ExecuteNonQuery();
using (SqlBulkCopy s = new SqlBulkCopy(dbConnection))
{
s.DestinationTableName = destinytablename;
//foreach (var column in csvFileData.Columns)
// s.ColumnMappings.Add(column.ToString(), column.ToString());
s.WriteToServer(csvFileData);
}
}
} // end of InsertDataIntoSQLServerUsingSQLBulkCopy
I spent 3 days trying to figure out... Any help is welcome.
Thank you.

Related

How to get excel rows one by one in command text using loop?oledb,c#

I am using oledb to read excel file and then i am trying to apply some conditions and then save in db, but i just want to read excel row one by one using loop.
var cmd = conn.CreateCommand();
cmd.CommandText = "select * from [الحيازات$]";// want to get this rows using loop so
//that only specific row is selected perform opertion blow then i will save in db,but
//its selecting all rows now
using (var rdr = cmd.ExecuteReader())
{
//LINQ query - when executed will create anonymous objects for each row
var query = (from DbDataRecord row in rdr
select row)
.Select(x =>
{
//dynamic item = new ExpandoObject();
Dictionary<string, string> item = new Dictionary<string, string>();
for (int i = 0; i < x.FieldCount; i++)
{
//code
}
});
}
Please check this
private void ReadExcel()
{
try
{
string connectionString = #"Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" + filePath + #";Extended Properties=""Excel 12.0;HDR=YES;IMEX=1;""";
using (OleDbConnection connection = new OleDbConnection())
{
connection.ConnectionString = connectionString;
using (OleDbCommand command = connection.CreateCommand())
{
command.CommandText = "SELECT * FROM [Sheet1$]";
connection.Open();
OleDbDataAdapter dataAdapter = new OleDbDataAdapter(command);
DataSet dataSet = new DataSet();
dataAdapter.Fill(dataSet);
if (dataSet != null && dataSet.Tables[0] != null)
{
foreach (DataRow row in dataSet.Tables[0].Rows)
{
string value = row["EmailColumnName"].ToString();
}
}
}
}
}
catch (Exception ex)
{
//Handle Exception
}
}

C# ASP.Net Processing Issue - Reading Table While Deleting Rows

Simply, I have an application that has one page that deletes and then re-adds/refreshes the records into a table every 30 seconds. I have another page that runs every 45 seconds that reads the table data and builds a chart.
The problem is, in the read/view page, every once in a while I get a 0 value (from a max count) and the chart shows nothing. I have a feeling that this is happening because the read is being done at the exact same time the delete page has deleted all the records in the table but has not yet refreshed/re-added them.
Is there a way in my application I can hold off on the read when the table is being refreshed?
Best Regards,
Andy
C#
ASP.Net 4.5
SQL Server 2012
My code below is run in an ASP.Net 4.5 built Windows service. It deletes all records in the ActualPlot table and then refreshes/adds new records from a text file every 30 seconds. I basically need to block (lock?) any user from reading the ActualPlot table while the records are being deleted and refreshed. Can you PLEASE help me change my code to do this?
private void timer1_Tick(object sender, ElapsedEventArgs e)
{
// Open the SAP text files, clear the data in the tables and repopulate the new SAP data into the tables.
var cnnString = ConfigurationManager.ConnectionStrings["TaktBoardsConnectionString"].ConnectionString;
SqlConnection conn = new SqlConnection(cnnString);
SqlConnection conndetail = new SqlConnection(cnnString);
SqlConnection connEdit = new SqlConnection(cnnString);
SqlCommand cmdGetProductFile = new SqlCommand();
SqlDataReader reader;
string sql;
// Delete all the records from the ActualPlot and the ActualPlotPreload tables. We are going to repopulate them with the data from the text file.
sql = "DELETE FROM ActualPlotPreload";
try
{
conn.Open();
SqlCommand cmd = new SqlCommand(sql, conn);
cmd.ExecuteNonQuery();
}
catch (System.Data.SqlClient.SqlException ex)
{
string msg = "Delete Error:";
msg += ex.Message;
Library.WriteErrorLog(msg);
}
finally
{
conn.Close();
}
sql = "DELETE FROM ActualPlot";
try
{
conn.Open();
SqlCommand cmd = new SqlCommand(sql, conn);
cmd.ExecuteNonQuery();
}
catch (System.Data.SqlClient.SqlException ex)
{
string msg = "Delete Error:";
msg += ex.Message;
Library.WriteErrorLog(msg);
}
finally
{
conn.Close();
}
// Read the SAP text file and load the data into the ActualPlotPreload table
sql = "SELECT DISTINCT [BoardName], [ProductFile], [ProductFileIdent] FROM [TaktBoards].[dbo].[TaktBoard] ";
sql = sql + "JOIN [TaktBoards].[dbo].[Product] ON [Product].[ProductID] = [TaktBoard].[ProductID]";
cmdGetProductFile.CommandText = sql;
cmdGetProductFile.CommandType = CommandType.Text;
cmdGetProductFile.Connection = conn;
conn.Open();
reader = cmdGetProductFile.ExecuteReader();
string DBProductFile = "";
string DBTischID = "";
string filepath = "";
string[] cellvalues;
DateTime dt, DateCheckNotMidnightShift;
DateTime ldSAPFileLastMod = DateTime.Now;
string MyDateString;
int FileRecordCount = 1;
while (reader.Read())
{
DBProductFile = (string)reader["ProductFile"];
DBTischID = (string)reader["ProductFileIdent"];
filepath = "c:\\inetpub\\wwwroot\\WebApps\\TaktBoard\\FilesFromSAP\\" + DBProductFile;
FileInfo fileInfo = new FileInfo(filepath); // Open file
ldSAPFileLastMod = fileInfo.LastWriteTime; // Get last time modified
try
{
StreamReader sr = new StreamReader(filepath);
FileRecordCount = 1;
// Populate the AcutalPlotPreload table from with the dates from the SAP text file.
sql = "INSERT into ActualPlotPreload (ActualDate, TischID) values (#ActualDate, #TischID)";
while (!sr.EndOfStream)
{
cellvalues = sr.ReadLine().Split(';');
if (FileRecordCount > 1 & cellvalues[7] != "")
{
MyDateString = cellvalues[7];
DateTime ldDateCheck = DateTime.ParseExact(MyDateString, "M/dd/yyyy", null);
DateTime dateNow = DateTime.Now;
string lsDateString = dateNow.Month + "/" + dateNow.Day.ToString("d2") + "/" + dateNow.Year;
DateTime ldCurrentDate = DateTime.ParseExact(lsDateString, "M/dd/yyyy", null);
string lsTischID = cellvalues[119];
if (ldDateCheck == ldCurrentDate)
{
try
{
conndetail.Open();
SqlCommand cmd = new SqlCommand(sql, conndetail);
cmd.Parameters.Add("#ActualDate", SqlDbType.DateTime);
cmd.Parameters.Add("#TischID", SqlDbType.VarChar);
cmd.Parameters["#TischID"].Value = cellvalues[119];
MyDateString = cellvalues[7] + " " + cellvalues[55];
dt = DateTime.ParseExact(MyDateString, "M/dd/yyyy H:mm:ss", null);
cmd.Parameters["#ActualDate"].Value = dt;
// Ignore any midnight shift (12am to 3/4am) units built.
DateCheckNotMidnightShift = DateTime.ParseExact(cellvalues[7] + " 6:00:00", "M/dd/yyyy H:mm:ss", null);
if (dt >= DateCheckNotMidnightShift)
{
cmd.ExecuteNonQuery();
}
}
catch (System.Data.SqlClient.SqlException ex)
{
string msg = "Insert Error:";
msg += ex.Message;
Library.WriteErrorLog(msg);
}
finally
{
conndetail.Close();
}
}
}
FileRecordCount++;
}
sr.Close();
}
catch
{ }
finally
{ }
}
conn.Close();
// Get the unique TischID's and ActualDate from the ActualPlotPreload table. Then loop through each one, adding the ActualUnits
// AcutalDate and TischID to the ActualPlot table. For each unique TischID we make sure that we reset the liTargetUnits to 1 and
// count up as we insert.
SqlCommand cmdGetTischID = new SqlCommand();
SqlDataReader readerTischID;
int liTargetUnits = 0;
string sqlInsert = "INSERT into ActualPlot (ActualUnits, ActualDate, TischID) values (#ActualUnits, #ActualDate, #TischID)";
sql = "SELECT DISTINCT [ActualDate], [TischID] FROM [TaktBoards].[dbo].[ActualPlotPreload] ORDER BY [TischID], [ActualDate] ASC ";
cmdGetTischID.CommandText = sql;
cmdGetTischID.CommandType = CommandType.Text;
cmdGetTischID.Connection = conn;
conn.Open();
readerTischID = cmdGetTischID.ExecuteReader();
DBTischID = "";
DateTime DBActualDate;
string DBTischIDInitial = "";
while (readerTischID.Read())
{
DBTischID = (string)readerTischID["TischID"];
DBActualDate = (DateTime)readerTischID["ActualDate"];
if (DBTischIDInitial != DBTischID)
{
liTargetUnits = 1;
DBTischIDInitial = DBTischID;
}
else
{
liTargetUnits++;
}
try
{
conndetail.Open();
SqlCommand cmd = new SqlCommand(sqlInsert, conndetail);
cmd.Parameters.Add("#ActualUnits", SqlDbType.Real);
cmd.Parameters.Add("#ActualDate", SqlDbType.DateTime);
cmd.Parameters.Add("#TischID", SqlDbType.VarChar);
cmd.Parameters["#TischID"].Value = DBTischID;
cmd.Parameters["#ActualDate"].Value = DBActualDate;
cmd.Parameters["#ActualUnits"].Value = liTargetUnits;
cmd.ExecuteNonQuery();
cmd.Parameters.Clear();
}
catch (System.Data.SqlClient.SqlException ex)
{
string msg = "Insert Error:";
msg += ex.Message;
Library.WriteErrorLog(msg);
}
finally
{
conndetail.Close();
}
}
conn.Close();
Library.WriteErrorLog("SAP text file data has been imported.");
}
If the data is being re-added right back after the delete (basically you know what to re-add before emptying the table), you could have both operation within the same SQL transaction, so that the data will be available to the other page only when it has been re-added.
I mean something like that :
public bool DeleteAndAddData(string connString)
{
using (OleDbConnection conn = new OleDbConnection(connString))
{
OleDbTransaction tran = null;
try
{
conn.Open();
tran = conn.BeginTransaction();
OleDbCommand deleteComm = new OleDbCommand("DELETE FROM Table", conn);
deleteComm.ExecuteNonQuery();
OleDbCommand reAddComm = new OleDbCommand("INSERT INTO Table VALUES(1, 'blabla', 'etc.'", conn);
reAddComm.ExecuteNonQuery();
tran.Commit();
}
catch (Exception ex)
{
tran.Rollback();
return false;
}
}
return true;
}
If your queries don't take too long to execute, you can start the two with a difference of 7.5 seconds, as there is a collision at every 90 seconds when the read/write finishes 3 cycles, and read/view finishes 2 cycles.
That being said, it's not a fool-proof solution, just a trick based on assumptions, in case you wan't to be completely sure that read/view never happens when read/write cycle is happening, try considering having a Read Lock. I would recommend reading Understanding how SQL Server executes a query and Locking in the Database Engine
Hope that helps.
I would try a couple of things:
Make sure your DELETE + INSERT operation is occurring within a single transaction:
BEGIN TRAN
DELETE FROM ...
INSERT INTO ...
COMMIT
If this isn't a busy table, try locking hints your SELECT statement. For example:
SELECT ...
FROM Table
WITH (UPDLOCK, HOLDLOCK)
In the case where the update transactions starts while your SELECT statement is running, this will cause that transaction to wait until the SELECT is finished. Unfortunately it will block other SELECT statements too, but you don't risk reading dirty data.
I was not able to figure this out but I changed my code so the program was not deleting all the rows in the ActualPlot table but checking to see if the row was there and if not adding the new row from the text file.

Get the Row values from a excel based on the column Value

I am trying to a get the Row values from a excel sheet, based on the column Value.
e.g. I have CutsomerID as lets say 5 , so I want First name 5, last Name 5 and Address 5
I am converting whole excel sheet into DataTable and then trying to read on each DataRow, when I get CustomerID as 5, I copy all the values and break from the loop
Here is my code and it is working fine as well, but I was wondering is there any way to optimise it.
Here is my Code.
public ExcelData GetDataByCustomerID(String excelFilePath, String customerID)
{
OleDbConnectionStringBuilder connectionStringBuilder = new OleDbConnectionStringBuilder();
connectionStringBuilder.Provider = "Microsoft.ACE.OLEDB.12.0";
connectionStringBuilder.DataSource = excelFilePath;
connectionStringBuilder.Add("Mode", "Read");
const string extendedProperties = "Excel 12.0;IMEX=1;HDR=YES";
connectionStringBuilder.Add("Extended Properties", extendedProperties);
String connectionString = connectionStringBuilder.ToString();
// Create connection object by using the preceding connection string.
using( var objConn = new OleDbConnection(connectionString))
{
objConn.Open();
// Get the data table contaning the schema guid.
DataTable excelSheetsDataTable = objConn.GetOleDbSchemaTable(OleDbSchemaGuid.Tables, null);
if (excelSheetsDataTable == null)
return null;
// get all the tables in the Sheet
List<String> excelSheets = (from DataRow row in excelSheetsDataTable.Rows select row["TABLE_NAME"].ToString()).ToList();
// Our data is on First sheet only
OleDbCommand _oleCmdSelect = new OleDbCommand(#"SELECT * FROM [" + excelSheets[0] + "]", objConn);
OleDbDataAdapter oleAdapter = new OleDbDataAdapter();
oleAdapter.SelectCommand = _oleCmdSelect;
DataTable newDataTable = new DataTable();
oleAdapter.FillSchema(newDataTable, SchemaType.Source);
oleAdapter.Fill(newDataTable);
if (newDataTable.Columns.Contains("CustomerID"))
{
foreach (DataRow rowValue in newTB.Rows)
{
if ((string) rowValue["CustomerID"] == customerID)
{
var data = new ExcelData
{
customerFirstName = rowValue["Customer_First_ Name"].ToString(),
customerLastName = rowValue["Customer_Last_Name"].ToString(),
customerAddress = rowValue["Customer_Address"].ToString(),
};
return data;
}
}
String message = String.Format("The CustomerID {0} not found in Excel file {1}", customerID, excelFilePath);
MessageBox.Show(message);
}
else
{
String message = String.Format("The Column CustomerID not found in Excel file {0}", excelFilePath);
MessageBox.Show(message);
}
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
return null;
}
public class ExcelData
{
public String customerID;
public String customerFirstName;
public String customerLastName;
public String customerAddress;
}
Change your select query to like below -
#"SELECT * FROM [" + excelSheets[0] + "] WHERE CustomerID=<your_value>"
Courtesy msdn web site:
Link: MSDN
DataTable dt;
private void button1_Click(object sender, EventArgs e)
{
try
{
openFileDialog1.ShowDialog();
string connectionString = string.Format("Provider = Microsoft.Jet.OLEDB.4.0;Data Source ={0};Extended Properties = Excel 8.0;", this.openFileDialog1.FileName);
var con = new OleDbConnection(connectionString);
var cmd = new OleDbCommand("select * from [sheet1$] where [MRN#]=#c", con);
cmd.Parameters.Add("#c", "33264");
con.Open();
var dr = cmd.ExecuteReader();
if (dr.HasRows)
{
dt = new DataTable();
dt.Load(dr);
}
dr.Close();
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
finally
{
DataGridView dv = new DataGridView();
this.Controls.Add(dv);
dv.DataSource = dt;
}
}
EDIT:
As per your code you should try the below lines of code:
OleDbCommand _oleCmdSelect = new OleDbCommand(#"SELECT * FROM [" + excelSheets[0] + "]" + " Where [CustomerID#] = #custID" , objConn);
_oleCmdSelect.Parameters.Add("#custID", customerID);
OleDbDataAdapter oleAdapter = new OleDbDataAdapter();
oleAdapter.SelectCommand = _oleCmdSelect;

Unidata UniObjects for .NET - Write amendments back to unidata from modified table

I'm trying to write data back into a file on Unidata, after the contents have been adjusted in a datagridview.
I've tried various option based around the code below, but with no luck.
Within the foreach section I want to update my file.
The file consists of 10 single value attributes.
I tried fl.write(),but get an error relating to writing to a null value...
try
{
DataTable modifiedTable = m_DS.Tables[0].GetChanges(DataRowState.Modified);
if (modifiedTable.Rows.Count > 0)
{
U2Connection con = GetConnection();
Console.WriteLine("Connected.................");
UniSession lUniSession = con.UniSession;
UniFile fl = lUniSession.CreateUniFile("myTableName");
UniDynArray udr3 = new UniDynArray(lUniSession);
foreach (DataRow item in modifiedTable.Rows)
{
}
con.Close();
}
}
Thank you for using UniObjects’s API of U2 Toolkit for .NET (formerly known as standalone UO.NET).
Yesterday (June 10th, 2014) , we have Released U2 Toolkit for .NET v2.1.0. Main features of U2 Toolkit for .NET v2.1.0 is
Native Visual Studio Integration
For other features, see this link
http://blog.rocketsoftware.com/2014/05/access-nosql-data-using-sql-syntax-u2-toolkit-net-v2-1-0-beta/
Can you please try the same code ( 10 single value attributes) using SELECT and UPDATE. For your information, SELECT and UPDATE behind the scene calls UniFile Read and Write. These Samples are part of the installation. Go to these directories.
C:\Program Files (x86)\Rocket Software\U2 Toolkit for .NET\U2 Database Provider\samples\C#\UniData\NativeAccess\Select_SQL_Syntax
C:\Program Files (x86)\Rocket Software\U2 Toolkit for .NET\U2 Database Provider\samples\C#\UniData\NativeAccess\Update_SQL_Syntax
SELECT
private static void Select()
{
try
{
Console.WriteLine(Environment.NewLine + "Start...");
ConnectionStringSettingsCollection settings = ConfigurationManager.ConnectionStrings;
ConnectionStringSettings cs = settings["u2_connection"];
U2Connection lConn = new U2Connection();
lConn.ConnectionString = cs.ConnectionString;
lConn.Open();
Console.WriteLine("Connected...");
U2Command cmd = lConn.CreateCommand();
//ID,FNAME,LNAME : Single Value
//SEMESTER: Multi Value
//COURSE_NBR,COURSE_GRD: MS
cmd.CommandText = string.Format("SELECT ID,FNAME,LNAME,SEMESTER,COURSE_NBR,COURSE_GRD FROM STUDENT WHERE ID > 0 ORDER BY ID");
U2DataAdapter da = new U2DataAdapter(cmd);
DataSet ds = new DataSet();
da.Fill(ds);
Console.WriteLine(Environment.NewLine);
ds.WriteXml(Console.Out);
lConn.Close();
Console.WriteLine(Environment.NewLine + "End...");
Console.WriteLine(SUCCESS_MSG);
}
catch (Exception e2)
{
string lErr = e2.Message;
if (e2.InnerException != null)
{
lErr += lErr + e2.InnerException.Message;
}
Console.WriteLine(Environment.NewLine + lErr);
Console.WriteLine(FAIL_MSG);
}
}
Update
private static void Update_Using_DataSet()
{
try
{
Console.WriteLine(Environment.NewLine + "Start...");
ConnectionStringSettingsCollection settings = ConfigurationManager.ConnectionStrings;
ConnectionStringSettings cs = settings["u2_connection"];
U2Connection lConn = new U2Connection();
lConn.ConnectionString = cs.ConnectionString;
lConn.Open();
Console.WriteLine("Connected...");
U2Command cmd = lConn.CreateCommand();
//ID,FNAME,LNAME : Single Value
//SEMESTER: Multi Value
//COURSE_NBR,COURSE_GRD: MS
cmd.CommandText = string.Format("SELECT ID,FNAME,LNAME,SEMESTER,COURSE_NBR,COURSE_GRD FROM STUDENT WHERE ID={0} ORDER BY ID",ID);
U2DataAdapter da = new U2DataAdapter(cmd);
U2CommandBuilder builder = new U2CommandBuilder(da);
da.UpdateCommand = builder.GetUpdateCommand();
DataSet ds = new DataSet();
da.Fill(ds);
DataTable dt = ds.Tables[0];
DataRowCollection lDataRowCollection = dt.Rows;
int i = 1;
foreach (DataRow item in lDataRowCollection)
{
item["FNAME"] = item["FNAME"] + "3";// modify single value
item["SEMESTER"] = item["SEMESTER"] + "$";//modify multi-value
item["COURSE_GRD"] = item["COURSE_GRD"] + "$";
i++;
}
da.Update(ds);//use DataAdapter's Update() API
//print modified value
cmd.CommandText = string.Format("SELECT ID,FNAME,LNAME,SEMESTER,COURSE_NBR,COURSE_GRD FROM STUDENT WHERE ID={0} ORDER BY ID", ID); ;
//verify the change
U2DataAdapter da2 = new U2DataAdapter(cmd);
DataSet ds2 = new DataSet();
da2.Fill(ds2);
Console.WriteLine(Environment.NewLine);
ds2.WriteXml(Console.Out);
//close connection
lConn.Close();
Console.WriteLine(Environment.NewLine + "End...");
Console.WriteLine(SUCCESS_MSG);
}
catch (Exception e2)
{
Console.WriteLine(FAIL_MSG);
string lErr = e2.Message;
if (e2.InnerException != null)
{
lErr += lErr + e2.InnerException.Message;
}
Console.WriteLine(Environment.NewLine + lErr);
}
}
You will need to modify the UniDynArray (the record) for each row value in the table and then write the UniDynArray to the file and specific record id:
for (Int32 i=0; i < modifiedTable.Rows.Count; i++)
{
DataRow item = modifiedTable.Rows[i];
//Modify each attribute in the record from the rows in the table
udr3.Replace(i+1, (String)item[0]);
}
//Write the modified record to the file
fl.Write("MyRecordId", udr3);
The reason you got the null reference exception is because you didn't assign a value to fl.RecordId or fl.Record before calling fl.Write(). As you can see above, I prefer to use the overload of the Write method that takes record id and record data as parameters instead of setting the properties on the instance of UniFile.

Dealing with huge amount of data when inserting into sql database

in my code the user can upload an excel document wish contains it's phone contact list.Me as a developer should read that excel file turn it into a dataTable and insert it into the database .
The Problem is that some clients have a huge amount of contacts like saying 5000 and more contacts and when i am trying to insert this amount of data into the database it's crashing and giving me a timeout exception.
What would be the best way to avoid this kind of exception and is their any code that can reduce the time of the insert statement so the user don't wait too long ?
the code
public SqlConnection connection = new SqlConnection(System.Configuration.ConfigurationManager.ConnectionStrings["ConnectionString"].ConnectionString);
public void Insert(string InsertQuery)
{
SqlDataAdapter adp = new SqlDataAdapter();
adp.InsertCommand = new SqlCommand(InsertQuery, connection);
if (connection.State == System.Data.ConnectionState.Closed)
{
connection.Open();
}
adp.InsertCommand.ExecuteNonQuery();
connection.Close();
}
protected void submit_Click(object sender, EventArgs e)
{
string UploadFolder = "Savedfiles/";
if (Upload.HasFile) {
string fileName = Upload.PostedFile.FileName;
string path=Server.MapPath(UploadFolder+fileName);
Upload.SaveAs(path);
Msg.Text = "successfully uploaded";
DataTable ValuesDt = new DataTable();
ValuesDt = ConvertExcelFileToDataTable(path);
Session["valuesdt"] = ValuesDt;
Excel_grd.DataSource = ValuesDt;
Excel_grd.DataBind();
}
}
protected void SendToServer_Click(object sender, EventArgs e)
{
DataTable Values = Session["valuesdt"] as DataTable ;
if(Values.Rows.Count>0)
{
DataTable dv = Values.DefaultView.ToTable(true, "Mobile1", "Mobile2", "Tel", "Category");
double Mobile1,Mobile2,Tel;string Category="";
for (int i = 0; i < Values.Rows.Count; i++)
{
Mobile1 =Values.Rows[i]["Mobile1"].ToString()==""?0: double.Parse(Values.Rows[i]["Mobile1"].ToString());
Mobile2 = Values.Rows[i]["Mobile2"].ToString() == "" ? 0 : double.Parse(Values.Rows[i]["Mobile2"].ToString());
Tel = Values.Rows[i]["Tel"].ToString() == "" ? 0 : double.Parse(Values.Rows[i]["Tel"].ToString());
Category = Values.Rows[i]["Category"].ToString();
Insert("INSERT INTO client(Mobile1,Mobile2,Tel,Category) VALUES(" + Mobile1 + "," + Mobile2 + "," + Tel + ",'" + Category + "')");
Msg.Text = "Submitied successfully to the server ";
}
}
}
You can try SqlBulkCopy to insert Datatable to Database Table
Something like this,
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(sqlConnection, SqlBulkCopyOptions.KeepIdentity))
{
bulkCopy.DestinationTableName = DestTableName;
string[] DtColumnName = YourDataTableColumns;
foreach (string dbcol in DbColumnName)//To map Column of Datatable to that of DataBase tabele
{
foreach (string dtcol in DtColumnName)
{
if (dbcol.ToLower() == dtcol.ToLower())
{
SqlBulkCopyColumnMapping mapID = new SqlBulkCopyColumnMapping(dtcol, dbcol);
bulkCopy.ColumnMappings.Add(mapID);
break;
}
}
}
bulkCopy.WriteToServer(YourDataTableName.CreateDataReader());
bulkCopy.Close();
}
For more Read http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlbulkcopy.aspx
You are inserting 1 row at a time, which is very expensive for this amount of data
In those cases you should use bulk insert, so the round trip to DB will be only once, if you need to roll back - all is the same transaction
You can use SqlBulkCopy which is more work, or you can use the batch update feature of the SqlAdpater. Instead of creating your own insert statement, then building a sqladapter, and then manually executing it, create a dataset, fill it, create one sqldataadpater, set the number of inserts in a batch, then execute the adapter once.
I could repeat the code, but this article shows exactly how to do it: http://msdn.microsoft.com/en-us/library/kbbwt18a%28v=vs.80%29.aspx
protected void SendToServer_Click(object sender, EventArgs e)
{
DataTable Values = Session["valuesdt"] as DataTable ;
if(Values.Rows.Count>0)
{
DataTable dv = Values.DefaultView.ToTable(true, "Mobile1", "Mobile2", "Tel", "Category");
//Fix up default values
for (int i = 0; i < Values.Rows.Count; i++)
{
Values.Rows[i]["Mobile1"] =Values.Rows[i]["Mobile1"].ToString()==""?0: double.Parse(Values.Rows[i]["Mobile1"].ToString());
Values.Rows[i]["Mobile2"] = Values.Rows[i]["Mobile2"].ToString() == "" ? 0 : double.Parse(Values.Rows[i]["Mobile2"].ToString());
Values.Rows[i]["Tel"] = Values.Rows[i]["Tel"].ToString() == "" ? 0 : double.Parse(Values.Rows[i]["Tel"].ToString());
Values.Rows[i]["Category"] = Values.Rows[i]["Category"].ToString();
}
BatchUpdate(dv,1000);
}
}
public static void BatchUpdate(DataTable dataTable,Int32 batchSize)
{
// Assumes GetConnectionString() returns a valid connection string.
string connectionString = GetConnectionString();
// Connect to the database.
using (SqlConnection connection = new SqlConnection(connectionString))
{
// Create a SqlDataAdapter.
SqlDataAdapter adapter = new SqlDataAdapter();
// Set the INSERT command and parameter.
adapter.InsertCommand = new SqlCommand(
"INSERT INTO client(Mobile1,Mobile2,Tel,Category) VALUES(#Mobile1,#Mobile2,#Tel,#Category);", connection);
adapter.InsertCommand.Parameters.Add("#Mobile1",
SqlDbType.Float);
adapter.InsertCommand.Parameters.Add("#Mobile2",
SqlDbType.Float);
adapter.InsertCommand.Parameters.Add("#Tel",
SqlDbType.Float);
adapter.InsertCommand.Parameters.Add("#Category",
SqlDbType.NVarchar, 50);
adapter.InsertCommand.UpdatedRowSource = UpdateRowSource.None;
// Set the batch size.
adapter.UpdateBatchSize = batchSize;
// Execute the update.
adapter.Update(dataTable);
}
}
I know this is a super old post, but you should not need to use the bulk operations explained in the existing answers for 5000 inserts. Your performance is suffering so much because you close and reopen the connection for each row insert. Here is some code I have used in the past that keeps one connection open and executes as many commands as needed to push all the data to the DB:
public static class DataWorker
{
public static Func<IEnumerable<T>, Task> GetStoredProcedureWorker<T>(Func<SqlConnection> connectionSource, string storedProcedureName, Func<T, IEnumerable<(string paramName, object paramValue)>> parameterizer)
{
if (connectionSource is null) throw new ArgumentNullException(nameof(connectionSource));
SqlConnection openConnection()
{
var conn = connectionSource() ?? throw new ArgumentNullException(nameof(connectionSource), $"Connection from {nameof(connectionSource)} cannot be null");
var connState = conn.State;
if (connState != ConnectionState.Open)
{
conn.Open();
}
return conn;
}
async Task DoStoredProcedureWork(IEnumerable<T> workData)
{
using (var connection = openConnection())
using (var command = connection.CreateCommand())
{
command.CommandType = CommandType.StoredProcedure;
command.CommandText = storedProcedureName;
command.Prepare();
foreach (var thing in workData)
{
command.Parameters.Clear();
foreach (var (paramName, paramValue) in parameterizer(thing))
{
command.Parameters.AddWithValue(paramName, paramValue ?? DBNull.Value);
}
await command.ExecuteNonQueryAsync().ConfigureAwait(false);
}
}
}
return DoStoredProcedureWork;
}
}
This was actually from a project where I was gathering emails for a restriction list, so kind of relevant example of what a parameterizer argument might look like and how to use the above code:
IEnumerable<(string,object)> RestrictionToParameter(EmailRestriction emailRestriction)
{
yield return ("#emailAddress", emailRestriction.Email);
yield return ("#reason", emailRestriction.Reason);
yield return ("#restrictionType", emailRestriction.RestrictionType);
yield return ("#dateTime", emailRestriction.Date);
}
var worker = DataWorker.GetStoredProcedureWorker<EmailRestriction>(ConnectionFactory, #"[emaildata].[AddRestrictedEmail]", RestrictionToParameter);
await worker(emailRestrictions).ConfigureAwait(false);

Categories