My SQL code inserts 10,000 records into a table from list. If record already exists, it updates a few fields.
Currently it taking more than 10 minutes and timing out unless I restrict the number of records to process. Is there anything in my code which I can do to solve this problem.
foreach(RMSResponse rmsObj in rmsList) {
try {
string connectionString = #"server=localhost\sqlexpress;" + "Trusted_Connection=yes;" + "database=MyDB; " + "connection timeout=30";
SqlConnection conn = new SqlConnection(connectionString);
conn.Open();
string str = "";
str += "SELECT * ";
str += "FROM RMSResponse ";
str += "WHERE LeadID = #RecruitID";
int LeadID;
Boolean IsPositive = true;
SqlCommand selectCommand = new SqlCommand(str, conn);
selectCommand.CommandType = CommandType.Text;
selectCommand.Parameters.Add(new SqlParameter("#RecruitID", rmsObj.RecruitID));
SqlDataReader sqlDataReader = selectCommand.ExecuteReader();
bool hasRows = sqlDataReader.HasRows;
while (sqlDataReader.Read()) {
LeadID = sqlDataReader.GetInt32(1);
IsPositive = sqlDataReader.GetBoolean(2);
IsPositive = (IsPositive == true) ? false : true;
Console.WriteLine("Lead ID: " + LeadID + " IsPositive: " + IsPositive);
}
sqlDataReader.Close();
if (hasRows) {
SqlCommand updateCommand = new SqlCommand("UPDATE RMSResponse set IsPositive=#IsPositive, OptOutDate=#OptOutDate where LeadID=#LeadID", conn);
updateCommand.Parameters.AddWithValue("#LeadID", rmsObj.RecruitID);
updateCommand.Parameters.AddWithValue("#IsPositive", IsPositive);
updateCommand.Parameters.AddWithValue("#OptOutDate", DateTime.Now);
updateCommand.ExecuteNonQuery();
sqlDataReader.Close();
}
if (!hasRows) {
SqlCommand insertCommand = new SqlCommand("INSERT INTO RMSResponse (LeadID, IsPositive, ReceivedDate) " + "VALUES(#LeadID, #IsPositive, #ReceivedDate)", conn);
insertCommand.Parameters.AddWithValue("#LeadID", rmsObj.RecruitID);
insertCommand.Parameters.AddWithValue("#IsPositive", true);
insertCommand.Parameters.AddWithValue("#ReceivedDate", DateTime.Now);
int rows = insertCommand.ExecuteNonQuery();
}
} catch (SqlException ex) {
Console.WriteLine(ex.Message);
}
}
You can move the update to SQL - all you are doing is setting the OptOutDate to be today. You can pass in the list of lead IDs to a batch update statement.
To insert records, you could bulk insert into a staging table, then execute SQL to insert the data for IDs that aren't already in the table.
There isn't much logic in your C#, so pulling the data out then putting it back in is making it unnecessarily slow.
If you don't want to go down this root, then other tips include:
Open one connection outside of the loop
Create one SqlCommand object outside of the loop and reuse it by resetting the paramenters
Change your select SQL to only select the columns you need, not *
A simple update or insert statement should never take 10 mins.
SqlServer is a very good database.
Create an index on LeadID on your table RMSResponse.
If your foreach is looping over many records, definitely consider a stored procedure, stored procedures will reduces a lot of the time taken.
If you want to Update or Insert(UPSERT) look at the Merge command added in SqlServer 2008.
Related
I have code that adds a value inside a SQL table :
SqlCommand command;
SqlDataAdapter adapter = new SqlDataAdapter();
String strSQL = "";
strSQL = "INSERT INTO tblTest (value1) VALUES ('" + strPLCData + "')";
command = new SqlCommand(strSQL, cnn);
adapter.InsertCommand = new SqlCommand(strSQL, cnn);
try
{
int rows = adapter.InsertCommand.ExecuteNonQuery();
txtStatusLogging.Text += "Inserted " + rows + " row(s) in the database." + Environment.NewLine;
}
catch (Exception ex)
{
MessageBox.Show("Failed to write to database : " + ex.Message);
}
command.Dispose();
But I'm a bit stuck when I want to add an unknown count of values in the database (according to a list of unknown size).
e.g. sometimes add only value1, other times add value1, value2 and value3 .... (depending on whats in a certain list).
How would I go about doing this?
There's absolutely no need for that SqlDataAdapter. If you want to add an arbitrary number of values to a table - use a straight INSERT SqlCommand and just loop over the list of values to insert.
Also: you should always use parametrized queries - no exceptions - and you should put your SqlConnection and SqlCommand objects in using () { ... } blocks - something like this:
Something like this:
public void InsertValues(List<int> values)
{
// define the insert query
string qryInsert = "INSERT INTO dbo.tblTest (value1) VALUES (#singleValue);";
using (SqlConnection conn = new SqlConnection(connectionString))
using (SqlCommand cmdInsert = new SqlCommand(qryInsert, conn))
{
// define parameter
cmdInsert.Parameter.Add("#singleValue", SqlDbType.Int);
conn.Open();
// loop over values
foreach (int aValue in values)
{
// set the parameter value, execute query
cmdInsert.Parameters["#singleValue"].Value = aValue;
cmdInsert.ExecuteNonQuery();
}
conn.Close;
}
}
I'm trying to retrieve multiple cells in different rows where the correct owner exists, but I'm only being able to retrieve the first match and it stops there, I've tried using it with a for, but I don't think .ExecuteScalar() is the way to do this. Maybe I'm just stupid and doing it completely wrong.
Code:
checkPlayerName = API.getPlayerName(player);
string checkOwnedCars = "SELECT COUNT(*) FROM [carOwners] WHERE Owner='" + checkPlayerName + "'";
con.Open();
SqlCommand checkCarsCount = new SqlCommand(checkOwnedCars, con);
int carsCountToVar = Convert.ToInt32(checkCarsCount.ExecuteScalar());
con.Close();
for (int i = 0; i < carsCountToVar; i++)
{
string displayCars = "SELECT LP FROM [carOwners] WHERE Owner='" + checkPlayerName + "'";
con.Open();
SqlCommand displayCarsCMD = new SqlCommand(displayCars, con);
string displayCarsToVar = displayCarsCMD.ExecuteReader().ToString();
API.sendChatMessageToPlayer(player, "Owned Vehicle: " + displayCarsToVar.ToString());
con.Close();
}
Table
For example, LP on 2nd and 3rd row are the ones that I want to store since both belong to the same owner, yet only first cell data (1337) is displaying.
You are not iterating the results you are getting from query.
Plus always use Parameterized queries to prevent SQL Injection Attacks
SqlCommand command = new SqlCommand("SELECT LP FROM [carOwners] WHERE Owner=#checkPlayerName", con);
command.Parameters.AddWithValue("#checkPlayerName",checkPlayerName);
using (SqlDataReader reader = command.ExecuteReader())
{
while (reader.Read())
{
Console.WriteLine(String.Format("{0}",reader["id"]));
//API.sendChatMessageToPlayer(player, "Owned Vehicle: " + reader["id"].ToString());
}
}
conn.Close();
I am not getting, how to do insert and update of the data in C# WinForms on single button click.
private void save_Click(object sender, EventArgs e)
{
SqlConnection cn = new SqlConnection();
cn.ConnectionString = "data source=Sai;database=kaur; user id=sa;password=azxc;";
cn.Open();
string gen;
if (radioButton1.Checked == true)
gen = "Male";
else
gen = "Female";
string clas = null;
clas = comboBox1.Text;
string section = null;
section = comboBox2.Text;
SqlCommand cmd = new SqlCommand();
cmd.CommandText = "insert into studetail values('" + textBox1.Text + "','" + textBox2.Text + "','" + gen + "','" + textBox3.Text + "','" + clas + "','" + section + "')";
cmd.Connection = cn;
int n = cmd.ExecuteNonQuery();
if (n > 0)
MessageBox.Show(n + " Row Inserted.");
else
MessageBox.Show("Insertion failed.");
SqlDataAdapter da = new SqlDataAdapter("select * from studetail ", cn);
DataTable dt = new DataTable();
da.Fill(dt);
dataGridView1.DataSource = dt;
You can add a deletion before the insertion:
private void save_Click(object sender, EventArgs e)
{
DeletePerson(id); // add this
SqlConnection cn = new SqlConnection();
...
}
public void DeletePerson(int id)
{
using(SqlConnection connection = new SqlConnection(credentials))
{
connection.Open();
SqlCommand cmd = new SqlCommand();
cmd.Connection = connection;
cmd.CommandText = "delete from studetail where someUniqeIdColumn = " + id;
cmd.ExecuteNonQuery();
}
}
Using responsible to dispose the connection.
Consider using Entity Framework or LINQ to SQL.
You are exposed to SQL injection.
First off the SQL query isn't quite right. It should look something like the following:
INSERT INTO studetail (columnName1, columnName2, ...columnNameN)
VALUES (value1, value2, ...valueN);
Where the column names are the columns where you want data to be inserted, and the values are the data you want inserted into said columns.
You should also be disposing the connection by wrapping the connection within a using statement.
using(var con = new SqlConnection(connectionString))
{
con.Open();
//rest of code that needs a connection here
}
Additionally, you need to be wary of SQL injection. I highly suggest reading this example from the MSDN website. It will give you an example of using an SQL Update and avoiding SQL injection with use of SqlCommand.Paramaters property.
You should also have a Primary Key in your database tables, if you don't already, so you can uniquely identify each record in a table.
To do an update and a save on the same button, you will need to check if a row already exists for the data that is being edited. This when a Primary comes in handy. You will want to check your database to see if a record already exists
SELECT 1 FROM studetail WHERE <Condition>
The WHERE condition will be the way you uniquely identify (a Primary Key) a row in your table. If the rows in the table are uniquely identified, the above SQL statement will return 1 if a value exists, which means you can UPDATE or 0 if no record exists, so you can INSERT
This is our code to prevent the same data from being added into SQL from our C# program but only the first same data will not be added in. The remaining ones adds the same data into SQL despite our prevention in our C# program. Can somebody help us troubleshoot?
in order not to duplicate data in database usually you set some constraints to your database. By having a unique field in database you can prevent multiple addition to your db.
Currently you are also fetching data from db to check if it exist already and that creates extra cost, just manipulate the design of db so that it won't accept the same column input twice
Count the value of data that is inserted
string constr = ConfigurationManager.ConnectionStrings["connstr"].ToString();
SqlConnection con = new SqlConnection(constr);
string sql1 = "SELECT COUNT (client_id) FROM client WHERE client_id = '" + txtid.Text + "' ";
SqlCommand cmd = new SqlCommand(sql1, con);
con.Open();
int temp = Convert.ToInt32(cmd.ExecuteScalar().ToString());
if (temp >0)
{
//show error message
}
You could check for the record you want to add, and if it doesn't exists, then add it to the table:
SqlConnection _cnt = new SqlConnection();
_cnt.ConnectionString = "Your Connection String";
SqlCommand _cmd = new SqlCommand();
_cmd.Connection = _cnt;
_cmd.CommandType = System.Data.CommandType.Text;
_cmd.CommandText = "SELECT id FROM myTable where Category=#Name";
_cmd.Parameters.Add("#Name", string);
_cmd.Parameters["#Name"].Value = newCatTitle;
_cnt.Open();
var idTemp = _cmd.ExecuteScalar();
_cmd.Dispose();
_cnt.Close();
_cnt.Dispose();
if (idTemp == null)
{
//Insert into table
}
else
{
//Message it already exists
}
I'm using MS Sql database in my application and when I do SELECT operation from database it stucks for a few minutes.
And it happens a few times per day. All other SELECTS take several seconds only.
Also I noticed if I close app while SELECT operation is in progress. It will not work at all on any next app starts UNTIL I restart database engine...
Why could it be?
Here is the code snippet:
using (SqlConnection myConnection = new SqlConnection(connString))
{
myConnection.Open();
SqlCommand command = myConnection.CreateCommand();
SqlTransaction transaction;
transaction = myConnection.BeginTransaction("LockTransaction");
command.Connection = myConnection;
command.Transaction = transaction;
try
{
int recordsAtOnce = maxToLock;
command.CommandText =
"SELECT TOP 1000 id, productName from Parts WHERE part_used is NULL;";
List<string> idList = new List<string>();
SqlDataReader myReader = command.ExecuteReader(CommandBehavior.Default);
while (myReader.Read())
{
string id = myReader.GetString(1);
string name = myReader.GetInt32(0).ToString();
idList.Add(id);
}
myReader.Close();
string idsStr = "";
for(int i = 0; i < idList.Count; i++)
{
if (i != 0)
{
idsStr += ", ";
}
idsStr += idList[i];
}
// lock record
command.CommandText = "UPDATE Parts SET part_used=\'rt\' WHERE id in (" + idsStr + ")";
command.Parameters.Clear();
command.CommandType = CommandType.Text;
command.ExecuteNonQuery();
transaction.Commit();
}
catch (Exception ex)
{
transaction.Rollback();
}
I think your reader values are being assigned to the wrong variables.
Try changing:
string id = myReader.GetString(1);
string name = myReader.GetInt32(0).ToString();
idList.Add(id);
To:
string id = myReader.GetInt32(0);
string name = myReader.GetString(1).ToString();
idList.Add(id);
This is most probably because of a lock. If another transaction is writing into the Parts table, your select needs to wait.
How big is the Parts table? That could cause performance problems. Or it could be another transaction locking before you.
Other things:
Call dispose on your connection, command and reader when done with them via using
Parameterize the second query rather than string concating. It's more secure and significantly more performant because of how Sql Server works internally.
minor: If you're looking at thousands of items, use StringBuilder instead of concatenating.