We Keep Coding

Concatenate SqlDataReader to string in C#

reader.Read() is sometimes false when looping and not all table rows are read. During this skipped state reader.Read() never becomes true. Waiting for it before iterating locks it up, of course. Is there a SQL connection status that will help mutex the loop? This happens 1 - 10% of the time during the loop.
** Answered: I was referencing the wrong sql column in my query. Adding reader.NextResult(); helped as well.
using System;
using System.Diagnostics;
using Microsoft.Data.SqlClient;
public string StringTableRows(string conn_str, int row_count)
{
string s = "";
Query query = new Query();
for (int i = 1; i < row_count;)
{
// **Fix: I was referencing the wrong column.
string dyn_query = query.RowQuery(ids[i]);
try
{
SqlConnection conn = new SqlConnection(conn_str);
using (conn)
{
SqlCommand cmd = new SqlCommand(dyn_query, conn);
using (cmd)
{
cmd.Connection.Open();
SqlDataReader reader = comm.ExecuteReader();
using (reader)
{
Debug.WriteLine("Outside: " + i);
while (reader.Read())
{
for (int j = 0; j < reader.FieldCount; j++)
{
s = s + reader.GetString(0);
Debug.WriteLine("Inside: " + i);
}
}
// **Fix: This is the fix per answer.
reader.NextResult();
}
}
}
}
catch (Exception ex)
{
Debug.WriteLine("Failed query: " + dyn_query);
Debug.WriteLine("Execute Exception: " + ex);
}
i++;
}
s = s.Insert(0,"<Tables>");
s = s + "</Tables>";
return s;
}

reader. Read() is sometimes false when looping and not all table rows are read
That will only happen if the additional rows are in a separate resultset because the CommandText has multiple SELECT statements or calls a stored procedure with multiple SELECT statements. In that case you have to call SqlDataReader.NextResult() before you call Read() again.

C# Mysql Insert Many Rows, perfomance problem

I wonder how can i bulk insert instead of execute this method everytime.
It's getting slow when i try to insert 1000 rows:
queryText = "INSERT INTO `player_items` (`player_id`, `item_id`, `count`) VALUES (#player_id, #item_id, #count)";
for (int i = 0; i < player.invenotrySize; i++)
{
Item item = player.inventory.GetItem[i];
MySqlParameter[] parameters = {
new MySqlParameter("player_id", 1),
new MySqlParameter("item_id", item.data.id),
new MySqlParameter("count", item.amount),
};
ExecuteNonQuery(queryText, parameters);
}
public int ExecuteNonQuery(string queryText, params MySqlParameter[] parameters)
{
int affectedRows = 0;
using (MySqlConnection mySqlConnection = CreateConnection())
{
using (MySqlCommand mySqlCommand = new MySqlCommand(queryText, mySqlConnection))
{
mySqlCommand.CommandType = CommandType.Text;
mySqlCommand.Parameters.AddRange(parameters);
affectedRows = mySqlCommand.ExecuteNonQuery();
}
}
return affectedRows;
}
I think the optimal way is to insert everything as a huge row. E.g
INSERT INTO tbl_name (a,b,c) VALUES(1,2,3),(4,5,6),(7,8,9);
But i have no idea how i can make a method to take care of this setup

You are opening and closing your connection for every single insert.
using (MySqlConnection mySqlConnection = CreateConnection())
This is a very expensive procedure, and therefore not really the way to work with a DB.
You should open your connection just once, and then close it when finished. Depending on what you app does this might be when you start your App (or before you do your first DB query) and then close it when exiting the App (or after you are certain there will be no more DB queries.
Then ideally you should also reuse the SqlCommand instance as well. But you need to make sure that you clear your parameters in between. So then you have something like this
int affectedRows = 0;
using (MySqlConnection mySqlConnection = CreateConnection())
{
string queryText = "INSERT INTO `player_items` (`player_id`, `item_id`, `count`) VALUES (#player_id, #item_id, #count)";
using (MySqlCommand mySqlCommand = new MySqlCommand(queryText, mySqlConnection))
{
mySqlCommand.CommandType = CommandType.Text;
for (int i = 0; i < player.invenotrySize; i++)
{
Item item = player.inventory.GetItem[i];
MySqlParameter[] parameters = {
new MySqlParameter("player_id", 1),
new MySqlParameter("item_id", item.data.id),
new MySqlParameter("count", item.amount)};
mySqlCommand.Parameters.Clear();
mySqlCommand.Parameters.AddRange(parameters);
affectedRows += mySqlCommand.ExecuteNonQuery();
}
}
}

It's not realy clean with your "ExecuteNonQuery" (do a multi row insert solution or just isolate/singleton the connection class like the solution above, will be better) but you can construct your whole query before execute instead of get/add connection, replace, execute foreach player.
queryText = "INSERT INTO `player_items` (`player_id`, `item_id`, `count`) VALUES";
for (int i = 0; i < player.invenotrySize; i++)
{
Item item = player.inventory.GetItem[i];
MySqlParameter[] parameters = {
new MySqlParameter("player_id_"+i, 1),
new MySqlParameter("item_id_"+i, item.data.id),
new MySqlParameter("count_"+i, item.amount),
};
queryText+= " (#player_id_"+i+", #item_id_"+i+", #count_"+i+"),";
}
//remove the last ,
queryText= queryText.Remove(queryText.Length - 1)+";";
ExecuteNonQuery(queryText, parameters);
Altnernate for to skip params if you are sure about your data.
Item item = player.inventory.GetItem[i];
queryText+= " (1, "+item.data.id+", "+item.amount+"),";

Efficient Way to Update a lot of Rows from C#

I have a program where I open a SqlConnection, load up a list of objects, modify a value on each object, then update the rows in the SQL Server database. Because the modification requires string parsing I wasn't able to do with with purely T-SQL.
Right now I am looping through the list of objects, and running a SQL update in each iteration. This seems inefficient and I'm wondering if there is a more efficient way to do it using LINQ
The list is called UsageRecords. The value I'm updating is MthlyConsumption.
Here is my code:
foreach (var item in UsageRecords)
{
string UpdateQuery = #"UPDATE tbl810CTImport
SET MthlyConsumption = " + item.MthlyConsumption +
"WHERE ID = " + item.Id;
SqlCommand update = new SqlCommand(UpdateQuery, sourceConnection);
update.ExecuteNonQuery();
}

Try this instead:
string UpdateQuery = #"UPDATE tbl810CTImport SET MthlyConsumption = #consumption WHERE ID = #itemId";
var update = new SqlCommand(UpdateQuery, sourceConnection);
update.Parameters.Add("#consumption", SqlDbType.Int); // Specify the correct types here
update.Parameters.Add("#itemId", SqlDbType.Int); // Specify the correct types here
foreach (var item in UsageRecords)
{
update.Parameters[0].Value = item.MthlyConsumption;
update.Parameters[1].Value = item.Id;
update.ExecuteNonQuery();
}
It should be faster because:
You don't have to create the command each time.
You don't create a new string each time (concatenation)
The query is not parsed at every iteration (Just changes the parameters values).
And it will cache the execution plan. (Thanks to #JohnCarpenter from the comment)

You can either use
SqlDataAdapter - See How to perform batch update in Sql through C# code
or what I have previously done was one of the following:
Tear down the ID's in question, and re-bulkinsert
or
Bulk Insert the ID + new value into a staging table, and update the table on SQL server:
update u
set u.MthlyConsumption = s.MthlyConsumption
from tbl810CTImport u
inner join staging s on
u.id = s.id

In a situation like this, where you can't write a single update statement to cover all your bases, it's a good idea to batch up your statements and run more than one at a time.
var commandSB = new StringBuilder();
int batchCount = 0;
using (var updateCommand = sourceConnection.CreateCommand())
{
foreach (var item in UsageRecords)
{
commandSB.AppendFormat(#"
UPDATE tbl810CTImport
SET MthlyConsumption = #MthlyConsumption{0}
WHERE ID = #ID{0}",
batchCount
);
updateCommand.Parameters.AddWithValue(
"#MthlyConsumption" + batchCount,
item.MthlyConsumption
);
updateCommand.Parameters.AddWithValue(
"#ID" + batchCount,
item.MthlyConsumption
);
if (batchCount == 500) {
updateCommand.CommandText = commandSB.ToString();
updateCommand.ExecuteNonQuery();
commandSB.Clear();
updateCommand.Parameters.Clear();
batchCount = 0;
}
else {
batchCount++;
}
}
if (batchCount != 0) {
updateCommand.ExecuteNonQuery();
}
}

It should be as simple as this . . .
private void button1_Click(object sender, EventArgs e)
{
SqlConnection con = new SqlConnection("Server=YourServerName;Database=YourDataBaseName;Trusted_Connection=True");
try
{
//cmd new SqlCommand( "UPDATE Stocks
//SET Name = #Name, City = #cit Where FirstName = #fn and LastName = #add";
cmd = new SqlCommand("Update Stocks set Ask=#Ask, Bid=#Bid, PreviousClose=#PreviousClose, CurrentOpen=#CurrentOpen Where Name=#Name", con);
cmd.Parameters.AddWithValue("#Name", textBox1.Text);
cmd.Parameters.AddWithValue("#Ask", textBox2.Text);
cmd.Parameters.AddWithValue("#Bid", textBox3.Text);
cmd.Parameters.AddWithValue("#PreviousClose", textBox4.Text);
cmd.Parameters.AddWithValue("#CurrentOpen", textBox5.Text);
con.Open();
int a = cmd.ExecuteNonQuery();
if (a > 0)
{
MessageBox.Show("Data Updated");
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
finally
{
con.Close();
}
}
Change the code to suit your needs.

TransactionScope appears not to be working

I'm trying to make sure that both my insert and delete below work completely or not at all. I have my connection object outside of my transaction scope which I believe is correct by not 100% sure.
I do know that this code is not working as I intent. After the first part (the insert runs) and then I abort by terminating on a break point, the rows are indeed inserted even though I never called scope.complete.
Please point out the flaw in my thinking and logic here.
sqlConnection.Open();
int numFound = 1;
int max = 99;
int iteration = 0;
while (iteration < max && numFound > 0)
{
iteration++;
var ids = new List<int>();
using (var sqlCommand0 = new SqlCommand(sql0, sqlConnection))
{
using (SqlDataReader reader1 = sqlCommand0.ExecuteReader())
{
while (reader1.Read())
{
ids.Add(reader1.GetInt32(0));
}
}
}
numFound = ids.Count;
if (numFound > 0)
{
using (var scope = new TransactionScope())
{
string whereClause = $"WHERE Id IN ({string.Join(",", ids)})";
string sql1 = string.Format(sqlTemplate1, whereClause);
using (var sqlCommand1 = new SqlCommand(sql1, sqlConnection))
{
sqlCommand1.ExecuteNonQuery();
}
// BREAK POINT HERE - ABORTED PROGRAM AND sql1 had been committed.
var sql2 = "DELETE FROM SendGridEventRaw " + whereClause;
using (var sqlCommand2 = new SqlCommand(sql2, sqlConnection))
{
sqlCommand2.ExecuteNonQuery();
}
scope.Complete();
total += numFound;
Console.WriteLine("deleted: " + whereClause);
}
}
}
}

I think it's because you open your connection before starting your transaction. You could try to fix your issue by first starting your transaction and then opening your connection.

Just from what I am seeing and from what I am assuming is what you intend to happen is this:
If your first query gets some records, then the next query executes, hence the statement:
if (numFound > 0)
If that is the case, and where you put your breakpoint is true, of course the insert statement will fire. Reason is:
using (var sqlCommand1 = new SqlCommand(sql1, sqlConnection))
{
sqlCommand1.ExecuteNonQuery();
}
is within that if statement. You're saying "if there are any rows, execute the insert query."
If you're trying to actually get the scope object to do the query, then you're going to have to have all of the query construction happening within the object and then having scope.complete() doing the execution.
For example:
//In TransactionScope class
public string Complete(var ids, int numFound, SqlConnection sqlConnection, string sqlTemplate1)
{
string whereClause = $"WHERE Id IN ({string.Join(",", ids)})";
string sql1 = string.Format(sqlTemplate1, whereClause);
using (var sqlCommand1 = new SqlCommand(sql1, sqlConnection))
{
sqlCommand1.ExecuteNonQuery();
}
var sql2 = "DELETE FROM SendGridEventRaw " + whereClause;
using (var sqlCommand2 = new SqlCommand(sql2, sqlConnection))
{
sqlCommand2.ExecuteNonQuery();
}
return whereClause;
}
//in your Main class
if (num > 0)
{
string whereClause = scope.Complete(ids, numFound, sqlConnection, sqlTemplate1);
Console.WriteLine("deleted" + whereClause"." );
}
I am of course just going off of the assumptions I stated above. If I am incorrect, please let me know.
Hope it helps.

Fastest way to update more than 50.000 rows in a mdb database c#

I searched on the net something but nothing really helped me. I want to update, with a list of article, a database, but the way that I've found is really slow.
This is my code:
List<Article> costs = GetIdCosts(); //here there are 70.000 articles
conn = new OleDbConnection(string.Format(MDB_CONNECTION_STRING, PATH, PSW));
conn.Open();
transaction = conn.BeginTransaction();
using (var cmd = conn.CreateCommand())
{
cmd.Transaction = transaction;
cmd.CommandText = "UPDATE TABLE_RO SET TABLE_RO.COST = ? WHERE TABLE_RO.ID = ?;";
for (int i = 0; i < costs.Count; i++)
{
double cost = costs[i].Cost;
int id = costs[i].Id;
cmd.Parameters.AddWithValue("data", cost);
cmd.Parameters.AddWithValue("id", id);
if (cmd.ExecuteNonQuery() != 1) throw new Exception();
}
}
transaction.Commit();
But this way take a lot of minutes something like 10 minutes or more. There are another way to speed up this updating ? Thanks.

Try modifying your code to this:
List<Article> costs = GetIdCosts(); //here there are 70.000 articles
// Setup and open the database connection
conn = new OleDbConnection(string.Format(MDB_CONNECTION_STRING, PATH, PSW));
conn.Open();
// Setup a command
OleDbCommand cmd = new OleDbCommand();
cmd.Connection = conn;
cmd.CommandText = "UPDATE TABLE_RO SET TABLE_RO.COST = ? WHERE TABLE_RO.ID = ?;";
// Setup the paramaters and prepare the command to be executed
cmd.Parameters.Add("?", OleDbType.Currency, 255);
cmd.Parameters.Add("?", OleDbType.Integer, 8); // Assuming you ID is never longer than 8 digits
cmd.Prepare();
OleDbTransaction transaction = conn.BeginTransaction();
cmd.Transaction = transaction;
// Start the loop
for (int i = 0; i < costs.Count; i++)
{
cmd.Parameters[0].Value = costs[i].Cost;
cmd.Parameters[1].Value = costs[i].Id;
try
{
cmd.ExecuteNonQuery();
}
catch (Exception ex)
{
// handle any exception here
}
}
transaction.Commit();
conn.Close();
The cmd.Prepare method will speed things up since it creates a compiled version of the command on the data source.

Small change option:
Using StringBuilder and string.Format construct one big command text.
var sb = new StringBuilder();
for(....){
sb.AppendLine(string.Format("UPDATE TABLE_RO SET TABLE_RO.COST = '{0}' WHERE TABLE_RO.ID = '{1}';",cost, id));
}
Even faster option:
As in first example construct a sql but this time make it look (in result) like:
-- declaring table variable
declare table #data (id int primary key, cost decimal(10,8))
-- insert union selected variables into the table
insert into #data
select 1121 as id, 10.23 as cost
union select 1122 as id, 58.43 as cost
union select ...
-- update TABLE_RO using update join syntax where inner join data
-- and copy value from column in #data to column in TABLE_RO
update dest
set dest.cost = source.cost
from TABLE_RO dest
inner join #data source on dest.id = source.id
This is the fastest you can get without using bulk inserts.

Performing mass-updates with Ado.net and OleDb is painfully slow. If possible, you could consider performing the update via DAO. Just add the reference to the DAO-Library (COM-Object) and use something like the following code (caution -> untested):
// Import Reference to "Microsoft DAO 3.6 Object Library" (COM)
string TargetDBPath = "insert Path to .mdb file here";
DAO.DBEngine dbEngine = new DAO.DBEngine();
DAO.Database daodb = dbEngine.OpenDatabase(TargetDBPath, false, false, "MS Access;pwd="+"insert your db password here (if you have any)");
DAO.Recordset rs = daodb.OpenRecordset("insert target Table name here", DAO.RecordsetTypeEnum.dbOpenDynaset);
if (rs.RecordCount > 0)
{
rs.MoveFirst();
while (!rs.EOF)
{
// Load id of row
int rowid = rs.Fields["Id"].Value;
// Iterate List to find entry with matching ID
for (int i = 0; i < costs.Count; i++)
{
double cost = costs[i].Cost;
int id = costs[i].Id;
if (rowid == id)
{
// Save changed values
rs.Edit();
rs.Fields["Id"].Value = cost;
rs.Update();
}
}
rs.MoveNext();
}
}
rs.Close();
Note the fact that we are doing a full table scan here. But, unless the total number of records in the table is many orders of magnitude bigger than the number of updated records, it should still outperform the Ado.net approach significantly...