We Keep Coding

Asynchronously updating multiple row updates 1/4 of rows instantly and then waits

I have a code to asynchronously update multiple rows in SQL Server's table. I tested it on updating 540 rows and 144 rows are updated in the table instanly, then it waits for about 5 minutes and then the rest is updated. At least this is how it looks when I check for updated rows with SELECT.. I'm wondering why is that.
The whole thing is triggered by button's click:
DialogResult res = MessageBox.Show($"Znaleziono {num} pasujących maszyn. Czy chcesz zaktualizować priorytet maszyny danymi z pliku?", "Potwierdź", MessageBoxButtons.YesNo, MessageBoxIcon.Question);
if(res == DialogResult.Yes)
{
await UpdatePriority();
MessageBox.Show("Updated!");
Here's UpdatePriority method that asynchronously call place.Edit() method for all places in the list of items:
public async Task<string> UpdatePriority()
{
List<Task<string>> UpdateTasks = new List<Task<string>>();
try
{
foreach (Place p in Items.Where(i => i.IsUpdated==true))
{
UpdateTasks.Add(Task.Run(()=> p.Edit()));
}
string response = "OK";
IEnumerable<string> res = await Task.WhenAll<string>(UpdateTasks);
}
catch (Exception ex)
{
throw;
}
return "Nie udało się zaktualizować danych żadnego zasobu..";
}
And here is Edit() method of place object. It basically updates place data in SQL server table:
public async Task<string> Edit()
{
string iSql = #"UPDATE JDE_Places
SET Priority=#Priority
WHERE PlaceId=#PlaceId";
string msg = "OK";
using (SqlCommand command = new SqlCommand(iSql, Settings.conn))
{
command.Parameters.AddWithValue("#PlaceId", PlaceId);
command.Parameters.AddWithValue("#Priority", Priority);
int result = -1;
try
{
result = await command.ExecuteNonQueryAsync();
IsUpdated = false;
}
catch (Exception ex)
{
msg = $"Wystąpił błąd przy edycji zasobu {Name}. Opis błędu: {ex.Message}";
}
}
return msg;
}
And here's Settings conn property that serves as reusable connection object:
public static class Settings
{
private static SqlConnection _conn { get; set; }
public static SqlConnection conn
{
get
{
if (_conn == null)
{
_conn = new SqlConnection(Static.Secrets.ConnectionString);
}
if (_conn.State == System.Data.ConnectionState.Closed || _conn.State == System.Data.ConnectionState.Closed)
{
try
{
_conn.Open();
}
catch (Exception ex)
{
MessageBox.Show("Nie udało się nawiązać połączenia z bazą danych.. " + ex.Message);
}
}
return _conn;
}
}
}
I realize it's probably better to keep the connection within using statement (instead of reusing it), but when I added it to place.Edit() method it worked even slower (and unreliably).
UPDATE: I ran few tests more and the time they took to add 540 rows varied from 15 seconds to 400 seconds.. Then I just changed result = await command.ExecuteNonQueryAsync() to result = command.ExecuteNonQuery() in Edit() of place object, ran few tests more, and all finished under 10 seconds! I don't know why async version of ExecuteNonQuery() was so much worse than non-async one, though. Single Edit() method was taking around 0,1 sec with ExecuteNonQuery() and 1 - 400 seconds with ExecuteNonQueryAsync(). Here are logs: ExecuteNonQuery() ExecuteNonQueryAsync()

Your issue here is your Settings class. You're essentially trying to use the same SqlConnection object in multiple Sqlcommands. SqlConnection is not threadsafe when used like this. You end up with multiple commands because your code is non-blocking and async. That is what is causing your code the "wait" (or deadlock). This is why when you run it sync (without the ExecuteNonQueryAsync, etc.) it works correctly.
You don't need this object at all anyway. ADO.Net handles connection pooling for you, so there is no advantage in re-using the same SqlConnection. Just create a new one for each SqlCommand:
public async Task<string> Edit()
{
using (SqlConnection conn = new SqlConnection(...))
using (SqlCommand command = new SqlCommand(iSql, conn))
{
...
}
}
and you should find that your "wait" goes away.

SQL connection closes during SqlBulkCopy

I'm trying to use SqlBulkCopy to import data into a temp table.
private bool CreateTempTable(IDbConnection conn, byte[] fileByteArray, string tempTableName)
{
try
{
Stream stream = new MemoryStream(fileByteArray);
using (var reader = new CsvReader(stream, false, System.Text.Encoding.UTF8))
{
// Bulk insert the data into a temporary table.
using (SqlBulkCopy bulkCopy = new SqlBulkCopy((SqlConnection)conn))
{
// Import the data into the temp table
bulkCopy.DestinationTableName = tempTableName;
bulkCopy.EnableStreaming = true;
bulkCopy.BatchSize = 20000;
bulkCopy.BulkCopyTimeout = 600;
bulkCopy.WriteToServer(reader);
}
}
}
catch (Exception ex)
{
Logger.Error(ex, $"CreateTempTable(): Exception while creating the TempTable \"{tempTableName}\" - {ex.Message}");
return false;
}
return true;
}
This method is called within another method that runs as async. The connection is passed and maintained by code that looks similar to this:
private async Task<bool> ConsumeAsyn(byte[] fileByteArray, string tempTableName)
{
using (var conn = (SqlConnection)OpenConnection())
{
if (CreateTempTable(conn, fileByteArray, tempTableName))
{
// success
}
else
{
// fail
}
}
}
This is basically how the ConsumeAsync is being called:
public async Task<bool> ProcessNextAsync2()
{
try
{
isIdle = false;
string filePath = ImportFilePickupPath + "\\" + Filename;
byte[] inStream = await LoadImportFileAsync(filePath);
bool consumeSuccess = await Consume(inStream, "##TempTable");
if (consumeSuccess)
{
// Delete the file
DeleteImportFile(filePath);
}
else
{
// Rescedule job
}
isIdle = true;
}
catch (Exception ex)
{
isIdle = true;
}
return isIdle;
}
If I use a small record set, of about a 100 rows in my csv, then everything works. The problem is if I have thousands of records, then the connection gets terminated during the bulk import process.
How can I pass the connection to the method and ensure that it remains open till the bulk import completes?

Maybe an answer, maybe not, but perhaps need to keep your IDisposables (at least/especially the SqlConnection) outside of the async tasks.
This answer over here makes it sound like .NET considers the IDiposables to be out of scope and so disposes them while you're still trying to use them:
SqlBulkCopy.WriteToServer() keep getting "connection is closed"

I were able to fix my problem by dropping support on a third party nuget lib that were dropping my connections. While it's unclear as to why it was happening, getting rid of all async calls and managing the connections myself seemed to have fixed my issue. Bulk imports was working fine however processing the data from the ##Temp table was slow. I decided to keep it all in memory and process it from there.
private bool Consume(byte[] fileByteArray, IDataProcess dataConsumer)
{
try
{
using (var conn = OpenConnection())
{
// Convert byte Array to a stream
Stream stream = new MemoryStream(fileByteArray);
// Create a reader from the stream
using (var reader = new CsvReader(stream, false, System.Text.Encoding.UTF8))
{
RecordEnumerator enumerator = reader.GetEnumerator();
enumerator.MoveNext();
do
{
// Proccess enumerator.Current with dataConsumer
} while (enumerator.MoveNext());
}
}
}
catch (Exception ex)
{
return false;
}
return true;
}
We run the risk of running out of memory but until we get a faster approach, this will have to do.

How to retry connecting to database after internet connection has been switched off after some time

i have an notification application built in c# for notifying any change in database . the application runs in the background . But the problem is after the start of the application if the internet is switched off the application throws an SQLException . i have used try catch to handle the exception . but i want my application to try connecting the database and when the connection is established it returns to the main code .
try
{
using (SqlConnection connection =
new SqlConnection(GetConnectionString()))
{
//i want to return here when the connection is reestablished
using (SqlCommand command =
new SqlCommand(GetListenerSQL(), connection))
{
connection.Open();
// Make sure we don't time out before the
// notification request times out.
command.CommandTimeout = NotificationTimeout;
SqlDataReader reader = command.ExecuteReader();
while (reader.Read())
{
messageText = System.Text.ASCIIEncoding.ASCII.GetString((byte[])reader.GetValue(13)).ToString();
// Empty queue of messages.
// Application logic could parse
// the queue data and
// change its notification logic.
}
object[] args = { this, EventArgs.Empty };
EventHandler notify =
new EventHandler(OnNotificationComplete);
// Notify the UI thread that a notification
// has occurred.
this.BeginInvoke(notify, args);
}
}
}
catch(SqlException e)
{
}
is it possible to do it without goto statement . i would prefer avoiding the goto statement .

I would move the retry logic out of the query method. I've seen a good retry library somewhere, but I can't find it just now.
public void StartListener()
{
var message = GetMessage();
//process message in some way
object[] args = { this, EventArgs.Empty };
EventHandler notify = OnNotificationComplete;
this.BeginInvoke(notify, args);
}
private const int TimeoutStep = 2000;
private const int MaxTimeout = 10000;
private string GetMessage(int timeout = 0)
{
//prevent loop of endless retries
if (timeout >= MaxTimeout)
{
//optional: define your own Exception class
throw new MaxTimeoutException();
}
try
{
Thread.Sleep(timeout);
return GetMessageFromDatabase();
}
catch (SqlException ex)
{
//log ex in debug mode at least
return GetMessage(timeout + TimeoutStep);
}
}
private string GetMessageFromDatabase()
{
string message = null;
using (var connection = new SqlConnection(GetConnectionString()))
{
using (var command = new SqlCommand(GetListenerSQL(), connection))
{
connection.Open();
command.CommandTimeout = NotificationTimeout;
using (var reader = command.ExecuteReader())
{
while (reader.Read())
{
message = System.Text.ASCIIEncoding.ASCII.GetString((byte[])reader.GetValue(13));
}
}
}
}
return message;
}

from the suggestion of #Archer i got the solution . in the catch block i call the method again which uses this connection after some suitable time . Something like
public void StartListener()
{
try
{
using (SqlConnection connection =
new SqlConnection(GetConnectionString()))
{
//i want to return here when the connection is reestablished
using (SqlCommand command =
new SqlCommand(GetListenerSQL(), connection))
{
connection.Open();
// Make sure we don't time out before the
// notification request times out.
command.CommandTimeout = NotificationTimeout;
SqlDataReader reader = command.ExecuteReader();
while (reader.Read())
{
messageText = System.Text.ASCIIEncoding.ASCII.GetString((byte[])reader.GetValue(13)).ToString();
// Empty queue of messages.
// Application logic could parse
// the queue data and
// change its notification logic.
}
object[] args = { this, EventArgs.Empty };
EventHandler notify =
new EventHandler(OnNotificationComplete);
// Notify the UI thread that a notification
// has occurred.
this.BeginInvoke(notify, args);
}
}
}
catch(SqlException e)
{
Thread.Sleep(2000);
StartListener();
}
}

If its an failure you should call an timer and the timer should call the backgroundworker.
Write the Functionality to check the connection in the backgroundworkers. if it goes for true you should stop the timer. And call the usual process

Strange behaviour on writing to db while using service broker

I am testing both service broker external activator and polling based client on behalf of process speed performances of each.
For external activator, I have built a command line application which is being notified when any change occur some table and writes to the same db. Code inside exe looks like as follows
private static void ProcessRequest()
{
using (var connection = new SqlConnection(ServiceConstant.ConnectionString))
{
connection.Open();
do
{
using (var tran = connection.BeginTransaction())
{
//Get a message from the queue
byte[] message = QueueProcessorUtil.GetMessage(ServiceConstant.QueueName, connection, tran, ServiceConstant.WaitforTimeout);
if (message != null)
{
MessageReceiving = true;
try
{
//Write it to the db
ProcessMessage(message);
}
catch (Exception ex)
{
logger.Write("Fail: " + ex);
}
tran.Commit();
}
else
{
tran.Commit();
MessageReceiving = false;
}
}
}
while (MessageReceiving);
}
}
When I insert 20 messages to the queue, total duration of insertion of all the messages is approx 10ms
When I extract the ProcessMessage function above which writes the messages to the db to an another separate console application and then call this function 20 times as follows, this time it takes approx 50ms
class Program
{
static void Main(string[] args)
{
for (var i = 1; i <= 20; i++)
{
string message = "mm";
ProcessMessaage(message);
}
}
}
ProcessMessage function
string sql = #"INSERT INTO [Workflow].[dbo].[TestOrderLog]([OrderId],[RecordTime])
VALUES (#orderId, GETDATE()) SELECT SCOPE_IDENTITY()";
using (SqlConnection con = new SqlConnection(ConfigurationManager.AppSettings["SqlConnection"].ToString()))
using (SqlCommand com = new SqlCommand(sql, con))
{
con.Open();
com.CommandType = CommandType.Text;
com.Parameters.AddWithValue("#orderId", 1);
try
{
var result = com.ExecuteScalar();
var id = (result != null) ? Convert.ToInt32(result) : 0;
}
catch (Exception ex)
{
throw ex;
}
con.Close();
}
I don't understand and am surprised although there are costly processing blocks (query a message) inside the loop of external activator code, it takes faster to write db than pure loop in console app code.
Why would pure insertion in a loop be slower than insertion inside the external activator exe instance's code?
Side note, in EAService.config file, <Concurrency min="1" max="1" />

It was an absurd mistake of mine, first one is compiled and deployed running code
and the second one is running with debugger inside visual studio, so the intervals became normal running without debugger.

Calling a MySQL Routine in C#

Please accept my apologies if I'm getting my verbiage wrong; I'm just now learning C# (my background is mostly Visual Basic and PHP).
What I'm trying to do is create a class / routine in C# (Windows Forms) for connecting and disconnecting to a MySQL database that can then be reused throughout the rest of my project without having to reiterate the code every time.
I've got my class / routine setup, but I'm stuck on trying to call it from the rest of my project. I know in Visual Basic this was a fairly simple task to do, but I just can't seem to figure it out how to do it in C#.
Any suggestions? Thank you in advance.
public void dbDisconnect(object sender, EventArgs e)
{
try
{
MySqlConnection connection = new MySqlConnection(Properties.Settings.Default.mysql_db_conn_string);
connection.Close();
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}

Here's a method I currently have in my app:
public static MySqlConnection CreateConnection(
string mysqlServer,
string mysqlUser,
string mysqlPassword,
string mysqlDatabase)
{
MySqlConnection mysqlConnection = null;
string mysqlConnectionString = String.Format(
"server={0};uid={1};pwd={2};database={3};DefaultCommandTimeout={4};",
mysqlServer, mysqlUser, mysqlPassword, mysqlDatabase, 120);
/**
** Workaround for MySQL 5.6 bug:
** http://stackoverflow.com/questions/30197699/reading-from-stream-failed-mysql-native-password-error
*/
int tryCounter = 0;
bool isConnected = false;
do
{
tryCounter++;
try
{
mysqlConnection = new MySqlConnection();
mysqlConnection.ConnectionString = mysqlConnectionString;
mysqlConnection.Open();
if (mysqlConnection.State == ConnectionState.Open)
{
isConnected = true;
}
}
catch (MySqlException ex)
{
if (tryCounter < 10)
{
DebugLog.Dump(ex.ToString(), DebugLog.MainLogFilePath);
Thread.Sleep(10000); // 10 seconds.
}
else
{
throw;
}
}
} while (!isConnected);
return mysqlConnection;
}
Usage:
using (MySqlConnection hostsDbConnection = HostsDbConnector.CreateConnection())
{
// Do something...
}
With using keyword you don't need to close the connection manually, it'll be closed automatically when it's no longer needed.