We Keep Coding

Transient Fault Retry logic best practices

Friends, I have a question about implementing a simple retry policy around the execution of the SQL command.
My question is: should the retry loop encapsulate the construction of the connection and transaction, or should it live inside the connection.
For example:
private void RetryLogSave(DynamicParameters parameters, int retries = 3)
{
int tries = 0;
using (var connection = new SqlConnection(_connectionString))
{
connection.Open();
using (var transaction = connection.BeginTransaction())
{
var logItemCommand = new CommandDefinition(commandText: Constants.InsertLogItem,
parameters: parameters, transaction: transaction, commandType: System.Data.CommandType.Text);
do
{
try
{
tries++;
connection.Execute(logItemCommand);
transaction.Commit();
break;
}
catch (Exception exc)
{
if (tries == retries)
{
transaction.Rollback();
throw exc;
}
Task.Delay(100 * tries).Wait();
}
}
while (true);
}
}
}
Is what I've done here appropriate and acceptable, or should the retry logic live on the outside of the SqlConnection construction?

Formalizing my comments as an answer.
should the retry logic live on the outside of the SqlConnection
construction?
Yes. If doing retry logic with keeping connection opened you're wasting resources. Someone else may use it while you're waiting N seconds for re-try. Opening/closing connections is usually (for most ODBC drivers) implemented on top of Connection Pooling mechanism. You do not actually close it - you allow connection to go back in pool to be reused by someone else. Keeping connections opened during re-try will force system to create more and more new physical connections (because they are not returned to the pool) and eventually your SQL Server will be exhausted.
Regarding re-try mechanism - to not reinvent the wheel, I usually use Polly library.
You can define somewhere static class with list of your polices:
public static class MyPolices
{
// Retry, waiting a specified duration between each retry
public static Policy RetryPolicy = Policy
.Handle<Exception>() // can be more specific like SqlException
.WaitAndRetry(new[]
{
TimeSpan.FromSeconds(1),
TimeSpan.FromSeconds(2),
TimeSpan.FromSeconds(3)
});
}
Then, simplify your method to
private void LogSave(DynamicParameters parameters)
{
using (var connection = new SqlConnection(_connectionString))
{
connection.Open();
using (var transaction = connection.BeginTransaction())
{
// make sure to not forget to dispose your command
var logItemCommand = new CommandDefinition(commandText: Constants.InsertLogItem,
parameters: parameters, transaction: transaction, commandType: System.Data.CommandType.Text);
try
{
// not sure if conn.Execute is your extension method?
connection.Execute(logItemCommand);
transaction.Commit();
}
catch (Exception exc)
{
transaction.Rollback();
throw;
}
}
}
}
and call it like this
MyPolices.RetryPolicy.Execute(() => LogSave(parameters));
This approach will make your code more SOLID keeping retry logic in isolation.

SqlCommand timeout in C# MMO application

I have a C# project that is working with TCP socket in an asynchronous way.
Every request comes from client and ask question from SQL Server stored procedure, opens and closes a SQL connection after ending of question.
I've used this code:
using (var con = new SqlConnection(setting.ConnectionString))
{
try
{
//some codes (edited)
SqlCommand command = new SqlCommand(con);
command.CommandText = "procedurename1";
command.CommandType = CommandType.StoredProcedure;
command.Parameters.Add(new SqlParameter("#name", sb.ToString()));
SqlDataAdapter adapter = new SqlDataAdapter(command);
try
{
adapter.Fill(dataSet);
}
catch (Exception ex)
{
con.Close();
con.Dispose();
throw ex;
}
finally {
con.Close();
con.Dispose();
}
}
catch(Exception ex)
{}
finally
{
con.close();
con.dispose();
}
}
I've used
netstat -a -n | find /c "1433"
to count SQL connections open and close.
Problem is SQL connections count increases and it rarely decreases and count down.
Main problem, is when my program works under lots of requests about 30 minutes, I get
SqlCommand timeout error (default 30 seconds passed)
and after restarting my C# program, the SqlCommand timeout will be gone.
Is this a problem of my program or SQL Server side?
Remember it always calls a stored procedure in SQL Server, not executing query
directly.
main method:
public void main()
{
Task.Factory.StartNew(() =>
{
allDone.Reset();
mySocket.AcceptAsync(e);
allDone.WaitOne();
});
}
public void e_Completed(object sender, SocketAsyncEventArgs e)
{
var socket = (Socket)sender;
ThreadPool.QueueUserWorkItem(HandleTcpRequest, e.AcceptSocket);
e.AcceptSocket = null;
socket.AcceptAsync(e);
}
public void HandleTcpRequest(object state)
{
//do some code and connection to SQL server
DLL.Request httprequest = new DLL.Request(dataSet.Tables[0], fileDt);
DLL.IHttpContext _context = new DLL.HttpContext(httprequest);
_context.GetResults();
}

Main problem, is when my program works under lots of requests about 30 minutes,
To isolate the root problem of the time-out, I suggest testing the sql query of the stored procedure independent of TCP socket calls for 30 minutes
and log the time-out exception details for inspection
Run the following query within 30 minutes to simulate your working environment:
public void RunQuery()
{
using (var con = new SqlConnection(setting.ConnectionString))
{
try
{
//some codes
}
catch(SqlException ex)
{
//test for timeout
if (ex.Number == -2) {
Console.WriteLine ("Timeout occurred");
// log ex details for more inspection
}
}
}
}
Read How to handle the CommandTimeout properly?
As you use async calls, I suggest you to try to use Asynchronous Database Calls With Task-based Asynchronous Programming Model (TAP)

I'm going to take a long-shot based on the way the limited Sql-related code we can see is written since we can't see "//some codes".
I'm going to guess that some of the disposable things like SqlCommand, DataReader, SqlDataAdapter, TransactionScope, etc are not in 'using' blocks, so are holding resources open on the database.
It may also be worth raising the possibility that this kind of problem could be in the code shown in the question or any other program accessing that database, including your own applications and SSMS (e.g. if a developer has an uncommitted transaction running in a window).
P.S. I would suggest deleting everything in the using block except the "//some codes" part.
UPDATE after more code was added
Here is your code after correction. this will ensure that the resources are disposed, which will prevent the leaking resources that are probably causing your problem.
using (var con = new SqlConnection(setting.ConnectionString))
{
//some codes (edited)
using (SqlCommand command = new SqlCommand(con))
{
command.CommandText = "procedurename1";
command.CommandType = CommandType.StoredProcedure;
command.Parameters.Add(new SqlParameter("#name", sb.ToString()));
using (var adapter = new SqlDataAdapter(command))
{
adapter.Fill(dataSet);
}
}
}
P.S. don't ever write "throw ex;" from inside a catch ever again. It causes the stack trace to be lost - just use "throw;".

Close connection when using a Transaction

I have the following method which uses a transaction.
private string getDocumentDetailsByNumber(string DocNumber)
{
SqlTransaction transaction = DALDBConnection.SqlConnection.BeginTransaction();
try
{
DataSet DocNum = new DataSet();
string sDocNumber = "";
string[] taleNamesDoc = new string[1];
taleNamesDoc[0] = "docnumber";
SqlParameter[] paramDoc = new SqlParameter[1];
paramDoc[0] = new SqlParameter("#DocumentNumber", DocNumber.ToString().Trim());
SqlHelper.FillDataset(transaction, CommandType.StoredProcedure, "spGetDocumentDetailsByNumber", DocNum, taleNamesDoc, paramDoc);
string docTitle = DocNum.Tables["docnumber"].Rows[0][0].ToString();
transaction.Commit();
return docTitle;
}
catch (Exception ex)
{
transaction.Rollback();
throw ex;
}
}
after running the method several times, user ended up getting the error message below.
the timeout period elapsed prior to obtaining a connection from the
pool
Error occurred because I haven't closed the connection and the connection pool has over flown.
I tried to close the connection before committing the transaction.
transaction.Connection.Close();
transaction.Commit();
Then got the following error.
This SqlTransaction has completed; it is no longer usable
How can I close the connection to avoid the error?

You cannot exhaust your pool by using a single connection. You need to close all connections you are using. Preferably after the transaction has ended one way or another. using blocks are your friend for almost all database related objects.
By the way:
throw ex;
This damages your exception by replacing the original stacktrace. Use:
throw;
to rethrow the exception you caught unchanged.

As mentioned, you should dispose of the connection properly. I've modified your code to demonstrate. Please note you will need to substitute the connection string with yours.
private string getDocumentDetailsByNumber(string DocNumber)
{
using (var connection = new SqlConnection("My Connection String"))
{
SqlTransaction transaction = connection.BeginTransaction();
DataSet DocNum = new DataSet();
string sDocNumber = "";
string[] taleNamesDoc = new string[1];
taleNamesDoc[0] = "docnumber";
SqlParameter[] paramDoc = new SqlParameter[1];
paramDoc[0] = new SqlParameter("#DocumentNumber", DocNumber.ToString().Trim());
SqlHelper.FillDataset(transaction, CommandType.StoredProcedure, "spGetDocumentDetailsByNumber", DocNum, taleNamesDoc, paramDoc);
string docTitle = DocNum.Tables["docnumber"].Rows[0][0].ToString();
transaction.Commit();
return docTitle;
} // Connection is disposed and cleaned up.
}
Opening new connections are cheap and should not be frowned upon. Each one you call the database you should open a new one like this. By maintaining a connection and not disposing of it, you are taking resources away from the database as well. It does not have an infinite amount of connections in its pool that can be used at once.
edit
Removed try/catch as mentioned in comments. If an exception is thrown while in the using block, a rollback will occur and the exception passed up the stack.

Have you considered CALLNIG CLOSE ON IT? Would be obvious to close a connection or?
Anything that implements IDIsposable should be disposed, btw., not just closed. And SqlConnection implements IDisposable. THis has nothing to do with SqlTransaction - you violae a fundamental rule of the .NET world by not disposing a disposable routine.

Using when working with SQLCommand/Connection

I`ve used a function to open sql connection and generate a command.So I don`t use using for them.
But what happens if I get an exception ?while catch the exception nedd I explicitly dispose them?

using translates into try-finally block. If you don't want to use using statement then you can mimic the same using try-finally, in your case with catch just have finally block and dispose the connection there. Something like:
{
SqlConnection conn = new SqlConnection("your connection string");
try
{
//your code
}
catch (SqlException se)
{
//handle particular exception first
}
catch (Exception ex)
{
//handle all other exceptions
}
finally
{
if (conn != null)
conn.Dispose();
}
}
That will ensure the disposal of connection in case of an exception and normal flow (when no exception occurs)

Understanding how transaction.complete function?

Hi All we have started to use transaction scope and below is the code snippet. What we need to understand is based on our understading after each of the using for connection the particular connection will be disposed/closed right? So since its closed how does the transaction.complete works?
using (TransactionScope transScope = new TransactionScope())
{
try
{
string myConnStringLocal = "User Id=***;Password=****;Host=" + globalSettings.settingLocalIP + ";Database=" + globalSettings.settingLocalDB;
using (MySqlConnection connectionLocal = new MySqlConnection(myConnStringLocal))
{
try{
connectionLocal.Open();
}
Catch(Exception e)
{
}
finally{
connectionLocal.Close();
}
}
string myConnStringCentral = "User Id=***;Password=*****;Host=" + globalSettings.settingCentralIP + ";Database=" + globalSettings.settingCentralDB;
using (MySqlConnection connectionCentral = new MySqlConnection(myConnStringCentral))
{
try{
connectionCentral.Open();
}
Catch(Exception e)
{
}
finally{
connectionCentral.Close();
}
}
string myConnStringCentralCopy = "User Id=*****;Password=*****;Host=" + globalSettings.settingCentralCopyIP + ";Database=" + globalSettings.settingCentralCopyDB;
using (MySqlConnection connectionCentralCopy = new MySqlConnection(myConnStringCentralCopy))
{
try{
connectionCentralCopy.Open();
}
Catch(Exception e)
{
}
finally{
connectionCentralCopy.Close();
}
}
transScope.Complete();
Console.WriteLine("Transaction is completed");
}
catch (Exception)
{
Console.WriteLine("Transaction is rolled back");
}
}

TransactionScope.Complete tells all of the Transaction Managers that they are good to commit this transaction. It is not a guarantee that everything will actually commit. After the Complete method is called all the transaction managers initiate a commit individually and if all of the transaction Managers succeed then the transaction is considered as completed successfully.
You may refer to this link for further details

When you connect to multiple databases in a single TransactionScope, the transaction is escalated to a distributed transaction and coordinated by MSDTC using 2-phase commit.
Regarding connection close - this is special case when connection is closed inside a TransactionScope, internally it is managed by System.Transactions infrastructure, so db session may still be open even though connection closed from your end.
See this note on MSDN:
Pending transactions started using Transact-SQL or BeginTransaction are automatically rolled back when the connection is reset if connection pooling is enabled. If connection pooling is off, the transaction is rolled back after SqlConnection.Close is called. Transactions started through System.Transactions are controlled through the System.Transactions infrastructure, and are not affected by SqlConnection.Close
EDIT Based on your comments, here is what you can do:
try
{
using (TransactionScope transScope = new TransactionScope())
{
string myConnStringLocal = ...;
using (var connectionLocal = new MySqlConnection(myConnStringLocal))
{
connectionLocal.Open();
// do your DB update
} //no need to close connection explicitly, the using() {..} statement does that for you
string myConnStringCentral = ...;
using (var connectionCentral = new MySqlConnection(myConnStringCentral))
{
connectionCentral.Open();
// do your DB update
} //no need to close connection explicitly, the using() {..} statement does that for you
string myConnStringCentralCopy = ...;
using (var connectionCentralCopy = new MySqlConnection(myConnStringCentralCopy))
{
connectionCentralCopy.Open();
// do your DB update
} //no need to close connection explicitly, the using() {..} statement does that for you
transScope.Complete();
Console.WriteLine("Transaction is completed");
} //no need to dispose transactionScope explicitly, the using() {..} statement does that for you
}
catch (Exception)
{
// If any exception occurs in the try block above transScope.Complete() line will be caught here
// and will automatically cause the transaction to rollback.
Console.WriteLine("Transaction is rolled back");
}
// You can then start new TransactionScope if you want to further update more than one DB in a transactional manner.
try
{
using (TransactionScope transScope = new TransactionScope())
{
//...
}
}
catch (Exception)
{
//...
}

When the Complete method is called, then everything in the scope will be committed, if no Exception is thrown. If the code gets out of scope without Complete, then no commit will occur. In short, if you call the Complete method, then, in case there are no Exceptions thrown, your transaction(s) in the scope of the given TransactionScope will be committed.
Also, I must add, that there might also be a hierarchy, a tree of TransactionScopes. You can also set the behavior of a TransactionScope for the case when a sub-scope of the TransactionScope rolled back.