A while back I asked a question about TransactionScope escalating to MSDTC when I wasn't expecting it to. (Previous question)
What it boiled down to was, in SQL2005, in order to use a TransactionScope, you can only instance and open a single SqlConnection within the life of the TransactionScope. With SQL2008, you can instance multiple SqlConnections, but only a single one can be open at any given time. SQL2000 will always escalate to DTC...we don't support SQL2000 in our application, a WinForms app, BTW.
Our solution to single-connection-only problem was to create a TransactionScope helper class, called LocalTransactionScope (aka 'LTS'). It wraps a TransactionScope and, most importantly, creates and maintains a single SqlConnection instance for our application. The good news is, it works - we can use LTS across disparate pieces of code and they all join the ambient transaction. Very nice. The trouble is, every root LTS instance created will create and effectively kill a connection from the connection pool. By 'Effectively Kill' I mean it will instance a SqlConnetion, which will open a new connection (for whatever reason, it never reuses a connection from the pool,) and when that root LTS is disposed, it closes and disposes the SqlConnection which is supposed to release the connection back to the pool so that it can be reused, however, it clearly never is reused. The pool bloats until it's maxed out, and then the application fails when a max-pool-size+1 connection is established.
Below I've attached a stripped down version of the LTS code and a sample console application class that will demonstrate the connection pool exhaustion. In order to watch your connection pool bloat, use SQL Server Managment Studio's 'Activity Monitor' or this query:
SELECT DB_NAME(dbid) as 'DB Name',
COUNT(dbid) as 'Connections'
FROM sys.sysprocesses WITH (nolock)
WHERE dbid > 0
GROUP BY dbid
I'm attaching LTS here, and a sample console application that you can use to demonstrate for yourself that it will consume connections from the pool and never re-use nor release them. You will need to add a reference to System.Transactions.dll for LTS to compile.
Things to note: It's the root-level LTS that opens and closes the SqlConnection, which always opens a new connection in the pool. Having nested LTS instances makes no difference because only the root LTS instance establishes a SqlConnection. As you can see, the connection string is always the same, so it should be reusing the connections.
Is there some arcane condition we're not meeting that causes the connections not to be re-used? Is there any solution to this other than turning pooling off entirely?
public sealed class LocalTransactionScope : IDisposable
{
private static SqlConnection _Connection;
private TransactionScope _TransactionScope;
private bool _IsNested;
public LocalTransactionScope(string connectionString)
{
// stripped out a few cases that need to throw an exception
_TransactionScope = new TransactionScope();
// we'll use this later in Dispose(...) to determine whether this LTS instance should close the connection.
_IsNested = (_Connection != null);
if (_Connection == null)
{
_Connection = new SqlConnection(connectionString);
// This Has Code-Stink. You want to open your connections as late as possible and hold them open for as little
// time as possible. However, in order to use TransactionScope with SQL2005 you can only have a single
// connection, and it can only be opened once within the scope of the entire TransactionScope. If you have
// more than one SqlConnection, or you open a SqlConnection, close it, and re-open it, it more than once,
// the TransactionScope will escalate to the MSDTC. SQL2008 allows you to have multiple connections within a
// single TransactionScope, however you can only have a single one open at any given time.
// Lastly, let's not forget about SQL2000. Using TransactionScope with SQL2000 will immediately and always escalate to DTC.
// We've dropped support of SQL2000, so that's not a concern we have.
_Connection.Open();
}
}
/// <summary>'Completes' the <see cref="TransactionScope"/> this <see cref="LocalTransactionScope"/> encapsulates.</summary>
public void Complete() { _TransactionScope.Complete(); }
/// <summary>Creates a new <see cref="SqlCommand"/> from the current <see cref="SqlConnection"/> this <see cref="LocalTransactionScope"/> is managing.</summary>
public SqlCommand CreateCommand() { return _Connection.CreateCommand(); }
void IDisposable.Dispose() { this.Dispose(); }
public void Dispose()
{
Dispose(true); GC.SuppressFinalize(this);
}
private void Dispose(bool disposing)
{
if (disposing)
{
_TransactionScope.Dispose();
_TransactionScope = null;
if (!_IsNested)
{
// last one out closes the door, this would be the root LTS, the first one to be instanced.
LocalTransactionScope._Connection.Close();
LocalTransactionScope._Connection.Dispose();
LocalTransactionScope._Connection = null;
}
}
}
}
This is a Program.cs that will exhibit the connection pool exhaustion:
class Program
{
static void Main(string[] args)
{
// fill in your connection string, but don't monkey with any pooling settings, like
// "Pooling=false;" or the "Max Pool Size" stuff. Doesn't matter if you use
// Doesn't matter if you use Windows or SQL auth, just make sure you set a Data Soure and an Initial Catalog
string connectionString = "your connection string here";
List<string> randomTables = new List<string>();
using (var nonLTSConnection = new SqlConnection(connectionString))
using (var command = nonLTSConnection.CreateCommand())
{
command.CommandType = CommandType.Text;
command.CommandText = #"SELECT [TABLE_NAME], NEWID() AS [ID]
FROM [INFORMATION_SCHEMA].TABLES]
WHERE [TABLE_SCHEMA] = 'dbo' and [TABLE_TYPE] = 'BASE TABLE'
ORDER BY [ID]";
nonLTSConnection.Open();
using (var reader = command.ExecuteReader())
{
while (reader.Read())
{
string table = (string)reader["TABLE_NAME"];
randomTables.Add(table);
if (randomTables.Count > 200) { break; } // got more than enough to test.
}
}
nonLTSConnection.Close();
}
// we're going to assume your database had some tables.
for (int j = 0; j < 200; j++)
{
// At j = 100 you'll see it pause, and you'll shortly get an InvalidOperationException with the text of:
// "Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool.
// This may have occurred because all pooled connections were in use and max pool size was reached."
string tableName = randomTables[j % randomTables.Count];
Console.Write("Creating root-level LTS " + j.ToString() + " selecting from " + tableName);
using (var scope = new LocalTransactionScope(connectionString))
using (var command = scope.CreateCommand())
{
command.CommandType = CommandType.Text;
command.CommandText = "SELECT TOP 20 * FROM [" + tableName + "]";
using (var reader = command.ExecuteReader())
{
while (reader.Read())
{
Console.Write(".");
}
Console.Write(Environment.NewLine);
}
}
Thread.Sleep(50);
scope.Complete();
}
Console.ReadKey();
}
}
The expected TransactionScope/SqlConnection pattern is, according to MSDN:
using(TransactionScope scope = ...) {
using (SqlConnection conn = ...) {
conn.Open();
SqlCommand.Execute(...);
SqlCommand.Execute(...);
}
scope.Complete();
}
So in the MSDN example the conenction is disposed inside the scope, before the scope is complete. Your code though is different, it disposes the connection after the scope is complete. I'm not an expert in matters of TransactionScope and its interaction with the SqlConnection (I know some things, but your question goes pretty deep) and I can't find any specifications what is the correct pattern. But I'd suggest you revisit your code and dispose the singleton connection before the outermost scope is complete, similarly to the MSDN sample.
Also, I hope you do realize your code will fall apart the moment a second thread comes to play into your application.
Is this code legal?
using(TransactionScope scope = ..)
{
using (SqlConnection conn = ..)
using (SqlCommand command = ..)
{
conn.Open();
SqlCommand.Execute(..);
}
using (SqlConnection conn = ..) // the same connection string
using (SqlCommand command = ..)
{
conn.Open();
SqlCommand.Execute(..);
}
scope.Complete();
}
Related
I need every thread to connect to one database and execute some query and maybe some thread in one time can execute the query.
I create a static class for connecting to SQL Server database with ADO.NET:
public static class ADOFire
{
static System.Data.SqlClient.SqlConnection Cnn = new System.Data.SqlClient.SqlConnection();
public static string CnnString { get; set; }
public static void CreateConnection()
{
if (Cnn.State == ConnectionState.Open)
return;
Cnn.ConnectionString = CnnString = ConfigurationManager.ConnectionStrings["CnnString"].ToString();
if (Cnn.State != System.Data.ConnectionState.Open)
Cnn.Open();
}
public static System.Data.DataTable GetTable(System.Data.SqlClient.SqlCommand Com, System.Data.SqlClient.SqlDataAdapter Ada, string ComText)
{
CreateConnection();
Com.Connection = Cnn;
Ada.SelectCommand = Com;
try
{
System.Data.DataTable T = new System.Data.DataTable();
Com.CommandText = ComText;
Ada.Fill(T);
return T;
}
catch { return null; }
}
}
And in here in each thread I call static function like this:
System.Data.SqlClient.SqlCommand Com = new System.Data.SqlClient.SqlCommand();
System.Data.SqlClient.SqlDataAdapter Ada = new System.Data.SqlClient.SqlDataAdapter();
Datatable dt = ADOFire.GetTable(Com, Ada, "Some Query 'select * from x'");
Based on this link, doesn't make much difference between open a new connection or use from existing connection
Whenever a user calls Open on a connection, the pooler looks for an available connection in the pool. If a pooled connection is available, it returns it to the caller instead of opening a new connection. When the application calls Close on the connection, the pooler returns it to the pooled set of active connections instead of closing it. Once the connection is returned to the pool, it is ready to be reused on the next Open call
My questions are:
Can a connection serve different threads at the same time? (one connection for all)
Isn't the problem of data clutter due to the static function?
No, ADO.NET does not two threads accessing the same.conection at the same time
Yes, but it isn't the statix method that is a problem - it is the static connection field that is a huge problem
What you should do is have the static method create (new) and return the connection to the caller, and remove the field completely. Typical usage:
using (var conn = CreateConnection()) {
//... Use it!
}
I also have grave concerns about:
- why you're passing in a command and command text and adapter
- the lack of parameters; suggests a huge security problem (SQL injection)
- the use of data table (which is almost never the most appropriate tool)
I was getting quite often the following error:
Npgsql.NpgsqlException: 'The connection pool has been exhausted, either raise MaxPoolSize (currently 100) or Timeout (currently 15 seconds)'
Then I looked for possible causes and solutions in here and found out that I should be applying the using statement. So I reviewed all my code and did that.
However, I keep getting that error while testing a button that gets information from my database, does some calculation and writes the results in a few textboxes. It usually crashes at the fifth-ish time I click it. A piece of the code follows:
private void CalcTemp(Cable cable)
{
string sqlString = "Server=172.19.2.40; Port=5432; User Id=postgres; Password=password; Database=PROLIG;";
using (NpgsqlConnection sqlCon = new NpgsqlConnection(sqlString))
{
string cmdString = #"SELECT tempamb, elevmaxonan, elevmaxonaf, elevmaxonaf2, topoil1_2, topoil1_4, especial1factor, especial1topoil,
especial2factor, especial2topoil, especial3factor, especial3topoil, especial4factor, especial4topoil,
especial5factor, especial5topoil, especial6factor, especial6topoil FROM correntes WHERE prolig_ofs_id = #id;";
NpgsqlCommand sqlCmd = new NpgsqlCommand(cmdString, sqlCon);
sqlCmd.Parameters.AddWithValue("id", StartOF.MyOF.id);
NpgsqlDataAdapter sqlDa = new NpgsqlDataAdapter(sqlCmd);
DataTable dt = new DataTable();
sqlDa.Fill(dt);
//does calculation
}
Any thoughts on why this is happening and how to fix it?
Thanks a lot.
Just add using to your command creation:
using (NpgsqlCommand sqlCmd = new NpgsqlCommand(cmdString, sqlCon))
{
Disposing all objects which implement IDisposable is a good practice.
Since your command is not disposed of in time, your connection is not closed and returned to the pool.
It is a lot of stuff on dispose which must be executed directly (or by using).
protected override void Dispose(bool disposing)
{
if (State == CommandState.Disposed)
return;
if (disposing)
{
// Note: we only actually perform cleanup here if called from Dispose() (disposing=true), and not
// if called from a finalizer (disposing=false). This is because we cannot perform any SQL
// operations from the finalizer (connection may be in use by someone else).
// We can implement a queue-based solution that will perform cleanup during the next possible
// window, but this isn't trivial (should not occur in transactions because of possible exceptions,
// etc.).
if (_prepared == PrepareStatus.Prepared)
_connector.ExecuteBlind("DEALLOCATE " + _planName);
}
Transaction = null;
Connection = null;
State = CommandState.Disposed;
base.Dispose(disposing);
}
So you need the following code:
private void CalcTemp(Cable cable)
{
string sqlString = "Server=172.19.2.40; Port=5432; User Id=postgres; Password=password; Database=PROLIG;";
using (NpgsqlConnection sqlCon = new NpgsqlConnection(sqlString))
{
string cmdString = #"SELECT * FROM correntes WHERE prolig_ofs_id = #id;";
using (NpgsqlCommand sqlCmd = new NpgsqlCommand(cmdString, sqlCon))
{
sqlCmd.Parameters.AddWithValue("id", StartOF.MyOF.id);
NpgsqlDataAdapter sqlDa = new NpgsqlDataAdapter(sqlCmd);
DataTable dt = new DataTable();
sqlDa.Fill(dt);
//does calculation
} //end using command (calls dispose on command, even if exception happens)
} //end using connection (calls dispose on connection object, even if exception happens)
}
Next advice - do not use data tables in case of large amount of data. Use DataReader instead.
I'm creating database, using ADO.NET. Basically, I'm executing SQL commands in next way:
private bool ExecuteSqlCommand(string command)
{
var success = true;
using (var connection = GetSqlConnection())
{
if (connection == null)
return false;
using (var myCommand = new SqlCommand("query", connection))
{
try
{
connection.Open();
myCommand.CommandText = command;
myCommand.ExecuteNonQuery();
}
catch (Exception ex)
{
success = false;
Log.LogMessage(string.Format("Unable to execute SQL command: {0}", ex.Message));
}
}
}
return success;
}
GetSqlConnection just gets some proper SqlConnection with connection string like
"Server={0}\\{1};User Id={2};Password={3};Application Name={4};"
It works well, and executes command properly, with one exception - according to Sql Manager Studio activity monitor, it remains as active connection even after method was executed.
According to question Why does my SqlConnection remain in the SSMS Processes list after closing it?, this is correct behavior, since connection might be reused. But the serius issue is, that later, READ_COMMITTED_SNAPSHOT command will be called for this database, while using different SqlConnection. Which leads to exception, since READ_COMMITTED_SNAPSHOT requires, that connection, which is used to call this command, should be the only connection to database.
I can't reuse this connection for further operations with database, since I use different connection string for them, with database specified as InitialCatalog (obviously, I can't use it, while database doesn't exist).
So, can I somehow remove this initial connection?
Execute SqlConnection.ClearPool This will mark all connections to be discarded instead of recycled.
I'm having a serious issue with my app. It builds a lot of MySql connections and then it's causing a crash.
I build every method like that:
MySqlConnection connect = new MySqlConnection(
local_connection_string
); //this is global variable.
protected void sample()
{
try
{
connect.Open();
MySqlCommand query = new MySqlCommand(
"here some mysql command"
, connect);
query.ExecuteNonQuery();
}
catch
{
}
finally
{
connect.Dispose();
connect.Close();
}
}
For some reason it's not closing any of these connections and when I keep refreshing it builds connections on the server, once limit is hit app is crashing. All connections are closed when app is shut down.
try this:
using(MySqlConnection conn = new MySqlConnetion(local_connection_string)
{
conn.open();
MySqlCommand query = new MySqlCommand(
"here some mysql command"
, connect);
query.ExecuteNonQuery();
}
using(resource){}: right way for IDisposable resource usage
probably need to add: Application.ApplicationExit event with MySqlConnection.ClearAllPools()
To ensure that connections are always closed, open the connection inside of a using block, as shown in the following code fragment. Doing so ensures that the connection is automatically closed when the code exits the block.
using (MySqlConnection connection = new MySqlConnection(connectionString))
{
connection.Open();
// Do work here; connection closed on following line.
}
MySQL counter part uses Connection pooling and does not close when you call close instead it puts it in the connection pool!
Make sure you First Close then Dispose the Reader, Command, and Connection object!
You can use ConnectionString Parameter "Pooling=false" or the static methods MySqlConnection.ClearPool(connection) and MySqlConnection.ClearAllPools()
and Using keyword is the right way to go with this kind of Scenario.
Just close first the connection , before calling the dispose...
finally
{
connect.Close();
connect.Dispose();
}
I have written a simple Mono C# application for writing to an SQLite database using the Mono implementation present in the Mono.Data.Sqlite package:
using System;
using Mono.Data.Sqlite;
class MainClass
{
public static void Main (string[] args)
{
using (var dbConnection = new SqliteConnection (#"Data Source=/var/log/gmblog;Version=3;"))
{
dbConnection.Open();
string sql = #"INSERT INTO ""queue"" (""data"") VALUES(""Test"")";
using (var insertCommand = new SqliteCommand (sql, dbConnection))
{
insertCommand.ExecuteNonQuery();
}
}
}
}
This works fine until I do an insert from another application such as sqlite3 and keeps this application running:
sqlite> insert into queue ("data") VALUES("test2");
Now the C# program hangs until it gives the following error:
Unhandled Exception: Mono.Data.Sqlite.SqliteException: The database file is locked
I don't have any problems writing to the table from other instances of sqlite3 or from a C++ application I created.
If I close the sqlite3 instance then the C# application works again.
Doing a lsof /var/log/gmblog shows that sqlite3 has obtained a reader lock after performing the INSERT:
sqlite3 15578 cup 3ur REG 8,17 13312 4988505 /var/log/gmblog
Before the INSERT it didn't have this lock:
sqlite3 15578 cup 3u REG 8,17 13312 4988505 /var/log/gmblog
But as I pointed out other applications do not have any problems written to the table while other applications are using the database.
Any ideas on what is wrong with my C# code? Is it a bug in the Mono implementation of SQLite?
Update 25/11
Note that it's the dbConnection.Open(); which results in the database locked error, not the insertCommand.ExecuteNonQuery();. I.e. the following code doesn't work either:
using System;
using Mono.Data.Sqlite;
class MainClass
{
public static void Main (string[] args)
{
using (var dbConnection = new SqliteConnection (#"Data Source=/var/log/gmblog;Version=3;"))
{
dbConnection.Open();
}
}
}
Please see the SQLiteConnection documentation: "If the SQLiteConnection goes out of scope, it is not closed. Therefore, you must explicitly close the connection by calling Close.". The example shows the connection being closed in a finally block. I can't see anywhere in your code sample where you close the connection.
You should also be able to use the "using" block which automatically calls close when the object goes out of scope:
using (var dbConnection = new SqliteConnection (#"..."))
{
dbConnection.Open();
string sql = #"INSERT INTO ""queue"" (""data"") VALUES(""Test"")";
using (var insertCommand = new SqliteCommand (sql, dbConnection))
{
insertCommand.ExecuteNonQuery();
}
}
This makes sure everything gets released properly and in a timely fashion.
I found a similar discussion on sqlite's website on multi-threading. I understand that you are not multi-threading there but instead trying to get a write-lock - but still the discussion have some useful hints that maybe helpful.
Each thread then proceeds to insert a number of records, let's say 1000. The problem you will encounter is the following: one thread will get control over the database by setting a lock on the file. This is fine, but the rest of the threads will keep on failing for each attempted INSERT while the lock is active.
Solution
Test for SQLITE_BUSY, which I didn't do originally. Here's some
pseudo-code to illustrate a solution:
while (continueTrying) {
retval = sqlite_exec(db, sqlQuery, callback, 0, &msg);
switch (retval) {
case SQLITE_BUSY:
Log("[%s] SQLITE_BUSY: sleeping fow a while...", threadName);
sleep a bit... (use something like sleep(), for example)
break;
case SQLITE_OK:
continueTrying = NO; // We're done
break;
default:
Log("[%s] Can't execute \"%s\": %s\n", threadName, sqlQuery, msg);
continueTrying = NO;
break;
}
}
You may also want to try busy_timeout parameter on connection string as shown here and here.
The busy_timeout parameter is implemented as a call to
sqlite(3)_busy_timeout. The default value is 0, which means to throw a
SqliteBusyException immediately if the database is locked.