I'm working on an ASP.NET MVC application which uses Linq to SQL to connect to one of about 2000 databases. We've noticed in our profiling tools that the application spends a lot of time making connections to the databases, and I suspect this is partly due to connection pool fragmentation as described here: http://msdn.microsoft.com/en-us/library/8xx3tyca(v=vs.110).aspx
Many Internet service providers host several Web sites on a single
server. They may use a single database to confirm a Forms
authentication login and then open a connection to a specific database
for that user or group of users. The connection to the authentication
database is pooled and used by everyone. However, there is a separate
pool of connections to each database, which increase the number of
connections to the server.
There is a relatively simple way to avoid this
side effect without compromising security when you connect to SQL
Server. Instead of connecting to a separate database for each user or
group, connect to the same database on the server and then execute the
Transact-SQL USE statement to change to the desired database.
I am trying to implement this solution in Linq to Sql so we have fewer open connections, and so there is more likely to be a connection available in the pool when we need one. To do that I need to change the database each time Linq to Sql attempts to run a query. Is there any way to accomplish this without refactoring the entire application? Currently we just create a single data context per request, and that data context may open and close many connections. Each time it opens the connection, I'd need to tell it which database to use.
My current solution is more or less like this one - it wraps a SqlConnection object inside a class that inherits from DbConnection. This allows me to override the Open() method and change the database whenever a connection is opened. It works OK for most scenarios, but in a request that makes many updates, I get this error:
System.InvalidOperationException: Transaction does not match
connection
My thought was that I would then wrap a DbTransaction object in a similar way to what I did with SqlConnection, and ensure that its connection property would point back to the wrapped connection object. That fixed the error above, but introduced a new one where a DbCommand was unable to cast my wrapped connection to a SqlConnection object. So then I wrapped DbCommand too, and now I get new and exciting errors about the transaction of the DbCommand object being uninitialized.
In short, I feel like I'm chasing specific errors rather than really understanding what's going on in-depth. Am I on the right track with this wrapping strategy, or is there a better solution I'm missing?
Here are the more interesting parts of my three wrapper classes:
public class ScaledSqlConnection : DbConnection
{
private string _dbName;
private SqlConnection _sc;
public override void Open()
{
//open the connection, change the database to the one that was passed in
_sc.Open();
if (this._dbName != null)
this.ChangeDatabase(this._dbName);
}
protected override DbTransaction BeginDbTransaction(IsolationLevel isolationLevel)
{
return new ScaledSqlTransaction(this, _sc.BeginTransaction(isolationLevel));
}
protected override DbCommand CreateDbCommand()
{
return new ScaledSqlCommand(_sc.CreateCommand(), this);
}
}
public class ScaledSqlTransaction : DbTransaction
{
private SqlTransaction _sqlTransaction = null;
private ScaledSqlConnection _conn = null;
protected override DbConnection DbConnection
{
get { return _conn; }
}
}
public class ScaledSqlCommand : DbCommand
{
private SqlCommand _cmd;
private ScaledSqlConnection _conn;
private ScaledSqlTransaction _transaction;
public ScaledSqlCommand(SqlCommand cmd, ScaledSqlConnection conn)
{
this._cmd = cmd;
this._conn = conn;
}
protected override DbConnection DbConnection
{
get
{
return _conn;
}
set
{
if (value is ScaledSqlConnection)
_conn = (ScaledSqlConnection)value;
else
throw new Exception("Only ScaledSqlConnections can be connections here.");
}
}
protected override DbTransaction DbTransaction
{
get
{
if (_transaction == null && _cmd.Transaction != null)
_transaction = new ScaledSqlTransaction(this._conn, _cmd.Transaction);
return _transaction;
}
set
{
if (value == null)
{
_transaction = null;
_cmd.Transaction = null;
}
else
{
if (value is ScaledSqlTransaction)
_transaction = (ScaledSqlTransaction)value;
else
throw new Exception("Don't set the transaction of a ScaledDbCommand with " + value.ToString());
}
}
}
}
}
I don't think that's going to work off a single shared connection.
LINQ to SQL works best with Unit of Work type connections - create your connection, do your atomically grouped work and close the connection as quickly as possible and reopen for the next task. If you do that then passing in a connection string (or using custom constructor that only passes a tablename) is pretty straight forward.
If factoring your application is a problem, you could use a getter to manipulate the cached DataContext 'instance' and instead create a new instance each time you request it instead of using the cached/shared instance and inject the connection string in the Getter.
But - I'm pretty sure this will not help with your pooling issue though. The SQL Server driver caches connections based on different connection string values - since the values are still changing you're right back to having lots of connections active in the connection string cache, which likely will result in lots of cache misses and therefore slow connections.
I think I figured out a solution that works for my situation. Rather than wrapping SqlConnection and overriding Open() to change databases, I'm passing the DBContext a new SqlConnection and subscribing to the connection's StateChanged event. When the state changes, I check to see if the connection has just been opened. If so, I call SqlConnection.ChangeDatabase() to point it to the correct database. I tested this solution and it seems to work - I see only one connection pool for all the databases rather than one pool for each db that has been accessed.
I realize this isn't the ideal solution in an ideal application, but for how this application is structured I think it should make a decent improvement for relatively little cost.
I think, that the best way is making UnitOfWork pattern with Repository pattern to work with Entity Framework. Entity Framework has FirstAsync, FirstOrDefaultAsync, this helped me to fix the same bug.
https://msdn.microsoft.com/en-us/data/jj819165.aspx
Related
I am using the EF 6.1.0
I have below custom DBContex object as DBEntites
public partial class DbEntities : DbContext
{
public DbEntities()
: base("name=DbEntities")
{
////Configuration.LazyLoadingEnabled = true;
////Configuration.ProxyCreationEnabled = false;
}
//// I have ALL Entites added as below
////public virtual IDbSet<CCode> CCodes { get; set; }
}
I have the below operations on context object
using (var context = new DbEntities())
{
var entitySet = context.Set<T>();
var res = entitySet.Where<T>(predicate).ToList();
if (context.Database.Connection.State == ConnectionState.Open)
{
context.Database.Connection.Close();
}
return res;
}
But after disposing the context object still i can see a active DB Connection. On connection state condition i can see that the connection is already closed(the connection had never true).
I am using the below query to see the connection on SQL.
select db_name(dbid) , count(*) 'connections count'
from master..sysprocesses
where spid > 50 and spid != ##spid
group by db_name(dbid)
order by count(*) desc
At the below statement a sql connection count increased. But it was never down even after disposing . (I mean after using block excuted it supposed to close the connection).
var res = entitySet.Where<T>(predicate).ToList();
Any help would be greatly appreciated.
As was figured out in comments, the reason is indeed connection pooling performed by .NET. .NET maintains a pool of connections for every connection string you use in your application, for perfomance reasons (because opening and closing connections often might be costly in terms of perfomance). That pool has certain minimum and maximum size (controlled by MinPoolSize and MaxPoolSize connection string parameters). When you open a connection (via SqlConnection.Open) - it might be taken out of the pool and not really opened afresh. When you close connection (which is also done by disposing EF context) - connection might be put into the pool instead, and not really closed. When connection is idle for certain time (about 5 minutes) - it might be removed from the pool.
If you (for some reason) want to avoid that, you can either set MaxPoolSize to 0 for your connection string, or clear pool explicitly by SqlConnection.ClearPool or SqlConnection.ClearAllPools.
Just to register my experience here. I had same problem with EF Core 6 + Postgres and the solution was just to disable "Pooling" in the connection string.
Pooling=false;
I have .NET application with Oracle Database and NHibernate.
I need to handle some "Connected" event which raised before NHibernate executed first DbCommand with currently used OracleConnection. This is due to the necessity of primary context initialization. I need to be sure that before executing any command context was initialized.
Is there such a possibility in NHibernate?
P.S. I can not use Oracle ON LOGON TRIGGER for this purpose
UPD. The solution is:
public class CustomConnectionProvider : DriverConnectionProvider
{
public override System.Data.IDbConnection GetConnection()
{
var conn = (OracleConnection)base.GetConnection();
//init context
return conn;
}
}
You can use a custom DriverConnectionProvider. Normally this is used to dynamically change the connection but it seems like a good fit for your scenario. Since it's responsible for creating and opening the connection, you are guaranteed that no operations have occurred before it executes.
I am designing a database wrapper for C#.
Below are the two options I have:
Option A:
class DBWrapper:IDisposable
{
private SqlConnection sqlConn;
public DBWrapper()
{
sqlConn = new SqlConnection("my connection string");
sqlConn.Open();
}
public DataTable RunQuery(string Sql)
{
implementation......
}
public Dispose()
{
if(sqlConn != null)
sqlConn.Close();
}
}
Option B:
class DBWrapper
{
public DBWrapper()
{
}
public DataTable RunQuery(string Sql)
{
SqlConnection sqlConn = new SqlConnection("my connection string");
.....implementation......
sqlConn.Close();
}
}
For option A connection is opened when class is instantiated. So no matter how many times the caller calls RunQuery the connection is always ready. But If the application instantiates DBWrapper early in the application, the connection will be just opened and doing nothing until the application is finished. Also, it could have many DBWrapper instantiated during the execution. So, it's kinda wasting resources.
For option B it doesn't have the problem option A has, but the a new connection has to be opened and closed everytime the caller calls RunQuery. I am not sure how much it will hurt the performance.
Please share your expertise. Thank you for reading.
For performance reasons, you'll definitely not want to go with Option B (at least in the cases I experienced.)
Let me suggest Option C:
class DBWrapper:IDisposable {
private SqlConnection sqlConn;
public void EnsureConnectionIsOpen()
{
if (sqlConn == null)
{
sqlConn = new SqlConnection("my connection string");
sqlConn.Open();
}
}
public DataTable RunQuery(string Sql)
{
EnsureConnectionIsOpen();
implementation......
}
public Dispose()
{
if(sqlConn != null)
sqlConn.Close();
}
}
You might consider using the singleton pattern to make sure there is only one instance of your DBWrapper.
A few comments worth considering:
In the approach where you manage a (perhaps) long-lived connection, it is important to check whether the connection is open before running a query. I've run into issues before where NETCF closed unused connections after a while.
In the approach where you open a new connection per-query, ensure that your connection, commands, and (if used) data readers are all properly wrapped in using statements or try/finally+dispose() blocks to free up connections and locks.
Happy coding!
Garbage collector is triggered under rather complex conditions but basically it is invoked when memory exceeds some limit, it is invoked periodically as well but the period is not constant. You never can be sure when exactly garbage collector disposes and consequently (in another run) destroys the object. One thing you can be sure is the fact that garbage collector will never dispose and destroy the object that still has references. For example object that is referenced via static variables on the class neither will be disposed nor destroyed.
Option B is more transactional, which has its advantages. ADO.NET uses implicit connection pooling, so you do not have to worry about creating new instances of SqlConnection frequently.
You should consider whether you are using a connected or disconnected data model; as the second approach lends itself better to a disconnected model.
But as i've said above, connection pooling means that it makes virtually no difference in practical terms.
You could have an option C where the database is opened on request in RunQuery (if it is not open) and closed on disposed (when it was opened). That way the database is only opened when really needed and will be opened only once.
So in pseudo code:
class DBWrapper
{
public DBWrapper()
{
}
SqlConnection sqlConn = null;
public DataTable RunQuery(string Sql)
{
if(sqlConn == null) sqlConn = new SqlConnection("my connection string");
.....implementation......
}
public Dispose()
{
if(sqlConn != null)
sqlConn.Close();
}
}
Also mind that the moment Dispose is called is not always directly after the object is not needed anymore (for example a function variable after the function is used). As far as I know it will be executed when the garbage collector collects the object (which is not directly). But I am not totally sure about this. This behaviour might also differ between web and non web application.
In my server application I want to use DB (SQL Server) but I am quite unsure of the best method. There are clients whose requests comes to threadpool and so their processing is async. Every request usually needs to read or write to DB, so I was thinking about static method which would create connection, execute the query and return the result. I'm only afraid whether opening and closing connection is not too slow and whether some connection limit could not be reached? Is this good approach?
IMHO the best is to rely on the ADO.NET connection pooling mechanism and don't try to handle database connections manually. Write your data access methods like this:
public void SomeMethod()
{
using (var connection = new SqlConnection(connectionString))
using (var command = connection.CreateCommand())
{
connection.Open();
command.CommandText = "SELECT Field1 FROM Table1";
using (var reader = command.ExecuteReader())
{
while(reader.Read())
{
// do something with the results
}
}
}
}
Then you can call this method from wherever you like, make it static, call it from threads whatever. Remember that calling Dispose on the connection won't actually close it. It will return it to the connection pool so that it can be reused.
Surprised that no one mentioned connection pooling. If you think you are going to have a large number of requests, why not just setup a pool with a min pool size set to say 25 (arbitrary number here, do not shoot) and max pool size set to say 200.
This will decrease the number of connection attempts and make sure that if you are not leaking connection handles (something that you should take explicit care to not let happen), you will always have a connection waiting for you.
Reference article on connection pooling: http://msdn.microsoft.com/en-us/library/8xx3tyca.aspx
Another side note, why the need to have the connection string in the code? Set it in the web.config or app.config for the sake of maintainability. I had to "fix" code that did such things and I always swore copiously at the programmer responsible for such things.
I have had exactly the same problem like you. Had huge app that i started making multithreaded. Benefit over having one connection open and being reused is that you can ask DB multiple times for data as new connection is spawned on request (no need to wait for other threads to finish getting data), and if for example you loose connection to sql (and it can happen when network goes down for a second or two) you will have to always check if connection is open before submitting query anyway.
This is my code for getting Database rows in MS SQL but other stuff should be done exactly the same way. Keep in mind that the sqlConnectOneTime(string varSqlConnectionDetails) has a flaw of returning null when there's no connection so it needs some modifications for your needs or the query will fail if sql fails to establish connection. You just need to add proper code handling there :-) Hope it will be useful for you :-)
public const string sqlDataConnectionDetails = "Data Source=SQLSERVER\\SQLEXPRESS;Initial Cata....";
public static string sqlGetDatabaseRows(string varDefinedConnection) {
string varRows = "";
const string preparedCommand = #"
SELECT SUM(row_count) AS 'Rows'
FROM sys.dm_db_partition_stats
WHERE index_id IN (0,1)
AND OBJECTPROPERTY([object_id], 'IsMsShipped') = 0;";
using (var varConnection = Locale.sqlConnectOneTime(varDefinedConnection))
using (var sqlQuery = new SqlCommand(preparedCommand, varConnection))
using (var sqlQueryResult = sqlQuery.ExecuteReader())
while (sqlQueryResult.Read()) {
varRows = sqlQueryResult["Rows"].ToString();
}
return varRows;
}
public static SqlConnection sqlConnectOneTime(string varSqlConnectionDetails) {
SqlConnection sqlConnection = new SqlConnection(varSqlConnectionDetails);
try {
sqlConnection.Open();
} catch (Exception e) {
MessageBox.Show("Błąd połączenia z serwerem SQL." + Environment.NewLine + Environment.NewLine + "Błąd: " + Environment.NewLine + e, "Błąd połączenia");
}
if (sqlConnection.State == ConnectionState.Open) {
return sqlConnection;
}
return null;
}
Summary:
Defined one global variable with ConnectionDetails of your SQL Server
One global method to make connection (you need to handle the null in there)
Usage of using to dispose connection, sql query and everything when the method of reading/writing/updating is done.
The one thing that you haven't told us, that would be useful for giving you an answer that's appropriate for you is what level of load you're expecting your server application to be under.
For pretty much any answer to the above question though, the answer would be that you shouldn't worry about it. ADO.net/Sql Server provides connection pooling which removes some of the overhead of creating connections from each "var c = new SqlConnection(connectionString)" call.
Please help!
Background info
I have a WPF application which accesses a SQL Server 2005 database. The database is running locally on the machine the application is running on.
Everywhere I use the Linq DataContext I use a using { } statement, and pass in a result of a function which returns a SqlConnection object which has been opened and had an SqlCommand executed using it before returning to the DataContext constructor.. I.e.
// In the application code
using (DataContext db = new DataContext(GetConnection()))
{
... Code
}
where getConnection looks like this (I've stripped out the 'fluff' from the function to make it more readable, but there is no additional functionality that is missing).
// Function which gets an opened connection which is given back to the DataContext constructor
public static System.Data.SqlClient.SqlConnection GetConnection()
{
System.Data.SqlClient.SqlConnection Conn = new System.Data.SqlClient.SqlConnection(/* The connection string */);
if ( Conn != null )
{
try
{
Conn.Open();
}
catch (System.Data.SqlClient.SqlException SDSCSEx)
{
/* Error Handling */
}
using (System.Data.SqlClient.SqlCommand SetCmd = new System.Data.SqlClient.SqlCommand())
{
SetCmd.Connection = Conn;
SetCmd.CommandType = System.Data.CommandType.Text;
string CurrentUserID = System.String.Empty;
SetCmd.CommandText = "DECLARE #B VARBINARY(36); SET #B = CAST('" + CurrentUserID + "' AS VARBINARY(36)); SET CONTEXT_INFO #B";
try
{
SetCmd.ExecuteNonQuery();
}
catch (System.Exception)
{
/* Error Handling */
}
}
return Conn;
}
I do not think that the application being a WPF one has any bearing on the issue I am having.
The issue I am having
Despite the SqlConnection being disposed along with the DataContext in Sql Server Management studio I can still see loads of open connections with :
status : 'Sleeping'
command : 'AWAITING COMMAND'
last SQL Transact Command Batch : DECLARE #B VARBINARY(36); SET #B = CAST('GUID' AS VARBINARY(36)); SET CONTEXT_INFO #B
Eventually the connection pool gets used up and the application can't continue.
So I can only conclude that somehow running the SQLCommand to set the Context_Info is meaning that the connection doesn't get disposed of when the DataContext gets disposed.
Can anyone spot anything obvious that would be stopping the connections from being closed and disposed of when the DataContext they are used by are disposed?
From MSDN (DataContext Constructor (IDbConnection)):
If you provide an open connection, the
DataContext will not close it.
Therefore, do not instantiate a
DataContext with an open connection
unless you have a good reason to do
this.
So basically, it looks like your connections are waiting for GC to finalize them before they will be released. If you have lots of code that does this, one approach might be to overide Dispose() in the data-context's partial class, and close the connection - just be sure to document that the data-context assumes ownership of the connection!
protected override void Dispose(bool disposing)
{
if(disposing && this.Connection != null && this.Connection.State == ConnectionState.Open)
{
this.Connection.Close();
this.Connection.Dispose();
}
base.Dispose(disposing);
}
Personally, I would happily give it (regular data-context, w/o the hack above) an open connection as long as I was "using" the connection (allowing me to perform multiple operations) - i.e.
using(var conn = GetConnection())
{
// snip: some stuff involving conn
using(var ctx = new FooContext(conn))
{
// snip: some stuff involving ctx
}
// snip: some more stuff involving conn
}
The SqlProvider used by the LINQ DataContext only closes the SQL connection (through SqlConnectionManager.DisposeConnection) if it was the one to open it. If you give an already-open SqlConnection object to the DataContext constructor, it will not close it for you. Thus, you should write:
using (SqlConnection conn = GetConnection())
using (DataContext db = new DataContext(conn))
{
... Code
}
I experienced the same issue using the Entity Framework. My ObjectContext was wrapped around a using block.
A connection was established when I called SaveChanges(), but after the using statement was out of scope, I noticed that SQL Management Studio still had a "AWAITING COMMAND" for the .NET SQL Client.
It looks like this has to do with the behavior of the ADO.NET provider which has connection pooling turned on by default.
From "Using Connection Pooling with SQL Server" on MSDN (emphasis mine):
Connection pooling reduces the number of times that new connections need to be opened. The pooler maintains ownership of the physical connection. It manages connections by keeping alive a set of active connections for each given connection configuration. Whenever a user calls Open on a connection, the pooler looks to see if there is an available connection in the pool. If a pooled connection is available, it returns it to the caller instead of opening a new connection. When the application calls Close on the connection, the pooler returns it to the pooled set of active connections instead of actually closing it. Once the connection is returned to the pool, it is ready to be reused on the next Open call.
Also ClearAllPools and ClearPool seems useful to explicitly close all pooled connections if needed.
I think the connection, while no longer referenced, is waiting for the GC to dispose of it fully.
Solution:
Create your own DataContext class which derives from the auto-generated one. (rename the base one so you don't have to change any other code).
In your derived DataContext - add a Dispose() function. In that - dispose the inner connection.
Well thanks for the help chaps, it has been solved now..
Essentially I took elements of most of the answers above and implemented the DataContext constructor as above (I already had overloaded the constructors so it wasn't a big change).
// Variable for storing the connection passed to the constructor
private System.Data.SqlClient.SqlConnection _Connection;
public DataContext(System.Data.SqlClient.SqlConnection Connection) : base(Connection)
{
// Only set the reference if the connection is Valid and Open during construction
if (Connection != null)
{
if (Connection.State == System.Data.ConnectionState.Open)
{
_Connection = Connection;
}
}
}
protected override void Dispose(bool disposing)
{
// Only try closing the connection if it was opened during construction
if (_Connection!= null)
{
_Connection.Close();
_Connection.Dispose();
}
base.Dispose(disposing);
}
The reason for doing this rather than some of the suggestions above is that accessing this.Connection in the dispose method throws a ObjectDisposedException.
And the above works as well as I was hoping!
The Dispose should close the connections, as MSDN points out:
If the SqlConnection goes out of
scope, it won't be closed. Therefore,
you must explicitly close the
connection by calling Close or
Dispose. Close and Dispose are
functionally equivalent. If the
connection pooling value Pooling is
set to true or yes, the underlying
connection is returned back to the
connection pool. On the other hand, if
Pooling is set to false or no, the
underlying connection to the server is
closed.
My guess would be that your problem has something to do with GetContext().