In my application, I am querying database with same sql every one second. I need to know what measure should I take. Will it ever cause any problem like " The
timeout period elapsed prior to obtaining a connection from the pool." or any other like that?
Currently, i am creating a new connection every second and the disposing it after it is used. Should I reuse connection in this case.
Whats the best approach?
IMHO best practice is to pump-and-dump connections as quickly as possible - use them to get access to what you need and close them right away. Your enemy performance-wise isn't the overhead it takes to create a connection (although there is some involved there) - it's locking a table in your database.
If you're looking to optimize your application, you should try to implement some sort of caching mechanism that saves you from having to make a round-trip to the database for each lookup. That would be to your benefit performance-wise.
Another thing you can do is use read-only connections where you can - they require less overhead than traditional ones and will improve your performance also.
You should definitely open and close the connection each time. Indeed, if your using block has much code after the last use of the connection, call Close() to get it back in the pool as soon as possible. That way the chance of another use not needing to open a completely new connection is reduced (see What does "opening a connection" actually mean? for a bit more on when a real connection is opened and when one is taken from the pool).
Is this "once a second" an average across different threads, or all on the one thread? If it's all on the one thread it doesn't matter, indeed it might even be slightly faster to keep the connection object open, because either way there won't be contention for it.
I would certainly consider caching results, though this has downsides in memory use, along with potentially complicated issues about when the cached results need to be refreshed - really this could be anywhere from trivial to impossible depending on just what you are doing.
It's also clearly a query to go that extra mile when optimising.
Why do you need to do this?
You could try caching the data to reduce the load on your database. Do you need data that is 1 second old, or is 5 seconds ok.
Closing the connection after each time you use it is OK. It does not really get closed, it just goes back into the connection pool.
If the library you're using does indeed pool the connections for you then it doesn't make a difference. If not, then it would be better to use the same connection multiple times. Creating a connection is time consuming.
A few questions...
What data are you getting from the database?
Could that data be stored in application memory?
There is no problem in such approach if you dispose connections like this:
using (SqlConnection cnn = new SqlConnection(ConnectionString))
{
SqlCommand cmd = new SqlCommand(commandText, cnn)
{
CommandType = CommandType.Text
};
DataSet ds = new DataSet();
SqlDataAdapter da = new SqlDataAdapter(cmd);
da.Fill(ds);
return ds;
}
The only problem may happen - is decreasing your db performance if your hardware isn't good enought.
Related
I have an application that connect to a SQL Server database with high frequency. Inside this service, there are many scheduled tasks that run every second, and each time I'm executing some query.
I don't understand which solution is better in this condition.
Opening a single SqlConnection and keeping it open while application is running and execute all query with that connection
Each time I want to execute query, opening a new connection and after query execution, close the connection (does this solution suitable for so many scheduled task that runs every 1 second?)
I tried second solution, but is there any better choice?
How do ORMs like EF manage connections?
As you see i have many service. I cant change interval and the interval is important for me. but the code makes so many calls and im following a better way manage connection over database. Also I'm making connection with Using Statement.
Is there any better solution?
you should use SQL Connection Pool feature for that.
It automatically manages in the background if a connection needs to be open or can be reused.
Documentation: https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/sql-server-connection-pooling?source=recommendations
Example copied from that page
using (SqlConnection connection = new SqlConnection(
"Integrated Security=SSPI;Initial Catalog=Northwind"))
{
connection.Open();
// Pool A is created.
}
using (SqlConnection connection = new SqlConnection(
"Integrated Security=SSPI;Initial Catalog=pubs"))
{
connection.Open();
// Pool B is created because the connection strings differ.
}
using (SqlConnection connection = new SqlConnection(
"Integrated Security=SSPI;Initial Catalog=Northwind"))
{
connection.Open();
// The connection string matches pool A.
}
By using the "using" statement, application checks if a connection in this pool can be reused before opening a new connection. So the overhead of opening and closing the connections disappears.
But after your last edit you seem to have other problems in your current architecture. Like the other poster recommends you can try to use the "with (nolock)" parameter in your sql statements. It creates dirty reads, but maybe that's ok for your application.
Alternatively if all your services use the same select statement maybe a stored procedure or a caching mechanism could help.
I assume that you are already opening/closing your SQL connections in either a "using" statement or explicitly in your code ( try/catch/finally ). If so you are already making use of connection pooling as it is enabled in ADO.Net by default ("By default, connection pooling is enabled in ADO.NET").
Therefore I don't think that your problem is so much a connection/resource problem as it is a database concurrency issue. I assume it to be either 1 of 2 issues :
Your code is making so many calls to the SQL server that it is exhausting all the available connections and nobody else can get one
Your code is locking tables in SQL that is causing other code/applications to timeout
If it is case 1, try and redesign your code to be "less chatty" to the database. Instead of making several inserts/updates per second, perhaps buffer the changes and make a single insert/update every 3-5 seconds in batch mode ( obvs if possible ). Or maybe your SQL statements are taking longer than 1 second to execute and you are calling them every second causing in a backlog scenario?
If it is case 2, try and redesign the SQL tables in such a way that the "reading" applications are not influenced by the "writing" application. Normally this involves a service that periodically writes aggregated data to a read-only table for viewing or at very least adding a "WITH(NOLOCK)" hint to the select clauses to allow dirty reads ( i.e. it wont lock the table to read, but may result in slightly out of date dataset i.e. eventual consistency )
Good luck
Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
This is the first ASP.net site I developed a long time ago, it has this code at the top of a lot of pages (and in a lot of methods).
cn = new SqlConnection(ConfigurationManager.ConnectionStrings["LocalSqlServer"].ToString());
cn.Open();
A lot of pages don't have:
cn.Close();
Also none of the pages do using(SqlConnection...), (although all the data readers are in using blocks).
First question is, is the the primary candiate for the error?
Second question is, is the best way to solve this refactoring/manually searching for unclosed connections? I know it's an ugly hack, but the time spent refactoring will be huge, but can we set a scheduled task to recycle the connection pool once a day at 3am or something?
Yes, that is the primary cause of the error. Currently, many of those connections will wait until the next GC to re-pool the underlying connection. You will exhaust the pool (and database connections) pretty quickly.
The best way of refactoring this is to add the missing using, such that the connection is scoped. Personally I'd also refactor that to a single method, i.e.
using(var cn = SomeUtilityClass.GetOpenConnection())
{...}
Closing the reader does little unless the reader is marked to close the connection; and having the data-reader close the connection (via a behaviour) sort of works, but it assumes you get as far as consuming the reader - it won't necessarily behave well in error scenarios.
Even I have encountered this error in the application that I once worked on. The problem that I identified was the same - no using statements and no close calls.
I would advise you to search the whole project for SqlConnection and then include the SqlConnection, SqlCommand and SqlDataAdapter in using statements and also do a connection.close within the sqlconnection using statement. Together with this in the config file increase the timeout of the connection within the connection string. You can also you CommandTimeout property of SqlCommand.
I am using old school ADO.net with C# so there is a lot of this kind of code. Is it better to make one function per query and open and close db each time, or run multiple queries with the same connection obect? Below is just one query for example purpose only.
using (SqlConnection connection = new SqlConnection(ConfigurationManager.ConnectionStrings["DBConnectMain"].ConnectionString))
{
// Add user to database, so they can't vote multiple times
string sql = " insert into PollRespondents (PollId, MemberId) values (#PollId, #MemberId)";
SqlCommand sqlCmd = new SqlCommand(sql, connection);
sqlCmd.Parameters.Add("#PollId", SqlDbType.Int);
sqlCmd.Parameters["#PollId"].Value = PollId;
sqlCmd.Parameters.Add("#MemberId", SqlDbType.Int);
sqlCmd.Parameters["#MemberId"].Value = Session["MemberId"];
try
{
connection.Open();
Int32 rowsAffected = (int)sqlCmd.ExecuteNonQuery();
}
catch (Exception ex)
{
//Console.WriteLine(ex.Message);
}
}
Well, you could measure; but as long as you are using the connections (so they are disposed even if you get an exception), and have pooling enabled (for SQL server it is enabled by default) it won't matter hugely; closing (or disposing) just returns the underlying connection to the pool. Both approaches work. Sorry, that doesn't help much ;p
Just don't keep an open connection while you do other lengthy non-db work. Close it and re-open it; you may actually get the same underlying connection back, but somebody else (another thread) might have made use of it while you weren't.
For most cases, opening and closing a connection per query is the way to go (as Chris Lively pointed out). However, There are some cases where you'll run into performance bottlenecks with this solution though.
For example, when dealing with very large volumes of relatively quick to execute queries that are dependent on previous results, I might suggest executing multiple queries in a single connection. You might encounter this when doing batch processing of data, or data massaging for reporting purposes.
Always be sure to use the 'using' wrapper to avoid mem leaks though, regardless of which pattern you follow.
If the methods are structured such that a single command is executed within a single method, then Yes: instantiate and dispose of the connection for each command.
If the methods are structured such that you have multiple commands executed in the same block of code, then the outer block needs to be the using clause for the connection.
ADO is very good about connection pooling so instantiating and disposing of the command object is going to be extremely fast and really won't impact performance.
As an example, we have a few pages that will execute update to 50 queries in order to compose the page. Because there is branching code to determine the queries to run, we have each of them wrapped with their own using (connection...) clauses.
We once ripped those out and grabbed one connection object and passed it to the individual methods. This had exactly zero performance improvement while complicating the hell out of the code with all the exception clauses every where to ensure the connection was properly disposed at the end. At the end of the test, we rolled back the code to how it was before. Much cleaner to know exactly what was going on and when a connection was being used.
Well, as always, it depends. If you have 5 database call to make within the same method call, you should probably use a single connection.
However, holding onto connection while nothing is happening isn't usually advised from a scalability standpoint.
ADO.NET is old school now? Wow, you just made me feel old. To me Rogue Wave ODBC using Borland C++ on Windows 3.1 is old school.
To answer, in general you want to understand how your data drivers work. Understand such concepts as connection pooling and learn to profile the transaction costs associate with connecting / disconnecting and executing queries. Then take that knowledge and apply it it your situation.
I have my business-logic implemented in simple static classes with static methods. Each of these methods opens/closes SQL connection when called:
public static void DoSomething()
{
using (SqlConnection connection = new SqlConnection("..."))
{
connection.Open();
// ...
connection.Close();
}
}
But I think passing the connection object around and avoiding opening and closing a connection saves performance. I made some tests long time ago with OleDbConnection class (not sure about SqlConnection), and it definitely helped to work like this (as far as I remember):
//pass around the connection object into the method
public static void DoSomething(SqlConnection connection)
{
bool openConn = (connection.State == ConnectionState.Open);
if (!openConn)
{
connection.Open();
}
// ....
if (openConn)
{
connection.Close();
}
}
So the question is - should I choose the method (a) or method (b) ? I read in another stackoverflow question that connection pooling saved performance for me, I don't have to bother at all...
PS. It's an ASP.NET app - connections exist only during a web-request. Not a win-app or service.
Stick to option a.
The connection pooling is your friend.
Use Method (a), every time. When you start scaling your application, the logic that deals with the state will become a real pain if you do not.
Connection pooling does what it says on the tin. Just think of what happens when the application scales, and how hard would it be to manually manage the connection open/close state. The connection pool does a fine job of automatically handling this. If you're worried about performance think about some sort of memory cache mechanism so that nothing gets blocked.
Always close connections as soon as you are done with them, so they underlying database connection can go back into the pool and be available for other callers. Connection pooling is pretty well optimised, so there's no noticeable penalty for doing so. The advice is basically the same as for transactions - keep them short and close when you're done.
It gets more complicated if you're running into MSDTC issues by using a single transaction around code that uses multiple connections, in which case you actually do have to share the connection object and only close it once the transaction is done with.
However you're doing things by hand here, so you might want to investigate tools that manage connections for you, like DataSets, Linq to SQL, Entity Framework or NHibernate.
Disclaimer: I know this is old, but I found an easy way to demonstrate this fact, so I'm putting in my two cents worth.
If you're having trouble believing that the pooling is really going to be faster, then give this a try:
Add the following somewhere:
using System.Diagnostics;
public static class TestExtensions
{
public static void TimedOpen(this SqlConnection conn)
{
Stopwatch sw = Stopwatch.StartNew();
conn.Open();
Console.WriteLine(sw.Elapsed);
}
}
Now replace all calls to Open() with TimedOpen() and run your program. Now, for each distinct connection string you have, the console (output) window will have a single long running open, and a bunch of very fast opens.
If you want to label them you can add new StackTrace(true).GetFrame(1) + to the call to WriteLine.
There are distinctions between physical and logical connections. DbConnection is a kind of logical connection and it uses underlying physical connection to Oracle. Closing/opening DbConnection doesn't affect your performance, but makes your code clean and stable - connection leaks are impossible in this case.
Also you should remember about cases when there are limitations for parallel connections on db server - taking that into account it is necessary to make your connections very short.
Connection pool frees you from connection state checking - just open, use and immediately close them.
Normally you should keep one connect for each transaction(no parallel computes)
e.g when user execute charge action, your application need find user's balance first and update it, they should use same connection.
Even if ado.net has its connection pool, dispatching connection cost is very low, but reuse connection is more better choice.
Why not keep only one connection in application
Because the connection is blocking when you execute some query or command,
so that means your application is only doing one db operation at sametime,
how poor performance it is.
One more issue is that your application will always have a connection even though your user is just open it but no operations.If there are many user open your application, db server will cost all of its connection source in soon while your users have not did anything.
The following code executes a simple insert command. If it is called 2,000 times consecutively (to insert 2,000 rows) an OleDbException with message = "System Resources Exceeded" is thrown. Is there something else I should be doing to free up resources?
using (OleDbConnection conn = new OleDbConnection(connectionString))
using (OleDbCommand cmd = new OleDbCommand(commandText, conn))
{
conn.Open();
cmd.ExecuteNonQuery();
}
The system resources exceeded error is not coming from the managed code, its coming from you killing your database (JET?)
You are opening way too many connections, way too fast...
Some tips:
Avoid round trips by not opening a new connection for every single command, and perform the inserts using a single connection.
Ensure that database connection pooling is working. (Not sure if that works with OLEDB connections.)
Consider using a more optimized way to insert the data.
Have you tried this?
using (OleDBConnection conn = new OleDBConnection(connstr))
{
while (IHaveData)
{
using (OldDBCommand cmd = new OldDBCommand())
{
cmd.Connection = conn;
cmd.ExecuteScalar();
}
}
}
I tested this code out with an Access 2007 database with no exceptions (I went as high as 13000 inserts).
However, what I noticed is that it is terribly slow as you are creating a connection every time. If you put the "using(connection)" outside the loop, it goes much faster.
In addition to the above (connecting to the database only once), I would also like to make sure you're closing and disposing of your connections. As most objects in c# are managed wrt memory, connections and streams don't have this luxury always, so if objects like this aren't disposed of, they are not guaranteed to be cleaned up. This has the added effect of leaving that connection open for the life of your program.
Also, if possible, I'd look into using Transactions. I can't tell what you're using this code for, but OleDbTransactions are useful when inserting and updating many rows in a database.
I am not sure about the specifics but I have ran across a similar problem. We utilize an Access database with IIS to serve our clients. We do not have very many clients but there are alot of connections being opened and closed during a single session. After about a week of work, we recieve the same error and all connection attempts fail. To correct the problem, all we had to do was restart the worker processes.
After some research, I found (of course) that Access does not perform well in this environment. Resources do not get released correctly and over time the executable will run out. To solve this problem, we are going to move to an Oracle database. If this does not fix the problem, I will keep you updated on my findings.
This could be occurring because you are not disposing the Connection and Command object created. Always Dispose the object at the end.
OledbCommand.Dispose();