I am working on some code that I would like to access an Oracle database inside of a Parallel.For loop. The loop will run for several minutes, and then result in the error:
"Attempted to read or write protected memory. This is often an
indication that other memory is corrupt."
There is no inner exception. Inside my Parallel.For loop, I am creating an opening the database connection as local objects. My code looks like this:
static void CheckSinglePath(Path p)
{
string sqlBase = "select * from table where hour = #HOUR#";
Parallel.For (1, 24, i =>
{
DBManager localdbm = new DBManager();
string sql = sqlBase;
sql = sql.Replace("#HOUR#", i.ToString());
OracleDataReader reader = db.GetData(sql);
if (reader.Read())
{
//do some stuff
}
reader.Close();
});
}
class DBManager
{
OracleConnection conn;
OracleCommand cmd;
public DBManager()
{
string connStr = "blahblahblah;Connection Timeout=600;";
conn = new OracleConnection(connStr);
conn.Open();
cmd = conn.CreateCommand();
}
public OracleDataReader GetData(string sql)
{
cmd.CommandText = sql;
return cmd.ExecuteReader();//EXCEPTION HERE!
}
}
What am I doing wrong? How can I create 24 parallel Oracle connections to process the data? I'm guessing there is some sort of race condition or memory leak that is going on here which I don't fully understand because it seems to be coming from inside the OracleConnection object. Is the database connection not threadsafe? I tried changing the connection string to use a connection pool and that didn't change anything.
Memory problems is always caused by wrong resources usage. You do not properly release your connections after the loop exit.
You need to implement IDisposable interface and after that you need rewrite your code in such manner with using keyword:
// dispose the connection after command finished
using (var localdbm = new DBManager())
{
var sql = sqlBase;
sql = sql.Replace("#HOUR#", i.ToString());
using (var reader = db.GetData(sql))
{
if (reader.Read())
{
//do some stuff
}
// no need to close reader
// as it's being disposed inside using directive
}
}
Related
Suppose I have a database method that looks like this:
public void insertRow(SqlConnection c)
{
using (var cmd = new SqlCommand("insert into myTable values(#dt)",c))
{
cmd.Parameters.Add(new SqlParameter("#dt",DbType.DateTime)).Value = DateTime.Now;
cmd.ExecuteNonQuery();
}
}
Now suppose I want to test this method. So, I write a test case that attempts to wrap this method inside a transaction, so that I can rollback the change after testing the result of the insertion:
public void testInsertRow()
{
SqlConnection c = new SqlConnection("connection.string.here");
SqlTransaction trans = c.BeginTransaction();
insertRow();
// do something here to evaluate what happened, e.g. query the DB
trans.Rollback();
}
This however fails to work, because:
ExecuteNonQuery requires the command to have a transaction when the connection assigned to the command is in a pending local transaction. The Transaction property of the command has not been initialized.
Is there a way to accomplish this, without having to rewrite every single database method to accept a transaction, and then rewrite every single call to pass null into the method for the transaction?
For example, this would work:
public void insertRow(SqlConnection c, SqlTransaction t)
{
using (var cmd = new SqlCommand("insert into myTable values(#dt)",c))
{
if (t != null) cmd.Transaction = t;
cmd.Parameters.Add(new SqlParameter("#dt",DbType.DateTime)).Value = DateTime.Now;
cmd.ExecuteNonQuery();
}
c.Close();
}
But then, I have to either rewrite each and every call to the database method to include that null parameter, or write override signatures for each and every database method that automatically pass in a null, e.g.
public void insertRow(SqlConnection c) { insertRow(c, null); }
What's the best way to allow transaction-based testing of database calls?
You can use a TransactionScope to add the connections automatically in to a transaction
public void testInsertRow()
{
using(TransactionScope scope = new TransactionScope(TransactionScopeOption.RequiresNew))
{
SqlConnection c = new SqlConnection("connection.string.here");
insertRow(c);
// do something here to evaluate what happened, e.g. query the DB
//do not call scope.Complete() so we get a rollback.
}
}
Now this will cause tests to block each other if you have multiple parallel tests running. If you database is set up to support it you could do Snapshot isolation so updates from concurrent tests won't lock each other out.
public void testInsertRow()
{
using(TransactionScope scope = new TransactionScope(TransactionScopeOption.RequiresNew,
new TransactionOptions(IsolationLevel = IsolationLevel.Snapshot))
{
SqlConnection c = new SqlConnection("connection.string.here");
insertRow(c);
// do something here to evaluate what happened, e.g. query the DB
//do not call scope.Complete() so we get a rollback.
}
}
I have following code in a model that I have made to work with all DB related activities so I don’t get to write the redundant piece of code every time I want to work with the Database, this model class looks like this:
Shared.cs:
private static string ConStr
{
get
{
Shared shrObj = new Shared();
return shrObj.DecryptString(ConfigurationManager.ConnectionStrings["constr"].ConnectionString);
}
}
public static SqlConnection SqlCon = new SqlConnection(ConStr);
public static SqlDataReader ORC(SqlCommand sqlCom)
{
SqlDataReader sqlReader=null;
try
{
SqlCon.Open();//ERROR HERE
//The ConnectionString property has not been initialized.
sqlReader = sqlCom.ExecuteReader();
}
catch (Exception ex)
{
WriteToFile(DateTime.Now.ToString("dd-MM-yyyy HH:mm:ss") + " | " + ex, 0);
return sqlReader;
}
return sqlReader;
}
//Functions for closing connections…
//Functions for reading scalar…
//etc.
And now from other models when I try to use these function like this:
User.cs
internal IEnumerable<User> GetUser()
{
var sqlCom = new SqlCommand("SELECT * FROM [user];", Shared.SqlCon);
using (var blgs = Shared.ORC(sqlCom))
{
// ……. … …
}
}
}
One more thing that confusing me is when this data read function is called for first time from here while logging in, it works fine it’s always the second time call which creates problem.
May be I am just not playing write with static properties. But I have to keep it static because its being used by the SqlConnection object which itself is static to be used directly from outside the class.
Kindly help.
In a web app, it's usually better to open and close a connection via a using statement like so:
using (SqlConnection conn = new SqlConnection(connectionString))
{
conn.Open();
// do some stuff
}
Don't reuse a sqlconnection as it can lead to unexpected timeouts in your webapp.
I'm looking to figure out the best way to execute a database query using the least amount of boilerplate code. The method suggested in the SqlCommand documentation:
private static void ReadOrderData(string connectionString)
{
string queryString = "SELECT OrderID, CustomerID FROM dbo.Orders;";
using (SqlConnection connection = new SqlConnection(connectionString))
{
SqlCommand command = new SqlCommand(queryString, connection);
connection.Open();
SqlDataReader reader = command.ExecuteReader();
try
{
while (reader.Read())
{
Console.WriteLine(String.Format("{0}, {1}", reader[0], reader[1]));
}
}
finally
{
reader.Close();
}
}
}
mostly consists of code that would have to be repeated in every method that interacts with the database.
I'm already in the habit of factoring out the establishment of a connection, which would yield code more like the following. (I'm also modifying it so that it returns data, in order to make the example a bit less trivial.)
private SQLConnection CreateConnection()
{
var connection = new SqlConnection(_connectionString);
connection.Open();
return connection;
}
private List<int> ReadOrderData()
{
using(var connection = CreateConnection())
using(var command = connection.CreateCommand())
{
command.CommandText = "SELECT OrderID FROM dbo.Orders;";
using(var reader = command.ExecuteReader())
{
var results = new List<int>();
while(reader.Read()) results.Add(reader.GetInt32(0));
return results;
}
}
}
That's an improvement, but there's still enough boilerplate to nag at me. Can this be reduced further? In particular, I'd like to do something about the first two lines of the procedure. I don't feel like the method should be in charge of creating the SqlCommand. It's a tiny piece of repetition as it is in the example, but it seems to grow if transactions are being managed manually or timeouts are being altered or anything like that.
edit: Assume, at least hypothetically, there's going to have to be a bunch of different types of data being returned. And consequently the solution can't be just one one-size-fits-all method, there will have to be a few different ones depending, at minimum, on whether ExecuteNonQuery, ExecuteScalar, ExecuteReader, ExecuteReaderAsync, or any of the others are being called. I'd like to cut down on the repetition among those.
Tried Dapper?
Granted this doesn't get you a DataReader but you might just prefer it this way once you've tried it.
It's about the lightest-weight an ORM can be while still being called an ORM. No more methods to map between DataReader and strong types for me.
Used right here on all the StackExchange sites.
using (var conn = new SqlConnection(cs))
{
var dogs = connection.Query("select name, age from dogs");
foreach (dynamic dog in dogs)
{
Console.WriteLine("{0} age {1}", dog.name, dog.age);
}
}
or
using (var conn = new SqlConnection(cs))
{
var dogs = connection.Query<Dog>("select Name, Age from dogs");
foreach (Dog dog in dogs)
{
Console.WriteLine("{0} age {1}", dog.Name, dog.Age);
}
}
class Dog
{
public string Name { get; set; }
public int Age { get; set; }
}
If you want to roll data access on your own, this pattern of help methods could be one way to remove duplication:
private List<int> ReadOrderData()
{
return ExecuteList<int>("SELECT OrderID FROM dbo.Orders;",
x => x.GetInt32("orderId")).ToList();
}
private IEnumerable<T> ExecuteList(string query,
Func<IDataRecord, T> entityCreator)
{
using(var connection = CreateConnection())
using(var command = connection.CreateCommand())
{
command.CommandText = query;
connection.Open();
using(var reader = command.ExecuteReader())
{
while(reader.Read())
yield return entityCreator(reader);
}
}
}
You'll have to add support for parameters and this might not compile, but the pattern is what I'm trying to illustrate.
What I typically do is use a custom class that I wrote a while back that accepts a SQL string, and optionally a list of parameters and it returns a DataTable.
Since the thing that changes between invocations is typically just the SQL that is optimal IMHO.
If you truly do need to use a DataReader you can do something like this:
public void ExecuteWithDataReader(string sql, Action<DataReader> stuffToDo) {
using (SqlConnection connection = new SqlConnection(connectionString)) {
using (SqlCommand command = new SqlCommand(sql, connection)) {
connection.Open();
using (SqlDataReader reader = command.ExecuteReader()) {
try {
while (reader.Read()) {
stuffToDo(reader);
}
}
finally {
reader.Close();
}
}
}
}
}
private static void ReadOrderData(string connectionString) {
string sql = "SELECT OrderID, CustomerID FROM dbo.Orders;";
ExecuteWithDataReader(sql, r => Console.WriteLine(String.Format("{0}, {1}", r[0], r[1])));
}
The first two line are the most important thing you need...
but if you still wish to do it, you can turn them to a database handler class, yes it will become more of code, but in refactoring concept, every thing will move to the related topic...
try to write a singleton class, that receive a command and do action, so return result of type SqlDataReader reader...
Doing this in comments was too much.
I would suggest that the boilerplate code around
using(conn = new sqlconnection)
using(cmd = new sqlcommand) {
// blah blah blah
}
isn't something to be lightly removed and instead would encourage that you keep it exactly where it's at. Resources, especially unmanaged ones, should be opened and released at the closest point to execution as possible IMHO.
In no small part due to the ease with which other developers will fail to follow the appropriate clean up conventions.
If you do something like
private SQLConnection CreateConnection()
{
var connection = new SqlConnection(_connectionString);
connection.Open();
return connection;
}
Then you are inviting another programmer to call this method and completely fail to release the resource as soon as the query is executed. I don't know what kind of app you are building, but in a web app such a thing will lead to memory / connection / resource errors of types that are difficult to debug, unless you've been through it before.
Instead, I'd suggest you look into a lightweight ORM such as Dapper.net or similar to see how they approached it. I don't use dapper, but I hear it's pretty good. The reason I don't use it is simply that we don't allow inline sql to be executed against our databases (but that's a very different conversation).
Here's our standard:
public static DataTable StatisticsGet( Guid tenantId ) {
DataTable result = new DataTable();
result.Locale = CultureInfo.CurrentCulture;
Database db = DatabaseFactory.CreateDatabase(DatabaseType.Clients.ToString());
using (DbCommand dbCommand = db.GetStoredProcCommand("reg.StatsGet")) {
db.AddInParameter(dbCommand, "TenantId", DbType.Guid, tenantId);
result.Load(db.ExecuteReader(dbCommand));
} // using dbCommand
return result;
} // method::StatisticsGet
We make heavy use of Enterprise Library. It's short, simple and to the point and very well tested. This method just returns a datatable but you could easily have it return an object collection.. or nothing.
I have a long-running service with several threads calling the following method hundreds of times per second:
void TheMethod()
{
using (var c = new SqlConnection("..."))
{
c.Open();
var ret1 = PrepareAndExecuteStatement1(c, args1);
// some code
var ret2 = PrepareAndExecuteStatement2(c, args2);
// more code
}
}
PrepareAndExecuteStatement is something like this:
void PrepareAndExecuteStatement*(SqlConnection c, args)
{
var cmd = new SqlCommand("query", c);
cmd.Parameters.Add("#param", type);
cmd.Prepare();
cmd.Parameters["#param"] = args;
return cmd.execute().read().etc();
}
I want reuse the prepared statements, preparing once per connection and executing them until the connection breaks. I hope this will improve performance.
Can I use the built-in connection pool to achieve this? Ideally every time a new connection is made, all statements should be automatically prepared, and I need to have access to the SqlCommand objects of these statements.
Suggest taking a slightly modified approach. Close your connection immedately after use. You can certainly re-use your SqlConnection.
The work being done at //some code may take a long time. Are you interacting with other network resources, disk resources, or spending any amount of time with calculations? Could you ever, in the future, need to do so? Perhaps the intervals between executing statement are/could be so long that you'd want to reopen that connection. Regardless, the Connection should be opened late and closed early.
using (var c = new SqlConnection("..."))
{
c.Open();
PrepareAndExecuteStatement1(c, args);
c.Close();
// some code
c.Open();
PrepareAndExecuteStatement2(c, args);
c.Close();
// more code
}
Open Late, Close Early as MSDN Magazine by John Papa.
Obviously we've now got a bunch of code duplication here. Consider refactoring your Prepare...() method to perform the opening and closing operations.
Perhaps you'd consider something like this:
using (var c = new SqlConnection("..."))
{
var cmd1 = PrepareAndCreateCommand(c, args);
// some code
var cmd2 = PrepareAndCreateCommand(c, args);
c.Open();
cmd1.ExecuteNonQuery();
cmd2.ExecuteNonQuery();
c.Close();
// more code
}
I have a class which uses both ADO.NET and LINQ to access a pair of databases on one server. The tables in the DBs are not extensive and so the entity objects are fairly light. I have written the code as best as I, using experience and net articles of course. For example...
http://dotnetperls.com/sqldataadapter
http://www.velocityreviews.com/forums/t71781-set-maximum-pool-size-in-web-config.html
http://msdn.microsoft.com/en-us/library/ms971481#adonetbest_topic5
The server which is running my code is only running my code and is not the same server as the db host. From looking at the connections going to the DB server from the my .NET app server (it's a Windows Service, but I don't want to dwell on that as it seem immaterial right now) the number of connections is around 200, but it should certainly be less than that; It should be around 10 as I have set the Max Pool Size to 10 in the appSettings.config.
Could anyone take a look at my connection code and tell me if I'm doing something which would cause the connections to shoot up, please?
Here is my LINQ DB context creation:
private const string ConnectionKey = "SQL2";
protected static string ConnectionString
{
get
{
return _connectionString = (_connectionString == null) ? ConfigurationManager.ConnectionStrings[ConnectionKey].ConnectionString : _connectionString;
}
}
private static string _connectionString = null;
protected static PricingDBDataContext ContextDB
{
get
{
if (_context == null)
{
_context = new PricingDBDataContext(ConnectionString);
}
return _context;
}
}
private static PricingDBDataContext _context = null;
Here is the ADO.NET side of things:
protected DataSet GetDataSet(bool isSproc, string cmdStr, params object[] args)
{
using (SqlConnection conn = new SqlConnection(ConnectionString))
{
using (SqlCommand cmd = new SqlCommand(cmdStr, conn))
{
cmd.CommandType = isSproc ? CommandType.StoredProcedure : CommandType.Text;
for (int index = 0; index < args.Length; index += 2)
cmd.Parameters.AddWithValue(args[index].ToString(), args[index + 1]);
conn.Open();
DataSet set = FillSet(cmd);
conn.Close();
return set;
}
}
}
private DataSet FillSet(SqlCommand cmd)
{
SqlDataAdapter adapter = new SqlDataAdapter(cmd);
DataSet set = new DataSet();
adapter.Fill(set);
return set;
}
Thanks,
Matt.
As a general rule, if you are doing anything explicit with the connection pool, you are probably doing it wrong. The default settings have served me well since the early days of .Net
Here are some things to keep in mind:
ADO will pool connections that have exactly the same connection string
Most ado.net objects implement IDisposable and should be disposed of either explicitly or in a using block. This includes connections, commands, datasets and data readers
Linq 2 SQL and EF data contexts also implement IDisposable and should be disposed