I have a problem with understanding transactions handling in System.Data.Sqlite connected with the following piece of code.
public static string ConnenctionString = "Data Source=C:\\DB\\Test.db";
public static void Test1()
{
SQLiteConnection connection = new SQLiteConnection(ConnenctionString);
connection.Open();
SQLiteTransaction transaction = connection.BeginTransaction();
for (int i = 0; i < 1000000; i++)
{
string sql = $"insert into table1 (id) values ({i})";
SQLiteCommand command = new SQLiteCommand(sql, connection);
command.ExecuteNonQuery();
}
transaction.Commit();
}
public static void Test2()
{
SQLiteConnection connection = new SQLiteConnection(ConnenctionString);
connection.Open();
string sql = "select count(*) from table1";
SQLiteCommand command = new SQLiteCommand(sql, connection);
SQLiteDataReader reader = command.ExecuteReader();
reader.Read();
Console.WriteLine(reader[0]);
}
public static void Test3()
{
for (int i = 0; i < 300; i++)
{
SQLiteConnection connection = new SQLiteConnection(ConnenctionString);
connection.Open();
SQLiteTransaction transaction = connection.BeginTransaction();
string sql = $"insert into table1 (id) values ({i})";
SQLiteCommand command = new SQLiteCommand(sql, connection);
command.ExecuteNonQuery();
transaction.Commit();
}
}
static void Main(string[] args)
{
//Everything is ok
Task.Factory.StartNew(() => Test1());
Thread.Sleep(1000);
Task.Factory.StartNew(() => Test2());
//Exception db is locked
Task.Factory.StartNew(() => Test3());
Thread.Sleep(1000);
Task.Factory.StartNew(() => Test2());
Console.ReadLine();
}
In one task I am executing some long running operation (in one transaction) - method Test1. In second task, I am trying to execute another query, without transaction - method Test2. Everything is running OK, I obtain a result that doesn't include changes from Test1 - because transaction hasn't been commited yet.
But when trying to execute in one task lots of queries - each in separate transaction, like in method Test3 and trying meanwhile to execute method Test2, I am obtaing exception: DB is locked. Is there a way to omit such behaviour modyfing connection string?
I would recommend using using statements (unfortunate pun):
using (var connection = new SqliteConnection("conn-string"))
{
connection.Open();
using (var transaction = connection.BeginTransaction())
{
for (int i = 0; i < 1000000; i++)
{
string sql = $"insert into table1 (id) values ({i})";
using (var command = new SqliteCommand(sql, connection))
{
command.ExecuteNonQuery();
}
}
transaction.Commit();
}
}
You can try to modify your database so it will use write-ahead logging. It will allow simultaneous access for multiple tasks without locking.
ExecuteQuery("pragma journal_mode=WAL");
immediately after creating the database. You need to execute this statement only once - it's persistent.
Related
I am using EF6 to query a backend database. User can customize a temporary table and query the data from the temporary table. I am using
DataTable result = context.Query(queryStatement);
to get the result and it has been working fine.
Now the query is needed among a serious of other sqlcommand and a transaction is needed. So I have
public static DataTable GetData()
{
using (MyDbContext context = new MyDbContext())
using (DbContextTransaction tran = context.Database.BeginTransaction())
{
try
{
int rowAffected = context.Database.ExecuteSqlCommand(
"UPDATE [MyDb].dbo.[TableLocks] SET RefCount = RefCount + 1 WHERE TableName = 'TESTTABLE1'");
if (rowAffected != 1)
throw new Exception("Cannot find 'TestTable1'");
//The following line will raise an exception
DataTable result = context.Query("SELECT TOP 100 * FROM [MyDb].dbo.[TestTable1]");
//This line will work if I change it to
//context.Database.ExecuteSqlCommand("SELECT TOP 100 * FROM [MyDb].dbo.[TestTable1]");
//but I don't know how to get the result out of it.
context.Database.ExecuteSqlCommand(
"UPDATE [MyDb].dbo.[TableLocks] SET RefCount = RefCount - 1 WHERE TableName = 'TestTable1'");
tran.Commit();
return result;
}
catch (Exception ex)
{
tran.Rollback();
throw (ex);
}
}
}
But this throws an exception while executing context.Query
ExecuteReader requires the command to have a transaction when the connection
assigned to the command is in a pending local transaction. The Transaction
property of the command has not been initialized.
And when I read this article: https://learn.microsoft.com/en-us/ef/ef6/saving/transactions
It says:
Entity Framework does not wrap queries in a transaction.
Is it the reason cause this issue?
How can I use context.Query() inside a transaction?
What else I can use?
I tried all other method, none of them work - because the return datatype cannot be predicted before hand.
I just realized that, the Query method is defined in MyDbContext!
public DataTable Query(string sqlQuery)
{
DbProviderFactory dbFactory = DbProviderFactories.GetFactory(Database.Connection);
using (var cmd = dbFactory.CreateCommand())
{
cmd.Connection = Database.Connection;
cmd.CommandType = CommandType.Text;
cmd.CommandText = sqlQuery;
using (DbDataAdapter adapter = dbFactory.CreateDataAdapter())
{
adapter.SelectCommand = cmd;
DataTable dt = new DataTable();
adapter.Fill(dt);
return dt;
}
}
}
May be you are missing this section -
you are free to execute database operations either directly on the
SqlConnection itself, or on the DbContext. All such operations are
executed within one transaction. You take responsibility for
committing or rolling back the transaction and for calling Dispose()
on it, as well as for closing and disposing the database connection
And then this codebase -
using (var conn = new SqlConnection("..."))
{
conn.Open();
using (var sqlTxn =
conn.BeginTransaction(System.Data.IsolationLevel.Snapshot))
{
try
{
var sqlCommand = new SqlCommand();
sqlCommand.Connection = conn;
sqlCommand.Transaction = sqlTxn;
sqlCommand.CommandText =
#"UPDATE Blogs SET Rating = 5" +
" WHERE Name LIKE '%Entity Framework%'";
sqlCommand.ExecuteNonQuery();
using (var context =
new BloggingContext(conn, contextOwnsConnection: false))
{
context.Database.UseTransaction(sqlTxn);
var query = context.Posts.Where(p => p.Blog.Rating >= 5);
foreach (var post in query)
{
post.Title += "[Cool Blog]";
}
context.SaveChanges();
}
sqlTxn.Commit();
}
catch (Exception)
{
sqlTxn.Rollback();
}
}
}
Specially this one -
context.Database.UseTransaction(sqlTxn);
Sorry guys, as mentioned above, I thought the Query method is from EF, but I examined the code and found it is actually coded by another developer, defined in class MyDbContext. Since this class is generated by EF, and I never think somebody have added a method.
It is
public DataTable Query(string sqlQuery)
{
DbProviderFactory dbFactory = DbProviderFactories.GetFactory(Database.Connection);
using (var cmd = dbFactory.CreateCommand())
{
cmd.Connection = Database.Connection;
cmd.CommandType = CommandType.Text;
cmd.CommandText = sqlQuery;
//And I added this line, then problem solved.
if (Database.CurrentTransaction != null)
cmd.Transaction = Database.CurrentTransaction.UnderlyingTransaction;
using (DbDataAdapter adapter = dbFactory.CreateDataAdapter())
{
adapter.SelectCommand = cmd;
DataTable dt = new DataTable();
adapter.Fill(dt);
return dt;
}
}
}
I am having a method like below where i perform save operations in 3 tables :
public void Do
{
using (var myConnection = new SqlConnection(connectionString))
{
for (int i = 0; i < 4; i++)
{
MyProcess.Execute(myConnection);//Dispose connection
}
if (myConnection.State == ConnectionState.Closed)
{
myConnection.Open();
using (var transaction = myConnection.BeginTransaction())
{
MyClass cls = new MyClass(myConnection,transaction);
// Here i have 3 insert operation
try
{
myRepo.Save1(myConnection,transaction);
myRepo.Save2(myConnection,transaction);
myRepo.Save3(myConnection,transaction);
transaction.Commit();
}
catch
{
transaction.Rollback();
}
}
}
}
}
Now in above approach i have created transaction so that if insert fails in 1 table then it shoudl roll back for other tables too.
Now in MyClass there are again some get data operations for which i need to pass transaction because if i dont do that i was getting below error :
ExecuteScalar requires the command to have a transaction when the
connection assigned to the command is in a pending local transaction.
The Transaction property of the command has not been initialized.
Above error was coming from GetOperations of MyClass and thats the reason why i have created transaction in the beginning and then passing to MyClass where i have some Get Operations from database tables
So my question is shall i restrict my transaction to Save1,Save2 and Save3 only and have all Get Operations inside MyClass to open and close their own connection?
Update :
public class MyClass
{
SqlConnection myConnection;
SqlTransaction transaction;
public MyClass(SqlConnection myConnection, SqlTransaction transaction)
{
this.myConnection = myConnection;
this.transaction = transaction;
}
public int GetEmployee(int id, int MockProfileId, string ProfileVersion)
{
var myEmp = new EmployeeRepo();
int basic = myEmp.GetEmployeeBasic(myConnection,transaction,100);
//Other code for get operations like GetEmployeeBasic
}
}
public class EmployeeRepo
{
public int GetEmployeeBasic(SqlConnection connection,SqlTransaction transaction,int id)
{
string query = "";
using (SqlCommand cmd = new SqlCommand(query, connection, transaction))
{
cmd.Parameters.AddWithValue("#id", id);
var data = cmd.ExecuteScalar();
if (data != null)
return (int)data;
else
return 0;
}
}
}
I have a C# application which has several methods which connect to a SQL Server database in order to execute a query.
Sometimes the connection fails and then the program exits.
A db administrator is looking on the database nevertheless I have to adapt the program in order to retry 2-3 times when a connection fails before to exiting.
I don't really know who doing this "smartly".
My connection code:
using (SqlConnection SqlCon = new SqlConnection(myParam.SqlConnectionString))
{
SqlCon.Open();
string requeteFou = "select XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX";
using (SqlCommand command = new SqlCommand(requeteFou, SqlCon))
{
using (SqlDataReader reader = command.ExecuteReader())
{
if (reader.HasRows)
{
while (reader.Read())
{
// do job
}
}
}
}
}
Since I use several methods, is there a simply way to overwrite the "connection" or "read" method in order to retry the connection 3 times for example ?
Best regards
I would use Polly for retry logic.
Very basic example retrying 3 times when there is a SqlException (not tested):
static void Main(string[] args)
{
var policy = Policy
.Handle<SqlException>()
.Retry(3);
try
{
policy.Execute(() => DoSomething());
}
catch (SqlException exc)
{
// log exception
}
}
private static void DoSomething()
{
using (SqlConnection conn = new SqlConnection(""))
{
conn.Open();
string requeteFou = "select XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX";
using (SqlCommand command = new SqlCommand(requeteFou, conn))
{
using (SqlDataReader reader = command.ExecuteReader())
{
if (!reader.HasRows) return;
while (reader.Read())
{
// do job
}
}
}
}
}
private static function()
{
DataTable dt = new DataTable();
string connectionString = "//your connection string";
String strQuery = //"Yourquery";
const int NumberOfRetries = 3;
var retryCount = NumberOfRetries;
var success = false;
while (!success && retryCount > 0)
{
try
{
SqlConnection conn = new SqlConnection(connectionString);
SqlCommand cmd = new SqlCommand();
cmd.CommandType = CommandType.Text;
cmd.CommandText = strQuery;
cmd.Connection = conn;
cmd.CommandTimeout = 180;
conn.Open();
SqlDataReader dr = cmd.ExecuteReader();
dt.Load(dr);
catch (Exception ex)
{
retryCount--;
Thread.Sleep(1000 * 60 * 15);
if (retryCount == 0)
{
//yourexception
}
}
}
}
Maybe wrap your using in a try block. Log a connection error in a catch block if you want. Put whole try{ }catch{ } in a for loop that will loop 3 times. If try block runs to the end of itself, break out of loop.
for(int i = 0; i < 3; i++)
{
try {
using (SqlConnection SqlCon = new SqlConnection(myParam.SqlConnectionString))
{
// your code
}
Thread.Sleep(1000); // wait some time before retry
break; // connection established, quit the loop
}
catch(Exception e) {
// do nothing or log error
}
}
You'd however have to handle differentiating SQL connection exception from other exceptions that you might encounter in your code.
I am trying to doing this:
Read a row from an SQLite db (in GetRuleByID() method)
Update the same row that I just read during (1) (See UpdateStatusForRuleID() method)
However my problem is that SQLite locks the database after the SELECT in GetRuleByID() so that update in UpdateStatusForRuleID() is only successful when called the first time.
I have tried enabling Write-Ahead-Logging in SQLite as well as PRAGMA read_uncommitted=1 in order to avoid SQLite locking the database for the SELECT, but this does not appear to work.
This should be simple but I have so far spent a complete night trying to solve this... Please help !
private static MicroRuleEngine.Rule GetRuleByID(int ruleID, SQLiteConnection connection, out Dictionary<string, string> dict)
{
dict = new Dictionary<string, string>();
string sql = String.Format("select * from rules WHERE ID = {0} ", ruleID.ToString());
SQLiteCommand command = new SQLiteCommand(sql, connection);
SQLiteDataReader reader = command.ExecuteReader();
if (reader.HasRows)
{
reader.Read();
// Convert row into a dictionary
for (int lp = 0; lp < reader.FieldCount; lp++)
{
dict.Add(reader.GetName(lp), reader.GetValue(lp) as string);
}
string json = dict["fulljson"];
MicroRuleEngine.Rule r = Newtonsoft.Json.JsonConvert.DeserializeObject<MicroRuleEngine.Rule>(json);
//command.Dispose();
return r;
}
}
internal static void UpdateStatusForRuleID(SQLConnectionManager DBMANAGER, int ruleID, bool status)
{
Dictionary<string, string> dict = null;
string dbVal = (status) ? "1" : "0";
MicroRuleEngine.Rule r = null;
string newJSON = null;
using (SQLiteConnection connection = DBMANAGER.CreateConnection())
{
r = GetRuleByID(ruleID, connection, out dict);
r.Active = (status);
newJSON = Newtonsoft.Json.JsonConvert.SerializeObject(r);
Thread.Sleep(1000);
string sql = "UPDATE rules SET active = #a, fulljson=#j WHERE ID = #i";
using (var command = new SQLiteCommand(sql, connection))
{
command.Parameters.Add(new SQLiteParameter("#a", dbVal));
command.Parameters.Add(new SQLiteParameter("#i", ruleID));
command.Parameters.Add(new SQLiteParameter("#j", newJSON));
command.ExecuteNonQuery(); // Database is locked here ???
}
connection.Close();
}
}
"Database is locked" means that some other connection (in the same or some other process) still has an active transaction.
You don't need multiple connections (unless you are using multiple threads); just use a single connection object for all database accesses.
Ensure that all command, reader, and transaction objects (and connections, if you decide to use temporary ones) are properly cleaned up, by using using:
using (var command = new SQLiteCommand(sql, connection))
using (var reader = command.ExecuteReader())
{
if (reader.HasRows)
...
}
Apparently, the code below works. I basically dropped the GetRuleByID() method (but then I had to re-write 4 other methods)
Thanks to all who provided input.
internal static void UpdateStatusForRuleID(SQLConnectionManager DBMANAGER, int ruleID, bool status)
{
string dbVal = (status) ? "1" : "0";
MicroRuleEngine.Rule r = null;
string newJSON = null;
using (SQLiteConnection conn = DBMANAGER.CreateConnection())
{
string sql = String.Format("select * from rules WHERE ID = {0} ", ruleID.ToString());
using (var command = new SQLiteCommand(sql, conn))
using (var reader = command.ExecuteReader())
{
if (reader.HasRows)
{
reader.Read();
string json = reader["fulljson"].ToString();
r = Newtonsoft.Json.JsonConvert.DeserializeObject<MicroRuleEngine.Rule>(json);
r.Active = (status);
newJSON = Newtonsoft.Json.JsonConvert.SerializeObject(r);
string sql2 = "UPDATE rules SET active = #a, fulljson=#j WHERE ID = #i";
using (var command2 = new SQLiteCommand(sql2, conn))
{
command2.Parameters.Add(new SQLiteParameter("#a", dbVal));
command2.Parameters.Add(new SQLiteParameter("#i", ruleID));
command2.Parameters.Add(new SQLiteParameter("#j", newJSON));
command2.ExecuteNonQuery();
}
}
}
}
}
I'm using .NET Connector to connect to MySQL.
In my application there are few threads using the same connection, so if a MySQLDataReader is not closed yet and some thread is trying execute a query it gives that error:
There is already an open DataReader associated with this Connection which must be closed first.
Will there ever be support in MySQL for multiple result sets or however it's called?
My database management class:
public class DatabaseConnection
{
private MySqlConnection conn;
public void Connect(string server, string user, string password, string database, int port = 3306)
{
string connStr = String.Format("server={0};user={1};database={2};port={3};password={4};charset=utf8",
server, user, database, port, password);
conn = new MySqlConnection(connStr);
conn.Open();
}
private MySqlCommand PrepareQuery(string query, object[] args)
{
MySqlCommand cmd = new MySqlCommand();
cmd.Connection = conn;
for (int i = 0; i < args.Length; i++)
{
string param = "{" + i + "}";
string paramName = "#DBVar_" + i;
query = query.Replace(param, paramName);
cmd.Parameters.AddWithValue(paramName, args[i]);
}
cmd.CommandText = query;
return cmd;
}
public List<Dictionary<string, object>> Query(string query, params object[] args)
{
MySqlCommand cmd = PrepareQuery(query, args);
MySqlDataReader reader = cmd.ExecuteReader();
List<Dictionary<string, object>> rows = new List<Dictionary<string, object>>();
while (reader.Read())
{
Dictionary<string, object> row = new Dictionary<string, object>();
for (int i = 0; i < reader.FieldCount; i++)
{
row.Add(reader.GetName(i), reader.GetValue(i));
}
rows.Add(row);
}
reader.Close();
return rows;
}
public object ScalarQuery(string query, params object[] args)
{
MySqlCommand cmd = PrepareQuery(query, args);
return cmd.ExecuteScalar();
}
public void Execute(string query, params object[] args)
{
MySqlCommand cmd = PrepareQuery(query, args);
cmd.ExecuteNonQuery();
}
public void Close()
{
conn.Close();
}
}
An example of how I use this:
DatabaseConnection Conn = new DatabaseConnection();
Conn.Connect("localhost", "root", "", "foogle");
var rows = conn.Query("SELECT * FROM `posts` WHERE `id` = {0}", postID);
foreach (var row in rows)
{
Console.WriteLine(row["title"]); // Writes the post's title (example)
}
Multiple result sets refers to a single query or query batch returning multiple row sets. Those results are accessed through the one and only DataReader for that connection.
What you're asking for is something quite different. You need the ability to performing multiple simultaneous queries of a single connection. Afaik .NET does not support that, not for SQL Server or any other driver.
Sharing a connection between multiple threads is a bad idea and totally unnecessary. .NET will use a connection pool to limit the total number of connections so It's perfectly safe to get a new connection for each (set of) queries you want to execute. Limit the scope of a connection to a thread and your problem will go away.