Can anyone tell me why the following code can deadlock? I'm simulating our webserver on multiple threads in a console app.
The console app has 5 threads and updates 250 records on each thread.
I am finding that transaction.Commit() is not enough, I will get deadlocks, so it clearly isn't releasing the locks at that point.
Unless I put the transaction.Dispose() in and the Sleep(50ms), I consistently get deadlocks on innodb. If I turn the code into a sproc, then the sleep needs to be bigger to avoid deadlocks. I'm not sure it does avoid them totally actually, need to run it with more threads.
Closing the connection after the transaction is more reliable but in the web app ideally we want to have a connection per request for performance.
Also putting transaction.Dispose() is far more reliable in terms of avoiding deadlocks, than using (var transaction = ...
We are using .NET currently, not .NET core.
I would bet if I write the same program using SqlClient for Sql/Server it will work - I'm going to try that tomorrow.
Can anyone explain this? What am I doing wrong?
static void Main(string[] args)
{
Console.WriteLine("GenerateBarcodesTestConsoleApp");
var connectionString = ConfigurationManager.ConnectionStrings["MyConnection"].ConnectionString;
var threads = Enumerable.Range(1, 5);
Parallel.ForEach(threads, t =>
{
GenerateBarcodes2(t, connectionString, 250);
});
Console.WriteLine("Press any key to exit...");
Console.ReadKey();
}
static void GenerateBarcodes2(int thread, string connectionString, int numberToGenerate)
{
using (var con = new MySqlConnection(connectionString))
{
con.Open();
var sql1 = "SELECT p.barcode, p..barcode_id " +
"FROM p_barcode p " +
"WHERE p.company_id = 1 " +
"AND SUBSTRING(p.barcode,1,2) = 'OK' " +
"AND players.in_use = 0 " +
"LIMIT 1 " +
"FOR UPDATE;";
var sql2 = "UPDATE p_barcode SET in_use = 1 WHERE company_id = 1 AND barcode_id = ?barcode_id AND in_use = 0";
for (int b = 0; b < numberToGenerate; b++)
{
using (var transaction = con.BeginTransaction(System.Data.IsolationLevel.RepeatableRead))
{
string barcode = string.Empty;
int barcodeId = 0;
using (var cmd = new MySqlCommand(sql1, con, transaction))
{
var rdr = cmd.ExecuteReader();
if (rdr.Read())
{
barcode = (string)rdr["barcode"];
barcodeId = (int)rdr["barcode_id"];
}
rdr.Close();
Console.WriteLine(barcode);
}
if (barcodeId != 0)
{
using (var cmd = new MySqlCommand(sql2, con, transaction))
{
cmd.Parameters.AddWithValue("barcode_id", barcodeId);
cmd.ExecuteNonQuery();
}
}
transaction.Commit();
System.Threading.Thread.Sleep(50);
}
//transaction.Dispose();
}
con.Close();
}
}
In MariaDb, SKIP LOCKED is the solution to prevent deadlocks.
There isn't a perfect solution to prevent deadlocks, without redesigning the system to avoid two threads trying to update the same record at the same time, however adding a small sleep after commit transaction appears to help massively. 20ms was about right on my dev machines.
Does this suggest that commit returns before the database has actually committed the transaction and released the locks? Either way, this behaviour is the same for INNODB and MARIADB.
Related
This one is related to spatilite also (not only SQLite)
I have a file database (xyz.db) which I am using by SQLiteconnection (SQLiteconnection is extends to spatialite).
I have so many records needs to update into database.
for (int y = 0; y < castarraylist.Count; y++)
{
string s = Convert.ToString(castarraylist[y]);
string[] h = s.Split(':');
SQLiteCommand sqlqctSQL4 = new SQLiteCommand("UPDATE temp2 SET GEOM = " + h[0] + "WHERE " + dtsqlquery2.Columns[0] + "=" + h[1] + "", con);
sqlqctSQL4.ExecuteNonQuery();
x = x + 1;
}
At above logic castarraylist is Arraylist which contains value which need to process into database.
When I checked above code updating around 400 records in 1 minute.
Is there any way by which I can able to improve performance ?
NOTE :: (File database is not thread-safe)
2. BEGIN TRANSACTION
Let's suppose I like to run two (or millions) update statement with single transaction in Spatialite.. is it possible ?
I read online and prepare below statement for me (but not get success)
BEGIN TRANSACTION;
UPDATE builtuparea_luxbel SET ADMIN_LEVEL = 6 where PK_UID = 2;
UPDATE builtuparea_luxbel SET ADMIN_LEVEL = 6 where PK_UID = 3;
COMMIT TRANSACTION;
Above statement not updating records in my database.
is SQLite not support BEGIN TRANSACTION ?
is there anything which I missing ?
And If I need to run individual statement then it's taking too much time to update as said above...
SQLite support Transaction, you can try below code.
using (var cmd = new SQLiteCommand(conn))
using (var transaction = conn.BeginTransaction())
{
for (int y = 0; y < castarraylist.Count; y++)
{
//Add your query here.
cmd.CommandText = "INSERT INTO TABLE (Field1,Field2) VALUES ('A', 'B');";
cmd.ExecuteNonQuery();
}
transaction.Commit();
}
The primary goal of a database transaction to get everything done, or nothing if something fails inside;
Reusing the same SQLiteCommand object by changing its CommandText property and execute it again and again might be faster, but leads to a memory overhead: If you have an important amount of queries to perform, the best is to dispose the object after use and create a new one;
A common pattern for an ADO.NET transaction is:
using (var tra = cn.BeginTransaction())
{
try
{
foreach(var myQuery in myQueries)
{
using (var cd = new SQLiteCommand(myQuery, cn, tra))
{
cd.ExecuteNonQuery();
}
}
tra.Commit();
}
catch(Exception ex)
{
tra.Rollback();
Console.Error.Writeline("I did nothing, because something wrong happened: {0}", ex);
throw;
}
}
I am trying to print out results from a MySQL query on a Visual C# Console application. I am able to get multiple columns on one line as you can see below, but I am wondering how I can get multiple results (rows). You see, my table holds more records that meet the query criteria. Can someone help me out?
class Program
{
static void Main(string[] args)
{
string ConnectionString = "Server=localhost; Database=world; Uid=root; Pwd=password"; // giving connection string
MySqlConnection connection = new MySqlConnection(ConnectionString);
MySqlCommand cmd = connection.CreateCommand();
cmd.CommandText = "SELECT name, population FROM city where population > 4000000";
try
{
connection.Open();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
MySqlDataReader reader = cmd.ExecuteReader();
while (reader.Read())
{
Console.WriteLine("City name is: " + reader["name"].ToString() + " " + reader["population"].ToString());
Console.Read();
}
}
Your call to Console.Read() is blocking the while loop, so you're only printing one line to console, and then waiting for user input.
Cheers
You want to remove that Console.Read();, it is blocking your application from continuing on until it reads another character input on the Console.
Other things to consider: Using statements ensure that the unmanaged resources consumed by the MySql objects are released when the objects are no longer in use. Use Parameterized queries (aka Prepared Statements) as they performs better and are more secure.
string sql = "SELECT name, population FROM city WHERE population > #population"
using (var conn = new MySqlConnection(/*Connection String*/))
{
conn.Open();
using (var cmd = new MySqlCommand(sql, conn))
{
cmd.Parameters.AddWithValue("#population", 4000000);
using (var reader = cmd.ExecuteReader())
{
while (reader.Read())
{
Console.WriteLine("City: {0} Population: {1}",
reader["name"], reader["population"]);
}
}
}
}
There may be other ways to do this, but the best way that I know of is loading the datareader into a datatable.
DataTable dt = new DataTable("City");
dt.Fill(reader);
foreach (DataRow row in dt.Rows){
Console.WriteLine("City name is: " + row["name"].ToString() + " " + row["population"].ToString());
Console.Read();
}
EDIT While answering, I realized that the Console.Read() may be the issue. This code will work, but an input to the console will need to be given for each line.
I have access to a particular API that enforces a speed limit of around 3 API calls per second. I'm creating a Windows Service using C#, and I figured if I just place a "Thread.Sleep(4000)" in between calls, that would delay each call by 4 seconds. Well, I had a "for" loop, looping 10 times, for testing. Within a second or so, it inserted all 10 records pulled from the API into my database. So, the Thread.Sleep(4000) wasn't being obeyed. I've read that Thread.Sleep() only works on the current thread. I don't know much about threading, but I was hoping one of you could tell me what I'm doing wrong, or at least suggest an alternative approach for obeying API traffic laws. Here's the relevant part of my code:
using (SqlConnection connection = new SqlConnection("Data Source=localhost;Initial Catalog=*****;Integrated Security=true;"))
{
for (int i = 0; i < 10; i++)
{
movie = api.GetMovieInfo(i);
Thread.Sleep(4000);
if (string.IsNullOrEmpty(movie.title))
continue;
string queryString = string.Format("insert into movie values ('{0}')", movie.title);
SqlCommand command = new SqlCommand(queryString, connection);
try
{
connection.Open();
command.ExecuteNonQuery();
connection.Close();
}
catch (Exception ex)
{
Debug.WriteLine(ex.Message);
}
}
}
It could be possible that you are seeing this behavior because your call to Thread.Sleep occurs in the using statement. Although I would still expect it to take at least four seconds and you stated that in only one second all of the 10 records were inserted. You could try to remove the call to Thread.Sleep from the using statement to see if the behavior improves...
see: C# exiting a using() block with a thread still running onthe scoped object
I do think that the Threading.Timer is a better option, but I would also think that Thread.Sleep should work as well:
for (int i = 0; i < 10; i++)
{
DoAPIWork();
Thread.Sleep(4000);
}
private void DoAPIWork()
{
using (SqlConnection connection = new SqlConnection("Data Source=localhost;Initial Catalog=*****;Integrated Security=true;"))
{
movie = api.GetMovieInfo(i);
if (string.IsNullOrEmpty(movie.title))
continue;
string queryString = string.Format("insert into movie values ('{0}')", movie.title);
SqlCommand command = new SqlCommand(queryString, connection);
try
{
connection.Open();
command.ExecuteNonQuery();
connection.Close();
}
catch (Exception ex)
{
Debug.WriteLine(ex.Message);
}
}
}
Try executing it on a new Thread
For example
for(int i = 0; i < 10; i++)
{
Thread executionThread = new Thread(() =>
{
//Call your api process here
DoWork();
});
executionThread.Start();
// This stops the current thread and wait for the executionThread
executionThread.Join();
}
and then let your DoWork() handle the sleep.
public void DoWork()
{
Thread.Sleep(4000);
// Do actual work here
}
Here's my test code, which seems to suggest that it's better to connect multiple times instead of connecting just once.
Am I doing something wrong?
int numIts = 100;
Stopwatch sw = new Stopwatch();
sw.Start();
using (SqlConnection connection = new SqlConnection(connectionParameters))
{
connection.Open();
for(int i = 0; i < numIts; i++)
{
SqlCommand command = new SqlCommand(sqlCommandName, connection);
command.CommandType = CommandType.StoredProcedure;
command.Parameters.AddWithValue(par1Name, par1Val);
command.Parameters.AddWithValue(par2Name, par2Val);
using(SqlDataReader reader = command.ExecuteReader())
{
}
}
}
sw.Stop();
TimeSpan durationOfOneConnectionManyCommands = sw.Elapsed;
Console.WriteLine(durationOfOneConnectionManyCommands);
sw.Reset();
sw.Start();
for(int i = 0; i < numIts; i++)
{
using (SqlConnection connection = new SqlConnection(connectionParameters))
{
connection.Open();
SqlCommand command = new SqlCommand(sqlCommandName, connection);
command.CommandType = CommandType.StoredProcedure;
command.Parameters.AddWithValue(par1Name, par1Val);
command.Parameters.AddWithValue(par2Name, par2Val);
using(SqlDataReader reader = command.ExecuteReader())
{
}
}
}
sw.Stop();
TimeSpan durationOfManyConnections = sw.Elapsed;
Console.WriteLine(durationOfManyConnections);
Output:
//output:
//00:00:24.3898218 // only one connection established
//00:00:23.4585797 // many connections established.
//
//output after varying parameters (expected much shorter):
//00:00:03.8995448
//00:00:03.4539567
Update:
OK, so those who said it would be faster w/ one connection have it. (although the difference is marginal, if any.)
Here's the revised code and output:
public void TimingTest()
{
numIts = 1000;
commandTxt = "select " + colNames + " from " + tableName;
OneConnection();
ManyConnections();
OneConnection();
}
private void ManyConnections()
{
Stopwatch sw = new Stopwatch();
sw.Start();
for (int i = 0; i < numIts; i++)
{
using (SqlConnection connection = new SqlConnection(connectionParameters))
{
connection.Open();
using (SqlCommand command = connection.CreateCommand())
{
command.CommandText = commandTxt;
using (SqlDataReader reader = command.ExecuteReader())
{
}
}
}
}
sw.Stop();
TimeSpan durationOfManyConnections = sw.Elapsed;
Console.WriteLine("many connections: " + durationOfManyConnections);
}
private void OneConnection()
{
Stopwatch sw = new Stopwatch();
sw.Start();
using (SqlConnection connection = new SqlConnection(connectionParameters))
{
connection.Open();
for (int i = 0; i < numIts; i++)
{
using (SqlCommand command = connection.CreateCommand())
{
command.CommandText = commandTxt;
using (SqlDataReader reader = command.ExecuteReader())
{
}
}
}
}
sw.Stop();
TimeSpan durationOfOneConnectionManyCommands = sw.Elapsed;
Console.WriteLine("one connection: " + durationOfOneConnectionManyCommands);
}
Output:
one connection: 00:00:08.0410024
many connections: 00:00:08.7278090
one connection: 00:00:08.6368853
one connection: 00:00:10.7965324
many connections: 00:00:10.8674326
one connection: 00:00:08.6346272
Update:
the difference is more striking if I use SQLConnection.ClearAllPools() after each function:
Output:
one connection: 00:00:09.8544728
many connections: 00:00:11.4967753
one connection: 00:00:09.7775865
By default, SqlConnection will use connection pooling. Therefore your code does most likely not actually open many connections in either case.
You can control if SqlConnection will use pooling by enabling or disabling the pool in the connectionstring, depending on what DB your connection string is for, the syntax will vary.
See here for some info if you use MSSQLServer. Try setting Pooling=false in the connection string and see if it makes a difference.
Definitively, it's better to have one connection. Maybe you are running your benchmark with small amount of data. Try increasing the number to 1,000 or 10,000.
Another point is that, depending on your app configuration, you might think you are running with multiple connections but .NET is pooling connections for you, so you are basically running with the same connections.
Since .NET reuses connections ("connection pooling"), there is not much overhead in creating a new instance of DbConnection several times in a row. ADO.NET will just reuse the connection under the hood. That's why it's good you are disposing the SqlConnection object each time, telling .NET that it can return it to the pool.
You can, however, increase performance of multiple inserts by using ADO.NET batching. In that case you can easily have several thousands of inserts per second. If performance is critical, you can even consider using SQLBulkCopy.
Also, your first pair of results is quite strange: 30s for 100 inserts?
In general, .NET's connection pooling should make it 'not matter' as it does a great job of recycling connections for you. But my practice is to use a single connection for a bunch of transactions I know will be taking place together. I think your timings are an indication of the connection pool doing its job and just plain variations in runs.
SqlClient will pool your connections. In your first case with one open, it will do the job of opening the connection. Every other run will use the pooled connection. If you reverse your order and do "many connections" first, I would expect you to see the opposite result.
I have run into a problem where the time to do a commit starts taking
longer and longer. We are talking on the orders of 250ms for a table
with ~ 20k lines and a disc size of around 2-3mb. And it just keeps getting worse. I have tracked the
performance problem down to something to do with indexs. It's almost
as if sqlite is creating the index on every commit. The commit consists of
100 INSERTS. I have made a as small program as I could where I can
reproduce the problem and have tried running this on Linux as well.
There the problem doesn't seem to occur. The problem exists with both
WAL and truncate journaling mode. The problem doesn't seem to exist
when I use a memory database instead of a file. I have tried both
version 3.6.23.1 and 3.7.6.3.
On Windows where I'm experiencing the problem I run sqlite in a C#
program. I have checked the implementation of transaction support in
the System.Date.Sqlite wrapper and it does absolutely nothing else
than simply to a COMMIT. Sadly I don't have a C compiler for Windows
so I can't check it when not running the wrapper, but it should be the
same.
System.IO.File.Delete("test.db");
var db_connection = new SQLiteConnection(#"Data Source=test.db");
db_connection.Open();
using (var cmd = db_connection.CreateCommand())
{
cmd.CommandText = "CREATE TABLE test (id integer primary key, dato integer)";
cmd.ExecuteNonQuery();
cmd.CommandText = "CREATE INDEX i on test(dato)";
cmd.ExecuteNonQuery();
}
SQLiteTransaction trans = null;
List<string> paths = new List<string>();
var random = new Random();
for (var j = 0; j < 150; ++j)
{
for (var i = 0; i < 1000; ++i)
{
if (i % 100 == 0)
{
trans = db_connection.BeginTransaction();
}
using (var cmd = db_connection.CreateCommand())
{
cmd.CommandText = String.Format("INSERT INTO test (dato) values ({0})", random.Next(1, 100000000));
cmd.ExecuteNonQuery();
}
if (i % 100 == 99 && trans != null)
{
var now = DateTime.Now;
trans.Commit();
trans.Dispose();
System.Console.WriteLine("commit {0}", (DateTime.Now - now).TotalMilliseconds);
}
}
}
Did you try reducing hard disk access, for example adding this command before creating any table:
cmd.CommandText = "PRAGMA locking_mode = EXCLUSIVE";
cmd.ExecuteNonQuery();
Providing your app allows exclusive locking of the database.
Also can help:
PRAGMA Synchronous=OFF