We Keep Coding

SQLite random database connection crash

I am programming a database export solution to convert a database. It is exporting and processing some hundred gigabytes of data in a multithreaded environment. All threads work in their own environment with own connections, but for orchestrating the export uses a sqlite "lookup" database that is used by all threads to distribute work and encode a specific field to a new id. This happens quite rarely (once every 50k rows exported) so it should not really slow the process even if locks are used.
For some reason about every 10-15min this lookup database throws an exception "could not open database" with errorcode 14. The exception randomly occurs on any of the ExecuteReader() methods. I tried to lock all methods accessing this database but still it crashes every 10-15min. Why? Actually when I just press on Resume in debugging mode everything works again, seems to be a temporary problem.
ExecuteLookup() is called by "main" exporting thread repeatedly.
DBQueryLookupDb() is called by any of the worker threads. (Something like INSERT INTO progress ...)
// using Microsoft.Data.Sqlite
private long ExecuteLookup(string value)
{
lock (this)
{
using (var cmd = GetSelectLookupCmd()) // "SELECT id,original FROM lookup_id WHERE original = #original", parameter is added, tried to re-use this command but same problem
{
cmd.Parameters["#original"].Value = value;
using (var res = cmd.ExecuteReader())
{
if (res.Read())
{
return res.GetInt64(0);
}
}
}
using (var cmd = GetInsertLookupCmd()) // INSERT INTO ...;SELECT last_insert_rowid();
{
cmd.Parameters["#original"].Value = value;
using (var res = cmd.ExecuteReader())
{
if (res.Read())
{
return res.GetInt64(0);
}
else { throw new Exception("Unexpected fail on lookup insert"); }
}
}
}
}
public void DBQueryLookupDb(string sql)
{
lock (this)
{
using (SqliteCommand c = new SqliteCommand())
{
c.Connection = lookupDb;
c.CommandType = System.Data.CommandType.Text;
c.CommandText = sql;
c.ExecuteNonQuery();
}
}
}

"PRAGMA JOURNAL_MODE ='PERSIST'" resolves the problem. TRUNCATE may work too. It may be a sync problem coming from the network drive (Sqlite deletes the journal and immediatley creates a new one before the network drive is ready again)

What is the best approach while polling database?

Currently I am doing an assignment in which I need to visit database of n number of servers to fetch some results.I have achieved this by iterating through the list of servers and raising tasks each one for each server in the collection. The task calls a function which basically makes an connection with the database,run query and disconnect from database.
My question is I am doing right by making a new connection on each polling with the database and closing it everytime or is this would be the best approach to keep a db connection open and fetch the result and then keep it open on next polling iteration.
PollingServerTimer() is being called by timer everytime.My polling timer is 3 sec.
Something like this :
private void PollingServerTimer(object sender, ElapsedEventArgs e)
{
foreach (var item in ServerOperationCollHandler)
{
if (item.RebootStatus != true)
{
PushItemIntoQueue(item);
}
}
}
public void PollingServerQueue()
{
while (isRunning)
{
this.waitHandle.WaitOne();
lock (syncRoot)
{
if (ServerQueue.Count > 0)
{
ServerOperationDataModel obj;
try
{
ServerQueue.TryDequeue(out obj);
Task GetCountFromDbTask = new Task(() => GetCountFromDb(obj));
GetCountFromDbTask.Start();
this.waitHandle.Reset();
}
catch (Exception ex)
{
MessageBox.Show("Problem encountered while finding iterim and recovery count");
isRunning = false;
break;
}
}
}
}
}
public void GetCountFromDb(ServerOperationDataModel obj)
{
ServerOperationDataModel serverObject = (ServerOperationDataModel)obj;
DataBaseHandler dbHandler = new DataBaseHandler(serverObject.DataBaseIP, serverObject.DataBasePort, serverObject.DataBaseName, serverObject.DataUserName, serverObject.DataUserPassword);
int attempts = 0;
do
{
try
{
dbHandler.connect();
}
catch (Exception ex)
{
break;
serverObject.DataBaseConnectionStatus = false;
log.Error("Connection attempt " + attempts + " failed.Retrying connection. Exception details :" + ex.ToString());
attempts++;
}
} while (attempts < _connectiontRetryAttempts && !dbHandler.isConnected());
if (dbHandler.isConnected())
{
/*Fetch Result and then get disconnect*/
dbHandler.disConnect();
}
else
{
//string msgLog = "Server : " + obj.ServerComponentIdentifier + " | " + obj.IPstring + "Connection cannot be established with the DB: " + obj.DataBaseIP + " | "+ obj.DataBasePort + " | " + obj.DataBaseName + " after a series of retries";
//LoggerUpdate.LogMessage(msgLog, LOGTYPE.POLLINGDATABASE, LoggerUpdate.ReturnLogDisplayObject(DateTime.Now, obj.ServerComponentIdentifier + "|" + obj.IPstring, Convert.ToInt16(LOGTYPE.POLLINGDATABASE), obj, msgLog));
}
}

I would not be concerned at all. Assuming you connect to the SQL Server (or a similar, enterprise DBMS), database connections are pooled at the client side which means that establishing the connection is costly only the first time the client connects to a particular db (formally: to a new, previously unseen connection string) and then each connection to the same database costs almost nothing.
If it hadn't been for pooling, applications servers would not be able to handle hundreds of concurrent browser connections that query the same data source. You would need much more than a connection every 3 seconds to cause any risk of depleting server or client resources.
You can read more on how pooling works
https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/sql-server-connection-pooling
A side note: You should polish your code a little bit.
For example, you have
GetCountFromDb(ServerOperationDataModel obj)
but then
ServerOperationDataModel serverObject = (ServerOperationDataModel)obj;
Why would you need to cast the obj to another variable of the very same type?
In the catch clause, you have break and some code below it which looks unreachable.

Take a look at the .NET SqlDependency object. This allows you to register a query with a database and, using an OnChange handler, receive notification whenever the result of the query changes.

SQLite DB Insert Very Slow

I am using an SQLite database, and inserting records into it. This takes a hugely long time! I have seen people who say they can process a couple thousand in a minute. I have around 2400 records. Each record takes 30s-2m to complete. Recreating the database is not an option. I have tried to create one transaction different ways. I need to use the timer, because I am using a ProgressBar to show me that something is happening. Here is the code I am using:
string con;
con = string.Format(#"Data Source={0}", documentsFolder);
SQLiteConnection sqlconnection = new SQLiteConnection(con);
SQLiteCommand sqlComm = sqlconnection.CreateCommand();
sqlconnection.Open();
SQLiteTransaction transaction = sqlconnection.BeginTransaction();
Timer timer2 = new Timer();
timer2.Interval = 1000;
timer2.Tick += (source, e) =>
{
URL u = firefox.URLs[count2];
string newtitle = u.title;
form.label1.Text = count2 + "/" + pBar.Maximum;
string c_urls = "insert or ignore into " + table + " (id,
url, title, visit_count, typed_count, last_visit_time, hidden) values (" + dbID + ",'" + u.url + "','"
+ newtitle + "',1,1, " + ToChromeTime(u.visited) + ", 0)";
string c_visited = "insert or ignore into " + table2 + " (id,
url,
visit_time, transition) values (" + dbID2 + "," + dbID + "," +
ToChromeTime(u.visited) + ",805306368)";
sqlComm = new SQLiteCommand(c_urls, sqlconnection);
sqlComm.ExecuteNonQuery();
sqlComm = new SQLiteCommand(c_visited, sqlconnection);
sqlComm.ExecuteNonQuery();
dbID++;
dbID2++;
pBar.Value = count2;
if (pBar.Maximum == count2)
{
pBar.Value = 0;
timer.Stop();
transaction.Commit();
sqlComm.Dispose();
sqlconnection.Dispose();
sqlconnection.Close();
}
count2++;
};
timer2.Start();
What am I doing wrong?

This is what I would address, in order. It may or may not fix the problem, but it won't hurt to see (and it might just do some magic):
Ensure the Database is not being contended with updates (from another thread, process, or even timer!). Writers will acquire locks and unclosed/over-long-running transactions can interact in bad ways. (For updates that take "30 seconds to 2 minutes" I would imagine there is an issue obtaining locks. Also ensure the media the DB is on is sufficient, e.g. local drive.)
The transaction is not being used (??). Move the transaction inside the timer callback, attach it to the appropriate SQLCommands, and dispose it before the callback ends. (Use using).
Not all SQLCommand's are being disposed correctly. Dispose each and every one. (The use of using simplifies this. Do not let it bleed past the callback.)
Placeholders are not being used. Not only is this simpler and easier to use, but it is also ever so slightly more friendly to SQLite and the adapter.
(Example only; there may be errors in the following code.)
// It's okay to keep long-running SQLite connections.
// In my applications I have a single application-wide connection.
// The more important thing is watching thread-access and transactions.
// In any case, we can keep this here.
SQLiteConnection sqlconnection = new SQLiteConnection(con);
sqlconnection.Open();
// In timer event - remember this is on the /UI/ thread.
// DO NOT ALLOW CROSS-THREAD ACCESS TO THE SAME SQLite CONNECTION.
// (You have been warned.)
URL u = firefox.URLs[count2];
string newtitle = u.title;
form.label1.Text = count2 + "/" + pBar.Maximum;
try {
// This transaction is ONLY kept about for this timer callback.
// Great care must be taken with long-running transactions in SQLite.
// SQLite does not have good support for (long running) concurrent-writers
// because it must obtain exclusive file locks.
// There is no Table/Row locks!
sqlconnection.BeginTransaction();
// using ensures cmd will be Disposed as appropriate.
using (var cmd = sqlconnection.CreateCommand()) {
// Using placeholders is cleaner. It shouldn't be an issue to
// re-create the SQLCommand because it can be cached in the adapter/driver
// (although I could be wrong on this, anyway, it's not "this issue" here).
cmd.CommandText = "insert or ignore into " + table
+ " (id, url, title, visit_count, typed_count, last_visit_time, hidden)"
+ " values (#dbID, #url, 'etc, add other parameters')";
// Add each parameter; easy-peasy
cmd.Parameters.Add("#dbID", dbID);
cmd.Parameter.Add("#url", u.url);
// .. add other parameters
cmd.ExecuteNonQuery();
}
// Do same for other command (runs in the same TX)
// Then commit TX
sqlconnection.Commit();
} catch (Exception ex) {
// Or fail TX and propagate exception ..
sqlconnection.Rollback();
throw;
}
if (pBar.Maximum == count2)
{
pBar.Value = 0;
timer.Stop();
// All the other SQLite resources are already
// cleaned up!
sqlconnection.Dispose();
sqlconnection.Close();
}

I'm not sure if this is your problem, but your general pattern of using ADO.NET is wrong - you shouldn't create new command(s) per each insert (and repeatedly pay for query preparation).
Instead, do the following:
Before the loop:
Create command(s) once.
Create appropriate bound parameters.
In the loop:
Just assign appropriate values to the bound parameters.
And execute the command(s).
You could also consider using less fine-grained transactions: try putting several inserts in the same transaction to minimize paying for transaction durability.
You might also want to take a look at this post.

You can try one of the following to improve performance :
Wrap all the inserts in a transaction - Can help in reducing the actual writes to the DB.
Use WAL - The Write-Ahead-Log is a journaling mode that speeds up writes and enables concurrency. (Not recommended if your DB is in a Network location).
Synchronous NORMAL - The Synchronous Mode dictates the the frequency at which data is actually flushed to the physical memory (fsync() calls). This can be time taking on some machines and hence the frequency at which this flush occurs is critical. Make sure to explicitly open connections with "Synchronous=NORMAL" ideal for most scenarios. There is a huge difference between Synchronous MODE as FULL and NORMAL (NORMAL is ~1000 times better).
Find more details in a similar post => What changed between System.Data.SQLite version 1.0.74 and the most recent 1.0.113?

Some tricky quick way to validate oracle db connection

My WCF service need to check is connection available now and can we work with it. We have many remote dbs. Their connection are weird sometimes and can't be used to query data or smth else.
So, for example this is regular connection string used:
User Id=user;Password=P#ssw0rd;Data Source=NVDB1;Connection Timeout=30
Here is service method, used for getting
public List<string> GetAliveDBs(string city)
{
if (String.IsNullOrEmpty(city))
return null;
List<string> cityDbs = (from l in alldbs where !String.IsNullOrEmpty(l.Value.city) && l.Value.city.ToUpper() == city.ToUpper() select l.Value.connString).ToList();
// There is no such city databases
if (cityDbs.Count == 0)
return null;
ReaderWriterLockSlim locker = new ReaderWriterLockSlim();
Parallel.ForEach(cityDbs, p =>
{
if (!IsConnectionActive(p.connString))
{
locker.EnterWriteLock();
try
{
cityDbs.RemoveAt(cityDbs.IndexOf(p));
}
finally
{
locker.ExitWriteLock();
}
}
});
return cityDbs;
}
static public bool IsConnectionAlive(string connectionString)
{
using (OracleConnection c = new OracleConnection(connectionString))
{
try
{
c.Open();
if ((c.State == ConnectionState.Open) && (c.Ping()))
return true;
else
return false;
}
catch (Exception exc)
{
return false;
}
}
}
I use devart components to communicate with Oracle DB.
Hope for your help, guys! Thanks in advance!

Try just executing a very low cost operation that should work no matter what schema you are connected to, e.g.
SELECT 1
(that statement works on MS SQL and MySQL... should work on Oracle too but I can't confirm that).
If you get the result you expect (in this case one row, with one column, containing a "1") then the connection is valid.
At least one connection pool manager uses this strategy to validate connections periodically.
UPDATE:
Here's a SQL Server version of your method. You can probably just replace "Sql" with "Oracle".
static public bool IsConnectionAlive(string connectionString)
{
try
{
using (SqlConnection conn = new SqlConnection(connectionString))
{
conn.Open();
using (SqlCommand cmd = new SqlCommand("SELECT 1", conn))
{
int result = (int)cmd.ExecuteScalar();
return (result == 1);
}
}
}
catch (Exception ex)
{
// You need to decide what to do here... e.g. does a malformed connection string mean the "connection isn't alive"?
// Maybe return false, maybe log the error and re-throw the exception?
throw;
}
}

If the goal is to simply determine if a server lives at the IP Address or host name then I'd recommend Ping (no 3 way handshake and has less overhead than a UDP message). You can use the System.Net.NetworkInformation.Ping class (see its documentation for an example) for this.
If you're looking to prove that there is actually something listening on the common Oracle port, I would suggest using either the System.Net.Sockets.TcpClient or System.Net.Sockets.Socket class (their documentation also provides examples) to provide this.
The simplest way to do this (by far) is to just open a connection using the Oracle API for C#. There is a very good tutorial that includes code here. It covers more than just the connection but you should be able to strip out the connection portion from the rest to fit your needs.

Oracle has products and software specifically for helping maintain high availability that can allow you to have dead connections removed from you connection pool through a setting called HA Events=true on the connection string. Your Oracle DBA will need to determine if your installation supports it.

TransactionScope helper that exhausts connection pool without fail - help?

A while back I asked a question about TransactionScope escalating to MSDTC when I wasn't expecting it to. (Previous question)
What it boiled down to was, in SQL2005, in order to use a TransactionScope, you can only instance and open a single SqlConnection within the life of the TransactionScope. With SQL2008, you can instance multiple SqlConnections, but only a single one can be open at any given time. SQL2000 will always escalate to DTC...we don't support SQL2000 in our application, a WinForms app, BTW.
Our solution to single-connection-only problem was to create a TransactionScope helper class, called LocalTransactionScope (aka 'LTS'). It wraps a TransactionScope and, most importantly, creates and maintains a single SqlConnection instance for our application. The good news is, it works - we can use LTS across disparate pieces of code and they all join the ambient transaction. Very nice. The trouble is, every root LTS instance created will create and effectively kill a connection from the connection pool. By 'Effectively Kill' I mean it will instance a SqlConnetion, which will open a new connection (for whatever reason, it never reuses a connection from the pool,) and when that root LTS is disposed, it closes and disposes the SqlConnection which is supposed to release the connection back to the pool so that it can be reused, however, it clearly never is reused. The pool bloats until it's maxed out, and then the application fails when a max-pool-size+1 connection is established.
Below I've attached a stripped down version of the LTS code and a sample console application class that will demonstrate the connection pool exhaustion. In order to watch your connection pool bloat, use SQL Server Managment Studio's 'Activity Monitor' or this query:
SELECT DB_NAME(dbid) as 'DB Name',
COUNT(dbid) as 'Connections'
FROM sys.sysprocesses WITH (nolock)
WHERE dbid > 0
GROUP BY dbid
I'm attaching LTS here, and a sample console application that you can use to demonstrate for yourself that it will consume connections from the pool and never re-use nor release them. You will need to add a reference to System.Transactions.dll for LTS to compile.
Things to note: It's the root-level LTS that opens and closes the SqlConnection, which always opens a new connection in the pool. Having nested LTS instances makes no difference because only the root LTS instance establishes a SqlConnection. As you can see, the connection string is always the same, so it should be reusing the connections.
Is there some arcane condition we're not meeting that causes the connections not to be re-used? Is there any solution to this other than turning pooling off entirely?
public sealed class LocalTransactionScope : IDisposable
{
private static SqlConnection _Connection;
private TransactionScope _TransactionScope;
private bool _IsNested;
public LocalTransactionScope(string connectionString)
{
// stripped out a few cases that need to throw an exception
_TransactionScope = new TransactionScope();
// we'll use this later in Dispose(...) to determine whether this LTS instance should close the connection.
_IsNested = (_Connection != null);
if (_Connection == null)
{
_Connection = new SqlConnection(connectionString);
// This Has Code-Stink. You want to open your connections as late as possible and hold them open for as little
// time as possible. However, in order to use TransactionScope with SQL2005 you can only have a single
// connection, and it can only be opened once within the scope of the entire TransactionScope. If you have
// more than one SqlConnection, or you open a SqlConnection, close it, and re-open it, it more than once,
// the TransactionScope will escalate to the MSDTC. SQL2008 allows you to have multiple connections within a
// single TransactionScope, however you can only have a single one open at any given time.
// Lastly, let's not forget about SQL2000. Using TransactionScope with SQL2000 will immediately and always escalate to DTC.
// We've dropped support of SQL2000, so that's not a concern we have.
_Connection.Open();
}
}
/// <summary>'Completes' the <see cref="TransactionScope"/> this <see cref="LocalTransactionScope"/> encapsulates.</summary>
public void Complete() { _TransactionScope.Complete(); }
/// <summary>Creates a new <see cref="SqlCommand"/> from the current <see cref="SqlConnection"/> this <see cref="LocalTransactionScope"/> is managing.</summary>
public SqlCommand CreateCommand() { return _Connection.CreateCommand(); }
void IDisposable.Dispose() { this.Dispose(); }
public void Dispose()
{
Dispose(true); GC.SuppressFinalize(this);
}
private void Dispose(bool disposing)
{
if (disposing)
{
_TransactionScope.Dispose();
_TransactionScope = null;
if (!_IsNested)
{
// last one out closes the door, this would be the root LTS, the first one to be instanced.
LocalTransactionScope._Connection.Close();
LocalTransactionScope._Connection.Dispose();
LocalTransactionScope._Connection = null;
}
}
}
}
This is a Program.cs that will exhibit the connection pool exhaustion:
class Program
{
static void Main(string[] args)
{
// fill in your connection string, but don't monkey with any pooling settings, like
// "Pooling=false;" or the "Max Pool Size" stuff. Doesn't matter if you use
// Doesn't matter if you use Windows or SQL auth, just make sure you set a Data Soure and an Initial Catalog
string connectionString = "your connection string here";
List<string> randomTables = new List<string>();
using (var nonLTSConnection = new SqlConnection(connectionString))
using (var command = nonLTSConnection.CreateCommand())
{
command.CommandType = CommandType.Text;
command.CommandText = #"SELECT [TABLE_NAME], NEWID() AS [ID]
FROM [INFORMATION_SCHEMA].TABLES]
WHERE [TABLE_SCHEMA] = 'dbo' and [TABLE_TYPE] = 'BASE TABLE'
ORDER BY [ID]";
nonLTSConnection.Open();
using (var reader = command.ExecuteReader())
{
while (reader.Read())
{
string table = (string)reader["TABLE_NAME"];
randomTables.Add(table);
if (randomTables.Count > 200) { break; } // got more than enough to test.
}
}
nonLTSConnection.Close();
}
// we're going to assume your database had some tables.
for (int j = 0; j < 200; j++)
{
// At j = 100 you'll see it pause, and you'll shortly get an InvalidOperationException with the text of:
// "Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool.
// This may have occurred because all pooled connections were in use and max pool size was reached."
string tableName = randomTables[j % randomTables.Count];
Console.Write("Creating root-level LTS " + j.ToString() + " selecting from " + tableName);
using (var scope = new LocalTransactionScope(connectionString))
using (var command = scope.CreateCommand())
{
command.CommandType = CommandType.Text;
command.CommandText = "SELECT TOP 20 * FROM [" + tableName + "]";
using (var reader = command.ExecuteReader())
{
while (reader.Read())
{
Console.Write(".");
}
Console.Write(Environment.NewLine);
}
}
Thread.Sleep(50);
scope.Complete();
}
Console.ReadKey();
}
}

The expected TransactionScope/SqlConnection pattern is, according to MSDN:
using(TransactionScope scope = ...) {
using (SqlConnection conn = ...) {
conn.Open();
SqlCommand.Execute(...);
SqlCommand.Execute(...);
}
scope.Complete();
}
So in the MSDN example the conenction is disposed inside the scope, before the scope is complete. Your code though is different, it disposes the connection after the scope is complete. I'm not an expert in matters of TransactionScope and its interaction with the SqlConnection (I know some things, but your question goes pretty deep) and I can't find any specifications what is the correct pattern. But I'd suggest you revisit your code and dispose the singleton connection before the outermost scope is complete, similarly to the MSDN sample.
Also, I hope you do realize your code will fall apart the moment a second thread comes to play into your application.

Is this code legal?
using(TransactionScope scope = ..)
{
using (SqlConnection conn = ..)
using (SqlCommand command = ..)
{
conn.Open();
SqlCommand.Execute(..);
}
using (SqlConnection conn = ..) // the same connection string
using (SqlCommand command = ..)
{
conn.Open();
SqlCommand.Execute(..);
}
scope.Complete();
}