We Keep Coding

How to improve sqlite write performance in C#

I'm using sqlite to save log and meet write performance issue.
string log = "INSERT INTO Log VALUES ('2019-12-12 13:43:06','Error','Client','This is log message')"
public int WriteLog(string log)
{
return ExecuteNoQuery(log);
}
public int ExecuteNoQuery(string command)
{
int nResult = -1;
try
{
using (SQLiteConnection dbConnection = new SQLiteConnection(ConnectString))
{
dbConnection.Open();
using (SQLiteCommand dbCommand = dbConnection.CreateCommand())
{
dbCommand.CommandText = command;
nResult = dbCommand.ExecuteNonQuery();
}
}
}
catch (Exception e)
{
// Output error message
}
return nResult;
}
Search in google, transaction could improve the write performance significantly, but unfortunately I don't know when a log message will come, I could not combine the log message. Is there any other way to improve my log write performance?
I tried to add a timer to my code and commit transaction automatically. But I don't think it's a good way to speed up log write performance.
public class DatabaseManager : IDisposable
{
private static SQLiteTransaction transaction = null;
private SQLiteConnection dbConnection = null;
private static Timer transactionTimer;
private long checkInterval = 500;
private DatabaseManager(string connectionString)
{
dbConnection = new SQLiteConnection(connectionString);
dbConnection.Open();
StartTransactionTimer();
}
public void Dispose()
{
if(transaction != null)
{
transaction.Commit();
transaction = null;
}
dbConnection.Close();
dbConnection = null;
}
private void StartTransactionTimer()
{
transactionTimer = new Timer();
transactionTimer.Interval = checkInterval;
transactionTimer.Elapsed += TransactionTimer_Elapsed;
transactionTimer.AutoReset = false;
transactionTimer.Start();
}
private void TransactionTimer_Elapsed(object sender, ElapsedEventArgs e)
{
StartTransation();
transactionTimer.Enabled = true;
}
public void StartTransation()
{
try
{
if (dbConnection == null || dbConnection.State == ConnectionState.Closed)
{
return;
}
if (transaction != null)
{
transaction.Commit();
transaction = null;
}
transaction = dbConnection.BeginTransaction();
}
catch(Exception e)
{
LogError("Error occurs during commit transaction, error message: " + e.Message);
}
}
public int ExecuteNoQuery(string command)
{
int nResult = -1;
try
{
using (SQLiteCommand dbCommand = dbConnection.CreateCommand())
{
dbCommand.CommandText = command;
nResult = dbCommand.ExecuteNonQuery();
}
}
catch (Exception e)
{
LogError("Error occurs during execute sql no result query, error message: ", e.Message);
}
return nResult;
}
}

This started out as a comment, but it's evolving to an answer.
Get rid of the GC.Collect(); code line.
That's not your job to handle garbage collection - and you're probably degrading performance by using it.
No need to close the connection, you're disposing it in the next line anyway.
Why are you locking? Insert statements are usually thread safe - and this one doesn't seem to be an exception of that rule.
You are swallowing exceptions. That's a terrible habit.
Since you're only ever insert a single record, you don't need to return an int - you can simply return a bool (true for success, false for failure)

Why you don't use the entity framework to do the communications with the database?
For me is the easiest way. It's a Microsoft library so you can sure that the performance is very good.
I made some work with entity framework and sqlite db's and everything works very well.
Here an example of use:
var context = new MySqliteDatabase(new SQLiteConnection(#"DataSource=D:\\Mydb.db;cache=shared"));
var author = new Author {
FirstName = "William",
LastName = "Shakespeare",
Books = new List<Book>
{
new Book { Title = "Hamlet"},
new Book { Title = "Othello" },
new Book { Title = "MacBeth" }
}
};
context.Add(author);
context.SaveChanges();
The type of MySqliteDatabase can be created automatically using database first approach or with Code First approach. You have a lot of information and examples on the internet.
Here the link to the official documentation:
https://learn.microsoft.com/en-us/ef/ef6/

How to check if remote database MYSQL is available

I need register the user access on my webpage aspx in MySQL remote Database.
But this MySQL remote Database it could be unavailable.
I have tried this code, but how to execute the RegisterUSer() method in the bool IsServerConnected() method ?
public bool IsServerConnected()
{
using (var l_oConnection =
new OdbcConnection(ConfigurationManager.ConnectionStrings["ConnMySQL"].ConnectionString))
{
try
{
l_oConnection.Open();
return true;
}
catch (OdbcException)
{
return false;
}
}
}
private void RegisterUSer()
{
using (OdbcConnection myConnectionString =
new OdbcConnection(ConfigurationManager.ConnectionStrings["ConnMySQL"].ConnectionString))
{
string sql = #String.Format(" INSERT IGNORE INTO tbl_user ");
sql += String.Format(" ... ");
using (OdbcCommand command =
new OdbcCommand(sql, myConnectionString))
{
try
{
command.Connection.Open();
command.ExecuteNonQuery();
}
catch (Exception ex)
{
throw ex;
}
finally
{
command.Connection.Close();
}
}
#Edit 01
Error :
The type or namespace name 'resultType' could not be found (are you
missing a using directive or an assembly reference?)

You could just do a "wrapper" method that calls first to IsServerConnected() and depending on the returned boolean then calls RegisterUSer() or throws an error if the database is not availiable.
Quick and dirty pseudocode
private resultType ChickenWrapMethod()
{
if (!IsServerConnected())
{
//Throw some error here and exit
}
RegisterUSer()
}
BTW...in my opinion you should consider opening the sql connection out of the methods so it can be shared by both operations

Try this in c#. I hope I was helpful.
using System.Net.NetworkInformation;
var ping = new Ping();
var reply = ping.Send("XX.XX.XX.XXX", 60 * 1000); // 1 minute time out (in ms)
if (reply.Status == IPStatus.Success)
{
Response.Write("Server XX.XX.XX.XXX is up");
RegisterUSer();
}
else
{
Response.Write("Server XX.XX.XX.XXX is down");
}

Calling a MySQL Routine in C#

Please accept my apologies if I'm getting my verbiage wrong; I'm just now learning C# (my background is mostly Visual Basic and PHP).
What I'm trying to do is create a class / routine in C# (Windows Forms) for connecting and disconnecting to a MySQL database that can then be reused throughout the rest of my project without having to reiterate the code every time.
I've got my class / routine setup, but I'm stuck on trying to call it from the rest of my project. I know in Visual Basic this was a fairly simple task to do, but I just can't seem to figure it out how to do it in C#.
Any suggestions? Thank you in advance.
public void dbDisconnect(object sender, EventArgs e)
{
try
{
MySqlConnection connection = new MySqlConnection(Properties.Settings.Default.mysql_db_conn_string);
connection.Close();
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}

Here's a method I currently have in my app:
public static MySqlConnection CreateConnection(
string mysqlServer,
string mysqlUser,
string mysqlPassword,
string mysqlDatabase)
{
MySqlConnection mysqlConnection = null;
string mysqlConnectionString = String.Format(
"server={0};uid={1};pwd={2};database={3};DefaultCommandTimeout={4};",
mysqlServer, mysqlUser, mysqlPassword, mysqlDatabase, 120);
/**
** Workaround for MySQL 5.6 bug:
** http://stackoverflow.com/questions/30197699/reading-from-stream-failed-mysql-native-password-error
*/
int tryCounter = 0;
bool isConnected = false;
do
{
tryCounter++;
try
{
mysqlConnection = new MySqlConnection();
mysqlConnection.ConnectionString = mysqlConnectionString;
mysqlConnection.Open();
if (mysqlConnection.State == ConnectionState.Open)
{
isConnected = true;
}
}
catch (MySqlException ex)
{
if (tryCounter < 10)
{
DebugLog.Dump(ex.ToString(), DebugLog.MainLogFilePath);
Thread.Sleep(10000); // 10 seconds.
}
else
{
throw;
}
}
} while (!isConnected);
return mysqlConnection;
}
Usage:
using (MySqlConnection hostsDbConnection = HostsDbConnector.CreateConnection())
{
// Do something...
}
With using keyword you don't need to close the connection manually, it'll be closed automatically when it's no longer needed.

Page_Load firing multiple times?

We have been dealing with an error for the last couple of days, so we created a small page (quick and dirty programming, my apologies in advance) that connects to the database, checks if a document exists, and displays some data related to the document. If there is an exception, an email is sent with the exception information and some log data.
Here's a simplified version of the code (short explanation below):
namespace My.Namespace
{
public partial class myClass : System.Web.UI.Page
{
private static SqlConnection conn = null;
private static SqlCommand command1 = null;
private static SqlCommand command2 = null;
private static string log = "";
protected void Page_Load(object sender, EventArgs e)
{
if (!Page.IsPostBack)
{
try
{
log += "START\n";
string docId = Request.QueryString["docId"];
if (!String.IsNullOrEmpty(docName))
{
bool docExists = doesDocExist(docId);
if (docExists == true)
{
string docMetadata = getMetadata(docId);
Response.Write(docMetadata);
}
}
else
{
// display error message
}
}
catch (sqlException sqlex)
{
// process exception
sendErrorMessage(sqlex.Message);
}
catch (Exception ex)
{
// process exception
sendErrorMessage(ex.Message);
}
}
}
private static bool doesDocExist(string docId)
{
log += "In doesDocExist\n";
bool docExists = false;
try
{
// open db connection (conn)
string cmd = String.Format("SELECT COUNT(*) FROM docs WHERE id='{0}'", docId);
command1 = new SqlCommand(cmd, conn);
conn.Open();
var val = command1.ExecuteScalar();
int numberOfRows = int.Parse(val.ToString());
if (numberOfRows > 0) { docExists = true; }
}
finally
{
// close db connection (conn)
}
return docExists;
}
protected string getMetadata(string docId)
{
log += "In getMetadata\n";
string docMetadata = "";
try
{
// open db connection (conn)
string cmd = String.Format("SELECT metadata FROM docs WHERE id='{0}'", docID);
command2 = new SqlCommand(cmd, conn);
conn.Open();
SqlDataReader rReader = command2.ExecuteReader();
if (rReader.HasRows)
{
while (rReader.Read())
{
// process metadata
docMetadata += DOCMETADATA;
}
}
}
return docMetadata;
}
public static void sendErrorMessage(string messageText)
{
HttpContext.Current.Response.Write(messageText);
// Send string log via email
}
}
}
I know it's too long, so here is a quick description of it. We have a class with the Page_Load method and three other methods:
doesDocExists: returns a bool value indicating if an document ID is in the database.
getMetadata: returns a string with metadata related to the document.
sendErrorMessage: sends an email with a log generated during the page.
From Page_Load we call doesDocExists. If the value returned is true, then it calls getMetadata and displays the value on the screen. If there's any error, it is caught in the Page_Load and sent as an email.
The problem is that when there's an error, instead of getting an email with the log (i.e.: START - In Function1 - In Function2), the log appears 100 times in the email (i.e.: START - In Function1 - In Function2 - Start - In Function1 - In Function2 - START... and so on), as if Page_Load was fired that many times.
We read online (http://www.craigwardman.com/blog/index.php/2009/01/asp-net-multiple-page-load-problem/) that it could be because of the PostBack. So, we added the condition if (!Page.IsPostBack), but the result is still the same.
Is there any reason why Page_Load would be triggered multiple times? Or is it that we are doing something wrong with the log variable and/or the try/catch that causes this behavior?

The log may be long because you are declaring the string log as static. Does it need to be static?

private static SqlConnection conn = null;
private static SqlCommand command1 = null;
private static SqlCommand command2 = null;
private static string log = "";
The problem is that log is Singleton along with other properties.
Whenever you access that page, you append text to log property which ends up being START - In Function1 - In Function2 - Start - In Function1 - In Function2 - START... and so on
Base on your scenario, you do not need to use Singleton inside myClass.
FYI: Since I do not know the rest of your code, ensure to instantiate conn, command1, command2.

If your page load functions are execute twice because post back is possible when you clicking on the button or link, so should check it and run by the below
if (!IsPostBack)
{
try
{
log += "START\n";
string docId = Request.QueryString["docId"];
if (!String.IsNullOrEmpty(docName))
{
bool docExists = doesDocExist(docId);
if (docExists == true)
{
string docMetadata = getMetadata(docId);
Response.Write(docMetadata);
}
}
else
{
// display error message
}
}
catch (sqlException sqlex)
{
// process exception
sendErrorMessage(sqlex.Message);
}
catch (Exception ex)
{
// process exception
sendErrorMessage(ex.Message);
}
}
}

How can I get the return value from Sql Server system message?

I am trying to verify the backup I have just done in c# using command against sql server Express
string _commandText = string.Format("RESTORE VERIFYONLY FROM DISK = '{0}'", backupLocation);
SqlDataReader _sqlDataReader = SqlHelper.ExecuteReader("BookssortedSQLDbConnection", CommandType.Text, _commandText);
If I execute the command in SSMS it returns 'The backup set on file 1 is valid.' but how can I get this message back into my code?
A reader wont work as there are no rows being returned.
NOTE: I have tried the SMO.Restore object to try and verify it but it doesn't work and that is why I am doing it this way.
_restore.SqlVerify(srv, out _errorMessage); //returns false even though bakcup is fine
BTW - Open to suggestions as I don't think this is the ideal way to achieve what I am trying to do

Informational messages (with severity less than 10) and PRINT output are returned to the client, and raised as InfoMessage events by the SqlConnection instance. Each event contains a collection of SqlError objects (this is the same class used in SqlException.Errors).
Here's a complete example that shows connection state changes, info messages and exceptions. Note that I use ExecuteReader instead of ExecuteNonQuery, but the info and exception results are the same.
namespace Test
{
using System;
using System.Data;
using System.Data.SqlClient;
public class Program
{
public static int Main(string[] args)
{
if (args.Length != 2)
{
Usage();
return 1;
}
var conn = args[0];
var sqlText = args[1];
ShowSqlErrorsAndInfo(conn, sqlText);
return 0;
}
private static void Usage()
{
Console.WriteLine("Usage: sqlServerConnectionString sqlCommand");
Console.WriteLine("");
Console.WriteLine(" example: \"Data Source=.;Integrated Security=true\" \"DBCC CHECKDB\"");
}
public static void ShowSqlErrorsAndInfo(string connectionString, string query)
{
using (SqlConnection connection = new SqlConnection(connectionString))
{
connection.StateChange += OnStateChange;
connection.InfoMessage += OnInfoMessage;
SqlCommand command = new SqlCommand(query, connection);
try
{
command.Connection.Open();
Console.WriteLine("Command execution starting.");
SqlDataReader dr = command.ExecuteReader();
if (dr.HasRows)
{
Console.WriteLine("Rows returned.");
while (dr.Read())
{
for (int idx = 0; idx < dr.FieldCount; idx++)
{
Console.Write("{0} ", dr[idx].ToString());
}
Console.WriteLine();
}
}
Console.WriteLine("Command execution complete.");
}
catch (SqlException ex)
{
DisplaySqlErrors(ex);
}
finally
{
command.Connection.Close();
}
}
}
private static void DisplaySqlErrors(SqlException exception)
{
foreach (SqlError err in exception.Errors)
{
Console.WriteLine("ERROR: {0}", err.Message);
}
}
private static void OnInfoMessage(object sender, SqlInfoMessageEventArgs e)
{
foreach (SqlError info in e.Errors)
{
Console.WriteLine("INFO: {0}", info.Message);
}
}
private static void OnStateChange(object sender, StateChangeEventArgs e)
{
Console.WriteLine("Connection state changed: {0} => {1}", e.OriginalState, e.CurrentState);
}
}
}

Its pretty difficult to retrieve the ssms message to the front end application . However you can write the message into a text file and then read the data from the file .
declare #cmd varchar(1000)
SET #cmd = 'osql -S YourServer -E -d YourDatabase -q "RESTORE VERIFYONLY FROM DISK=''c:\yourBackup.bkp''" -o c:\result.txt'
EXEC master.dbo.xp_cmdshell #cmd
You can execute the above sql statements from your application and then read the result from the result.txt file