We Keep Coding

How to improve sqlite write performance in C#

I'm using sqlite to save log and meet write performance issue.
string log = "INSERT INTO Log VALUES ('2019-12-12 13:43:06','Error','Client','This is log message')"
public int WriteLog(string log)
{
return ExecuteNoQuery(log);
}
public int ExecuteNoQuery(string command)
{
int nResult = -1;
try
{
using (SQLiteConnection dbConnection = new SQLiteConnection(ConnectString))
{
dbConnection.Open();
using (SQLiteCommand dbCommand = dbConnection.CreateCommand())
{
dbCommand.CommandText = command;
nResult = dbCommand.ExecuteNonQuery();
}
}
}
catch (Exception e)
{
// Output error message
}
return nResult;
}
Search in google, transaction could improve the write performance significantly, but unfortunately I don't know when a log message will come, I could not combine the log message. Is there any other way to improve my log write performance?
I tried to add a timer to my code and commit transaction automatically. But I don't think it's a good way to speed up log write performance.
public class DatabaseManager : IDisposable
{
private static SQLiteTransaction transaction = null;
private SQLiteConnection dbConnection = null;
private static Timer transactionTimer;
private long checkInterval = 500;
private DatabaseManager(string connectionString)
{
dbConnection = new SQLiteConnection(connectionString);
dbConnection.Open();
StartTransactionTimer();
}
public void Dispose()
{
if(transaction != null)
{
transaction.Commit();
transaction = null;
}
dbConnection.Close();
dbConnection = null;
}
private void StartTransactionTimer()
{
transactionTimer = new Timer();
transactionTimer.Interval = checkInterval;
transactionTimer.Elapsed += TransactionTimer_Elapsed;
transactionTimer.AutoReset = false;
transactionTimer.Start();
}
private void TransactionTimer_Elapsed(object sender, ElapsedEventArgs e)
{
StartTransation();
transactionTimer.Enabled = true;
}
public void StartTransation()
{
try
{
if (dbConnection == null || dbConnection.State == ConnectionState.Closed)
{
return;
}
if (transaction != null)
{
transaction.Commit();
transaction = null;
}
transaction = dbConnection.BeginTransaction();
}
catch(Exception e)
{
LogError("Error occurs during commit transaction, error message: " + e.Message);
}
}
public int ExecuteNoQuery(string command)
{
int nResult = -1;
try
{
using (SQLiteCommand dbCommand = dbConnection.CreateCommand())
{
dbCommand.CommandText = command;
nResult = dbCommand.ExecuteNonQuery();
}
}
catch (Exception e)
{
LogError("Error occurs during execute sql no result query, error message: ", e.Message);
}
return nResult;
}
}

This started out as a comment, but it's evolving to an answer.
Get rid of the GC.Collect(); code line.
That's not your job to handle garbage collection - and you're probably degrading performance by using it.
No need to close the connection, you're disposing it in the next line anyway.
Why are you locking? Insert statements are usually thread safe - and this one doesn't seem to be an exception of that rule.
You are swallowing exceptions. That's a terrible habit.
Since you're only ever insert a single record, you don't need to return an int - you can simply return a bool (true for success, false for failure)

Why you don't use the entity framework to do the communications with the database?
For me is the easiest way. It's a Microsoft library so you can sure that the performance is very good.
I made some work with entity framework and sqlite db's and everything works very well.
Here an example of use:
var context = new MySqliteDatabase(new SQLiteConnection(#"DataSource=D:\\Mydb.db;cache=shared"));
var author = new Author {
FirstName = "William",
LastName = "Shakespeare",
Books = new List<Book>
{
new Book { Title = "Hamlet"},
new Book { Title = "Othello" },
new Book { Title = "MacBeth" }
}
};
context.Add(author);
context.SaveChanges();
The type of MySqliteDatabase can be created automatically using database first approach or with Code First approach. You have a lot of information and examples on the internet.
Here the link to the official documentation:
https://learn.microsoft.com/en-us/ef/ef6/

c# hidden input not populating on page load when failed

Very odd one this - when i run some code on page_load and it fails, i want to populate a hidden text box with a value. The code fires and steps through in debug but the value still turns out blank. The code runs fine and hits this:
if (strResults[0] != "Success") {
fail.Value = "Failed";
}
so fail's value should say "Failed" if the submit was a failure (Success if it worked) but it doesn't.
Here' the code (edited)
try {
if (Request.QueryString["bla"] != null) {
if (Request.QueryString["blabla"] == "yes")
}
if (Request.QueryString["blablablabla"] != null) {
if (Request.QueryString["bla"] == "yes") {
string strResponse = "";
try {
if ());
if (intTransactionCodeAlreadyExists > 0) {
} else {
//do stuff
);
string[] strResults = strResponse.Split('|');
if (strResults[0] != "Success") {
fail.Value = "Failed";
}
}
}
} catch (Exception ex) {
//do stuff
}
}
}
} catch (Exception ex) {
//do stuff
}

This was a bit messed up in my setup. After a re-install of visual studio, the code started working as expected. Not 100% sure what the issue was here but all fixed. Thanks for the help.

C# Recursive function approach?

I have a recursive function in a windows service. This function upon completion rewinds itself as it has been repeated multiple times in recursion. Isn't that an overhead ?
Is there any way to avoid unwinding ? Is there any better approach?
Edit : In this method, I get 100 records from DB and then process them and then get another 100 and so on till all the records in DB have been processed.
Also, there is no limit of how many total records there might be in the db so this function can repeat itself quite a lot.
public void ServiceFunctionality()
{
try
{
// Get Data From WEBAPI
HttpClient client = new HttpClient();
HttpResponseMessage response = response = client.GetAsync("webapi url link").Result;
Response<ServiceWrapper> objResponse = response.Content.ReadAsAsync<Response<ServiceWrapper>>().Result;
if (objResponse != null)
{
if (objResponse.isSuccess == true)
{
listContact = objResponse.data.lContact;
int MaxPKinSelectedRecords = objResponse.data.MaxPKinSelectedRecords;
int MaxPKinTotalRecords = objResponse.data.MaxPKinTotalRecords;
if (listContact != null && listContact.Count>0)
{
try
{
Parallel.ForEach(listContact, contact =>
{
// some code...
});
// Recursive Call
if (MaxPKinTotalRecords != MaxPKinSelectedRecords)
{
ServiceFunctionality();
}
}
catch (Exception ex)
{
// Logging
}
}
}
else
{
// Logging
}
}
else
{
// Logging
}
}
catch (Exception ex)
{
// Logging
}
}

You can always unwind to a while loop. Because your calls aren't altering state, this is trival.
public void ServiceFunctionality()
{
bool done = false;
while(!done) {
try
{
done = true; //if we don't reset this, we're done.
// Get Data From WEBAPI
HttpClient client = new HttpClient();
HttpResponseMessage response = response = client.GetAsync("webapi url link").Result;
Response<ServiceWrapper> objResponse = response.Content.ReadAsAsync<Response<ServiceWrapper>>().Result;
if (objResponse != null)
{
if (objResponse.isSuccess == true)
{
listContact = objResponse.data.lContact;
int MaxPKinSelectedRecords = objResponse.data.MaxPKinSelectedRecords;
int MaxPKinTotalRecords = objResponse.data.MaxPKinTotalRecords;
if (listContact != null && listContact.Count>0)
{
try
{
Parallel.ForEach(listContact, contact =>
{
// some code...
});
// set loop variable
if (MaxPKinTotalRecords != MaxPKinSelectedRecords)
{
done = false;
}
}
catch (Exception ex)
{
// Logging
}
}
}
else
{
// Logging
}
}
else
{
// Logging
}
}
catch (Exception ex)
{
// Logging
}
}
}

Do not use recursion for calling a function whenever you have alternate suitable solution. I personally almost never do
I have tried to keep it same other than using a while..
Do not forget to break your loop. I tried to handle this thing but still
Just to be very careful, never take a risk of infinite loop on server I took maxPossibleIterations. So that in case of any mistake your web service server would not have to go for infinite iterations
public void ServiceFunctionality()
{
long maxPossibleIterations = 999999;
try
{
while (true)
{
maxPossibleIterations++;
// Get Data From WEBAPI
HttpClient client = new HttpClient();
HttpResponseMessage response = response = client.GetAsync("webapi url link").Result;
Response<ServiceWrapper> objResponse = response.Content.ReadAsAsync<Response<ServiceWrapper>>().Result;
if (objResponse != null)
{
if (objResponse.isSuccess == true)
{
listContact = objResponse.data.lContact;
int MaxPKinSelectedRecords = objResponse.data.MaxPKinSelectedRecords;
int MaxPKinTotalRecords = objResponse.data.MaxPKinTotalRecords;
if (listContact != null && listContact.Count>0)
{
try
{
Parallel.ForEach(listContact, contact =>
{
// some code...
});
if (MaxPKinTotalRecords == MaxPKinSelectedRecords)
{
break;
}
}
catch (Exception ex)
{
// Logging
}
}
else
break; //Important
}
else
{
// Logging
break;
}
}
else
{
// Logging
break;
}
} // End while
}
catch (Exception ex)
{
// Logging
}
}

Method runs for every instance of the method

I am trying to set my ASP row and cell color based off my method in my data class. I have the color set and the method running. The method returns true or false based on the conditions below. How can I prevent the method from running multiple times?
For instance, it runs once and returns correctly then it runs again with "" in the lineNum and brandNum fields with the pack count as the same number as the first run. It processed to run once for every instance of my method call. Then when I actually call the method again it runs everything again. Why is it running more than once and how can I fix this?
output:
lineNum:123 brandNum:456 packCount:15
second run (not asked for)
lineNum:"" brandNum:"" packCount:15
Method to determine if true:
SqlParameter[] parameters = new SqlParameter[]
{
new SqlParameter("#GenNum6", itemNum),
new SqlParameter("#GenTxt9", brandNum)
};
try
{
reader = App_Code.DBHelper.executeQuery(dbConn,
sqlString.ToString(), parameters);
if (reader.HasRows)
{
while (reader.Read())
{
PackCount = reader["PackCount"].ToString();
}
}
reader.Close();
reader.Dispose();
dbConn.Close();
dbConn.Dispose();
}
catch (Exception ex)
{
throw ex;
}
finally
{
if (dbConn != null)
{
try { dbConn.Close(); dbConn.Dispose(); }
catch { }
}
if (reader != null)
{
try { reader.Close(); reader.Dispose(); }
catch { }
}
}
if (Convert.ToInt32(PackCount) <= 12)
{
return true;
}
else
{
return false;
}
Call:
if (lineData.PacksLeft1(L1.Text,L2.Text))
{
myTable.Rows[1].Cells[0].BgColor = "#FE2E2E";
myTable.Rows[1].Cells[1].BgColor = "#FE2E2E";
myTable.Rows[1].Cells[2].BgColor = "#FE2E2E";
myTable.Rows[1].Cells[3].BgColor = "#FE2E2E";
}

There is no need for the finally statement as it performs the same action that is performed in the try statement. I believe that after seeing this piece, if you go back through where it is being called from, you may have it running in a try catch, doing the exact same thing. I would suggest taking the
reader.Close();
reader.Dispose();
dbConn.Close();
dbConn.Dispose();
out of the try block and leaving the finally statement. This way, no matter the outcome, these items are performed.
Finally is only used when you want the code to run something after it has preformed the try statement. Such as:
Try{
//Try this statement, if it works, run it.
}
catch{
//error occurred
}
finally{
//run this code whether the try failed or not.
}

Weird error on transactions -> using NonQuery getting Reader must be closed

I'm developing a client in C# and POSTGRESQL.
It needs to parse some texts and insert data in the tables correctly, so we have a parsed which gives me a Dictionary for each tables (4 at the moment).
So we have a thread which insert in a queue those dictionaries on a ConcurrentQueue.
Now, we have two timers:
1) every 10 seconds commit an opened transaction and recreates one
these are the methods:
void transactionTimer_Elapsed(object sender, System.Timers.ElapsedEventArgs e)
{
try
{
Commit();
}
catch (Exception ex)
{
Logger.Log(Logger.LogType.ERROR, ex);
Rollback();
}
Transaction();
}
}
public void Commit()
{
if (trans != null)
{
trans.Commit();
trans.Dispose();
trans = null;
}
}
public void Transaction()
{
if (dbCon == null || (dbCon != null && dbCon.State != ConnectionState.Open))
dbCon = Connection;
if (trans == null)
trans = dbCon.BeginTransaction();
}
public void Rollback()
{
if (trans != null)
{
trans.Rollback();
trans.Dispose();
trans = null;
}
}
2) pick one hundred of data on the queue and do a huge insert ( not 100* insert by just one by using paramereters like this:
insert into Tabletest1_HandsData( handDataId, handData) values( #handDataId0, #handData0),( #handDataId1, #handData1),( #handDataId2, #handData2),( #handDataId3, #handData3) ....
by using this helper
public bool Insert(String tableName, Dictionary<String, object> data, bool usingTransaction = false)
{
Boolean returnCode = true;
var sql = GetSQL(tableName, data);
try
{
int rowsUpdated = -1;
var conn = (!usingTransaction) ? Connection : dbCon;
DbCommand mycommand = GetCommand(conn, sql);
GetCommandByDictionary(mycommand, data);
rowsUpdated = mycommand.ExecuteNonQuery();
if (!usingTransaction)
{
conn.Close();
conn.Dispose();
}
}
catch (Exception ex)
{
Logger.Log(Logger.LogType.ERROR, ex);
returnCode = false;
}
return returnCode;
}
protected override void GetCommandByDictionary(DbCommand cmd, Dictionary<string, object> data)
{
foreach (var val in data)
(cmd as NpgsqlCommand).Parameters.AddWithValue(val.Key.ToString(), val.Value);
}
So, we parse and insert on a queue, then every 3 seconds we pick 100 of these and insert, and every 10 second a transaction is committed and recreated
My error is that ExecuteNonQuery is giving me:
There is already an open DataReader associated with this Command which must be closed first.
Why is that happening?
I may take this opportunity to ask you, how can I do better? please feel free to insult me, I have tried a lot ot stuffs
Thanks
Luca