We Keep Coding

Database is locked: SQLite exception thrown on my android and ios xamarin app

The database was very stable until I did all the package and bundle signing and deploy the app to the app center for distributed testing, then I started getting all kind of errors from the SQLite database. I did some research and got to know that the database is not threading safe, but how can I make it thread safe, all my database connections and queries are through async.
This is my database connection
private static object collisionLock = new object();
//local SQLite database.
public SQLiteAsyncConnection Db;
private Uri url;
private MemberDataStore membersStore;
private AdvantageMemberDataStore advMembersStore;
private EventDataStore eventDataStore;
public ConnectionsUtil()
{
Db = DependencyService.Get<IDatabaseConnection>().DbConnection();
}
This is the code that does inserts and updates, I have three methods like the one below that access the database and many queries that uses the Db within the app
public async Task GetAdvantageMembersAsync(AdvantageMemberDataStore store)
{
await Db.QueryAsync<AdvantageMember>("DROP TABLE IF EXISTS AdvantageMember").ContinueWith(t =>
{
Db.CreateTableAsync<AdvantageMember>();
Console.WriteLine("AdvantageMember table created!");
});
url = new Uri("http://54.39.180.209/benefits");
Console.WriteLine("Gideon: Attempting to get advantage members.");
try
{
string json;
//Gideon: get json string containing advantage members.
using (WebClient sr = new WebClient())
{
Console.WriteLine("Gideon: Retrieving advantage Members....");
sr.DownloadStringCompleted += async (s, e) =>
{
Console.WriteLine("Gideon: Advantage Members Downloaded. Processing advantage Members....");
json = e.Result;
lock (collisionLock)
{
Db.InsertAllAsync(JsonConvert.DeserializeObject<IEnumerable<AdvantageMember>>(json)).ContinueWith(async t =>
{
await store.UpdateItemsAsync(Db.QueryAsync<AdvantageMember>("SELECT * FROM AdvantageMember GROUP BY Title").Result);
});
}
Console.WriteLine("Processing Members benefits for android");
advMembersStore = store;
};
await Task.Run(() => sr.DownloadStringAsync(url));
}
}
catch (Exception e)
{
Console.WriteLine("Gideon: An error occured while trying to dbect. ERROR: " + e);
}
}
**This is the code that specify the folder for the sqlite connection**
public class DatabaseConnection_iOS: Services.IDatabaseConnection
{
public SQLiteAsyncConnection DbConnection()
{
var dbName = "MyDb.db3";
string personalFolder =
Environment.
GetFolderPath(Environment.SpecialFolder.Personal);
string libraryFolder =
Path.Combine(personalFolder, "..", "Library");
var path = Path.Combine(libraryFolder, dbName);
return new SQLiteAsyncConnection(path);
}
}

In my case, I was put SaveContext inside foreach
var transaction = _context.Database.BeginTransaction();
try
{
foreach (var item in obj)
{
_context.EntrevistaNacionals.Add(item);
await _context.SaveChangesAsync(); //--wrong
}
transaction.Commit();
return true;
}

UI will be blocked while using async and await in c#

Here I need to export some data from a local server into Excel. I am using an async call for that method. But still, it is blocking the UI while exporting the data.
May I know the reason why the UI is being blocking?
Also, I need some clarification
1)await keyword is using to wait some time until the process is
completed, then what is the difference b/w synchronous and
asynchronous process.
2)If one method is declared as Async task, then all inner methods
are performing as asynchronous?
3)While Executing inner methods(method1,method2,method3), method3 will
be depends upon method1. So, put await keyword for method1. Am I
right?
Code snippet:
private async void ConvertExcel(object sender)
{
var excelEngine = new ExcelEngine();
var application = excelEngine.Excel;
var newbookWorkbook = application.Workbooks.Create(1);
var work = newbookWorkbook.Worksheets[0];
var openDialog = new SaveFileDialog
{
FilterIndex = 2,
Filter = "Excel 97 to 2003 Files(*.xls)|*.xls|Excel 2007 to 2013 Files(*.xlsx)|*.xlsx"
};
if (openDialog.ShowDialog() == true)
{
newbookWorkbook.Version = openDialog.FilterIndex == 1 ? ExcelVersion.Excel97to2003 : ExcelVersion.Excel2013;
}
else
return;
try
{
// This method is used to get the data from server.
var table = GetFullDataAsync();
work.ImportDataTable(table, true, 1, 1, true);
using (Stream stream = openDialog.OpenFile())
{
newbookWorkbook.SaveAs(stream);
}
newbookWorkbook.Close();
excelEngine.Dispose();
}
catch (Exception exception)
{
Console.WriteLine("Exception message: {0}",ex.Message);
}
}
internal async Task<DataTable> GetFullDataAsync()
{
DataTable dataTable = new DataTable();
dataTable = GetDataFromServer(DataEngine.Query,DataEngine.Connection);
dataTable.Locale = CultureInfo.InvariantCulture;
return dataTable;
}
public DataTable GetDataFromServer(string query, DbConnection connection)
{
if (connection.State != ConnectionState.Open)
{
connection.Open();
}
var command = connection.CreateCommand();
command.CommandText = query;
command.Connection = connection;
var reader = command.ExecuteReader();
var dataTable = new DataTable
{
Locale = System.Globalization.CultureInfo.InvariantCulture
};
dataTable.Load(reader);
return dataTable;
}

1) await keyword is using for wait some time until the process is completed, then what is the difference b/w synchronous and asynchronous process.
synchronous: waiting blocks the thread, so locks the ui
asynchronous: waiting frees up the thread, so does not block the ui
2) If one method is declared as Async task, then all inner methods are performing as asynchronous?
Don't assume that is true. You probably want that to be the case, but it is up to you as the developer to code it that way.
The complier does not enforce this for you.
In your code above, the answer is no; your code is not asynchronous. Notice that you never use the await keyword.
You could need to access the data in an asynchronous way first (returning a Task), otherwise async/await keywords can't help you.
3) While Executing inner methods(method1,method2,method3), method3 will be depends upon method1. So, put await keyword for method1. Am i right?
Yes. If each method is returning a Task, then await them all.
Typically when you starting using await in one place, it will bubble up through the rest of your code, so you keep adding more awaits as you have descibed.

have you tried
GetFullDataAsync().ContinueWith((table)=>{
work.ImportDataTable(table, true, 1, 1, true);
using (Stream stream = openDialog.OpenFile())
{
newbookWorkbook.SaveAs(stream);
}
newbookWorkbook.Close();
excelEngine.Dispose();
})

You can change your code to use async communication with Sql server like that:
private async Task ConvertExcel(object sender)
{
var excelEngine = new ExcelEngine();
var application = excelEngine.Excel;
var newbookWorkbook = application.Workbooks.Create(1);
var work = newbookWorkbook.Worksheets[0];
var openDialog = new SaveFileDialog
{
FilterIndex = 2,
Filter = "Excel 97 to 2003 Files(*.xls)|*.xls|Excel 2007 to 2013 Files(*.xlsx)|*.xlsx"
};
if (openDialog.ShowDialog() == true)
{
newbookWorkbook.Version = openDialog.FilterIndex == 1 ? ExcelVersion.Excel97to2003 : ExcelVersion.Excel2013;
}
else
return;
try
{
// This method is used to get the data from server.
var table = await GetFullDataAsync();
work.ImportDataTable(table, true, 1, 1, true);
var saveWorkbookTask = Task.Run(() => {
using (Stream stream = openDialog.OpenFile())
{
newbookWorkbook.SaveAs(stream);
}
});
await saveWorkbookTask;
newbookWorkbook.Close();
excelEngine.Dispose();
}
catch (Exception exception)
{
Console.WriteLine("Exception message: {0}",ex.Message);
}
}
internal async Task<DataTable> GetFullDataAsync()
{
DataTable dataTable = new DataTable();
dataTable = await GetDataFromServer(DataEngine.Query,DataEngine.Connection);
dataTable.Locale = CultureInfo.InvariantCulture;
return dataTable;
}
public async Task<DataTable> GetDataFromServer(string query, DbConnection connection)
{
if (connection.State != ConnectionState.Open)
{
connection.Open();
}
var command = connection.CreateCommand();
command.CommandText = query;
command.Connection = connection;
var reader = await command.ExecuteReaderAsync();
var loadTableTask = Task.Run(() => {
var dataTable = new DataTable
{
Locale = System.Globalization.CultureInfo.InvariantCulture
};
dataTable.Load(reader);
return dataTable;
});
return await loadTableTask;
}
But you still have blocking IO operations. So if your file is large than the UI will be blocked when the file will be written to disk.

Microsoft.Azure.NotificationHub - Async call in sync method

I currently have to provide a sync as async method in my API: Please find the code below. The only problem is that I don’t have a
sync method in the backend. I use Azure.NotificationHub client. That client has only *Async methods. Is my way reasonable?
public PushHubNotificationResult SendPushMessage(string userId, string message)
{
PushHubNotificationResult result = new PushHubNotificationResult();
try
{
result = SendPushMessageAsync(userId, message).GetAwaiter().GetResult();
} catch (Exception ex)
{
result.Status = PushHubNotificationResultType.Error;
result.Error = ex.Message;
result.Exception = ex;
}
return result;
}
public async Task<PushHubNotificationResult> SendPushMessageAsync(string userId, string message)
{
PushHubNotificationResult result = new PushHubNotificationResult();
// EnableTestSend see: https://azure.microsoft.com/en-us/documentation/articles/notification-hubs-push-notification-fixer/#self-diagnose-tips
// Create a new Notification Hub client.
Microsoft.Azure.NotificationHubs.NotificationHubClient hub =
Microsoft.Azure.NotificationHubs.NotificationHubClient.CreateClientFromConnectionString(NotificationHub, NotificationHubName);
// Sending the message so that all template registrations that contain "messageParam"
// will receive the notifications. This includes APNS, GCM, WNS, and MPNS template registrations.
Dictionary<string, string> templateParams = new Dictionary<string, string>();
templateParams["messageParam"] = message;
string userTag = "_UserId:" + userId; // That line sets the IMEI or SerialNo (WLAN only device) == userId to which the push message is sent
try
{
// Send the push notification and log the results.
NotificationOutcome outcome = await hub.SendTemplateNotificationAsync(templateParams, userTag);
result.Status = PushHubNotificationResultType.Success;
foreach (RegistrationResult hubResult in outcome.Results)
{
result.PushNotificationHub = hubResult.ApplicationPlatform;
result.RegistrationId = hubResult.RegistrationId;
result.Outcome = hubResult.Outcome;
}
}
catch (System.Exception ex)
{
result.Status = PushHubNotificationResultType.Error;
result.Error = ex.Message;
result.Exception = ex;
}
return result;
}
thanks for any advice,
Eric

If you want to use sync-over-async, it's very important that you use ConfigureAwait(false) in your async code, otherwise you are very likely to get a deadlock.
NotificationOutcome outcome =
await hub.SendTemplateNotificationAsync(templateParams, userTag).ConfigureAwait(false);
The async method already converts exceptions to PushHubNotificationResultType.Error, why does the sync version do it too?

Nested SqlConnection.Open throws an exception inside TransactionScope

I am using TransactionScope in my repository unit tests to rollback any changes made by tests.
Setup and teardown procedures for tests look like this:
[TestFixture]
public class DeviceRepositoryTests {
private static readonly string ConnectionString =
ConfigurationManager.ConnectionStrings["TestDB"].ConnectionString;
private TransactionScope transaction;
private DeviceRepository repository;
[SetUp]
public void SetUp() {
transaction = new TransactionScope(TransactionScopeOption.Required);
repository = new DeviceRepository(ConnectionString);
}
[TearDown]
public void TearDown() {
transaction.Dispose();
}
}
Problematic test consists of code which inserts records to database and CUT that retrieves those records.
[Test]
public async void GetAll_DeviceHasSensors_ReturnsDevicesWithSensors() {
int device1Id = AddDevice();
AddSensor();
var devices = await repository.GetAllAsync();
// Asserts
}
AddDevice and AddSensor methods open sql connection and insert a row into a database:
private int AddDevice() {
var sqlString = "<SQL>";
using (var connection = CreateConnection())
using (var command = new SqlCommand(sqlString, connection)) {
var insertedId = command.ExecuteScalar();
Assert.AreNotEqual(0, insertedId);
return (int) insertedId;
}
}
private void AddSensor() {
const string sqlString = "<SQL>";
using (var connection = CreateConnection())
using (var command = new SqlCommand(sqlString, connection)) {
var rowsAffected = command.ExecuteNonQuery();
Assert.AreEqual(1, rowsAffected);
}
}
private SqlConnection CreateConnection() {
var result = new SqlConnection(ConnectionString);
result.Open();
return result;
}
GetAllAsync method opens a connection, executes query, and for each fetched row opens new connection to fetch child objects.
public class DeviceRepository {
private readonly string connectionString;
public DeviceRepository(string connectionString) {
this.connectionString = connectionString;
}
public async Task<List<Device>> GetAllAsync() {
var result = new List<Device>();
const string sql = "<SQL>";
using (var connection = await CreateConnection())
using (var command = GetCommand(sql, connection, null))
using (var reader = await command.ExecuteReaderAsync()) {
while (await reader.ReadAsync()) {
var device = new Device {
Id = reader.GetInt32(reader.GetOrdinal("id"))
};
device.Sensors = await GetSensors(device.Id);
result.Add(device);
}
}
return result;
}
private async Task<List<Sensor>> GetSensors(int deviceId) {
var result = new List<Sensor>();
const string sql = "<SQL>";
using (var connection = await CreateConnection())
using (var command = GetCommand(sql, connection, null))
using (var reader = await command.ExecuteReaderAsync()) {
while (await reader.ReadAsync()) {
// Fetch row and add object to result
}
}
return result;
}
private async Task<SqlConnection> CreateConnection() {
var connection = new SqlConnection(connectionString);
await connection.OpenAsync();
return connection;
}
}
The problem is that when GetSensors method calls SqlConnection.Open I get following exception:
System.Transactions.TransactionAbortedException : The transaction has aborted.
----> System.Transactions.TransactionPromotionException : Failure while attempting to promote transaction.
----> System.Data.SqlClient.SqlException : There is already an open DataReader associated with this Command which must be closed first.
----> System.ComponentModel.Win32Exception : The wait operation timed out
I could move code that fetches child object out of the first connection scope (this would work), but let's say I don't want to.
Does this exception mean that it is impossible to open simultaneous connections to DB inside single TransactionScope?
Edit
GetCommand just calls SqlCommand contructor and do some logging.
private static SqlCommand GetCommand(string sql, SqlConnection connection, SqlParameter[] parameters) {
LogSql(sql);
var command = new SqlCommand(sql, connection);
if (parameters != null)
command.Parameters.AddRange(parameters);
return command;
}

The issue is that two DataReader objects can't be open at the same time against the database (unless MARS is enabled). This restriction is by design. As I see it you have a few options:
Enable MARS on your connection string; add this MultipleActiveResultSets=True
Don't use the DataReader if it's really not necessary. But the way you've got your code written, it's pretty necessary.
Populate the Sensor property after loading the devices.
Use Dapper, it can do all of this (including populate the Sensor) and likely faster.
Using Dapper you could do something like this (and you wouldn't need GetSensors):
public async Task<List<Device>> GetAllAsync() {
var result = new List<Device>();
const string sql = "<SQL>";
using (var connection = await CreateConnection())
using (var multi = connection.QueryMultiple(sql, parms)) {
result = multi.Read<Device>().ToList();
var sensors = multi.Read<Sensors>().ToList();
result.ForEach(device => device.Sensors =
sensors.Where(s => s.DeviceId = device.Id).ToList());
}
return result;
}
Here your sql would look like this:
SELECT * FROM Devices
SELECT * FROM Sensors
See the Multi Mapping documentation for Dapper.

Threading issues with SQLite3 and C# async

I am trying to save some data to a SQLite3 database. If I do not use async, I can save the data without any problems. As soon as I try to use the following code however, I receive the following error:
{Unable to evaluate expression because the code is optimized or a native frame is on top of the call stack.}
From my UI, I invoke the following SyncDomainTablesAsync method:
private readonly IDataCoordinator _coordinator;
public Configuration(IDataCoordinator coordinator)
{
_coordinator = coordinator;
}
public async Task<int> SyncDomainTablesAsync(IProgress<string> progress, CancellationToken ct, DateTime? lastDateSynced=null, string tableName = null)
{
//Determine the different type of sync calls
// 1) Force Resync (Drop/Create Tables and Insert)
// 2) Auto Update
var domainTable = await GetDomainTablesAsync(progress,ct,lastDateSynced, tableName);
var items = domainTable.Items;
int processCount = await Task.Run<int>( async () =>
{
int p = 0;
progress.Report(String.Format("Syncing Configurations..."));
foreach (var item in items)
{
progress.Report(String.Format("Syncing {0} Information",item.Key));
var task = await SyncTableAsync(item.Value); // INVOKED BELOW
if (task) progress.Report(String.Format("Sync'd {0} {1} records", item.Value.Count,item.Key));
if (ct.IsCancellationRequested) goto Cancelled;
p += item.Value.Count;
}
Cancelled:
if (ct.IsCancellationRequested)
{
//Update Last Sync'd Records
progress.Report(String.Format("Canceling Configuration Sync..."));
ct.ThrowIfCancellationRequested();
}
else
progress.Report(String.Format("Syncing Configurations Compleleted"));
return p;
},ct);
return processCount;
}
private async Task<bool> SyncTableAsync(IEnumerable<object> items, bool includeRelationships = false)
{
try
{
//TODO: Replace with SaveObjects method
var i = await Task.Run(() => _coordinator.SaveObjects(items, includeRelationships));
if (i == 0)
return false;
}
catch(Exception ex)
{
return false;
}
return true;
}
The UI invokes the SyncDomainTablesAsync method. I then create a new Task and loop through the items that were returned from the GetDomainTablesAsync method. During each iteration I await until the SyncTableAsync method completes. Within the SyncTableAsync I am calling a SaveObject method inside of a class that implements my IDataCoordinator interface.
public override int SaveObjects(IEnumerable<object> items, Type underlyingType, bool saveRelationships = true)
{
int result = 0;
if (items == null)
throw new ArgumentNullException("Can not save collection of objects. The collection is null.");
else if (items.Count() == 0)
return 0;
// Check if table exists.
foreach (var item in items)
this.CreateTable(item.GetType(), saveRelationships);
using (SQLiteConnection connection = new SQLiteConnection(this.StorageContainerPath))
{
connection.BeginTransaction();
foreach (var item in items)
{
result += ProcessSave(item, saveRelationships, connection);
}
try
{
connection.Commit();
}
catch (SQLiteException ex)
{
connection.Rollback();
throw ex;
}
}
return result;
}
public override int CreateTable(Type type, bool createRelationalTables = false)
{
if (this.TableExists(type) == 1)
return 1;
using (SQLiteConnection cn = new SQLiteConnection(this.StorageContainerPath))
{
try
{
// Check if the Table attribute is used to specify a table name not matching that of the Type.Name property.
// If so, we generate a Sql Statement and create the table based on the attribute name.
//if (Attribute.IsDefined(type, typeof(TableAttribute)))
//{
// TableAttribute attribute = type.GetAttribute<TableAttribute>();
// Strongly typed to SQLiteCoordinator just to get a SqlQuery instance. The CreateCommand method will create a table based on 'type'
var query = new SqlQuery<SQLiteCoordinator>().CreateCommand(DataProviderTypes.Sqlite3, type);
query = query.TrimEnd(';') + ";";
cn.Execute(query);
//}
// Otherwise create the table using the Type.
//else
//{
// cn.CreateTable(type);
//}
// If we are to create relationship tables, we cascade through all relationship properties
// and create tables for them as well.
if (createRelationalTables)
{
this.CreateCascadingTables(type, cn);
}
}
catch (Exception ex)
{
return 0;
}
}
return 1;
}
The flow of the code goes
UI->SyncDomainTablesAsync->SyncTableAsync->SaveObjects->SaveTable(type)
The issue that I have is within Save Table. If I just use SaveTable synchronously I have no issues. Using it in my async method above, causes a thread abort exception. The exception is thrown within the SQLite.cs file included with SQLite.net (within the . The weird thing is that the table is created in the database, even though the exception is thrown. The error is thrown some times when the Prepare() function is called and the rest of the time when the SQLite3.Step() function is called.
public int ExecuteNonQuery ()
{
if (_conn.Trace) {
Debug.WriteLine ("Executing: " + this);
}
var r = SQLite3.Result.OK;
var stmt = Prepare (); // THROWS THE ERRROR
r = SQLite3.Step(stmt); // THROWS THE ERRROR
Finalize(stmt);
if (r == SQLite3.Result.Done) {
int rowsAffected = SQLite3.Changes (_conn.Handle);
return rowsAffected;
} else if (r == SQLite3.Result.Error) {
string msg = SQLite3.GetErrmsg (_conn.Handle);
throw SQLiteException.New (r, msg);
} else {
throw SQLiteException.New (r, r.ToString ());
}
}
I assume that because my foreach statement awaits the return of SyncTableAsync that none of the threads are closed. I am also getting a system transaction critical exception that says "attempting to access a unloaded app domain".
Am I using await/async incorrectly with Sqlite3 or is this an issue with Sqlite3 that I am not aware of.
Attached is a photo of the Parallel's stack and the exception.
EDIT
When I try to run the code above as well in unit tests, the unit tests process never dies. I have to exit Visual Studio in order to get the process to die. I am assuming something in SQLite.dll is grabbing a hold of the process when the exception is thrown and not letting go, but I am not sure.
EDIT 2
I can modify the initial method SyncDomainTablesAsync to the following and the code runs without error. The issue is my use of async and await I believe.
public async Task<int> SyncDomainTablesAsync(IProgress<string> progress, CancellationToken ct, DateTime? lastDateSynced=null, string tableName = null)
{
var domainTable = await GetDomainTablesAsync(progress,ct,lastDateSynced, tableName);
var items = domainTable.Items;
foreach (var item in items)
{
_coordinator.SaveObjects(item.Value, typeof(object), true);
}
return 1;
}