We Keep Coding

SQL Server CE two way sync with remote Access database

I'm working on a pretty special, legacy project where I need to build an app for PDA devices under Windows Mobile 6.5. The devices have a local database (SQL Server CE) which we are supposed to sync with a remote database (Microsoft Access) whenever they are docked and have network access.
So the local database using SQL Server CE works fine, but I can’t figure out a way to sync it to the Access database properly.
I read that ODBC and OLEDB are unsupported under Windows Mobile 6.5, most ressources I find are obsolete or have empty links, and the only way I found was to export the local database relevant tables in XML in the hope to build a VBA component for Access to import them properly. (and figure out backwards sync).
Update on the project and new questions
First of all, thanks to everyone who provided an useful answer, and to #josef who saved me a lot of time with the auto path on this thread.
So a remote SQL Server is a no go for security reasons (client is paranoid about security and won't provide me a server). So I'm tied to SQL Server CE on the PDA and Access on the computer.
As for the sync:
The exportation is fine: I'm using multiple dataAdapters and a WriteXML method to generate XML files transmitted by FTP when the device is plugged back in. Those files are then automatically imported into the Access database. (see code at the end).
My problem is on the importation: I can acquire data through XML readers from an Access-generated file. This data is then inserted in a dataset (In fact, I can even print the data on the PDA screen) but I can't figure out a way to do an "UPSERT" on the PDA's database. So I need a creative way to update/insert the data to the tables if they already contains data with the same id.
I tried two methods, with SQL errors (from what I understood it's SQL Server CE doesn't handle stored procedures or T-SQL). Example with a simple query that is supposed to update the "available" flag of some storage spots:
try
{
SqlCeDataAdapter dataAdapter = new SqlCeDataAdapter();
DataSet xmlDataSet = new DataSet();
xmlDataSet.ReadXml(localPath +#"\import.xml");
dataGrid1.DataSource = xmlDataSet.Tables[1];
_conn.Open();
int i = 0;
for (i = 0; i <= xmlDataSet.Tables[1].Rows.Count - 1; i++)
{
spot = xmlDataSet.Tables[1].Rows[i].ItemArray[0].ToString();
is_available = Convert.ToBoolean(xmlDataSet.Tables[1].Rows[i].ItemArray[1]);
SqlCeCommand importSpotCmd = new SqlCeCommand(#"
IF EXISTS (SELECT spot FROM spots WHERE spot=#spot)
BEGIN
UPDATE spots SET available=#available
END
ELSE
BEGIN
INSERT INTO spots(spot, available)
VALUES(#spot, #available)
END", _conn);
importSpotCmd.Parameters.Add("#spot", spot);
importSpotCmd.Parameters.Add("#available", is_available);
dataAdapter.InsertCommand = importSpotCmd;
dataAdapter.InsertCommand.ExecuteNonQuery();
}
_conn.Close();
}
catch (SqlCeException sql_ex)
{
MessageBox.Show("SQL database error: " + sql_ex.Message);
}
I also tried this query, same problem SQL server ce apparently don't handle ON DUPLICATE KEY (I think it's MySQL specific).
INSERT INTO spots (spot, available)
VALUES(#spot, #available)
ON DUPLICATE KEY UPDATE spots SET available=#available
The code of the export method, fixed so it works fine but still relevant for anybody who wants to know:
private void exportBtn_Click(object sender, EventArgs e)
{
const string sqlQuery = "SELECT * FROM storage";
const string sqlQuery2 = "SELECT * FROM spots";
string autoPath = System.IO.Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().GetName().CodeBase); //get the current execution directory
using (SqlCeConnection _conn = new SqlCeConnection(_connString))
{
try
{
SqlCeDataAdapter dataAdapter1 = new SqlCeDataAdapter(sqlQuery, _conn);
SqlCeDataAdapter dataAdapter2 = new SqlCeDataAdapter(sqlQuery2, _conn);
_conn.Open();
DataSet ds = new DataSet("SQLExport");
dataAdapter1.Fill(ds, "stock");
dataAdapter2.Fill(ds, "spots");
ds.WriteXml(autoPath + #"\export.xml");
}
catch (SqlCeException sql_ex)
{
MessageBox.Show("SQL database error: " + sql_ex.Message);
}
}
}

As Access is more or less a stand-alone DB solution I strongly recommend to go with a full flavored SQL Server plus IIS to setup a Merge Replication synchronisation between the SQL CE data and the SQL Server data.
This is described with full sample code and setup in the book "Programming the .Net Compact Framework" by Paul Yao and David Durant (chapter 8, Synchronizing Mobile Data).
For a working sync, all changes to defined tables and data on the server and the CE device must be tracked (done via GUIDs, unique numbers) with there timestamps and a conflict handling has to be defined.
If the data is never changed by other means on the server, you may simply track Device side changes only and then push them to the Access database. This could be done by another app that does Buld Updates like described here.
If you do not want to go the expensive way to SQL Server, there are cheaper solutions with free SQLite (available for CE and Compact Framework too) and a commercial Sync tool for SQLite to MSAccess like DBSync.
If you are experienced, you may create your own SQLite to MS ACCESS sync tool.

Get the execution time of a ADO.NET SQL Command

I have been searching over to find if there is any easy way to get the Execution time of a ADO.NET command object.
I know i can manually do a StopWatch start and stop. But wanted to if there are any easy way to do it in ADO.NET

There is a way, but using SqlConnection, not command object. Example:
using (var c = new SqlConnection(connectionString)) {
// important
c.StatisticsEnabled = true;
c.Open();
using (var cmd = new SqlCommand("select * from Error", c)) {
cmd.ExecuteReader().Dispose();
}
var stats = c.RetrieveStatistics();
var firstCommandExecutionTimeInMs = (long) stats["ExecutionTime"];
// reset for next command
c.ResetStatistics();
using (var cmd = new SqlCommand("select * from Code", c))
{
cmd.ExecuteReader().Dispose();
}
stats = c.RetrieveStatistics();
var secondCommandExecutionTimeInMs = (long)stats["ExecutionTime"];
}
Here you can find what other values are contained inside dictionary returned by RetrieveStatistics.
Note that those values represent client-side statistics (basically internals of ADO.NET measure them), but seems you asked for analog of Stopwatch - I think that's fine.

The approach from the answer of #Evk is very interesting and smart: it's working client side and one of the main key of such statistics is in fact NetworkServerTime, which
Returns the cumulative amount of time (in milliseconds) that the
provider spent waiting for replies from the server once the
application has started using the provider and has enabled statistics.
so it includes the network time from the DB server to the ADO NET client.
An alternative, more DB server oriented, would be running SET STATISTICS TIME ON and then retrieve the InfoMessage.
A draft of the code of the delegate (where I'm simply writing to the debug console, but you may want to replace it with a StringBuilder Append)
internal static void TrackInfo(object sender, SqlInfoMessageEventArgs e)
{
Debug.WriteLine(e.Message);
foreach (var element in e.Errors) {
Debug.WriteLine(element.ToString());
}
}
and usage
conn.InfoMessage += TrackInfo;
using (var cmd = new SqlCommand(#"SET STATISTICS TIME ON", conn)) {
cmd.ExecuteNonQuery();
}
using (var cmd = new SqlCommand(yourQuery, conn)) {
var RD = cmd.ExecuteReader();
while (RD.Read()) {
// read the columns
}
}

I suggest you move to SQL Server 2016 and use the Query Store feature. This will track execution time and performance changes over time for each query you submit. Requires no changes in your application. Track all queries, including those executed inside stored procedures. Track any application, not only your own. Is available in all editions, including Express, and including the Azure SQL DB Service.
If you track on the client side, you must measure the time yourself, using a wall clock. I would add and expose performance counters and then use the performance counters infrastructure to capture and store the measurements.
As a side not, simply tracking the execution time of a batch sent to SQL Server yields very coarse performance info and is seldom actionable. Read How to analyse SQL Server performance.

Slow opening SQLite connection in C# app using System.Data.SQLite

Edit 3:
I guess my issue is resolved for the moment... I changed both my service and test app to run as the SYSTEM account instead of the NetworkService account. It remains to be seen if the benefits of changing the user account will persist, or if it will only be temporary.
Original Question:
I've noticed that my small 224kB SQLite DB is very slow to open in my C# application, taking anywhere from some small number of ms, to 1.5 seconds or more. Below is my code, with all the extra debugging statements I've added this afternoon. I've narrowed it down to the call to cnn.Open(); as shown in the logs here:
2014-03-27 15:05:39,864 DEBUG - Creating SQLiteConnection...
2014-03-27 15:05:39,927 DEBUG - SQLiteConnection Created!
2014-03-27 15:05:39,927 DEBUG - SQLiteConnection Opening...
2014-03-27 15:05:41,627 DEBUG - SQLiteConnection Opened!
2014-03-27 15:05:41,627 DEBUG - SQLiteCommand Creating...
2014-03-27 15:05:41,627 DEBUG - SQLiteCommand Created!
2014-03-27 15:05:41,627 DEBUG - SQLiteCommand executing reader...
2014-03-27 15:05:41,658 DEBUG - SQLiteCommand executed reader!
2014-03-27 15:05:41,658 DEBUG - DataTable Loading...
2014-03-27 15:05:41,767 DEBUG - DataTable Loaded!
As you can see, in this instance it took 1.7 SECONDS to open the connection. I've tried repeating this, and cannot predict whether subsequent connections will open nearly immediately, or be delayed like this.
I've considered using some form of connection pooling, but is it worthwhile to pursue that for a single-instance single-threaded application? Right now, I'm creating an instance of my SQLiteDatabase class, and calling the below function for each of my queries.
public DataTable GetDataTable(string sql)
{
DataTable dt = new DataTable();
try
{
Logging.LogDebug("Creating SQLiteConnection...");
using (SQLiteConnection cnn = new SQLiteConnection(dbConnection))
{
Logging.LogDebug("SQLiteConnection Created!");
Logging.LogDebug("SQLiteConnection Opening...");
cnn.Open();
Logging.LogDebug("SQLiteConnection Opened!");
Logging.LogDebug("SQLiteCommand Creating...");
using (SQLiteCommand mycommand = new SQLiteCommand(cnn))
{
Logging.LogDebug("SQLiteCommand Created!");
mycommand.CommandText = sql;
Logging.LogDebug("SQLiteCommand executing reader...");
using (SQLiteDataReader reader = mycommand.ExecuteReader())
{
Logging.LogDebug("SQLiteCommand executed reader!");
Logging.LogDebug("DataTable Loading...");
dt.Load(reader);
Logging.LogDebug("DataTable Loaded!");
reader.Close();
}
}
cnn.Close();
}
}
catch (Exception e)
{
throw new Exception(e.Message);
}
return dt;
}
Edit:
Sure, dbConnection is the connection string, set by the following function. inputFile is just the string path of the filename to open.
public SqLiteDatabase(String inputFile)
{
dbConnection = String.Format("Data Source={0}", inputFile);
}
And at this point, I think sql is irrelevant, as it's not making it to that point when the cnn.Open() stalls.
Edit 2:
Ok, I've done some more testing. Running the testing locally, it completes a 1000 iteration loop in ~5 seconds, for about 5ms per call to cnn.Open(). Running the test from the same windows installer that I did on my local PC, it completes in ~25 minutes, averaging 1468ms per call to cnn.Open().
I made a small test program that only calls the TestOpenConn() function from the service program (same exact code that is running in the Windows service), running against a copy of the file located in a test directory. Running this on the server or my local PC results in acceptable performance (1.95ms per call on the server, 4ms per call on my local PC):
namespace EGC_Timing_Test
{
class Program
{
static void Main(string[] args)
{
Logging.Init("log4net.xml", "test.log");
var db = new SqLiteDatabase("config.sqlite");
db.TestOpenConn();
}
}
}
Here's the test function:
public void TestOpenConn()
{
// TODO: Remove this after testing loop of opening / closing SQLite DB repeatedly:
const int iterations = 1000;
Logging.LogDebug(String.Format("Running TestOpenConn for {0} opens...", iterations));
var startTime = DateTime.Now;
for (var i = 0; i < iterations; i++)
{
using (SQLiteConnection cnn = new SQLiteConnection(dbConnection))
{
Logging.LogDebug(String.Format("SQLiteConnection Opening, iteration {0} of {1}...", i, iterations));
var startTimeInner = DateTime.Now;
cnn.Open();
var endTimeInner = DateTime.Now;
var diffTimeInner = endTimeInner - startTimeInner;
Logging.LogDebug(String.Format("SQLiteConnection Opened in {0}ms!", diffTimeInner.TotalMilliseconds));
cnn.Close();
}
}
var endTime = DateTime.Now;
var diffTime = endTime - startTime;
Logging.LogDebug(String.Format("Done running TestOpenConn for {0} opens!", iterations));
Logging.LogInfo(String.Format("{0} iterations total:\t{1}", iterations, diffTime));
Logging.LogInfo(String.Format("{0} iterations average:\t{1}ms", iterations, diffTime.TotalMilliseconds/iterations));
}

I guess my issue is resolved for the moment... I changed both my service and test app to run as the SYSTEM account instead of the NetworkService account. It remains to be seen if the benefits of changing the user account will persist, or if it will only be temporary.

I'm assuming you're using the open source System.Data.SQLite library.
If that's the case, it's easy to see through the Visual Studio Performance Profiler that the Open method of the SQLiteConnection class has some serious performance issues.
Also, have a lookthrough the source code for this class here: https://system.data.sqlite.org/index.html/artifact/97648754af51ffd6
There's an awful lot of disk access being made to read XML configuration and Windows environment variable(s).
My suggestion is to try and call Open() as seldom as possible, and try and keep a reference to this open SQLiteConnection object around in memory.
A performance ticket is raised with SQLite Forum

Having had the same problem, I was looking into this and it seems to be related to permissions on the file or it's parent folders, who created it, and/or how it was created. In my case, the SQLite database file was being created by a script run as a regular user, and then an IIS-hosted service would access the file under a different domain service account.
Every time the service opened a connection, it took over 1.5 seconds, but otherwise operated correctly (it could eventually access the file). A stand-alone program running as the regular user could open a connection to the same file in the same place in a few milliseconds.
Analysis of a procmon trace revealed that in the case of the service, we were getting several ACCESS DENIED logs on the file over the course of about 1.5 seconds, that were not present in the trace when running as the regular user.
Not sure what's going on there. The service worked fine and was able to eventually query the data in the file, albeit slowly.
When we made the service account the owner of the parent folder of the file, and gave him write permission, the ACCESS DENIED logs disappeared and the service operated at full speed.

You can add "Modify" permissions of appropriate user to folder with your database.
Right Click on folder > Properties > Security > Edit > Add (I added IIS_Users) > Select "Modify" checkbox > OK

SQL Server CE not picking up updates from another process?

I've got two processes with connections to the same SQL CE .sdf database file. One inserts items into a table and the other reads all the records from the table. After the insert I can confirm the rows are there with the Server Explorer but my query from the second process does not show them:
this.traceMessages.Clear();
SqlCeCommand command = new SqlCeCommand("SELECT AppName, Message, TraceId FROM Messages", this.connection);
using (var reader = command.ExecuteReader())
{
while (reader.Read())
{
this.traceMessages.Add(
new TraceMessage
{
AppName = reader.GetString("AppName"),
Message = reader.GetString("Message"),
TraceId = reader.GetString("TraceId")
});
}
}
It can generally load up correctly the first time but doesn't pick up updates, even after restarting the process. The connection string just has a simple Data Source that I've confirmed is pointing to the same file on both processes.
Anyone know why this is happening? Is there some setting I can enable to get updates from separate processes to work?

This is because unlike "traditional" databases, the data that you write is not flushed to disk immediately, it is deferred and happens some time later.
You have two choices in the writing program:
1) Add the Flush Interval parameter to your connection string and set it to 1. This will have a lag of up to a second before the data is flushed to the sdf.
2) When you call Commit, use the parameterized overload that allows you to specify CommitMode.Immediate. This will flush data to disk immediately.

Changing SQL Server settings programmatically

I request you to read my question carefully.
You might know when you install VS2005/2008 with SQL Server Express edition, the SQL Server operates in Windows authentication mode by default. You can use the SQL Server Management Studio to change the mode to Mixed mode (Windows and SQL Server Authentication mode).
Similarly to allow the SQL Server remote connection through TCP/IP, you need to use SQL Server Configuration Manager then select Protocol for SQLEXPRESS and then change the setting for Tcp/IP option.
What i need is to automate this process programmatically using C#. That is, i need to write a c# program to change the mode or change the tcp/ip settings etc.
Can anyone provide me help on this, how could i do that?
Thank you for sharing your valuable time.

You should use SQL Server Management Objects (SMO) - this is an API for managing SQL Server programmatically.
UPDATE:
Proves to be a bit tricky: Server.LoginMode (read/write), Server.TcpEnabled and Server.NamedPipesEnabled (get only, unfortunately). In order to modify protocols, you need to examine Microsoft.SqlServer.Management.Smo.Wmi namespace (hence going from 'the other end'):
ServerProtocol - represents server protocol
ServerProtocolCollection - a collection of all protocols defined on a given server

This function in C# will enable TCP/IP Protocol and set the Login mode to Mixed mode.
See complementary information here.
here is the code:
private static bool SetServerProperties()
{
#region standardize Connection String
string tempCatalog = "master";
string temp = #"Data Source=" + dataSource + ";Initial Catalog=" + tempCatalog + ";Integrated Security=True;MultipleActiveResultSets=True";
#endregion
SqlConnection sqlconnection = new SqlConnection(temp);
SqlCommand cmd = new SqlCommand("select ##ServerName", sqlconnection);
sqlconnection.Open();
string serverName = "";
try
{
SqlDataReader dr = cmd.ExecuteReader();
while (dr.Read())
serverName = dr[0].ToString();
}
catch
{
MessageBox.Show("Failed to Set SQL Server Properties for remote connections.");
}
Server srv = new Server(serverName);
srv.ConnectionContext.Connect();
srv.Settings.LoginMode = ServerLoginMode.Mixed;
ManagedComputer mc = new ManagedComputer();
try
{
Service Mysvc = mc.Services["MSSQL$" + serverName.Split('\\')[1]];
if (Mysvc.ServiceState == ServiceState.Running)
{
Mysvc.Stop();
Mysvc.Alter();
while (!(string.Format("{0}", Mysvc.ServiceState) == "Stopped"))
{
Mysvc.Refresh();
}
}
ServerProtocol srvprcl = mc.ServerInstances[0].ServerProtocols[2];
srvprcl.IsEnabled = true;
srvprcl.Alter();
Mysvc.Start();
Mysvc.Alter();
while (!(string.Format("{0}", Mysvc.ServiceState) == "Running"))
{
Mysvc.Refresh();
}
return true;
}
catch
{
MessageBox.Show("TCP/IP connectin could not be enabled.");
return false;
}
}

What about modifying the registry?
Client Protocol Settings are stored here:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSSQLServer\Client\SNI9.0
Check out ProtocolOrder.
Authentication Mode is stored here:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL.1\MSSQLServer\LoginMode
See:
Authentication Settings

I was able to do this with a small footprint by executing this stored procedure from C#:
USE [master]
EXEC xp_instance_regwrite N'HKEY_LOCAL_MACHINE', N'Software\Microsoft\MSSQLServer\MSSQLServer', N'LoginMode', REG_DWORD, 2
GO
It doesn't look like much but works flawlessly and instantly, without restarting services.

I think you could solve your problem making a silent installation of SQL Server Express edition using a configuration file for the install process.
In this link you can find the command line parameters for the installation.
In this one you can find how to make your configuration file.