We Keep Coding

C# Program - Implementing Local Storage to complement SQL Server Storage

My team has a tool that we use for storing and analyzing our main product's test data. The test data is stored on an SQL Server by a loader program. We then have a client program that is used to view and analyze the data. The client program queries the database and displays it to the user.
We want to implement a way for the client program to store this information locally without having a server instance or additional software installed. We may need to use the tool on customer computers where can't easily install software (no admin). Thus installing SQL Express locally won't work for our needs. We just want to run the EXE on a PC.
Meanwhile we want to minimize the amount of re-work to enable this capability. Currently the tool queries the DB utilizing a connection string and stores the data into a DataTable object. I doubt it can be as easy as setting up a local object and changing the connection string, but that would be nice. Also, I don't mind changing how the querying works if it allows interfacing with both options. What I'm trying to avoid is having to maintain two completely separate methods for interfacing with the data.

If you can't install the database, then embed it.
Just use the old reliable SQLite; export the data you need to a testdatabase.db or a dbgenerator.sql.
SQLServer dialect and SQLite are for the most part compatible.
And if you're using Entity Framework then just plug in Microsoft.EntityFrameworkCore.Sqlite.

Embedded (In memory) database or Stand-Alone Database

I will build an C# WPF application that intakes USB/RS232 data (~ rate: 10ms) for hours and have to store all these data to a database. User can save all the RS232 data and setting data (e.g. TextBoxes) to a database and also output them to a file. Once they output all the data to a file, the corresponding data in the database will be erased. The user can later, import (or open), the file and load it into the application, and all its data will shown on the display and store back to the database again. I am looking for embedded database solution where user does not need to install a database server separately.
Can embedded (in memory) database, such as SQLite, accomplish such a job?

Yes SQLite can do this - there are a couple of points specific to your situations:
In order to use an in-memory database you must specify :memory: as the data source on your connection string.
In order to copy the in-memory database to disk you must use the SQLite backup API (see this question for more details) - these are not exposed by the System.Data.SQLite data adapter (the one I recommend you use), and so you will need to craft yourself some PInvoke calls.

Why not? There is now SQL Compact Edition 4 with Visual Studio 2010.

SQLite does not require installation of a database server. It can also operate in-memory: http://www.sqlite.org/inmemorydb.html
For using C# with SQLite, see: Is there a .NET/C# wrapper for SQLite?

SQLite doesnot require any installation but developing according to SQLite is cumbersome.
For example, even above statement gives error.
CREATE TABLE IF NOT EXISTS firma (firm_id INTEGER PRIMARY KEY,firm_name TEXT,firma_owner TEXT,counter INTEGER);

How to change database design in a deployed application?

Situation
I'm creating a C#/WPF 4 application using a SQL Compact Edition database as a backend with the Entity Framework and deploying with ClickOnce.
I'm fairly new to applications using databases, though I don't suspect I'll have much problem designing and building the original database. However, I'm worried that in the future I'll need to add or change some functionality which will require me to change the database design after the database is already deployed and the user has data in the database.
Questions
Is it even possible to push an updated database design out to users via a clickonce update in the same way it is for code changes?
If I did, how would the user's data be affected?
How is this sort of thing done in real situations? What are some best-practices?
I figure that in the worst case, I'd need to build some kind of "version" number into the database or program settings and create some routine to migrate the user's current version of the database to the new one.
I appreciate any insight into my problem. Thanks a lot.

There are some 'tricks' that are employed when designing databases to allow for design changes.
Firstly, many database designers create views to code against, rather than coding directly to the tables. This allows tables to be altered (split or merged, etc) while only requiring that the views are updated. You may want to investigate database refactoring techniques for this.
Secondly, you can indeed add versioning information to the database (commonly done as a 'version' table with a single field). Updating the database can be done through code or through scripts. One system I worked on would automatically check the database version and then progressively update the schema through versions in code until it matched the required version for the runtime. This was quite an undertaking.

I think your "worst" case is actually a pretty good route to go in this situation. Maintain a database version in the DB and have your application check and update the DB as necessary. If you build your updater correctly, it should be able to maintain the user's data. Depending on the update this might involve creating temporary tables to hold the existing data and repopulating new versions of the tables from them. You might be able to include a new SDF file with the new schema in place in the update process and simply transfer the data. It might be slightly easier that way -- you could use file naming to differentiate versions and trigger the update code that way.

Unfortunately version control and change management for databases is desperately, desperately far from what you can do with the rest of your code.
If you have an internal-only environment there are a number of tools which will help you (DBGhost, Red Gate has a newish app, some deployment management apps) but all of them are less than full solutions imho, but they are mostly good enough.
For client-shipped solutions you really don't have anything better than your worst case I'm afraid. Just try and design with flexibility in mind - see Dr.Herbie's answer.
This is not a solved problem basically.

"Smart Client Deployment with ClickOnce" by Brian Noyes has an excellent chapter on this issue. (Chapter 5)
ISBN 978-0-32-119769-6
He suggests something like this:
if(ApplicationDeployment.CurrentDeployment.IsFirstRun) {
MigrateData();
}
private void MigrateData() {
string previousDb = Path.Combine(ApplicationDeployment.CurrentDeployment.DataDirectory, #".\pre\mydb.sdf");
if(!File.Exists(previousDb))
return;
string oldConnString = #"Data Source=|DataDirectory|\.pre\mydb.sdf";
string newConnString = #"Data Source=|DataDirectory|\mydb.sdf";
//If you are using datasets perform any migration here, with the old and new table adapters.
//Otherwise use an .sql data migration script.
//Store the version of the database in the database, and check that in the beginning of your update script and GOTO the correct line in the SQL script.
}

A common solution is to include a version number somewhere in the database. If you have a table with miscellaneous system data, throw it in there, or create a table with one record just to hold the DB version number. Then whenever the program starts up, check if the database version is less than the expected version. If so, execute the required SQL CREATE, ALTER, etc, commands to bring it up to speed. Have a script or function for each version change. So if you see the database is currently at version 6 and the code expects version 8, execute the 6 to 7 update and the 7 to 8 update.
Another method we used on one project I worked was to ship a schema-only, no data database with the code. Every time you installed a new version the installer would also install the latest copy of this new blank database. Then when the program started it up it would compare the user's current database schema with the new database schema, and determine what database changes were needed on the fly. Like, if in the "reference schema" table Foo had a column named Bar, and there was no column Bar in the user's current database, we would generate a "alter table Foo add Bar ..." and execute it. While writing the first draft of the program to do this was a fair amount of work, once we'd done it there was pretty much zero maintenance to keep the DB schema up to date. The conversion was just done on the fly.
Note that this scheme doesn't handle DB changes that require changing data values, like if you add a new column that must be initially populated by doing some computation on data from other tables or some such. But if you can generate new data from old data, that must mean that the new data is redundant and your database is not normalized. I don't think the situation ever came up for us.

I had the same issue with an app in Android with an SQLite database adding a table. I changed the name of the database to include a version extension, like: theDataBaseV1, deleted the previous one and the app works fine.
I just changed the name of the database and the name in this line of code
private static final String DATABASE_NAME = "busesBogotaV2.db";
in the DBManager when its going to open.
Does anybody knows if this trivial solution has any unintended consequences?

A GUI which creates and uses a database and installes easily

I want to create a GUI with C++ (QT4). The GUI should work on Windows and should be able to
create a database
use the database created by it (I should use an existing DBMS, in order not to worry for queries)
database should be specific to the GUI, other software should not be able to use that database (the database may be for example encoded)
the gui with its ability of working with database should be easily installed on the other computers, that is I don't won't to ask user to change some options on his computer manually
So my questions are:
What kind of database can help me to do this, what I should learn connected with database to be able to perform this task?
Should I encode the database by my GUI, or databases have such command to save them on disk already encoded?
Thanks!

You could try looking into SQLite. The library can be used with C++. It will not need an external DBMS. SQLite is embedded into your application, and you can access you database through it. Also, the database files it produces can be encoded, so it will be accessible to your application only.

first, you should decide what are the scenario your system going to be applied.
then only proceed to source for database provider (MySQL, Postgres, etc).
you can't really jump to UI implementation straight away because all of the database mentioned above can do what you need.

OleDB vs ODBC: does one of them NOT require driver installation FOR ALL Oracle, MySQL, SQL Server?

I want my application to be able to work with multiple db vendors. In other words, depending on the DB infrastructure of the client, the db schema will be deployed on one of Oracle, MySQL, SQL Server. I am between using ODBC and OleDB, and the following key requirements that must be taken into account for the selection:
the DB schema must be created from within the application (I was told that ODBC may be problematic in this case, is this true?)
it is strongly desirable that the end users are not required to install any additional software (i.e. Oracle Instant Client etc). So, the driver should be preferably either:
already bundled with Windows. Does Windows have a generic version of ODBC / OleDB?
be able to be bundled in a way with the application. I.E. in Java, I can bundle the JDBC driver for MySQL as a .jar file with my application. Can this be done by including a .dll file?
it is strongly desirable that the end users are not required to make any external configuration, such as creating ODBC datasources
Cheers!

You do need to have an appropriate, database specific driver installed.
Once this is installed the connection string will also be, to some degree, database dependent.
Using ADO.NET much database interaction in code can be independent by using the common interfaces (e.g. IDbCommand) rather than the provider specific subclasses, but a provider specific factory will still be needed.
Even then, SQL dialects and datatypes have significant variation.
Expect to need to be very familiar with building your own abstracts, inversion of control (IoC) and debugging each different database. The latter would strongly indicate debugging actively across multiple database platforms from the project start to avoid sudden need for significant porting effort.

Most ODBC/OLEDB drivers are using a "common language" which still requires some kind of native device driver or "client install" provided by the vendor to properly connect to the database.
What you want to look for is a proper ADO.NET driver, which will have all the required libraries built into it, or it may only require a second DLL to go with it that doesn't not require a "client install". This will also allow for easy use of the Connectin String in your app.config file and all the goodness that ADO.NET provides.
Here are some links to the common ones you need:
MySQL - MySQL Connector Oracle -
Oracle Data Provider (ODP) SQL
Server - Already built in (naturally)
SQLite - SQL.Data.Sqlite (just in case you want that available)

Both need the database drivers. If not the ODBC layer does not know how to connect to remote database.

Well, yes but you need neither ODBC or OLEDB to do this. You can get 100% native ADO.NET providers for SQL Server, MySQL and Oracle from here http://www.datadirect.com/products/net/index.ssp.