migration synch developmental and production databases - c#

I am using MVC 5 with NET Framework 4.5.1. with Code-first. I am also using Migrations with the SQL 2012 server and (localdb)\v11.0.
I am in the middle of developing a project using C# and MVC5. During development, I created a lot of new tables in my developmental computer and changed a the "Name" field which I believe the system makes an index for. I added it and deleted it several times.
After that, I added a lot iof new unrelated tables, but for some reason, my migrations started giving me foreign constraint errors due to the indexes for the "Name" field. These errors kept multiplying as I fixed them, so, I decided to revert back to an initial state in the migration, and reset using the current position as a new starting point. I was hoping, that the production table would look at this new starting point in the development db, and resynch itself to the developmental state. I thought that I had read somewhere that the production db matches itself to the developmental db and updates itself. I believe that there is a migration file in the production db which would match itself to the file in the developmental db -that file was clearly out of synch. I have considered deleting the data in it, but I am holding off till I get advice.
Anyway, I changed the name of the migrations directory in the Dev computer and excluded it from the project. Then I reinitialized my tables (using a new db name in my local db) on the dev computer and re-loaded it with the initialization data. It all worked.
Now, I had a new problem, my production db and my developmental db were different, and my migration in the dev computer was setup to create new files whereas the one in the production state was expecting the older migration. Every time I tried to update the production db using the development computer, I kept getting an error that the files existed - which of course they did.
So, I commented out all the create files in my migration file and re-tried. Now, the production db would start, but would not run because the updated code had new fields it was referring to which did not get created in the production db. So, on my production db I started to get errors of all the fields that were missing. I tried to make automatic migrations true as well, that did not work. I am guessing, the only way to fix this is to go in manually and synch the fields one by one.
QUESTION 1: Is there an automatic way to synch (using migrations) the production db and the developmental db so that they become the same same as the developmental db?
QUESTION 2: Keeping in view the above scenario, what would have been a better way to go about to re-set the miggrations with a production db out as well?

I found a solution. The folks at Red-Gate have a great SQL tool called the SQL Compare. It compares the database file structures and even makes them EXACTLY the same, at a click of a button.
But, before you use it, be sure you ONLY compare "tables", as opposed to everything which includes "users" and "roles" and a lot more. That is because when you run the software, it backups, deletes and re-creates, and if the roles or users get deleted, the software can no longer access the database and everything gets deleted! Also... MAKE A BACKUP! (I lost all my test data on my first try)
http://www.red-gate.com/products/sql-development/sql-compare/
(This is not a sales plug for the folks at Red-Gate. I dont know them, but their tool helped me immensely - its a good tool, easy to use, and FREE for 14 days! - and I list it here so that anyone else, and I am sure there are many, who may be stuck like me can be helped.)
April 24 2015
Ok. There is more to it after you synch both the databases so that they look exactly alike.
Create a Back up of your production data *
Delete the Migration folder in your developmental folder.
Enable Migrations again
Add an initial migration
Update the local database
Now you have your local completely set up *
Go to the host database
Find the table called "__MigrationHistory" in Host/Production
Delete all the data (you want to purge it) ("__MigrationHistory" (Host))
Now copy all the data from the local "__MigrationHistory" to the hosted "__MigrationHistory"
(There will be your one single line i.e. the initial one you created above")
Now the data has been saved and every thing will be synched and it will work.
You can begin development again.

Related

Entity Framework 6 just deleted my database?

I'm using MVC 5 with Entity Framework 6, automatic migrations are disabled, no seed is used and I'm scripting out my updates to production.
I'll hold up my hands and say I'm fairly new to MVC and EF.
I just uploaded a new codebase to production, and forgot that the model had changed. When I went to login to the new version, it threw and error page. Fine, I forgot to script the latest migration - I did that, hooked up to SSMS and found that the database was gone.
Slight panic, restore from backup (I'm not that daft!) and apply the migration and everything starts working as it should.
Why did my code delete the entire database when the model had changed? As said, automatic migrations are disabled and this behaviour doesn't happen on development machines, they throw an error about the model having been changed.
Edit to add: Whilst looking at the Initialisation patterns in Fernanda's answer, none of them say "Drop the database and then do nothing else". My database was dropped, the MDF and LDF gone, nothing is SSMS and nothing recreated in its place. If a blank database was created in its place I'd understand a bit more. That said, the user account had DBO on the database but not master, so would have been able to drop the database but not create a new one?
Read this about Database Initialization Strategies in Code-First:
http://www.entityframeworktutorial.net/code-first/database-initialization-strategy-in-code-first.aspx
Check your dbcontext initialization and be sure that the option DropCreateDatabaseIfModelChanges is comment
//Database.SetInitializer(new DropCreateDatabaseIfModelChanges());

Is it OK to update a production database with EF migrations?

According to this blog post most companies using EF Migrations are supposedly not updating the database schema of production databases with EF migrations. Instead the blog post's author recommends to use Schema update scripts as part of the deployment process.
I've used Schema update scripts for a few years now and while they work, I was planning to use EF migrations instead in the future for the following reasons:
Faster deployment, less downtime
A simpler deployment procedure
Much easier migration of existing data than it would be possible with T-SQL
A more comprehensible syntax of the changes waiting to be applied (DbMigration class with clean C# syntax vs. clunky T-SQL Migration script in a traditional environment).
There is an easy and fast downgrade path to the old db schema if the deployment of the new software version should fail
One reason I can think of that would prohibit the use of EF to migrate a production DB would be if the DB schema was only altered by the DBAs as opposed to the Developers. However, I am both DBA and Developer, so this does not matter in my case.
So, what are the risks of updating a production database using EF?
Edit: I would like to add that, as solomon8718 already suggested, I am always pulling a fresh copy of the production database to my staging server and test the EF Migrations to be applied on the staging server before applying them to a production server. IMO this is essential for any schema update to a production system, whether I'm using EF migrations or not.
Well, I'll try and answer anyhow. I would say No, there's no reason not to use Code First Migrations in production. After all, what's the point of this easy to use system if you can't take it all the way?
The biggest problems I see with it are all problems that you can have with any system, which you've noted already. As long as the whole team (DBA included if applicable) is on board with it, I think allowing EF to manage the schema through migrations is less complex, and hence less error-prone than traditional script-based management. I would still take a backup before performing a migration on a production system, but then you'd do that anyhow.
There's nothing that says a DBA can't perform a migration from Visual Studio, either. The access could still be locked down with privileges at the database level, and he/she could review the migration (in a helpful SQL export format using -Script, if desired) before performing the actual operation. Then they're still in control, but you can use code-first migrations. Hell, they might even end up liking it!
Update: since SPROCs and TVFs were brought up, we handle those in migrations as well, although they are actually done with straight-up SQL statements using a DbMigration.Sql() call in the Up(), and the reverse of them in the Down() (You can also use CreateStoredProcedure and DropStoredProcedure for simple SPROCs, but I think you still have to define the body itself in SQL). I guess you could say that's a caveat; there isn't yet a way for an entire, comprehensive database to be written purely in C#. However, you can use migrations which include SQL scripts to manage the entire schema. One benefit we've found from this process is you can use the C# config file for schema object names (different server names for production vs dev for example) with a simple String.Format, combined with XML Transformation for the config files themselves.
Yes there are good reasons not to use an automated system such as Code First Migrations to make production database changes. But as always there are exceptions to the rules.
One reason which has been mentioned would be access permissions, which would be directly related to your organization's change management rules and security policies.
Another reason would be your level of trust in the Migrations tool itself. Are we sure the tool doesn't have a bug in it? What happens if the tool fails midway through? Are you certain you have up-to-date backups and a process to roll-back if need be?
The change scripts may execute unexpected or inefficient scripts. I've experienced cases where the sql generated copied the data into a temp table, dropped the original table, then recreated the original table for things like adding a new column if you accidentally (or purposefully) change the order in which the column appears, or when you rename the table. If millions of records are involved this could cause serious performance issues.
My recomendation:
Assuming you have a Staging database that mirrors your production schema, use the Migrations tool to generate its change scripts against that system. We usually restore our stage database from a fresh production copy before running. We then examine the change scripts manually to check for issues. After that we run the scripts against our stage database to make sure it executes properly and that all the changes expected took place. Now we are sure that the scripts are both safe to run in production and perform the expected changes. This process would address all three issues I listed above.
One other caveat I found: If you have several websites using the same data context, you need to make sure that all of them are updated at the same time. Otherwise there might be a constant database update / downgrade fight between the websites. Other than that, it worked fine for me.
EDIT: My own perspective one year after starting to use EF Migrations in production:
EF Migrations is actually pretty cool, even for production use, provided that you
Test the migrations on a staging system. I test all migrations by migrating all the way down and up again on my CI server before running integration tests.
Do not trigger migrations automatically, but with a batch file that is launched by an admin. This is essentially the same as running the sql for a migration manually in SSMS.
I use it in production for a couple of projects. Once you get the hang of it I think it's fine.
During development you can keep auto migrations on but at the end you can connect to the live db right from package manager console and generate a migration. It will give you one migration for all the changes.
But always always always use the -script option with update-database and fire the SQL yourself.
I would also advice not using the update db option from web deploy. That way there is no way to tell how much of the migration has already been fired on error. I've ran into trouble with that a few times. So best to get the SQL and fire it manually.

Verify that target database schema complies with what's in Entity Framework?

We have a process where our database guys script changes (and version them using Juneau) to our application's database out-of-band with our code base. They're good at accounting for new columns being null, and not wiping existing data, but occasionally a column rename sneaks in that isn't fully communicated. So they will make some changes to the database schema on a testing server, we'll update Entity Framework to work with those changes, and then commit our code. This process works okay, except for when it's time to deploy.
We have TFS set up to deploy the successful build to the appropriate servers, but there's no guarantee that the database for that environment has been updated. We don't care if extra fields/tables/views/etc. exist in the target database, but we want change the build to check that the database contains at least everything EF is aware of.
I looked at this question, but I don't need the schema to match exactly. Plus, we don't want it creating/modifying the database directly. And this question seems like it's trying to achieve a similar ideal, but still not quite what we're looking to achieve. We just want a integration test of sorts to verify our version of EF will work with the target schema.
I wonder why you try to deploy your application without changes to database. Your application is dependent on the database so the deployment should always be done after the database. It looks like you are going to invest a lot of time to develop validation to fix your incorrect deployment process (where fixing the process itself is the correct solution).
Anyway you can create some "validation" of the database but it will take some time. If you are using EDMX file you can open it as XML and read its SSDL part which describes all expected tables, columns, relations, views (in form of SELECT SQL queries), stored procedures and functions. You can parse this XML part and use system database views (sys.tables, sys.columns, ...) to query if these objects exists in the database.
Another approach can be using database diff. tool to compare your current test database with the target one. This will require the tool which can be executed from command line and you will have to parse its output to find breaking changes.

How to evolve SQL Server database of MVC3 sites in the future and avoid data lost

I'm getting ready to develop a MVC 3 website with C#, Entity Framework and SQL Server.
This website is built for critical jobs and data lost is something absolutely not allowed ! In my knowledge I had no experience of evolving database, but I know this project should be able to evolve while using incremental development methodology. May I know is there any guideline to follow and how do I evolve it without any single error? In term of database initial design or anything. Just, 0 Data Lost is highest priority requirement.
I need answer for this 2 question and hope some experience could guide me in this issue
How to update database include table, column without affect other data in the same table
How to update remote database (for example C# window apps and database is not with me)
For the 1. question the database is located at my web server but question 2 the database is staying with user end.
The answer is: it is your duty to design upgrade process in the way your requirements are met. There is no auto magic which will do this for you.
The process usually involves creation of upgrade SQL script which will modify database structure and if needed it can also move data to temporary tables while structure of main tables are changed so that data are not lost. You can also maintain database version in some special table and check it before you run the update so that you ensure that update script is run on expected old version.
There are tools like RedGate SQL Compare and Visual Studio Database tools (only Premium and Ultimate version) which are able to take old database, a new database and create difference script for you so that old database schema can be upgraded to a newer one. This works for most scenarios but you must always very carefully test result in your testing environment. It is best to test in on backup of your production database if possible.
How to avoid data loss if anything goes wrong? There is only one very simple way BACKUP THE DATABASE before you do any changes and restore the old database if anything goes wrong. Backup can be even scripted with SQL. Without successful backup never touch your production database.
How to upgrade client side database? You will use the same process but you will wrap it all in some installation package (.msi) for example created with WiX.

How to change database design in a deployed application?

Situation
I'm creating a C#/WPF 4 application using a SQL Compact Edition database as a backend with the Entity Framework and deploying with ClickOnce.
I'm fairly new to applications using databases, though I don't suspect I'll have much problem designing and building the original database. However, I'm worried that in the future I'll need to add or change some functionality which will require me to change the database design after the database is already deployed and the user has data in the database.
Questions
Is it even possible to push an updated database design out to users via a clickonce update in the same way it is for code changes?
If I did, how would the user's data be affected?
How is this sort of thing done in real situations? What are some best-practices?
I figure that in the worst case, I'd need to build some kind of "version" number into the database or program settings and create some routine to migrate the user's current version of the database to the new one.
I appreciate any insight into my problem. Thanks a lot.
There are some 'tricks' that are employed when designing databases to allow for design changes.
Firstly, many database designers create views to code against, rather than coding directly to the tables. This allows tables to be altered (split or merged, etc) while only requiring that the views are updated. You may want to investigate database refactoring techniques for this.
Secondly, you can indeed add versioning information to the database (commonly done as a 'version' table with a single field). Updating the database can be done through code or through scripts. One system I worked on would automatically check the database version and then progressively update the schema through versions in code until it matched the required version for the runtime. This was quite an undertaking.
I think your "worst" case is actually a pretty good route to go in this situation. Maintain a database version in the DB and have your application check and update the DB as necessary. If you build your updater correctly, it should be able to maintain the user's data. Depending on the update this might involve creating temporary tables to hold the existing data and repopulating new versions of the tables from them. You might be able to include a new SDF file with the new schema in place in the update process and simply transfer the data. It might be slightly easier that way -- you could use file naming to differentiate versions and trigger the update code that way.
Unfortunately version control and change management for databases is desperately, desperately far from what you can do with the rest of your code.
If you have an internal-only environment there are a number of tools which will help you (DBGhost, Red Gate has a newish app, some deployment management apps) but all of them are less than full solutions imho, but they are mostly good enough.
For client-shipped solutions you really don't have anything better than your worst case I'm afraid. Just try and design with flexibility in mind - see Dr.Herbie's answer.
This is not a solved problem basically.
"Smart Client Deployment with ClickOnce" by Brian Noyes has an excellent chapter on this issue. (Chapter 5)
ISBN 978-0-32-119769-6
He suggests something like this:
if(ApplicationDeployment.CurrentDeployment.IsFirstRun) {
MigrateData();
}
private void MigrateData() {
string previousDb = Path.Combine(ApplicationDeployment.CurrentDeployment.DataDirectory, #".\pre\mydb.sdf");
if(!File.Exists(previousDb))
return;
string oldConnString = #"Data Source=|DataDirectory|\.pre\mydb.sdf";
string newConnString = #"Data Source=|DataDirectory|\mydb.sdf";
//If you are using datasets perform any migration here, with the old and new table adapters.
//Otherwise use an .sql data migration script.
//Store the version of the database in the database, and check that in the beginning of your update script and GOTO the correct line in the SQL script.
}
A common solution is to include a version number somewhere in the database. If you have a table with miscellaneous system data, throw it in there, or create a table with one record just to hold the DB version number. Then whenever the program starts up, check if the database version is less than the expected version. If so, execute the required SQL CREATE, ALTER, etc, commands to bring it up to speed. Have a script or function for each version change. So if you see the database is currently at version 6 and the code expects version 8, execute the 6 to 7 update and the 7 to 8 update.
Another method we used on one project I worked was to ship a schema-only, no data database with the code. Every time you installed a new version the installer would also install the latest copy of this new blank database. Then when the program started it up it would compare the user's current database schema with the new database schema, and determine what database changes were needed on the fly. Like, if in the "reference schema" table Foo had a column named Bar, and there was no column Bar in the user's current database, we would generate a "alter table Foo add Bar ..." and execute it. While writing the first draft of the program to do this was a fair amount of work, once we'd done it there was pretty much zero maintenance to keep the DB schema up to date. The conversion was just done on the fly.
Note that this scheme doesn't handle DB changes that require changing data values, like if you add a new column that must be initially populated by doing some computation on data from other tables or some such. But if you can generate new data from old data, that must mean that the new data is redundant and your database is not normalized. I don't think the situation ever came up for us.
I had the same issue with an app in Android with an SQLite database adding a table. I changed the name of the database to include a version extension, like: theDataBaseV1, deleted the previous one and the app works fine.
I just changed the name of the database and the name in this line of code
private static final String DATABASE_NAME = "busesBogotaV2.db";
in the DBManager when its going to open.
Does anybody knows if this trivial solution has any unintended consequences?

Categories