We have a project for school in which we have to create a web application. We have access to an online database, but the problem is that we have to connect through VPN before we can connect to it.
For this reason, we're looking for a possibility in which we have a local database (which would be in the project, I suppose?) that we can all use (the project is on a subversion server). But when we deploy the project on our deployment server, we want it to use the real database connection.
I think I've seen it before, but after searching for hours I couldn't find anything relevant.
Is this possible?
EDIT:
We use MVC5 with Entity Framework.
People typically do this in one of several ways depending on what you want to achieve.
Pull down the database as an mdf file and store it in your repository. You can then have a manual step during setup where you ask people to import it into their database (I recommend localdb for local development but sure, you can use sql server or something).
Advantages: Very simple to set up for the person arranging this.
Disadvantages: Manual step is difficult for beginners. If the database is large it will swell your repository. If one developer changes the database (for example by adding a column) then you have to let everyone know to blow away their copy and restore from backup. Also, there is no real explicit history of how your database changes and your test database is not integrated with whatever you have to do for deployment.
Pull down the database as an mdf file. Include this file in your project and set it's properties as Content/Copy if Newer. Then use it directly using a connection string to attachDb such as Server=(localdb)\v11.0;AttachDbFilename=.\MyDataFile.mdf;Database=dbname;.
Advantages: No manual step, everything just works
Disadvantages: Obviously you'd want to use relative paths for AttachDbFilename and I'm not 100% that this is supported. Also, same as above but instead of having to let everyone know when their db needs restoration it just restores behind the scenes. This can mean users suddenly see their data disappear with no notice. It can also fail sometimes due to things like a locked database file and everyone just has to get good at keeping an eye out for that.
Maintain a sql script that can recreate your database in localdb. Provide people with a powershell or batch script (also in source control) to run it easily. Optionally use a post-build script that determines if you need to recreate the database and runs it.
Advantages: Everything is very explicit. Reasonably small size in the repository (which should be able to store text efficiently). You can use the same script as part of deployment.
Disadvantages: More work to set up. Still no real way to deploy changes to existing databases.
Use Entity Framework Database first. I can't speak to what the process looks like exactly when doing this but I know that it is possible.
Advantages: I guess.
Disadvantages: Ewwww EF database first
Use Entity Framework Code First with Migrations. Use explicit migrations (not the silly auto-generate-my-entire-db cruft) and write a proper Seed method to populate your data.
Advantages: This is what professional developers do and is based on tested patterns used frequently by Rails, Django, and many other frameworks. It is very flexible and explicit and supports changing existing databases.
Disadvantages: Can be quite difficult to set up if you don't have experience and especially if you're unaware of the migrations pattern. There's some naming difficulties that make it kind of hard to google (database first EF vs code-first EF, explicit migrations vs auto-generate-the-db, several different Seed methods that depend on your initializer).
You can create a local database using Entity Framework, which saves the database file on the local filesystem, which you can push to your version control server to share with your colleagues. If you decide to deploy the database, you can generate an sql script that you can run on the production database. You can do so by connecting to the local database using SQL Management Studio. You will just need to modify the connection string of the published application after deployment.
You used to be able to use Sql Server Express but it has changed to LocalDB and can be installed and run locally.
http://blogs.msdn.com/b/sqlexpress/archive/2011/07/12/introducing-localdb-a-better-sql-express.aspx
https://msdn.microsoft.com/en-us/library/ms233763.aspx
Related
According to this blog post most companies using EF Migrations are supposedly not updating the database schema of production databases with EF migrations. Instead the blog post's author recommends to use Schema update scripts as part of the deployment process.
I've used Schema update scripts for a few years now and while they work, I was planning to use EF migrations instead in the future for the following reasons:
Faster deployment, less downtime
A simpler deployment procedure
Much easier migration of existing data than it would be possible with T-SQL
A more comprehensible syntax of the changes waiting to be applied (DbMigration class with clean C# syntax vs. clunky T-SQL Migration script in a traditional environment).
There is an easy and fast downgrade path to the old db schema if the deployment of the new software version should fail
One reason I can think of that would prohibit the use of EF to migrate a production DB would be if the DB schema was only altered by the DBAs as opposed to the Developers. However, I am both DBA and Developer, so this does not matter in my case.
So, what are the risks of updating a production database using EF?
Edit: I would like to add that, as solomon8718 already suggested, I am always pulling a fresh copy of the production database to my staging server and test the EF Migrations to be applied on the staging server before applying them to a production server. IMO this is essential for any schema update to a production system, whether I'm using EF migrations or not.
Well, I'll try and answer anyhow. I would say No, there's no reason not to use Code First Migrations in production. After all, what's the point of this easy to use system if you can't take it all the way?
The biggest problems I see with it are all problems that you can have with any system, which you've noted already. As long as the whole team (DBA included if applicable) is on board with it, I think allowing EF to manage the schema through migrations is less complex, and hence less error-prone than traditional script-based management. I would still take a backup before performing a migration on a production system, but then you'd do that anyhow.
There's nothing that says a DBA can't perform a migration from Visual Studio, either. The access could still be locked down with privileges at the database level, and he/she could review the migration (in a helpful SQL export format using -Script, if desired) before performing the actual operation. Then they're still in control, but you can use code-first migrations. Hell, they might even end up liking it!
Update: since SPROCs and TVFs were brought up, we handle those in migrations as well, although they are actually done with straight-up SQL statements using a DbMigration.Sql() call in the Up(), and the reverse of them in the Down() (You can also use CreateStoredProcedure and DropStoredProcedure for simple SPROCs, but I think you still have to define the body itself in SQL). I guess you could say that's a caveat; there isn't yet a way for an entire, comprehensive database to be written purely in C#. However, you can use migrations which include SQL scripts to manage the entire schema. One benefit we've found from this process is you can use the C# config file for schema object names (different server names for production vs dev for example) with a simple String.Format, combined with XML Transformation for the config files themselves.
Yes there are good reasons not to use an automated system such as Code First Migrations to make production database changes. But as always there are exceptions to the rules.
One reason which has been mentioned would be access permissions, which would be directly related to your organization's change management rules and security policies.
Another reason would be your level of trust in the Migrations tool itself. Are we sure the tool doesn't have a bug in it? What happens if the tool fails midway through? Are you certain you have up-to-date backups and a process to roll-back if need be?
The change scripts may execute unexpected or inefficient scripts. I've experienced cases where the sql generated copied the data into a temp table, dropped the original table, then recreated the original table for things like adding a new column if you accidentally (or purposefully) change the order in which the column appears, or when you rename the table. If millions of records are involved this could cause serious performance issues.
My recomendation:
Assuming you have a Staging database that mirrors your production schema, use the Migrations tool to generate its change scripts against that system. We usually restore our stage database from a fresh production copy before running. We then examine the change scripts manually to check for issues. After that we run the scripts against our stage database to make sure it executes properly and that all the changes expected took place. Now we are sure that the scripts are both safe to run in production and perform the expected changes. This process would address all three issues I listed above.
One other caveat I found: If you have several websites using the same data context, you need to make sure that all of them are updated at the same time. Otherwise there might be a constant database update / downgrade fight between the websites. Other than that, it worked fine for me.
EDIT: My own perspective one year after starting to use EF Migrations in production:
EF Migrations is actually pretty cool, even for production use, provided that you
Test the migrations on a staging system. I test all migrations by migrating all the way down and up again on my CI server before running integration tests.
Do not trigger migrations automatically, but with a batch file that is launched by an admin. This is essentially the same as running the sql for a migration manually in SSMS.
I use it in production for a couple of projects. Once you get the hang of it I think it's fine.
During development you can keep auto migrations on but at the end you can connect to the live db right from package manager console and generate a migration. It will give you one migration for all the changes.
But always always always use the -script option with update-database and fire the SQL yourself.
I would also advice not using the update db option from web deploy. That way there is no way to tell how much of the migration has already been fired on error. I've ran into trouble with that a few times. So best to get the SQL and fire it manually.
I'm getting ready to develop a MVC 3 website with C#, Entity Framework and SQL Server.
This website is built for critical jobs and data lost is something absolutely not allowed ! In my knowledge I had no experience of evolving database, but I know this project should be able to evolve while using incremental development methodology. May I know is there any guideline to follow and how do I evolve it without any single error? In term of database initial design or anything. Just, 0 Data Lost is highest priority requirement.
I need answer for this 2 question and hope some experience could guide me in this issue
How to update database include table, column without affect other data in the same table
How to update remote database (for example C# window apps and database is not with me)
For the 1. question the database is located at my web server but question 2 the database is staying with user end.
The answer is: it is your duty to design upgrade process in the way your requirements are met. There is no auto magic which will do this for you.
The process usually involves creation of upgrade SQL script which will modify database structure and if needed it can also move data to temporary tables while structure of main tables are changed so that data are not lost. You can also maintain database version in some special table and check it before you run the update so that you ensure that update script is run on expected old version.
There are tools like RedGate SQL Compare and Visual Studio Database tools (only Premium and Ultimate version) which are able to take old database, a new database and create difference script for you so that old database schema can be upgraded to a newer one. This works for most scenarios but you must always very carefully test result in your testing environment. It is best to test in on backup of your production database if possible.
How to avoid data loss if anything goes wrong? There is only one very simple way BACKUP THE DATABASE before you do any changes and restore the old database if anything goes wrong. Backup can be even scripted with SQL. Without successful backup never touch your production database.
How to upgrade client side database? You will use the same process but you will wrap it all in some installation package (.msi) for example created with WiX.
I have a C# app (in VisualStudio 2010) that used SqlServer 2005 accessed through TableAdapters in C#.
I haven't discovered a good way to manage DB changes. For example, in my next release I have a bunch of db schema changes. I made all of my DB changes in Sql Server Management Studio. But now I have to manually make these changes on the production servers in turn after I deploy the new application code (slow and buggy).
Furthermore, if I decide to roll back my release to a previous version, I have to manually go through and undo all my db changes before I can deploy the old code (and now I am under time constraints because the app is down). Again, this is also very error prone.
Oh, and lets hope that one of my errors doesn't cause massive destruction to the production DB, otherwise I now have to pull the most recent backup out of storage and try again (very time consuming).
I have heard of things like Migrations from Rails (and ORMs like SubSonic). I think that the new ORM style (define your schema in c# code) helps alleviate a lot of this, but unfortunately, as I am using TableAdapters, I don't see how I could implement something like migrations.
How do people deal with this?
Release management for DBs usually involves migrations of static data and running of scripts to update/create programmability elements (sprocs, UDFs, triggers, etc) and modify existing schema definitions. Looks to me like you're missing the scripts. If you're making changes manually to your development DB and not creating scripts that mirror those changes, you will need to repeat the same manual steps against your test/production environments, which as you say is error prone and dangerous.
SQL Server Management Studio makes it easy to save scripts that reflect changes to any database objects. In the toolbar there should be an icon called "Generate change script", which gives you the option to save the SQL file to disk. You can then use this to perform the same change against another server. You can also manually script any or all stored procs, UDFs, triggers and so on, and run those against a server as well (just right-click on them).
As to rollback, that's normally achieved by restoring a backup of the database made just before the deployment process begins.
This whole process tends to be different for each company, but that's generally how it's done.
ORMs that auto-generate schemas have always seemed evil to me, not to mention pretty much impossible to use against a production box, but I guess there's also an option.
The easiest way to deal with this problem is to buy software that can detect db schema by comparing two databases changes and generate a change script that can update your target database. I am using Visual Studio Ultimate 2010 for that, but there's also cheaper software that can do the same. This works for me 99% of the time (the only instance where this did not work properly for me is when I renamed a table column).
If you don't have such a piece of software, it is crucial to generate your SQL change scripts by hand. Whenever you do a change to the database schema, keep track of the SQL you used for that changed and add it to one big file of db schema changes for the next version of your software. It's a bit tedious at the beginning, but you'll get used to it pretty quickly.
Then when you are ready to deploy the software, proceed as follows:
Take the website offline
Make a backup of your current production database.
Make a backup of your current production website.
Upload your new code to the server
Run the DB changes script you previously created (either by hand or with the software mentioned above)
Take the website back online and see if it works. If it doesn't and you can't easily fix the problem, revert to the previous website and db version until you have fixed the bug.
All of these steps can be easily automated using batch files and the SQL server agent or SQLCMD.
Generally you should deploy to a staging server first, then test your website very thoroughly and only then move on to the production server. This way you avoid longer downtimes on your production server and minimize the risk of losing any vital data.
Here at Red Gate Software we're currently tackling this exact issue. Please take a look at our SSMS add-in, SQL Source Control, in combination with SQL Compare Pro. We're also working on a 'Migrations' feature, due out later this year, allowing custom migrations scripts to be defined for specific version transitions. As we're still in the early stages of the project, there's still time to give us feedback and help us design a great solution. We'd love to speak to your further about your requirements!
Currently, changes to the database are made through the SQL Server Management program. IF a table changes, sqlmetal is run to regenerate the linqtosql classes and development continues. However, this makes deployment a pain, as you have to go through and manually update the deployment database (and any other databases used in the development cycle). It would be nice if we could use C# to generate these changes, as it would help eliminate human error and have the added benefit of being able to keep the database structure in git. Right now, the only representation of the database is in the generated linqtosql classes.
I've been looking around for a nice library that can handle this sort of thing, but the main solutions seem to be: keep a sql generation script, or embed sql in C# classes that can be run to make changes to the database. Both of these seem to be very non-ideal situations, as you lose the nice strong-typing that C# provides. It seems like there should be a way to do this using pure C#.
I've seen hints of being able to do things like generate databases from POCOs using both the entity framework and linqtosql, but I'm having a hardtime finding specific examples of that being used. Additionally, I haven't been able to discover if those have a graceful (i.e. data preserving) way of handling changes to the database after the initial table generation.
Are there any projects out there that solve this problem?
There exist several tools that help you with schema (and data) migrations of your database: RikMigrations, Migrator.Net and Machine.Migrations. Hope that helps.
Wizardby looks also promising: It provides database independent DDL scripts and automated migrations between different versions of a schema.
VS2010 can operate version control on your database schema through a Database Project. There are other tools out there for DB development that offer version control, you'll need to search to find them and compare pricing.
I prefer to version using sql scripts. Works pretty well, is free, supports updates, easy to version, works well with traditional source control methods.
First,
Create your DB
Use the Database Publishing Wizard to publish the database as a .sql script
Add a version number to the script
Add to your solution
Check into source control.
As updates are made,
Script updates to the previous schema as .sql files
Add a version number to the script that is incremented from the previous version
Add to your solution
Check into source control.
It sounds like you need a tool like Migrator.NET to manage your database migrations. We use it with a call from our site start-up to migrate the database as needed for any particular version.
I have toyed with an idea for creating a cleaner interface and someday hope to get around to implementing it, but other priorities have pushed that back. For now we are using raw sql strings in our migrations because there isn't a sybase driver implementation (outside of a very ugly hack I have written to manage the versioning table).
Redgate software offers something that may be really useful for you. It's called SQL Packager and it does it's job pretty well.
Features:
Easy roll-out of database updates across your client base
Script and compress your schema and data accurately and quickly
Package any pre-existing SQL script as a .exe, or launch as a C# project
Simplify deployments and updates for SQL Server 2000, 2005 and 2008
They also offer SQL Source Control which also may be useful to keep things nice and easy.
As an addon to MadBoy, SQL Packager can also launch the package as a C# project.
Red-Gate's SQL Compare is excellent as well, and as some of the banners on SO indicates, there is new SQL Source Control available as well.
Then they have their SQL Comparison SDK.
The trick here is to rely on the database being the single source of truth for your Linq schema, not the generated classes.
We use Linq to SQL extensively in our dev shop, and work as follows:
1. Create your database (working copy) from version control (baseline).
2. Modify your database any which way you like.
3. Generate Linq to SQL classes from the (working) database.
4. Create patches to update your baseline database to your working copy.
5. Check in and share these patches with all developers.
For a very quick and easy way of generating baseline and working copy databases, try DBSourceTools. http://dbsourcetools.codeplex.com
Have fun.
I am currently working on a project that include the use of SQLServer. I would like to know what strategy I should use when I install the software to build the database? I need to set up the tables, the stored procedures and the users.
Does my software need to make a check on start up to see if the database exist and then if it doesn't, create it up?
Is there any way that I could automate this when I install SQLServer?
Thank you.
EDIT
Ok right now I have plenty of nice solution, but I am really looking for a solution (free or open source would be awesome) that would allow me to deploy a new application that needs SQLServer to be freshly installed and setuped to the needs of the software.
RedGate software offers SQL Packager which gives you option to produce a script/.exe to deploy whole db including everything (stored procedures, tables etc) from one single .exe. If you can afford it and want to have easy way to do it (without having to do it yourself) it's the way to go ;-)
Easy roll-out of database updates across your client base
Script and compress your schema and data accurately and quickly
Package any pre-existing SQL script as a .exe, or launch as a C# project
Simplify deployments and updates for SQL Server 2000, 2005 and 2008
You could use migration framework like Migrator.Net, then you could run the migrations every time your application starts. The good thing about this approach is that you can update your database when you release a new version of your software.
Go take a look at their Getting started page. This might clear up the concept.
I have succesfully used this approach to solve the problem you are confronted with.
You do all of that with the SQL scripts. And then your installation program runs them against the customer's SQL Server.
You can write a T-SQL script that only creates objects when they do not exist. For a database:
if db_id('dbname') is null
create database dbname
For a stored procedure (see MSDN for a list of object types):
if object_id('spname', 'P') is null
exec ('create procedure dbo.spname as select 1')
go
alter procedure dbo.spname
as
<procedure definition>
The good thing about such scripts is that running them multiple times doesn't create a problem- all the objects will already exist, so the script won't do anything a second time.
Setting up the server is pretty straight forward if you're using MS SQL Server. As for creating the database and tables, you generally only do this once. The whole point of a database is that the data is persistent, so if there's a chance that the database won't exist you've either got a) major stability problems, or b) no need for an actual database.
Designing the database, tables, and procedures is an entire component of the software development process. When I do this I usually keep all of my creation scripts in source control. After creation you will write the program in such a way that it assumes the database already exists - checking for connectivity is one thing, but the program should never think that there is no database at all.
you can make a script from all of objects that exist in your db. after that you can run this script from your code.
when you create your db script with script wizard in sql server, in "choose script options" section, set "Include If Not Exist" to yes. with this work done only if db not exists.