I have an application that uses an MS-SQL-Server to store it's data. We will roll out the application in several steps, so users can test first basic functionalities and we add functions over time.
Probably, this will cause changes to the database. In early stages, we can just drop the whole database and create everything anew with a script. But eventually, users want to keep their test data. For example, if I add a new column to a table, I don't want to delete and create anew the whole table, loosing all the data the user has stored so far in the process.
So, what I need is a script, that updates myDatabase v1.0 to myDatabase v2.0.
My question is: What is the best way to create such an update script? Do I have to manually keep track of all the changes and then assemble the update script myself? Or is there a function which could (semi)automatically create the update script from the old and the new database?
And finally, what is the best way to apply this patch? I have a ASP.NET Web-API so, I could create a controller api/updates/v1.0-to-v2.0
How would the code for applying the script at the server would look like?
Thanks in advance,
Frank
I'm working on a solution to this very problem, check out dbpatcher.com the software that I've created will help to make migrating database changes easier. I'm putting together the website at the moment so would welcome feedback. The program itself isn't yet available, as I'm trying to figure out the details for publishing.
If this is an on-going concern (corporate), you should really consider different environments, ie. test, staging and production. This way you can test your deployments and database scripting changes in a pristine environment (something that looks exactly like production).
Given that, to answer your question, there really isn't a good way to do this. I've seen people use diff tools to detect the differences between schema's, and it creates scripts to sync two schema's, but it's not fool proof.
I find that scripting the changes and combining that with version control and an installation procedure (manual or automated) is the only way to get consistent results, and even that fails sometimes.
Code first entity framework is attempting to solve that issue, but it's not an option for a number of shops.
I would love to see a good tool to manage this, but the set of diverse frameworks and human error are the biggest problems here.
In terms of down time, there really is no such thing as live in-place upgrades of web applications. There are ways to mitigate it to a minimum, like update one set of a load balanced Web/App server at a time and then fail users to the new software. If your doing table alterations, the likelyhood that your not going to lock the table and interrupt your users are pretty low.
Thanks for your replies!
I have different environments, my concern is how to change the database schema without loosing data in the production environment (if possible, of course). Downtime is not so much a problem. As I am not productive yet, I simply create a script to recreate the whole database, but when the user has live data stored, he would probably be a bit ... upset.
There is a nice tool from redgate, that seems to solve that problem, but I did not checked it yet.
Related
I recently started a new personal project to learn Entity Framework. My end goal is to make a desktop game that uses SQL compact for data management and uses Entity Framework for the game objects. Not actually knowing there were multiple ways to start EF (model first, code first, db first) I went with the most obvious choice of model first.
I've been working with it successfully now, however one thing concerns me, especially post-development. My goal with the game is that users can update to the latest version without losing any of their existing data. The current issue is that all the generation scripts are destructive by nature (dropping everything then recreating it) - that means I can't run those against the user SQLCE DBs out in "production", so I need to come up with an alternative plan of action.
That said, does anyone have recommended solutions on best practices? In previous desktop apps, I've traditionally used XML/binary to store data, which allows me to easily update the "schema" without affecting existing data (versioning in the app tailors the Load() according to the version, while the Save() always saves in the latest version).
What are some recommendations on handling this problem using SQLCE?
What you need, if understood right, is to utilize migrations which come with EF. Since the question is general this link should best guide you to what you need I think...
http://blogs.msdn.com/b/adonet/archive/2012/02/09/ef-4-3-code-based-migrations-walkthrough.aspx
With migrations which you can tailor manually if needed (and come in the shape of code which is applied at each point of change, incrementally) and you can also supply your 'seeding' if required.
i.e. you should be able to do most of what you require, delete, remove old incompatible data - and seed the new one that you have - and all related to a particular migration step you have.
How would that work with your app deployment specifically, that's a bit more complex I guess, but this should get you started, and then with each db version-breaking change your new code update would contain all the migrations since the previous update (or just one usually is enough, i.e. make it be one with each update) and the code to tear-down or create new things.
hope this helps,
I want to be able to maintain a count and a last accessed date across application loads for a web service polling application. I'm not too sure what the best way to do this is. I dont like the idea of storing that data in a database as I would have to create one specifically for the purpose. What other options do I have and are there any particularly nice ways of keeping application state between subsequent runs of the app?
Persisting data eh? I suggest a database or file.
File solutions you can just XML serialize to a file and load it again when the app starts.
If the data is shared or might ever grow, then a database is probably the best solution. You can find one that fits your need among the many free projects if you wish:
couchdb
mysql
postgres
mangodb
membase
sqlite
etc
You could roll your own solution that doesn't involve a database, but most likely there is one that fits your needs and learning it would be useful beyond just the project at hand.
Don't be afraid to make a 'configuration' style table for your website, that simply has only a few rows and let's you store runtime information as needed.
Perfectly fine.
I have a C# app (in VisualStudio 2010) that used SqlServer 2005 accessed through TableAdapters in C#.
I haven't discovered a good way to manage DB changes. For example, in my next release I have a bunch of db schema changes. I made all of my DB changes in Sql Server Management Studio. But now I have to manually make these changes on the production servers in turn after I deploy the new application code (slow and buggy).
Furthermore, if I decide to roll back my release to a previous version, I have to manually go through and undo all my db changes before I can deploy the old code (and now I am under time constraints because the app is down). Again, this is also very error prone.
Oh, and lets hope that one of my errors doesn't cause massive destruction to the production DB, otherwise I now have to pull the most recent backup out of storage and try again (very time consuming).
I have heard of things like Migrations from Rails (and ORMs like SubSonic). I think that the new ORM style (define your schema in c# code) helps alleviate a lot of this, but unfortunately, as I am using TableAdapters, I don't see how I could implement something like migrations.
How do people deal with this?
Release management for DBs usually involves migrations of static data and running of scripts to update/create programmability elements (sprocs, UDFs, triggers, etc) and modify existing schema definitions. Looks to me like you're missing the scripts. If you're making changes manually to your development DB and not creating scripts that mirror those changes, you will need to repeat the same manual steps against your test/production environments, which as you say is error prone and dangerous.
SQL Server Management Studio makes it easy to save scripts that reflect changes to any database objects. In the toolbar there should be an icon called "Generate change script", which gives you the option to save the SQL file to disk. You can then use this to perform the same change against another server. You can also manually script any or all stored procs, UDFs, triggers and so on, and run those against a server as well (just right-click on them).
As to rollback, that's normally achieved by restoring a backup of the database made just before the deployment process begins.
This whole process tends to be different for each company, but that's generally how it's done.
ORMs that auto-generate schemas have always seemed evil to me, not to mention pretty much impossible to use against a production box, but I guess there's also an option.
The easiest way to deal with this problem is to buy software that can detect db schema by comparing two databases changes and generate a change script that can update your target database. I am using Visual Studio Ultimate 2010 for that, but there's also cheaper software that can do the same. This works for me 99% of the time (the only instance where this did not work properly for me is when I renamed a table column).
If you don't have such a piece of software, it is crucial to generate your SQL change scripts by hand. Whenever you do a change to the database schema, keep track of the SQL you used for that changed and add it to one big file of db schema changes for the next version of your software. It's a bit tedious at the beginning, but you'll get used to it pretty quickly.
Then when you are ready to deploy the software, proceed as follows:
Take the website offline
Make a backup of your current production database.
Make a backup of your current production website.
Upload your new code to the server
Run the DB changes script you previously created (either by hand or with the software mentioned above)
Take the website back online and see if it works. If it doesn't and you can't easily fix the problem, revert to the previous website and db version until you have fixed the bug.
All of these steps can be easily automated using batch files and the SQL server agent or SQLCMD.
Generally you should deploy to a staging server first, then test your website very thoroughly and only then move on to the production server. This way you avoid longer downtimes on your production server and minimize the risk of losing any vital data.
Here at Red Gate Software we're currently tackling this exact issue. Please take a look at our SSMS add-in, SQL Source Control, in combination with SQL Compare Pro. We're also working on a 'Migrations' feature, due out later this year, allowing custom migrations scripts to be defined for specific version transitions. As we're still in the early stages of the project, there's still time to give us feedback and help us design a great solution. We'd love to speak to your further about your requirements!
I began recently a new job, a very interesting project (C#,.Net 4, Linq, VS 2010 and SQL Server). And immediately I got a very exciting challenge: I must implement either a new tool or integrate the logic when program start, or whatever, but what must happen is the following: the customers have previous application and database (full with their specific data). Now a new version is ready and the customer gets the update. In the mean time we made some modification on DB (new table, columns, maybe an old column deleted, or whatever). I’m pretty new in Linq and also SQL databases and my first solution can be: I check the applications/databases version and implement all the changes step by step comparing all tables, columns, keys, constrains, etc. (all this new information I have in my dbml and the old I asked from the existing DB). And I’ll do this each time the version changed. But somehow I feel, this is NOT a smart solution so I look for a general solution of this problem.
Is there a way to update customers DB from the dbml file? To create a new one is not a problem (CreateDatabase with DataContext), is there any update/alter database methods? I guess I’m not the only one who search for such a solution (I found nothing in internet – or I looked for bad keywords). How did you solve this problem? I look also for an external tool, but first for a solution with C#, Linq or something similar.
For any idea thank you in advance!
Best regards,
Emil
What I always do is use Red Gate's SQL Compare to compare the schema of the new database to the schema of the old database. It will generate a change script for you and then you can run that script in code.
We have a table that has a single row in it for program setup information. One of the columns in this table is the database version number. This will instantly tell us what database version the customer has when we do an update. Then we run every script that will update them to the latest version they need to be running. Whenever we release a new version (with database changes), we run the SQL Compare and make a script to go from the previous version to the next. We don't do any scripts that will skip versions, just in case of strange conflicts that may arise from that.
This also gives us the opportunity to do any data massaging we may have to do in between versions by writing a custom script and inserting that into the update scripts. Every update script changes that database version field as well.
This allows us to do a lot of automated updating. Having that database version allows the client to take a peek at that version before the user has a chance to use the application. If it's different and the application needs an update, it will go out to our ftp site and download the update and run the setup automatically.
Basically what you want to be able to do is to script the changes - to be able to run "something" that allows you to update one version of the database to the next and also to make any necessary changes to the data required by that change in the schema.
Good news is that you can do this with SQL, you can write DDL statements to create and modify a database schema.
My solution is to put my database schema maintenance entirely in code, I think this is the best version of the writeup I've done so far:
How to create "embedded" SQL 2008 database file if it doesn't exist?
Why in code? Because it works. May not be the best solution but its one I have had some success with and the results are consistent and repeatable. Oh and its version controlled too.
The big problem you may have in this specific instance is that you need to establish a baseline - to make sure that the existing databases are consistent in terms of their schema. This is where more complex and clever tools may serve you better - being able to do a schema diff and then update has a lot of appeal as a concept for example but equally you're somewhat dependent on having your reference database perfect and that raises other issues.
I am working on a Sometimes Connected CRUD application that will be primarily used by teams(2-4) of Social Workers and Nurses to track patient information in the form of a plan. The application is a revisualization of a ASP.Net app that was created before my time. There are approx 200 tables across 4 databases. The Web App version relied heavily on SP's but since this version is a winform app that will be pointing to a local db I see no reason to continue with SP's. Also of note, I had planned to use Merge Replication to handle the Sync'ing portion and there seems to be some issues with those two together.
I am trying to understand what approach to use for the DAL. I originally had planned to use LINQ to SQL but I have read tidbits that state it doesn't work in a Sometimes Connected setting. I have therefore been trying to read and experiment with numerous solutions; SubSonic, NHibernate, Entity Framework. This is a relatively simple application and due to a "looming" verion 3 redesign this effort can be borderline "throwaway." The emphasis here is on getting a desktop version up and running ASAP.
What i am asking here is for anyone with any experience using any of these technology's(or one I didn't list) to lend me your hard earned wisdom. What is my best approach, in your opinion, for me to pursue. Any other insights on creating this kind of App? I am really struggling with the DAL portion of this program.
Thank you!
If the stored procedures do what you want them to, I would have to say I'm dubious that you will get benefits by throwing them away and reimplementing them. Moreover, it shouldn't matter if you use stored procedures or LINQ to SQL style data access when it comes time to replicate your data back to the master database, so worrying about which DAL you use seems to be a red herring.
The tricky part about sometimes connected applications is coming up with a good conflict resolution system. My suggestions:
Always use RowGuids as your primary keys to tables. Merge replication works best if you always have new records uniquely keyed.
Realize that merge replication can only do so much: it is great for bringing new data in disparate systems together. It can even figure out one sided updates. It can't magically determine that your new record and my new record are actually the same nor can it really deal with changes on both sides without human intervention or priority rules.
Because of this, you will need "matching" rules to resolve records that are claiming to be new, but actually aren't. Note that this is a fuzzy step: rarely can you rely on a unique key to actually be entered exactly the same on both sides and without error. This means giving weighted matches where many of your indicators are the same or similar.
The user interface for resolving conflicts and matching up "new" records with the original needs to be easy to operate. I use something that looks similar to the classic three way merge that many source control systems use: Record A, Record B, Merged Record. They can default the Merged Record to A or B by clicking a header button, and can select each field by clicking against them as well. Finally, Merged Records fields are open for edit, because sometimes you need to take parts of the address (say) from A and B.
None of this should affect your data access layer in the slightest: this is all either lower level (merge replication, provided by the database itself) or higher level (conflict resolution, provided by your business rules for resolution) than your DAL.
If you can install a db system locally, go for something you feel familiar with. The greatest problem I think will be the syncing and merging part. You must think of several possibilities: Changed something that someone else deleted on the server. Who does decide?
Never used the Sync framework myself, just read an article. But this may give you a solid foundation to built on. But each way you go with data access, the solution to the businesslogic will probably have a much wider impact...
There is a sample app called issueVision Microsoft put out back in 2004.
http://windowsclient.net/downloads/folders/starterkits/entry1268.aspx
Found link on old thread in joelonsoftware.com. http://discuss.joelonsoftware.com/default.asp?joel.3.25830.10
Other ideas...
What about mobile broadband? A couple 3G cellular cards will work tomorrow and your app will need no changes sans large pages/graphics.
Excel spreadsheet used in the field. DTS or SSIS to import data into application. While a "better" solution is created.
Good luck!
If by SP's you mean stored procedures... I'm not sure I understand your reasoning from trying to move away from them. Considering that they're fast, proven, and already written for you (ie. tested).
Surely, if you're making an app that will mimic the original, there are definite merits to keeping as much of the original (working) codebase as possible - the least of which is speed.
I'd try installing a local copy of the db, and then pushing all affected records since the last connected period to the master db when it does get connected.