Hopefully what I'm after is possible at all. I'm currently looking at a way to do what Entity Framework Code First migrations does when you use Package Manager Console in Visual Studio but do it in runtime. The two processes I'm after are Add-Migration and Update-Database.
To explain a bit more, I want to be able to construct custom meta data for the Model and compare that against existing or non-existing database. This should output the same migration file that you get from Add-Migration command in Package Manager console.
After I've got one or more migration files I want to run the Update-Database command again in run-time, programatically to apply Up / Down changes to the database.
The key point is to be able to provide a completely custom (virtual?) model meta data to the Add-Migration procedure, not one based on actual poco classes.
If anyone has done something similar before, I'd appreciate any pointers on where to look as I didn't have much luck with Googling / EF documentation.
EDIT:
To address the WHY: I want my users to be able to define their structures in run-time. They would define the structure as part of a changeset and then be able to apply one or more changesets to the underlying database. Essentially, users would do from web-ui what a developer does from visual studio, except that the classes won't be .cs files but instead a class description in a relational database.
EF seems like a perfect fit to do this because it already does everything I need it to do. I've done schema comparison and run-time change mechanisms in the past. Since EF already does this, I don't want to reinvent the wheel. The migration files that EF Code First outputs are also a perfect fit fro what I'm trying to do.
The reason I haven't yet dug into the EF source is I wanted to check if someone else has attempted this before and if they can share and pointers. Some concrete questions:
What class / method do I need to be able to run schema compare?
How do I hook into that process to supply my own meta-data for "POCO" classes?
What are the options for outputting the detected schema changes (the Up, Down and Seed methods)? Ideally I'd like to compile a change set into a satellite assembly when user chooses to apply changes.
How can one run DbMigration instances in run-time?
I'm after some concrete examples.
Related
I am working on a project using Entity Framework 5 with MVC5. My Project is currently running.
I am trying to add a column in a table. But as we know that in EF when we add a field in model it drop and recreate the database, which i can`t do.
One method for this is found code migration. But my manager is not allow me to use that(because its a big database project).
Please help me and suggest something for it.
When I start using code first with Entity Framework, I was in the same situation as you. I was always running Update-Database -F and then watching all my tables get dropped and recreated, even for something as simple as renaming a field.
Versioning databases is hard, but it's much easier with named migrations (which I think it what you mean when you refer to code migrations). I know your boss is against the idea, but it's very flexible.
Essentially you run Add-Migration -Name xxx in your Package Manager Console and Entity Framework will scaffold a configuration class for you with the default commands (both for versioning Up() and Down()) it will execute when you Update-Database. If you don't like the commands, you can change them. You can even move data around if you need to (it's a bit fiddly though).
I think you have four options available to you;
Use code-first automatic migrations: This is what you have at the moment, and doesn't give you enough control over what happens when you update your database. It's good for getting started in the earlier stages of a project, but becomes unwieldy after production.
Use code-first named migrations: Gives you the control you need via Configurations - but your boss has prevented use from using it.
Use a database-first approach: Database First allows you to reverse engineer a model from an existing database. So if you need to make a change, you would change your database first, and then regenerate your models using EF. This is usually favoured by DBA's, but it may mean that you have reimplement some aspects of your existing project.
Dont use entity framework: It's possible that you could revert back to SQL queries, which your boss might accept and gives you the flexibility you need - but who needs that kind of pain?
Let me know if I can help further.
Could someone explain the concept of Migrators (specifically fluentmigrator)?
Here are the (possibly confused) facts Ive gleaned on the subject:
Is it a way to initially create then maintain updates for a database
by way of versioning.
The first migration (or initial version of the
database) would contain all the tables, relationships and properties
required (done either fluently or using a chunk of sql in a script).
When you want to push a change to a database, you would create a new
migration method (Up and Down), something like add a new table or modify a field.
To deploy one of these migrations, you would use a
command line specifying the dll containing the migration, the
connection string and the required version.
If you had a rather complex set of data models, wouldn't it be rather difficult and time consuming to create a migration definition for all of that?
I know with nHibernate/fluent you can easily generate tables for a database without having to define anything other than the models and map files. Is there a way to make this configuration compatible with the Migrator/Versioning?
When nhibernate/fluent is in charge of generating a database, I do not necessarily need to define every thing aspect of the tables. Its done either via convention or via the mapping files. With the migrators I would need to define this level of detail?
Lots of questions here. I'll answer the questions with a focus on FluentMigrator.
Is it a way to initially create then maintain updates for a database
by way of versioning.
FluentMigrator is a way to version control your database schema. Everyone does it in some way. Either manually, with sql scripts, with a tool like SqlCompare or a Visual Studio Database project. All these methods are easy to mess up. It is so easy to make a mistake when releasing a new version and cause the system to crash. Migrations is a better way to handle this.
FluentMigrator allows you to define a change to the schema as code and this is usually checked in to your source control with the other code changes. Meaning that you can say version 1.XX of your system should have version 123 of the database. It means if you roll back your code to the previous version you also know what version of the database to rollback to as well.
It can be used both to create the database schema from the beginning or to start with version control of the schema for an existing database.
A Migration is a way to describe a change to your database schema. FluentMigrator creates a VersionInfo table and stores the unique id (version number) of the Migration after is has been applied.
For example, if I have two Migrations one with Id 1 and one with Id 2. If then I execute the first Migration then Id 1 will be stored in the VersionInfo table and I can look there and know that the version of the database is 1 and that version 2 has not been applied yet.
Being able to know which version the database schema is very useful when pushing changes from Test to Production or if you have multiple copies of the database in Production. For example, I have a customer with offices all around the world and each office has their own copy of the database and all of them are on different versions. Without knowing the database version it would be very difficult to update them safely.
Most of the time I do not need to actually look in the VersionInfo table, FluentMigrator handles this automatically. It compares the assembly with Migrations to the VersionInfo table and figures out which changes have not been applied yet and then executes those.
The first migration (or initial version of the database) would contain
all the tables, relationships and properties required (done either
fluently or using a chunk of sql in a script).
The starting point is up to you. You can have a first migration that is an sql script that you have generated from the current database. You could could also use one of the contrib projects like FluentMigrator.T4 to generate a Fluent Migration. Or you could just decide that the existing database is the starting point and save a copy of it to be able to restore it as version 1.
I have introduced FluentMigrator to a lot of legacy databases without any major problems.
When you want to push a change to a database, you would create a new
migration method (Up and Down), something like add a new table or
modify a field.
Yes, Up is used to apply the change specified in the Migration and Down rolls it back. So Up could be to create a table and Down could be to drop the table.
To deploy one of these migrations, you would use a command line
specifying the dll containing the migration, the connection string and
the required version.
There are three runners available to execute migrations. The command line runner, the Nant task and the MSBuild task. There are usually executed as part of a build script.
The MigrationRunner class can also be used in code. You might do this if you wanted to build your own runner or if you have other needs (like building databases dynamically or automatically updating the database if a new migration is added.)
If you had a rather complex set of data models, wouldn't it be rather
difficult and time consuming to create a migration definition for all
of that?
I have mostly answered this already. It is usually quite easy to generate an sql script for a database. For Sql Server it takes less than a minute to generate the script even for large databases. This script can be saved in a .sql file and executed as the first migration using the Execute.EmbeddedSqlScript expression. It works a treat.
I know with nHibernate/fluent you can easily generate tables for a
database without having to define anything other than the models and
map files. Is there a way to make this configuration compatible with
the Migrator/Versioning?
At the moment, there is no such integration and in practise I, at least, don't miss it. There was some discussion about connecting Fluent NHibernate and FluentMigrator but it would be a lot of work. It would enable scaffolding to generate changes to the model like EF Code First migrations do. It's not on the roadmap at the moment however.
When nhibernate/fluent is in charge of generating a database, I do not
necessarily need to define every thing aspect of the tables. Its done
either via convention or via the mapping files. With the migrators I
would need to define this level of detail?
Yes, you would need to define at that level of detail. FluentMigrators' migrations are a DSL (own little language) for defining schema changes that are translated to sql. You can write sql directly as well using the Execute.Sql expression. Entity Frameworks migrations have that sort of integration which has both advantages and disadvantages.
Check out the wiki or one of the tutorials here, here (part 1) or here (part 2) for more help getting started.
I'm reading up on EF Migrations and starting to play around with it.
I use Git for source control, and every time I start work on a new feature I create a branch. The reason for this is that if I need to quickly jump to a different feature, or do an important bugfix that needs to be pushed out, I can just change branches and continue working.
Currently I have my app setup so that if I'm not in a production environment (web.config appsetting) it nukes the database and re-sets it up if there have been data model changes (Code-first). I thought that migrations would be good in that I would not have to completely nuke my data on the slightest model change.
Unfortunately, I am seeing some signs that it isn't as version control friendly as I had hoped. For instance, if I add a new column to my data model, update the database, then decide to revert my changes, EF is still aware of the model changes because it stores them in the database itself.
It seems like there is no clean way to tell it to downgrade db version when the branch first happened, and then migrate up to where the db is on the branch I'm deciding to work on without a LOT of remembering and manual tracking.
Does anyone have a strategy for how to utilize migrations in a non-trivial development scenario with version control?
Because each branch will contain migrations that the others clearly cannot know about, there's no way it could know how to transform the database to match (without the use of automatic migrations, which I doubt would work well anyway).
The best strategy for support different branches is for each to target it's own copy of the database, depending on how many branches we're talking about. That way you're free to jump between them as often as you like and they'll only touch their own database as changes are made/merged in.
Alternatively ensure your core data is part of your migration scripts, and blow the database away like you currently do.
Edit: After merging changes between branches you may need to regen their metadata using the add-migration command.
We have a process where our database guys script changes (and version them using Juneau) to our application's database out-of-band with our code base. They're good at accounting for new columns being null, and not wiping existing data, but occasionally a column rename sneaks in that isn't fully communicated. So they will make some changes to the database schema on a testing server, we'll update Entity Framework to work with those changes, and then commit our code. This process works okay, except for when it's time to deploy.
We have TFS set up to deploy the successful build to the appropriate servers, but there's no guarantee that the database for that environment has been updated. We don't care if extra fields/tables/views/etc. exist in the target database, but we want change the build to check that the database contains at least everything EF is aware of.
I looked at this question, but I don't need the schema to match exactly. Plus, we don't want it creating/modifying the database directly. And this question seems like it's trying to achieve a similar ideal, but still not quite what we're looking to achieve. We just want a integration test of sorts to verify our version of EF will work with the target schema.
I wonder why you try to deploy your application without changes to database. Your application is dependent on the database so the deployment should always be done after the database. It looks like you are going to invest a lot of time to develop validation to fix your incorrect deployment process (where fixing the process itself is the correct solution).
Anyway you can create some "validation" of the database but it will take some time. If you are using EDMX file you can open it as XML and read its SSDL part which describes all expected tables, columns, relations, views (in form of SELECT SQL queries), stored procedures and functions. You can parse this XML part and use system database views (sys.tables, sys.columns, ...) to query if these objects exists in the database.
Another approach can be using database diff. tool to compare your current test database with the target one. This will require the tool which can be executed from command line and you will have to parse its output to find breaking changes.
In writing an application that runs on Fluent Nhibernate/Nhibernate, something has me a bit concerned. I suppose this would be true of any ORM (and even without using an ORM), but what is the ... I guess the word is 'field of study' that relates to the best practices and methods for updating a database after deployment?
In nHibernate, I establish a SessionFactory and have an initial run where it writes the database out based on the mappings. That's fine and good, I can even write the database out manually. But what about when my client comes back and wants something new added? Can I append to the database without losing my data? I am completely new to all of this and it has been troubling me since the start of this project, and I really do not know what direction to go to make sure I can manage the program after it is deployed.
I have looked at other stack overflow questions that I could find regarding this topic - one of which did not even have an accepted answer (though the question itself was kind of vague), but I did discover the tool http://www.red-gate.com/products/sql-development/sql-compare/ from the question
Tool to upgrade SQL Express database after deployment though I am wondering just how good of a 'strategy' that is.
There are a couple of options, use the AutoMapping feature in Fluent NHibernate to minimize the mapping code you write. If your schema changes comply with the AutoMap conventions then you only need to work with the corresponding domain object changes.
Another less optimal option is to take a database first approach and have something like MyGeneration automatically generate the domain classes and NHibernate mapping files from the schema. This works if you have complete control of the database schema and it can be made to implement a good domain model design (both conditions which very rarely ever happen...)
In either approach, these tools can help handle the database scripting needed to "migrate" the schema changes to a new version
from my experience, after deployment you have to manually keep your db structure up-to-date.
that means that whenever you add / change your db structure, you do so using a script with DDL commands.
when you're ready to deploy, you just run those DDL scripts against your production db.
for example, if you add a 'bar' column to your 'foo' table, your script would be something like (pseudo-code):
ALTER TABLE foo ADD COLUMN 'bar' int(32) not null default(0);