So I am using Neo4J and thought about how to update existing databases with schema changes e.g. a Node has new properties, edges changed, values changed, etc.
So far I was only using SQL Server and the Entity Framework includes a migration tool that can automatically write migrations when the schema changes.
How is this done in Neo4J?
E.g. a customer has v. 1.0 from our software and neo4j database. We make changes for v.1.1 and now we need to update the db from all our customers.
I am using Neo4J in C#. I have so far only found something similar for Ruby: https://neo4jrb.readthedocs.io/en/8.2.x/Migrations.html
You can use: https://www.liquigraph.org/.
Where you can create the cypher queries you need to migrate database data.
Since neo4j is basically "schema-less" (in the relational DB sense), there is no tool for migrating to a new neo4j "schema".
You will have to write code to update the DB.
I do actively maintain Neo4j-Migrations:
https://github.com/michael-simons/neo4j-migrations
It's basically an automated script runner, that stores schema versions inside the same database as a subgraph. It is also able to store them separately in a Neo4j server supporting multi databases.
As you mention that you are using C#: This tool is Java based, but I do also provide native binaries for all 3 major OSses availabe on GitHub. You might can integrate them into your workflow.
Related
I have data stored in a MySQL database and I'm accessing it through entity framework 6. What I want to do is to extract the data and store it in a server-independent manner, so that I can use this data in the future for bootstraping of a larger database (where the data from MySQL is only a small portion of). The target database will not be MySQL.
The MySQL data was there before, so I used Database First. For the new database I use Code First. The MySQL data also needs some clean-ups, what is done in code. So the workflow is:
Read data from MySQL with Entity Framework database first
Optimize/enhance/correct it in code
Store in another database using code first
So my first thought was using SQLite, but sadly, Code First is not supported.
Using some other format (i.e. XML/DataSets/CSV) to store the data would require me to reimplement the import code.
Right now I didn't find another embedded database with Entity Framework support. As Entity Framework is meant for relational databases, I would say that Redis (or other NoSQL dbs) won't do the trick.
I also thought if "converting" H2 using IKVM.NET and then using H2 as SQLite replacement would be an option -- but there's the risk that this might fail due to some internal dependency not supported by IKVM.NET.
So I would be glad to get answers to the following questions:
Would the H2 way work? Is there even a stable/working ADO.NET provider?
I wouldn't care much about the local storage format (could also be JSON etc.) -- is there anything not-SQL, but providing Enitity Framework support?
Did I miss something? Is there a way I just didn't see yet?
Here are some related questions I stumbled upon which deal with the possibility of using Entity Framework to use files:
Entity Framework with text files (no database!)
A list of Entity Framework providers for various databases
Entity Framework with XML Files
Microsoft Entity Framework using a flat file as a data source
There is no complete solution, but there are many ready-made parts:
This project wraps the compiled Jar of H2 after IKVM.Net
with classes that implement the ADO.Net interface
to allow for easy use in .Net projects:
https://code.google.com/p/h2sharp/
EF provider for SharpHSQL (which doesn't work, because, SharpHSQL is too old, it's port of Hypersonic 1.4):
https://github.com/ArsenShnurkov/SharpHSQL/tree/master/src/SharpHsql.Linq
here is the description of build process:
https://groups.google.com/forum/#!topic/h2-database/QAvFqbyd4_0
https://code.google.com/p/h2sharp/wiki/BuildingH2Sharp
The authors of above code did a great job of writing such huge amount, so i think it's nothing wrong with writing some more to finish your task.
I just started to use LINQ-to-SQL, so my problem may be trivial.
We have a central database running on SQL Server 2005. There are distributed desktop .NET 4 applications which save measurement data into a local SQL Server Compact database. These local database (SDF) files are regularly transported to the server, where they are imported into the central database using SqlBulkCopy.
The distributed desktop applications use LINQ-to-SQL to handle data and to create their local SDF database. The ORM is currently done by a manually written DataContext subclass, compiled as a separate library, which contains a nested class for every table of the central database. I wrote the DataContext subclass by hand simply because I wanted to avoid code generators before I more-or-less understand how LINQ-to-SQL works.
The central database is simple at the moment, but it will structurally expand soon, by adding new tables and adding new versions of existing tables. The problem is, it would be nice to automate the generation of the DataContext subclass. In an ideal situation, this could be done as part of the daily build process. This way after the database team changes the database, the application developer team would get the new version of the ORM library. (Old code would not break, since every old table would stay in the database. Old versions of the tables will be deleted only when none of the distributed application versions use them.)
So my question is, what is the best way to generate a DataContext subclass for an existing database? I would prefer a command line tool or an API. Thank you for your help in advance!
Our product supports multiple databases. At the moment, we support FireBird and MSSql and there is future support coming for Oracle.
When we push out an update for our product there is the potential that we also need to update the client's database schema as well.
Traditionally we have scripts that are flavored to the DB version which would do things like "Alter table add column" which are executed in order to bring the database up to the correct version. This is becoming a hassle because we have to maintain two sets of sql scripts (with more on the way if we add Oracle to the mix).
We use the Entity Framework in our db layer. The EF already contains a schema of the database. I wonder - is there a slick way to use the EF and it's knowledge of the schema to handle the updates to the client DB?
EDIT -
This is EF 4.0
You haven't mentioned which version of EF you are using. In V4, you can use the "Code First" model. Check out ScottGu's post. I'm not 100% sure if it is supported across all the DBs you need though. [edit]This will only create new schemas, not update existing schemas[/edit]
It might be some work, but it may be worth switching to nHibernate, which supports a wider variety of DBs and auto-updates the schema.
Currently, changes to the database are made through the SQL Server Management program. IF a table changes, sqlmetal is run to regenerate the linqtosql classes and development continues. However, this makes deployment a pain, as you have to go through and manually update the deployment database (and any other databases used in the development cycle). It would be nice if we could use C# to generate these changes, as it would help eliminate human error and have the added benefit of being able to keep the database structure in git. Right now, the only representation of the database is in the generated linqtosql classes.
I've been looking around for a nice library that can handle this sort of thing, but the main solutions seem to be: keep a sql generation script, or embed sql in C# classes that can be run to make changes to the database. Both of these seem to be very non-ideal situations, as you lose the nice strong-typing that C# provides. It seems like there should be a way to do this using pure C#.
I've seen hints of being able to do things like generate databases from POCOs using both the entity framework and linqtosql, but I'm having a hardtime finding specific examples of that being used. Additionally, I haven't been able to discover if those have a graceful (i.e. data preserving) way of handling changes to the database after the initial table generation.
Are there any projects out there that solve this problem?
There exist several tools that help you with schema (and data) migrations of your database: RikMigrations, Migrator.Net and Machine.Migrations. Hope that helps.
Wizardby looks also promising: It provides database independent DDL scripts and automated migrations between different versions of a schema.
VS2010 can operate version control on your database schema through a Database Project. There are other tools out there for DB development that offer version control, you'll need to search to find them and compare pricing.
I prefer to version using sql scripts. Works pretty well, is free, supports updates, easy to version, works well with traditional source control methods.
First,
Create your DB
Use the Database Publishing Wizard to publish the database as a .sql script
Add a version number to the script
Add to your solution
Check into source control.
As updates are made,
Script updates to the previous schema as .sql files
Add a version number to the script that is incremented from the previous version
Add to your solution
Check into source control.
It sounds like you need a tool like Migrator.NET to manage your database migrations. We use it with a call from our site start-up to migrate the database as needed for any particular version.
I have toyed with an idea for creating a cleaner interface and someday hope to get around to implementing it, but other priorities have pushed that back. For now we are using raw sql strings in our migrations because there isn't a sybase driver implementation (outside of a very ugly hack I have written to manage the versioning table).
Redgate software offers something that may be really useful for you. It's called SQL Packager and it does it's job pretty well.
Features:
Easy roll-out of database updates across your client base
Script and compress your schema and data accurately and quickly
Package any pre-existing SQL script as a .exe, or launch as a C# project
Simplify deployments and updates for SQL Server 2000, 2005 and 2008
They also offer SQL Source Control which also may be useful to keep things nice and easy.
As an addon to MadBoy, SQL Packager can also launch the package as a C# project.
Red-Gate's SQL Compare is excellent as well, and as some of the banners on SO indicates, there is new SQL Source Control available as well.
Then they have their SQL Comparison SDK.
The trick here is to rely on the database being the single source of truth for your Linq schema, not the generated classes.
We use Linq to SQL extensively in our dev shop, and work as follows:
1. Create your database (working copy) from version control (baseline).
2. Modify your database any which way you like.
3. Generate Linq to SQL classes from the (working) database.
4. Create patches to update your baseline database to your working copy.
5. Check in and share these patches with all developers.
For a very quick and easy way of generating baseline and working copy databases, try DBSourceTools. http://dbsourcetools.codeplex.com
Have fun.
For some integration tests I want to use LINQ to SQL to drop/re-create the test database. I've had this working fine before, however in this project the database is split up into several schemas.
When I try to run the ctx.CreateDatabase() command I'm getting this exception:
The specified schema name "xyz" either
does not exist or you do not have
permission to use it.
The login I'm using to do this has the role dbcreator - Does it need further permissions? Surely a login with persmissions to create a database should be able to create everything contained in that database also?
Update:
Since it looks like there isn't a solution to this problem using LINQtoSQL, does anyone have recommendations of any similiar tools to generate a db that are preferably free? Ideally I don't want to have to muck about hand writing sql build scripts.
From what I've read, the CreateDatabase() method is limited in what it can reproduce of the original database. It won't recreate things like triggers and check constraints, and I'm guessing it doesn't create custom schemas either. You may want to look into creating the database using a SQL Server .mdf file instead to work around this issue. See this blog entry for more details on some of the limitations of CreateDatabase().
I generally do this sort of work in NAnt to create, initialize the database, create users, add logins, etc....and also roll back capabilities. I have written on this topic quite a bit if you are interested:
Build automation with NAnt
Continuous integration with CruiseControl.NET
I will have to see if I can get LINQ to SQL to work in the way you are trying to use it...that sounds like what we used to do with NHibernate.
The dbcreator fixed server role grants you the permission to create a database. If you create a database, you are the dbo of said database and as dbo you have absolute power in the database, includding the power to create, alter and drop any schema and any object contained in any schema.
the problem with LINQ's CreateDatabase() is not permission, is code quality. The generated SQL code simply does not create the needed schema, so the Create table statements fail because the schema does not exist.
Your best choice, if you can afford it, is to add a VSTS Database Edition GDR R2 project to your solution and declare all your database objects in the Database Edition project (part of your solution). You'll be also getting the added benefit of storing all your database objects in a proper source control solution. The output of the Database project would be a .dbschema file containing the definition of your database. At deployment time (test or real) you would run the VSDBCMD Deployment and Schema Import tool to import your .dbschema into the target server. The tool is capable of doing initial deployment of your schema, as well as further upgrades (deploy only differences). The VSDB solution would allow you to controll all your database objects: tables, indexes, views, schemas, field contraints, table constraints, triggers, procedures, users, permissions, logins etc etc. It really covers all the objects that can be defined in SQL Server.
Actually LINQ to SQL does support schemas, but not every Sql Server edition does. To enable CreateDatabase() to generate them the DataContext must be aware that the target database does support them. It can be done by setting the provider on the DataContext:
[Provider(typeof(Sql2008Provider))]
public class CustomDataContext : DataContext {
...
}
Your user also requires db_dlladmin for that database.
I would definately look at Entity Framework, which I am beginning to look into these days. It's an OR/M, and will most definately suit your needs, and alot more once the next version is released.
Entity Framework is also a brain-child of Microsoft and can be found here: http://msdn.microsoft.com/en-us/library/aa697427(VS.80).aspx
One thing to remember between LINQ to SQL and LINQ to Entities is that you are programming against a model, and not the database.