We use SubSonic as an ORM of sorts(really more of a query-helper). For one reason or another, we have a bit of a dynamic schema and as such certain tables have generated column names and such. Well, this has been all fine and dandy until now. Now, our production generated columns don't match up with our development generated columns. The first work around that came to mind is just regenerate the subsonic files before deploying to the production servers, but that seems a bit messy. Is there any better way than this?
Note, that these generated columns are never actually used from the ORM but we do sometimes pull down entire rows using
var data=MyData.DynamicTable.SingleOrDefault(x => x.id==1);
That would throw an error in production though using the development generated subsonic files when trying to load GeneratedColumn10 or whatever, which exists in development but not in production.
I have actually ended up modifying the SQLServer provider so that it checks each column name against a regex to decide if the column should be included or not. It is important to note that you must do this for both columns and for where it looks up the Foreign Keys
You could create your DAL against the production DB and use it with your developement server. Unless your additional columns in the dev environment are not nullable and have no default value specified your update and insert queries will work, too.
Related
I apologize if this is duplicative; I could find nothing directly pertaining.
The difficulty involves EF Core (v 3.1.8, if it matters), but is not specific or restricted thereto. I am doing code first, creating a number of entities, but the key point is that I am getting my initial data set from an app that I am trying to replace. My new app has a number of structural differences in every corresponding entity, but the data in the old app is still critical, so I will be transferring it to my new database. (Old db is hosted by MS SQL 2008; new db is hosted by MS SQL 2019, if it matters).
Most of the key fields are GUIDs, and the problem is that in EF Core, at the point in the future when I want to use the new app to do more data entry, I will also want the database to choose the GUID. In EF Core Fluent API parlance, that would be, for example:
modelBuilder.Entity("ReplaceOldApp.Models.Address", b =>
{
b.Property<Guid>("AddressID")
.ValueGeneratedOnAdd()
.HasColumnType("uniqueidentifier");
}
However, if I inform EF Core that I want the database to create the key, then it will create the tables such that when I try to transfer the data from the old database (whether using EF or some other means), the new database will ignore the old GUID and create a new, unrelated one. (Or at least, that's what I think will happen. I'm not ready to try it yet.) If that happens, then all of the data from, say, the old Person entity (such as the above-implied Address entity), will no longer be related between their corresponding entities in the new database, because all records will have shiny new GUIDs. I will have all the information, and no way to actually use it.
Obviously I can tell EF Core to inform the database that it will not be creating the GUIDs, and I can then read, unmunge and transfer the data from the old database to the new without fear of data loss (God willing). But then going forward, for any new data entry, the GUIDs will not be automatically genned. I can of course then mod my IEntityTypeConfiguration Fluent API classes for the various entities and do a second migration, re-genning the affected tables, but I'm worried that EF Core will decide that it needs to DROP the tables to accommodate such a change. (Again, I do not know for sure because I have not tried it: sorry.)
So my question is: How would you approach such a situation? Should I ignore EF and do something clever with MS SQL Studio? Should I do two migrations with a transfer in-between? Should I tell the database, even though it has been told to gen the keys, somehow to accept the old keys without changing things, perhaps via LINQ?
============== Edit:
I'm sure SSIS would work to transfer the data from old to new databases, but the learning curve appears daunting, and I am only trying to solve one problem, not gain a new career. Powershell ditto, although it may be a bit more of a hacker's tool, and as such knowledge of it might assist tweaking or help to solve a diverse set of one-time SQL Server headaches. However, again, as would you, I prefer to use what I know, or failing that, learn or learn more about a tool which promises to serve me consistently into the future.
With the very welcome new (to me) information about IDENTITY_INSERT, and information gained from Linq To Sql and identity_insert, I believe I should not use LINQ to SQL because it may assume that IDENTITY_INSERT is OFF and simply filter out the crucial GUID, failing therefore to provide it to the target server. Rather, it seems I can use C# to produce a series of generated SQL statements, and then run each one on the target server inside a TransactionScope(). Because each such insert will thereby run 'in the same connection', the state of IDENTITY_INSERT will be preserved for that entire insert transaction, and (creek don't rise) it should work.
Again, I appreciate your answer, Randy in Marin. It has, it seems, led me to an approach that will work within the potential constraints of my context (EF Core), while allowing me to preserve the crucial existing IDENTITY information. Peace.
Not being an EF programmer, I don't know if there is an option for identity insert that you can enable for a migration. You might search the term to see if it comes up.
Our team support database migrations. We can do it a number of ways. I would not even consider EF because it's not designed for data migrations - or for database design. (And because we tend to use what we know.)
This is not the way I would do it, but it might be better than SSIS if you have not used SSIS. If the tables are in the same database or in databases on the same server, you can use T-SQL to load each table one at a time. Even if not on the same server, a linked server would allow a distributed transaction. (I avoid linked servers like the plague, but for a one time thing like a migration I would tolerate it. I would rather restore a copy of the source database to the destination server to use as a source. Distributed transactions gone wrong have forced me to reboot critical servers.)
Each table can have a 4 part name. If the server part (e.g., using a linked server name) is not present, the local instance is used. If the database part is not present, the current database is used. This is the format I assume for the "src_table" and "dst_table".
[myserver\myinstance].[mydatabase].[myschema].[mytable]
Each table is loaded with T-SQL as follows:
TRUNCATE TABLE dst_table
SET IDENTITY_INSERT dst_table ON
INSERT dst_table (...) SELECT ... FROM src_table
SET IDENTITY_INSERT dst_table OFF -- must be turned off - only 1 table can have this ON
If there are foreign keys, some tables (e.g., def tables) would need to be loaded first.
If the table does not have an IDENTITY column (EF code creates all values), you don't use the IDENTITY_INSERT stuff. It will fail if you use it and there is not an identity column. It will fail if you don't use it and try to insert into an identity column.
If there is a lot of data in a table, the transaction might be too big or slow. Inserting in batches might be called for.
If it was something to run on a schedule, I would likely create a SSIS package to do the load.
If I wanted to try something new, I would use powershell and the DBATools module cmdlets to see if extracting to csv and importing the csv would be efficient. The import cmdlet has a column mapping parameter, among many others. PowerShell could be used to do transformation, but I think this crosses over into SSIS territory.
I have dealt with migrations where the GUIDs and IDs no longer related after the move. Using queries joining the new data to the old data, we were able to fix the related values. It's likely more work to fix it after than to plan for it to be correct from the start.
We have a system that will use the same code to communicate with different client databases. These databases will use the same EF Model, but different connection strings.
Our problem is, not every site will be using the same version of our database structure; some might be missing a few columns or contain a few old columns.
If we upgrade the system to the current version, now the database model now has an extra EmergencyContact column. All older databases will now fail, because EF is trying to insert into this column (even though we have not set a value for this property).
Is there a way of telling EF to only use columns for which we have a value for, when it generates the INSERT INTO query?
EF will be fine if your schema has missing columns that are in the real database, but it will not work if you have columns in the schema that are not in the database, and there is no way to fix that.
Your only choice is to use different schemas for different databases, and write code that manages them (ie, only instantiates the version of the context you need).
In the case where your model does not match your database schema, EF will only insert/update the columns in the model. However, if the unknown columns are not null, EF will throw an exception. Also, if you created relational constraints on the unknown columns, of course those will not be created as they are not yet known.
If the persistence layer per site is the only part that changes then I would extract your EF model into it's own version e.g.
DbV1.dll
DbV2.dll
You could then load in the appropriate DLL based on some setting from the client i.e. you could pass information as a custom header e.g.
db-version: 1
There are other more reliable ways, however, I don't know what your current setup is like so it's difficult to answer.
We have a process where our database guys script changes (and version them using Juneau) to our application's database out-of-band with our code base. They're good at accounting for new columns being null, and not wiping existing data, but occasionally a column rename sneaks in that isn't fully communicated. So they will make some changes to the database schema on a testing server, we'll update Entity Framework to work with those changes, and then commit our code. This process works okay, except for when it's time to deploy.
We have TFS set up to deploy the successful build to the appropriate servers, but there's no guarantee that the database for that environment has been updated. We don't care if extra fields/tables/views/etc. exist in the target database, but we want change the build to check that the database contains at least everything EF is aware of.
I looked at this question, but I don't need the schema to match exactly. Plus, we don't want it creating/modifying the database directly. And this question seems like it's trying to achieve a similar ideal, but still not quite what we're looking to achieve. We just want a integration test of sorts to verify our version of EF will work with the target schema.
I wonder why you try to deploy your application without changes to database. Your application is dependent on the database so the deployment should always be done after the database. It looks like you are going to invest a lot of time to develop validation to fix your incorrect deployment process (where fixing the process itself is the correct solution).
Anyway you can create some "validation" of the database but it will take some time. If you are using EDMX file you can open it as XML and read its SSDL part which describes all expected tables, columns, relations, views (in form of SELECT SQL queries), stored procedures and functions. You can parse this XML part and use system database views (sys.tables, sys.columns, ...) to query if these objects exists in the database.
Another approach can be using database diff. tool to compare your current test database with the target one. This will require the tool which can be executed from command line and you will have to parse its output to find breaking changes.
Which ORM will give me compile-tested queries?
Is linqtosql compile time tested?
Edit:
Say I write a query that references a column named 'TotalSales'. I then rename the column in my database to TotalSales2 (and any other config file like: Employee.cfg.xml in nHibernate).
When I compile the project, I want Visual Studio to tell me the column 'totalSales' doesn't exist and then I will go and change it.
There aren't any as far as I'm aware. They will often let you create a LINQ query that cannot be translated into SQL for example. Also, I am not aware of any compile time checking that your mappings map to your database correctly.
You can, and should in my opinion, perform all these checks within tests. Most ORMs make this easy to do.
I use LLBLGen but it has to be "refreshed" when data model changes are made. I don't think you'll get an ORM that will AT COMPILE TIME check for modifications against the database. You're asking for quite a bit there.
In DataObjects.Net properties marked by [Field] attribute are always bound to field in database, so you can be sure that query will be translated. If you use not persistent field or another not supported statement, query translator will fail in runtime or performs such operation on fetched objects (on client).
Generally compile time validation is impossible or theoretically can be performed with special post-build tasks, that will scan compiled code, find all queries and validate them. But such checks will seriously slow down compilation process.
Perhaps not exactly what you're looking for but if using the Entity Framework and selecting "Update Model From Database" from the designer you will get messages saying the fields are no longer mapped if you change the names.
This doesn't happen automatically when you build a project.
Basically, you need 2 features together:
Compile-time checked queries (= an ORM with LINQ implementation). This is normally not a problem - at least some tools support this.
Pre-build step updating your entities based on database schema. AFAIK this is rarely implemented this way - normally you must explicitly update the model by the schema. Note that this part is normally rather costly.
SubSonic can do that if you include the code generation step as a pre-build event.
I used a Java tool called DODS, which was developed ca. 2000 with the Enhydra application server. DODS is still around here: http://www.enhydra.org/tech/dods/
The way DODS works, and which meets your goal of compile-time validation, is that it's a code generation tool. It generates Java classes corresponding to the tables in your database. Object instances of these classes have getters and setters for each column in the table. Of course if you change your database structure, you have to re-generation the Java code using DODS.
As long as you keep the generated code up to date with the structure of your database, it provides compile-time validation that any application code that uses these classes is querying valid tables and columns.
Anyway, I realize you tagged your question with C# and ASP.NET. A tool that generates Java code isn't going to be that helpful for you. But there could be another tool more specifically for .NET that works on the same principle of generating code that maps to database structure. So I'd suggest narrowing your search to .NET ORM tools that say something about code generation.
I am working on a project that requires the use of multiple databases that for the most part are completely identical but some columns might be missing. How do you get NHibernate to handle this for instance i have a table with 4 columns an index and 2 data coloumns that will always be availible but a singe customer does not want the column in their database.
as this is part of a legacy application migration i do not have the luxury of dictating the database format or even change the databases. anybody have any ideas of how to do this. I cannot get NHibernate shards to work with this either.
KR
Nicky
I don't know of a way to tell NHibernate to ignore columns that are otherwise mapped.
I would look at creating multiple mappings files for the different databases and then depending on your environment configure your SessionFactory using the correct mapping files.
This may seem like a little more work to setup initially but it makes it very clear that in database X you have columns A-B-C and in database Y you only have columns A-B.