We Keep Coding

Using Entity Framework with Existing Models

I am working on a MVC4 project which will need to use a number of different databases, each with a few stored procedures for searching. The site is an asset search tool which needs to query various existing systems. If I allow the EF to generate models on its own, I will end up with a Model for each procedure I use in each database.
What I would prefer is to have my own POCO model already defined and the EF maps its results to that Model. So regardless of what database the data is taken from it maps back to that same Model. The column names in each database differ slightly so it would really need to be mapping columns to model properties.
There is no writing back to the database, it purely selects data out.
On the 'Edit Function Import' form I can create a model based on the results. There is also an option to view 'Function Import Mapping' but it does not appear to do what I am looking for.
Has anyone else tried this?
Added an image to help explain the issue
The closest to this I have managed so far is to have EDMX1 query 2 databases. This only works because they are on the same Db server. I had to fully qualify the Db names in the stored procedure. I could then use 1 EF Model as a return type for the 2 queries. That Model still is not usable in another EDMX though, so if I need to connect to a different Db server, I still cannot share the Model. So the problem is not solved.
Here is image of current progress.

Function Import Mapping is for mapping stored procedure / function calls to EF code. It's not really relevant here, unless you're using stored procs (which is not the way to go 90% of the time with EF - only use stored procs for more complex procedures).
An EF context, by its very nature, can only have a single database associated with it. You need to create multiple contexts in order to access multiple databases at once.
What I would do in your case is create a database-first schema (.edmx) file for each database, then write a service layer abstraction above it that allows you to flatten the data into your expected model. This is the kind of thing I do all the time, regardless of how many databases I'm working on at once. You've almost outlined this in your first diagram. The service layer may have multiple classes (for example, for a blog website you might have BlogService, UserService, CommentService etc), each of which contain methods that you call from you application layer.
I've put a quick diagram together that might help to explain
http://www.gliffy.com/go/publish/image/4818386/L.png
The service layer does all of your EF work, and your application layer (or business layer, whatever you want to call it) will do all of your business logic.
This setup lends itself well to TDD and Dependency Injection / IoC. Everything is neat and nicely separated.

Strategies for replacing legacy data layer with Entity framework and POCO classes

We are using .net C# 4.0, VS 2010, EF 4.1 and legacy code in this project we are working on.
I'm working on a win form project where I have made a decision to start using entity framework 4.1 for accessing an ms sql db. The code base is quite old and we have an existing data layer that uses data adapters. These data adapters are used all over the place (in web apps and win form apps) My plan is to replace the old db access code with EF over time and get rid for the tight coupling between UI layers and data layer.
So my idea is to more or less combine EF with the legacy data access layer and slowly replace the legacy data layer with a more modern take on things using EF. So for now we need to use both EF and the legacy db access code.
What I have done so far is to add a project containing the edmx file and context. The edmx is generated using database first approach. I have also added another project that contains the POCO classes (by using ADO.NET POCO Entity Generator). I have more or less followed Julia Lerman's approach in her book "Programming Entity Framework" on how to split the model and the generated POCO classes. The database model has been set for years and it's not an option the change the table and the relationships, triggers, stored procedures, etc, so I'm basically stuck with the db model as it is.
I have read about the repository pattern and unit of work and I kind of like the patterns, but I struggle to implement them when I have both EF and the legacy db access code to deal with. Specially when I don't have the time to replace all of the legacy db access code with a pure EF implementation. In an perfect world I would start all over again with a fresh take one the data model, but that is not an option here.
Is the repository and unit of work patterns the way to go here? In order to use the POCO classes in my business layer, I sometimes need to use both EF and the legacy db code to populate my POCO classes. In another words, I can sometimes use EF to retrieve a part of the data I need and the use the old db access layer to retrieve the rest of the data and then map the data to my POCO classes. When I want to update some data I need to pick data from the POCO classes and use the legacy data access code to store the data in the database. So I need to map the data retrieved from the legacy data access layer to my POCO classes when I want to display the data in the UI and vice versa when I want to save data to the data base.
To complicate things we store some data in tables that we don't know the name of before runtime (Please don't ask me why:-) ). So in the old db access layer, we had to create sql statements on the fly where we inserted the table and column names based on information from other tables.
I also find that the relationships between the POCO classes are somewhat too data base centric. In another words, I feel that I need to have a more simplified domain model to work with. Perhaps I should create a domain model that fits the bill and then use the POCO classes as "DAO's" to populate the domain model classes?
How would you implement this using the Repository pattern and Unit of Work pattern? (if that is the way to go)

Alarm bells are ringing for me! We tried to do something similar a while ago (only with nHibernate not EF4). We had several problems running ADO.NET along side an ORM - database concurrency being a big one.
The database model has been set for
years and it's not an option the
change the table and the
relationships, triggers, stored
procedures, etc, so I'm basically
stuck with the db model as it is.
Yep. Same thing! The problem was that our stored procs contained a lot of business logic and weren't simple CRUD procs so keeping the ORM updated with the various updates performed by a stored procedure was not easy at all - Single Responsibility Principle - not a good one to break!
My plan is to replace the old db
access code with EF over time and get
rid for the tight coupling
between UI layers and data layer.
Maybe you could decouple without the need for an ORM - how about putting a service/facade layer infront of your UI layer to coordinate all interactions with the underlying domain and hide it from the UI.
If your database is 'king' and your app is highly data driven I think you will always be fighting an uphill battle implementing the patterns you mention.
Embrace ado.net for this project - use EF4 and DDD patterns on your next green field proj :)

EDMX + POCO class generator results in EFv4 code, not EFv4.1 code but you don't have to bother with these details. EFv4.1 offers just different API which does exactly the same (and it is only wrapper around EFv4 API).
Depending on the way how you use datasets you can reach some very hard problems. Datasets are representation of the change set pattern. They know what changes were done to data and they are able to store just these changes. EF entities know this only if they are attached to the context which loaded them from the database. Once you work with detached entities you must make a big effort to tell EF what has changed - especially when modifying relations (detached entities are common scenario in web applications and web services). For those purposes EF offers another template called Self-tracking entities but they have another problems and limitations (for example missing lazy loading, you cannot apply changes when entity with the same key is attached to the context, etc.).
EF also doesn't support several features used in datasets - for example unique keys and batch updates. It's fun that newer MS APIs usually solve some pains of previous APIs but in the same time provide much less features then previous APIs which introduces new pains.
Another problem can be with performance - EF is slower then direct data access with datasets and have higher memory consumption (and yes there are some memory leaks reported).
You can forget about using EF for accessing tables which you don't know at design time. EF doesn't allow any dynamic behavior. Table names and the type of database server are fixed in mapping. Another problems can be with the way how you use triggers - ORM tools don't like triggers and EF has limited features when working with database computed values (possibility to fill value in the database or in the application is disjunctive).
The way of filling POCOs from EF + Datasets sounds like this will not be possible when using only EF. EF has some allowed mapping patterns but possibilities to map several tables to single POCO class are extremely limited and constrained (if you want to have these tables editable). If you mean just loading one entity from EF and another entity from data adapter and just make reference between them you should be OK - in this scenario repository sounds like reasonable pattern because the purpose of the repository is exactly this: load or persist data. Unit of work can be also usable because you will most probably want to reuse single database connection between EF and data adapters to avoid distributed transaction during saving changes. UoW will be the place responsible for handling this connection.
EF mapping is related to database design - you can introduce some object oriented modifications but still EF is closely dependent on the database. If you want to use some advanced domain model you will probably need separate domain classes filled from EF and datasets. Again it will be responsibility of repository to hide these details.

From how much we have implemented, I have learned following things.
POCO and Self Tracking objects are difficult to deal with, as if you do not have easy understanding of what goes inside, there will be number of unexpected behavior which may have worked well in your previous project.
Changing pattern is not easy, so far we have been managing simple CRUD without unit of work and identity map pattern. Now lot of legacy code that we wrote in past does not consider these new patterns and the logic will not work correctly.
In our previous code, we were simply using transactions and single insert/update/delete statement that was directly sent to database assuming transactions on server side will take care of all operations.
In such conditions, we were directly dealing with IDs all the time, newly generated IDs were immediately available after single insert statement, however this is not case with EF.
In EF, we are not dealing with IDs, we are dealing with navigation properties, which is a huge change from earlier ADO.NET programming methods.
From our experience we found that only replacing EF with earlier data access code will result in chaos. But EF + RIA Services offer you a completely new solution where you will probably get everything you need and your UI will very easily bind to it. So if you are thinking about complete rewriting using UI + RIA Services + EF, then it is worth, because lot of dependency in query management reduces automatically. You will be focusing only on business logic, but this is a big decision and the amount of man hours required in complete rewriting or just replacing EF is almost same.
So we went UI + RIA Services + EF way, and we started replacing one one module. Mostly EF will easily co-exist with your existing infrastructure so there is no harm.

'Uncomplete' POCO on client - Can I fill up the collections later?

I have a setup with Client -> WCF -> POCO -> EF4.
Say I have a list with A entities. The A entity contain among other properties a huge list of B entities that isn't loaded by default. When a certain action is done on the client, it may need to know the list of B entities...
If I load the B entities for the A entity and attach them to the collection, the A entity is in effect changed, and I guess when saving the entity it will also save these 'new' B entities to the A entity?
I could wire up a GetEntityWithAllDetails function, but then I would get some data that I already have, and if there were other collections I didn't want to load, it would be a complete mess.
The question can be boiled down to how can I recomplete the POCO on the client side when I only have a partial POCO to start with and want to avoid loading data twice and still being able to rely on EF4 to save the entity correctly?

That is a complex task and EF doesn't handle it - it is your responsibility. When you use detached entities the change tracking is up to you.
Your solution currently probably is:
Client sends request to WCF service
WCF uses EF to get data, close context and return POCO graph or partial graph back to client
Client modifies the POCO graph / partial graph and sends modified data back to WCF service
WCF creates new EF context and saves the POCO graph
Sounds easy but it doesn't. In the last step you must manually explain to the new context what has changed. It generally means heavy interaction with ObjectStateManager (in case of ObjectContext API) or DbChangeTracker (in case of DbContext API). It also means that you must pass information about changes from the client.
For example suppose that you are modifing Order entity. Order entity is dependent on Customer entity and it has dependent OrderItem entities. To make this interesting suppose that OrderItems must be processed by different warehouses so each warehouse has access only items assigned to it.
In the step one you will request Order from one warehouse
In the step two you will retireve Order without Customer and with a supset of OrderItems.
In the step three the warehouse modifies sevaral OrderItems as processed. Deletes single OrderItem because of discontinued product and inserts another OrderItem for replacement of discontinued product. Because of insufficient supplies some items will be unchanged. The warehouse sends Order back to the server.
What will you do in the step four? You must apply some knowledge. The first knowledge is that cutomer was not send to client so you can't modify a customer relation. In case of foreign key relation it means that CustomerId can't be modified. Now you must explicitly say which OrderItem was updated (= exists in DB), which was unchanged (= doesn't need any action), which was inserted (= must be inserted) and the worst part which was deleted (if you don't send some information about deletion from the client you can't know it without reloding the entity graph from the database).
At least you can be happy that EF will not delete anything you explicitly don't mark for deletion. So order items related to other warehouses and their relations to the order will be unchanged.
There are two general approaches how to deal with it:
Load entity graph first and merge changes into the graph. Then save the attached (loaded) graph. You will simply compare the loaded entity graph with the received entity graph and process all required updates, deletes, inserts.
Use self tracking entities instead of POCOs which are implementations of Change set pattern and are able to track changes on the client. STEs have some disadvantages which make them useless in certain scenarios.
There is also completely separate architecture approach using DTOs instead of direct EF POCOs but it results in same complications as you have at the moment.

Welcome to n-tier development.
This sort of situation is exactly why many architected enterprise scale solutions use data transfer objects between tiers.
I would recommend avoiding domain entity propagation from the service (business) tier to the client. If you go down the track of having entities become aware of whether they are fully loaded, or what tier they are currently on, they are hardly "POCO" are they?
So you write a service method "GetEntityWithAllDetails". It should take a GetEntityWithAllDetailsRequest object and return a GetEntityWithAllDetailsResponse object containing whatever the caller of the service expects, and no more.
Obviously there is a far bit of mapping to be done between between DTO's and domain objects - libraries such as Automapper (and others) can help with that.
Propagating domain entities to the client also retricts your flexibiltiy with regards to lazy or eager loading of entities and leaves you having to deal with re-attaching/merging entities, which is problem with EF becuase it will not re-attach entity graphs - you must walk the graph manually.
I will try and say it really plainly. Propagating domain entities from the service to the client is the road to programming hell and very quickly leads to objects having all sorts of responsibilties that are orthoganol to their purpose.

EF4 Custom Entities within the same edmx

I have an edmx file that reflects 100% my DB schema.
Along with this I need to create some custom Entities based on the existent, pretty much like a view that will aggregate fields of several entites into a single one.
The problem is that this isn't quite working. I tried several approaches but it always gave me conflicts with the actual entites already on the edmx.
I need to have those entities that reflect my DB schema, so do I have to create another edmx file to hold my custom entites and avoid colisions?
I also though of:
create a stored procedure but then if
I need to filter the SP result I
eather have to add support for serach
on the SP of get all the rows and
filter with Linq2Objects... won't do
this!
create a View, and this one would
work pretty well but I want to try
to do this making use of the EF4
designer and keep everything in one
place.
Could anyone point me to some examples?

I think what you are describing is a view so this is probably the right way to go.
You can store the view code in the edmx using the DefiningQuery node. You don't need to create the view in the database. However there is no designer support for this feature, you will need to hand edit the edmx. The changes you make should be persisted if you refresh the edmx from the database using the designer.
Details here:
http://msdn.microsoft.com/en-us/library/cc982038.aspx
http://blogs.msdn.com/b/davidebb/archive/2010/10/01/the-easy-way-to-create-an-entity-framework-defining-query.aspx
It would be easier just to create the views in the database, and let the designer find and model them.
This entity will be read-only, but of course you can then assign SPs for UPDATE/INSERT/DELETE if you want to support modifications via this view.

You can't define two entities based on same table except special cases (table splitting, hiearchy mapping). In this case you have to use DefiningQuery as #James suggested or QueryView. The difference is that DefiningQuery is defined in storage model and it is common SQL. QueryView is defined in conceptual model and it is ESQL defined on top of already existing entities. QueryView supports only some features of ESQL (for example it doesn't support aggregate functions). In both cases you have to modify EDMX directly (XML), these features are not supported in designer.
Database view mentioned by #James is also an option if you don't want to use these advanced EF features. You can as well simply expose predefined queries on your object context and map return projection to custom type.
Be aware that neither of these methods will allow you to modify, insert or delete data.

Linq to entities and business logic

Whenever I read an article on linq to entities, I sounds like I should embed my business logic in the generated entity-classes.
This would means that the "user" of the business-objects (controller, service-layer, ...) would have to know about the datacontext-object that's needed to use linq.
It would also mean that the DAL-logic and the business-logic will be mixed up.
A lot of microsoft examples use some kind of DTO-approach. I'm not a big fan of the DTO pattern.
So should I let the business-object encapsulate the linq-entity and provide access to it using properties, or should I stick with the DTO pattern?
What are your suggestions?
Thanks

The entity model generates partial classes. In a project of mine I'm using an entity model inside a class library and added a few .cs files to it that add functionality to the default entity classes. (Mostly functions related to error and message logging to a database table and a method to import/export all data to XML.)
But true business logic is located in a second class library that refers to this entity class library.
Let's explain. First I created an entity model from a database. It contains a list of company names and addresses. I do this by choosing "New project|Class library" and then add an ADO.NET Entity Data Model to this library. The Entity model will be linked to my SQL Server database, importing the tables I need and auto-creates classes for the tables I want to access.
I then add a second .cs file for every table that I want to expand. This would be a few raw methods that are strongly linked to the database. (Mostly import/export methods and error logging.) This I will call Entity.Model, which will compile to Entity.Model.dll.
Then I add a second project which will contain the business logic. Again, I use "New project|Class library" to create it and add Entity.Model.dll to it's references. I then start writing classes which will translate the database-specific classes to more logical classes. In general, I would not have to make many changes except that I will protect or hide certain fields and add a few calculated fields. The business logic would only expose the functionality that I want to access from my client application and not a single method more. Thus, if I don't allow the client to delete companies then the "Delete" function in the entity model will not be exposed in the business layer. And maybe I want to send a notification when a company changes their address, so I'll add an event that gets triggered when the address field of the company is changed. (Which will write a message log or whatever.) I'll call this business.logic and it compiles to Business.Logic.dll.
Finally, I'll create the client application and will add a reference to Business.Logic.dll. (But NOT to the entity model.) I can now start to write my application and access the database through the business layer. The business layer will do validations, trigger a few events and do whatever else in addition to modifying the database through the entity model. And the entity model just serves to keep the database relations simple, allowing me to "walk through" the data through all foreign links in the database.

I wouldn't edit the generated files as they are likely to change.
What you could do is wrap them some query object and pass that around.
ayende makes a very good point about where the DAL should really live
Also, you should be a fan of viewmodels/dtos ;)

I prefer to wrap calls to the entity classes in business classes. (Call it BC for short.) Each BC has several constructors, one or more of which allows the context to be passed to it. This allows one BC to call another BC and load related data in the same context. This also makes working with collections of BCs that need a shared context simpler.
In the constructors that don't take a context, I create a new context in the constructor itself. I make the context available through a read-only property that other BCs can access.
I write using the MVVM pattern. The nice thing about this approach is that so far I have been able to write my viewmodels without ever refering to the data context, just the BCs that form my models. Hence if I need to modify my data access (replace entity framework, or upgrade to version 4 when it is out of beta) my views and viewmodels are isolated from the changes required in the BCs.
Not sure this is the best approach, but so far I have liked the results. It does take tweaking to get the BCs designed correctly, but I end up with a flexible, extensible, and maintainable set of classes.