We Keep Coding

Seperation of models in a web api application

My team devolops a web api application using entity framework,
The Gui is developed by a seperate team.
My question is how should the models be defined? Should we have two projects - one for domain models (database entities) and one for Dtos which are serializable?
Where should the parsing from Dto to domain models should happen and when should it happen the opposite way?
Moreover, sometimes all the data is needed to be sent to the clients.. Should a Dto be created for those cases as well? Or should I return a domain model?

Generally speaking, it's a good idea to not let your entities (database models) leak out of your database layer. However, as with everything in software - this can have its downfalls. One such downfall being is that it starts to increase complexity of your data layer as it involves mapping your entities to their DTO within your database layer, ultimately leaving repositories that are full of similar methods returning different DTO types.
Some people also feel that exposing IQueryables from your data layer is also a bad thing as you start to leak abstractions to different layers - though this has always seemed a little extreme.
Personally, I favour what I feel is a more pragmatic approach and I prefer to use a tool like AutoMapper to automatically map my entities to my DTOs within the business logic layer.
For example:
// Initial configuration loaded on start up of application and cached by AutoMapper
AutoMapper.Mapper.CreateMap<BlogPostEntity, BlogPostDto>();
// Usage
BlogPostDto blogPostDto = AutoMapper.Mapper.Map<BlogPostDto>(blogPostEntity);
AutoMapper also has the ability to configure more complex mapping, though you should try and avoid this if possible by sticking to flatter DTOs.
In addition, another great feature of AutoMapper is the ability to automatically project your entities to DTOs. This results in much cleaner SQL where only the columns within your DTO are queried:
public IEnumerable<BlogPostDto> GetRecentPosts()
{
IEnumerable<BlogPostDto> blogPosts = this.blogRepository.FindAll().Project(this.mappingEngine).To<BlogPostDto>().ToList();
return blogPosts;
}
Moreover, sometimes all the data is needed to be sent to the clients.. Should a Dto be created for those cases as well? Or should I return a domain model?
DTOs should be created for those. Ultimately you don't want your client depending on your data schema, which is exactly what will happen if you expose your entities.
Alternatives: Command/Query Segregation
It behooves me to also highlight that there are also some other alternatives to a typical layered architecture, such as the Command/Query Segregation approach where you model your commands and queries via a mediator. I won't go into it in too much detail as it's a whole other subject but it's one I would definitely favour over a layered approach discussed above. This would result in you mapping your entities to your DTOs directly within the modelled command or query.
I would recommend taking a look at Mediatr for this. The author, Jimmy Bogard who also created AutoMapper also has this video talking about the same subject.

I've had similar requirements in several projects and in most cases we separated at least three layers:
Database Layer
The database objects are simple one-to-one representations of the database tables. Nothing else.
Domain Layer
The domain layer defines entity objects which represent a complete business object. In our defintion an entity aggregates all data which is directly associated to the entity and can not be regarded as a dedicated entity.
An exmaple: In an application which handles invoices you have a table invoice and invoice_items. The business logic reads both tables and combines the data into a entity object Invoice.
Application Layer
In the application layer we define models for all kind of data we want to send to the client. Pass-through of domain entity objects to save time is tempting but strictly prohibited. The risk to publish any data which shouldn't be published is too high. Furthermore you gain more freedom regarding the design of your API. That's what helps you to fit your last requirement (send all data to the client): Just built a new model which aggregates the data of all domain objects you need to send.
This is the minimum set of layers we use in all projects. There were hundreds of cases where we've been very happy to have several abstraction layers which gave us enough possibilities to enhance and scale an application.

Handling Mismatch Between Domain and Code First Entities

I am using Code First approach and there are some mismatch between my model for code first approach (DAL) and my domain model (BLL). I imagine my Data Model to have annotations, properties, configurations, etc related to database only and not the same for my Domain model entities and vice versa to obey separation of concerns.
How do I go about handling this situation in my application? This is more logical then technical I guess. Asked before in many places but no concrete lead yet. Hope some suggestion from SO will help.

In my experience because of the rich mapping possibilities of the Entity Framework you don't have to separate a Data access and Business logic layer at all, you just have to use the Fluent API of the Entity Framework. In one of my current project we have more than 150 classes with inheritance hierarchies and all but we can still use it without "duplicating" the objects.
Some good introductions about the fluent API can be found here:
http://msdn.microsoft.com/en-us/data/hh134698.aspx
http://msdn.microsoft.com/en-us/magazine/hh852588.aspx
About the separation: we simply use a Domain project and a Persistence.EntityFramework project where the latter contains all the mappings thus the Domain does not reference the EntityFramework.dll at all.
And if you have some specific mapping questions e.g. the ones you mentioned that are the reasons you created two layers one for DAL and the other for BL just ask them.

I would go with AutoMapper. It can help you to reduce the boilerplate code needed to convert from one object to another.
You can find it here:
https://github.com/AutoMapper/AutoMapper
Edit:
Put your domain models either in BLL or in a separate project, add reference to the BLL or this separate project in the DAL (also reference the new project in BLL), and use the AutoMapper in the DAL. So only domain models will leave the DAL.

You normally have:
A domain model (entities), which is what the O/RM (Entity Framework) uses;
A Data Transfer Objects (DTO) model, used for sending data to a view (in MVC) or by web services, etc.
I agree with Andras: the best way to go from one (domain) to the other (DTO) is by using Automapper. Of course, you can also do it by hand.
One thing that you need to realize is that there is no need for a 1-1 mapping between the domain and the DTO, the DTO can contain denormalized or calculated properties as well.

Data Layer Architecture with multiple datasources

I am trying to create a system that allows you to switch multiple data sources, e.g. switching from Entity Framework to Dapper. I am trying to find the best approach to do this.
At the moment I have different projects for different data layers, e.g. Data.EF for Entity Framework, Data.Dapper for Dapper. I have used a database approach but when it creates the models the information generated is coupled together and not easy to refactor, e.g. separation of models.
I have a project called models, this holds domain and view models, and I was thinking of creating Data.Core and follow the repository pattern. But then, doing this will add an extra layer so I would have Presentation / Business / Repository / Data.
I would like to know the best structure for this approach. Should I also do a code-first approach to create my database? This helps separate concerns and improve abstraction. This is quite a big application so getting the structure right is essential.

I'd suggest factoring your data interfaces either to the model through repository interfaces for your entities or to an infrastructure project. (I think the latter was your rationale behind creating a Data.Core project.)
Each data source will then implement the very same set of interfaces, and you can easily switch between them, even dynamically using dependency injection.
For instance, using repositories:
Model
\_ Entities
Entity
\_ Repositories
IEntityRepository
Data.EF
EntityRepository : Model.IEntityRepository
Data.Dapper
EntityRepository : Model.IEntityRepository
Then in your business you won't need to even reference Data.EF or Data.Dapper: you can work with IEntityRepository and have that reference injected dynamically.

I think you approach is correct. I'd say Presentation / business / repository / data is pretty standard these days.
I'd say the code first approach using POCOs is the preferred option today in the industry. I would advise to start creating a project containing your POCO data structures with any logic in it and take it from there. The advantage of this is that your objects model the domain more naturally. If you start with a db centric approach the problem is that, if you are not careful, you may end with objects more akin to SQL relational databases than to the real model. This was painfully evident in the first versions of .net where it was encouraged to use Datasets tighly coupled with the db and that often caused problems to work with in the business layer.
If needed you can do any complex mapping between the business objects and the db objects in the repository layer. You can use a proxy and/or a unit of work if you need to.

I would suggest you create your domain objects, use the code-first approach and also apply the repository pattern

Yes the repository pattern does bring in an extra layer. Have a look at this post for more detail information Difference between Repository and Service Layer?
RE: code-first approach to create my database
It doesn't matter how big your application is, it is a question of what else you intend to use the database for. If this database is simply a repository for this application then using code-first is fine as you are simply storing your code objects. However if you are using this database as an integration point between applications then you may wish to design the database seperately to the application models.

Dal (with Entity Framework) and Model layers into MVC

First of all , I use EF into Dal layer (separeted project from MVC, same solution). The generated models from EF's EDMX file are the actual models from Model layer?? If so, how do I access these models to work in MVC's View layer? I think it's wrong accessing data layer directly from view to work with these models, and If I make a Model layer with "my models" and translate Dal's models into mine models... it'll be duplicated code.
Probably I'm getting something wrong, but most of egs. are with code first approach, and I can't figure out this.

You are right in your idea of not accessing access the Models in your DAL directly from your presentation layer.
To avoid duplicating code when translating your DAL objects into the Models used by your views, you could use something like AutoMapper, which is supposed to do the heavylifting for you in exactly that scenario.

I think it's wrong accessing data layer directly from view to work with these models...
That's right, the appropriate method is using View Model
When you have dozens of distinct values to pass to a view, the same flexibility that allows you to
quickly add a new entry, or rename an existing one, becomes your worst enemy .You are left on your
own to track item names and values; you get no help from Microsoft IntelliSense and compilers .
The only proven way to deal with complexity in software is through appropriate design. So defining an object model for each view helps you track what that view really needs. I suggest you define a
view-model class for each view you add to the application.
-- "Programming Microsoft ASP.NET MVC" by Dino Esposito
A ViewModel provides all information that your view requires to make itself. To transfer data from ViewModel and a business entity you can use AutoMapper.
Don't worry about duplication, those are two different concept and should be separated from each other; it makes your application easy to maintain.

I may be mistaking but to me, using generation from EDMX provides you with the DbContext which could be considered as the DAL and entities which could be considered as the Model.
So you might directly manipulate entity instances as your business object. Manipulation of the base through the DbContext should appear in the BLL layer. Additionally, you might implement DTOs where needed.
This, of course, assumes you want to use entity framework code generation. Other options like using POCOs might be more relevant considering your overall architecture.

I use a view model in my local project and the models in the other project. Then put references to the models im gonna use on the page in my view model. Then reference the view model on my page. Let me know if that sounds like something you want to do and I can edit in some code.

Entity Framework and 3 layer architecture

I have a three layer architecture program. The questions are:
1. Data access is the layer of EF?
2. If i want to use an entity generated by EF from Presentation Layer, then i reference the Data Access, but this violates the principles of 3 layered architecture.

Microsoft Spain released a pretty good documentation, guide and sample application for N-layered applications on codeplex, you can look it up here:
http://microsoftnlayerapp.codeplex.com/
You will find many directions and helpful implementation patterns there.
hth.

Yes EF would be your Data Access Layer.
With EF you can use T4 templates with POCO support, you can then extract these POCO into a seperate dll and this will be reference from all of your layers.

What type of application are you building? If you are building an ASP.NET MVC 3 application, you can have your View be the presentation layer, your Model is your data access (which can use EF) and the controller and / or Action Filters can contain your business logic and in this scenario you will be using your EF Model in the presentation layer but still satisfy the separation of concerns principle.

EF does two things: -
1) Generates an domain model for you (optional, but commonly used)
2) Gives you the ability to query / modify your database via that domain model.
This can give the appearance of blurring the lines between domain model and data access but the two are indeed separate.
As long as you're not doing stuff like creating object contexts and writing queries directly in your presentation tierthen IMHO you are not breaking abstraction - the only thing you are "breakin"g is the fact that you will need to reference System.Data.Objects (or whatever the EF dll is) in your presentation project(s) (which is just a physical artifact) unless you go down the route suggested by Jethro to generate your domain model into a separate project.

For the three tier architecture. I would consider doing Abstraction using Domain Model and Data model pattern rather then doing direct EF from Presentation Layer.
So the idea is that you have your Data Model which has EF POCO classes with Repositories which knows how to access these Classes for various CRUDs.
Your Domain Model would have models related to your Client (so you can put various ViewModels or Validation related code), It can be a WPF or MVC web app.
Now between these two there is a business which talks to both Domain and Data models.
Your Presentation Layer does know nothing about the EF/Data Layer/Repository. When you want to introduce new Data Framework or database, you just need to write new repository classes and data models classes (which prob. be with some sort of code gen).
This also allows your code to be Unit testable as well.