We Keep Coding

Seperation of models in a web api application

My team devolops a web api application using entity framework,
The Gui is developed by a seperate team.
My question is how should the models be defined? Should we have two projects - one for domain models (database entities) and one for Dtos which are serializable?
Where should the parsing from Dto to domain models should happen and when should it happen the opposite way?
Moreover, sometimes all the data is needed to be sent to the clients.. Should a Dto be created for those cases as well? Or should I return a domain model?

Generally speaking, it's a good idea to not let your entities (database models) leak out of your database layer. However, as with everything in software - this can have its downfalls. One such downfall being is that it starts to increase complexity of your data layer as it involves mapping your entities to their DTO within your database layer, ultimately leaving repositories that are full of similar methods returning different DTO types.
Some people also feel that exposing IQueryables from your data layer is also a bad thing as you start to leak abstractions to different layers - though this has always seemed a little extreme.
Personally, I favour what I feel is a more pragmatic approach and I prefer to use a tool like AutoMapper to automatically map my entities to my DTOs within the business logic layer.
For example:
// Initial configuration loaded on start up of application and cached by AutoMapper
AutoMapper.Mapper.CreateMap<BlogPostEntity, BlogPostDto>();
// Usage
BlogPostDto blogPostDto = AutoMapper.Mapper.Map<BlogPostDto>(blogPostEntity);
AutoMapper also has the ability to configure more complex mapping, though you should try and avoid this if possible by sticking to flatter DTOs.
In addition, another great feature of AutoMapper is the ability to automatically project your entities to DTOs. This results in much cleaner SQL where only the columns within your DTO are queried:
public IEnumerable<BlogPostDto> GetRecentPosts()
{
IEnumerable<BlogPostDto> blogPosts = this.blogRepository.FindAll().Project(this.mappingEngine).To<BlogPostDto>().ToList();
return blogPosts;
}
Moreover, sometimes all the data is needed to be sent to the clients.. Should a Dto be created for those cases as well? Or should I return a domain model?
DTOs should be created for those. Ultimately you don't want your client depending on your data schema, which is exactly what will happen if you expose your entities.
Alternatives: Command/Query Segregation
It behooves me to also highlight that there are also some other alternatives to a typical layered architecture, such as the Command/Query Segregation approach where you model your commands and queries via a mediator. I won't go into it in too much detail as it's a whole other subject but it's one I would definitely favour over a layered approach discussed above. This would result in you mapping your entities to your DTOs directly within the modelled command or query.
I would recommend taking a look at Mediatr for this. The author, Jimmy Bogard who also created AutoMapper also has this video talking about the same subject.

I've had similar requirements in several projects and in most cases we separated at least three layers:
Database Layer
The database objects are simple one-to-one representations of the database tables. Nothing else.
Domain Layer
The domain layer defines entity objects which represent a complete business object. In our defintion an entity aggregates all data which is directly associated to the entity and can not be regarded as a dedicated entity.
An exmaple: In an application which handles invoices you have a table invoice and invoice_items. The business logic reads both tables and combines the data into a entity object Invoice.
Application Layer
In the application layer we define models for all kind of data we want to send to the client. Pass-through of domain entity objects to save time is tempting but strictly prohibited. The risk to publish any data which shouldn't be published is too high. Furthermore you gain more freedom regarding the design of your API. That's what helps you to fit your last requirement (send all data to the client): Just built a new model which aggregates the data of all domain objects you need to send.
This is the minimum set of layers we use in all projects. There were hundreds of cases where we've been very happy to have several abstraction layers which gave us enough possibilities to enhance and scale an application.

What should be CRUD?

I read many documentation about CRUD and I still don't understand what exactly should be CRUDable! It seems most of people are talking about CRUD entities but they architecture doesn't show any Create, Read, Update or Delete methods in their entities. They implements these CRUD operations in a separate class. I like to call these kind of classes CRUD controllers.
Is it correct to create POCO entites with CRUD controller? What should be CRUD?

My take is that you should have a repository which performs the CRUD operations.
Then a controller should call the appropriate CRUD method in the repository, possibly via an intermediate service layer.
Read more about the repository pattern here and here.

These classes are usually called repositories. A repository provides access to your entities with means of adding, updating, removing and retrieving one or more entities. So the repository Creates, Reads, Updates and Deletes (CRUD).
When using a database your POCO is normally an object your database entity is converted into, e.g. with AutoMapper in the repository.

In architectural terms, CRUD means that you have entities without business rules. This entities, have, at most, some simple validations. When you have this kind of entity you speak of CRUD because you can modify the data without worrying about anything else (but validations). This can be used for example for maintaining a list of contacts: name + phone no. + address: at most you can validate that the name is not empty, the phone no. is valid, and the address is valid. But there are no business rules in there.
If there are involved business rules, you should avoid using CRUD to make sure that the business rules are respected. For example you should not allow CRUD for an order detail, because there are business rules involved: perhaps you cannot change an order detail if the order is already paid or sent, or confirmed to the customer. Besides the total order amount depends on the order details. In this case you should use the order with its details as a whole, and read / write / update it all at once. (In DDD this is called "aggregate").
Speaking about CRUD is not a question of how you implement it (repository, ORM like DbContext or NHibernate, or wichever you want to use), but a more philosophical question.
Implementing CRUD is much faster than implementing any other architecture which involves business rules (for example DDD). If you can use CRUD for an entity, is advisable to use it, but not in the other cases.
As to your comment:
but they architecture doesn't show any Create, Read, Update or Delete methods in their entities
That's natural... you can do CRUD with EF for example, without explicitly declaring the CRUD methods. Create an entity in the context, or remove it or modify it and the CRUD operations will be implicitly executed on the SaveChanges.

Confusion about where to put business logic when using Entity framework

I've just started working with the Entity framework and I'm confused about how the classes normally in the business layer fit in with the entities that are created by the Entity Framework.
When working with classic ADO.NET, I would have a class called Customer for example and then another class called DALCustomer to handle database interaction, in this structure I would have put the code to do calculations, filtering and delcare an instance of the DAL withing Customer for saving, updating and deleting in the Customer class.
With the Entity Framework, if you have a table called Customer, the Entity framework creates an entity called Customer and this is where my confusion begins, does this entity remove the need for a Customer in the business layer? So in essence all the fields and methods that normally go in the business layer go in the entity generated by the Entity Framework? Or should a class still exist in the business layer called CustomerBL for example, that still contains the fields and methods needed to accomplish the business logic required for calculations, filtering and still needs an instance of the EF DAL declared to handle data access?
If there should be a business class, in this case CustomerBL, one other question jumps to mind, should the fields that are created in the customer entity be recreated in CustomerBL or should an instance of the Customer entity be declared in CustomerBL so there would be no need to have the fields declared in 2 locations?

Entity framework was designed with separation between data model and conceptual model in mind. It supports inheritance, entity splitting (not EF core), table splitting, complex types or owned types, and transparent many-to-many associations (not EF core), all of which allow molding the domain model to one's needs without being constrained too much by the data store model. EF core supports shadow properties which can be used to hide cross-cutting concerns from the exposed class model.
The code-first approach allows working with POCOs in which only a few properties are mapped to data store columns and others serve domain goals. Model-first and Database-first generate partial classes, allowing one to extend the generated code.
Of course this separation of conceptual model and store model can only succeed to a certain extent. Some things work against this goal of persistence ignorance. For instance -
If lazy loading is desirable, it is necessary to declare navigation properties as virtual, so EF can override them in proxy types. Domain-driven design (DDD) would encourage using virtual only when polymorphism is required.
It is very convenient to have primitive foreign key properties (say, ParentId) accompanying the "real" associations (a Parent reference). Purists consider this a violation of DDD principles.
The EF class model will is part of a data access layer and should primarily serve that goal. Therefore, it will contain many reciprocal relationships, in order to benefit from navigation properties as much as possible when writing LINQ queries. These mutual relationships are another violation of DDD principles.
There is a large number of differences between LINQ-to-objects and LINQ-to-entities. You just can't ignore the fact that you are LINQ-ing against a totally different universe than objects in memory. This is referred to as tight coupling, or leaky abstraction.
EF can only map concrete classes, no interfaces.
But then... generally I'm happy with using generated EF classes or POCOs from a code-first model as domain classes. So far, I've never seen a frictionless transition from one data store or ORM to another, if it happens at all. Persistence ignorance is a fiction. Idiosyncrasies from the DAL easily leave a footprint in the domain. Only when you have to code for different data stores/models or when stores/models are expected to change relatively often it pays off to minimize this footprint as much as possible or abstract it away completely.
Another factor that may promote EF classes as domain classes is that many applications today have multiple tiers, where (serialized) different view models or DTOs are sent to a client. Using domain classes in UIs hardly ever fits the bill. You may as well use the EF class model as the domain and have services return dedicated models and DTOs as required by a UI or service consumers. Another abstraction layer may be more of a burden than a blessing, if only performance-wise.

In my opinion the whole point of using POCOs as entities that can be persisted is to remove the distinction between "database entities" and "business entities". "Entities" are supposed to be "business entities" that directly can be persisted to and loaded from a data store and therefore act as "database entities" at the same time. By using POCOs the business entities are decoupled from the specific mechanism to interact with a database.
You can move the entities into a separate project - for example - that has no references to any EF assembly and yet use them in a database layer project to manage persistence.
This does not mean that you can design your business entities completely without having the requirements for EF in mind. There are limitations you need to know to avoid trouble when you come to the point to map the business entities to a database schema using EF, for instance:
You must make navigation properties (references or collections of references to other entities) virtual to support lazy loading with EF
You cannot use IEnumerable<T> for collections that have to be persisted. It must be ICollection<T> or a more derived type.
It's not easy to persist private properties
The type char is not supported by EF and you can't use it if you want to persist its values
and more...
But an additional set of entities is - in my opinion - an additional layer of complexity that should be justified to be really needed if the mentioned limitations are too tight for your project.
YA2C (Yet another 2 cents :))

I don't know if it's considered a good practice by others but personally this is how i handled this in the past:
The classes generated by EF are your DAL, and then for BL create a complementary set of classes in which you will have the structure you require (like maybe merging data from related entities in a one to one relationship) and other business logic concerns are handled (custom validation like implementing IDataErrorInfo to make it play nice with the UI in WPF for instance) and also create classes that would contain all the business layer methods relating to a type of entity, that use the BL instances and convert to and from EF entities to the BL objects.
So, for instance, you have Customer in your db. EF will generate a class Customer, and in the BL there will be a Customer (prefix, suffix, etc.) class and a CustomerLogic class. In the BL Customer class you can do whatever is needed to satisfy requirements without having to tamper with the EF entities and in the CustomerLogic class you would have BL methods (load most valued customers, save customer with extra data, etc.).
Now, this enables you to be loosely coupled to the datasource implementation. Another example of why this has benefited me in the past (in a WPF project) is that you can do stuff like implement IDataErrorInfo and implement validation logic in the CustomerBL classes so that when you bind the entity to a create/edit form on the UI you will benefit from the built in functionality provided by WPF.
...My 2 cents, i am also curious to find out what is the best practice or what other solutions/points of view are.
Also perhaps related to this topic - Code-first vs Model/Database-first

This topic maybe a bit old but this may help. Andras Nemes pointed out in his blog the concern of using DDD (domain driven design) over the technology driven design such as EF, MVC, etc.
http://dotnetcodr.com/2013/09/12/a-model-net-web-service-based-on-domain-driven-design-part-1-introduction/

I used the business logic to write my methods and return the results in its created view like:
namespace Template.BusinessLogic
{
public interface IApplicantBusiness
{
List<Template.Model.ApplicantView> GetAllApplicants();
void InsertApplicant(Template.Model.ApplicantView applicant);
}
}

'Uncomplete' POCO on client - Can I fill up the collections later?

I have a setup with Client -> WCF -> POCO -> EF4.
Say I have a list with A entities. The A entity contain among other properties a huge list of B entities that isn't loaded by default. When a certain action is done on the client, it may need to know the list of B entities...
If I load the B entities for the A entity and attach them to the collection, the A entity is in effect changed, and I guess when saving the entity it will also save these 'new' B entities to the A entity?
I could wire up a GetEntityWithAllDetails function, but then I would get some data that I already have, and if there were other collections I didn't want to load, it would be a complete mess.
The question can be boiled down to how can I recomplete the POCO on the client side when I only have a partial POCO to start with and want to avoid loading data twice and still being able to rely on EF4 to save the entity correctly?

That is a complex task and EF doesn't handle it - it is your responsibility. When you use detached entities the change tracking is up to you.
Your solution currently probably is:
Client sends request to WCF service
WCF uses EF to get data, close context and return POCO graph or partial graph back to client
Client modifies the POCO graph / partial graph and sends modified data back to WCF service
WCF creates new EF context and saves the POCO graph
Sounds easy but it doesn't. In the last step you must manually explain to the new context what has changed. It generally means heavy interaction with ObjectStateManager (in case of ObjectContext API) or DbChangeTracker (in case of DbContext API). It also means that you must pass information about changes from the client.
For example suppose that you are modifing Order entity. Order entity is dependent on Customer entity and it has dependent OrderItem entities. To make this interesting suppose that OrderItems must be processed by different warehouses so each warehouse has access only items assigned to it.
In the step one you will request Order from one warehouse
In the step two you will retireve Order without Customer and with a supset of OrderItems.
In the step three the warehouse modifies sevaral OrderItems as processed. Deletes single OrderItem because of discontinued product and inserts another OrderItem for replacement of discontinued product. Because of insufficient supplies some items will be unchanged. The warehouse sends Order back to the server.
What will you do in the step four? You must apply some knowledge. The first knowledge is that cutomer was not send to client so you can't modify a customer relation. In case of foreign key relation it means that CustomerId can't be modified. Now you must explicitly say which OrderItem was updated (= exists in DB), which was unchanged (= doesn't need any action), which was inserted (= must be inserted) and the worst part which was deleted (if you don't send some information about deletion from the client you can't know it without reloding the entity graph from the database).
At least you can be happy that EF will not delete anything you explicitly don't mark for deletion. So order items related to other warehouses and their relations to the order will be unchanged.
There are two general approaches how to deal with it:
Load entity graph first and merge changes into the graph. Then save the attached (loaded) graph. You will simply compare the loaded entity graph with the received entity graph and process all required updates, deletes, inserts.
Use self tracking entities instead of POCOs which are implementations of Change set pattern and are able to track changes on the client. STEs have some disadvantages which make them useless in certain scenarios.
There is also completely separate architecture approach using DTOs instead of direct EF POCOs but it results in same complications as you have at the moment.

Welcome to n-tier development.
This sort of situation is exactly why many architected enterprise scale solutions use data transfer objects between tiers.
I would recommend avoiding domain entity propagation from the service (business) tier to the client. If you go down the track of having entities become aware of whether they are fully loaded, or what tier they are currently on, they are hardly "POCO" are they?
So you write a service method "GetEntityWithAllDetails". It should take a GetEntityWithAllDetailsRequest object and return a GetEntityWithAllDetailsResponse object containing whatever the caller of the service expects, and no more.
Obviously there is a far bit of mapping to be done between between DTO's and domain objects - libraries such as Automapper (and others) can help with that.
Propagating domain entities to the client also retricts your flexibiltiy with regards to lazy or eager loading of entities and leaves you having to deal with re-attaching/merging entities, which is problem with EF becuase it will not re-attach entity graphs - you must walk the graph manually.
I will try and say it really plainly. Propagating domain entities from the service to the client is the road to programming hell and very quickly leads to objects having all sorts of responsibilties that are orthoganol to their purpose.

Adding custom navigation properties using SPs in .NET Entity Framework 4

My company uses stored procs for all SELECT operations so it's making it rather difficult for me to create sensible navigation properties. I'm not too concerned at this point whether they're lazy loaded or not.
So for example I created an entity for Customer then created a FunctionImport to map GetAllCustomersSP to return a collection of Customer entities. But I want a navigation property "Orders" on each Customer entity.
But if I use the Customer entity partial class to just add this property, the problem is that I don't have access to the original Context, so I can't call the GetCustomerOrdersSP either explicitly or deferred.
The only option I can see is to modify my repository to add these properties in explicitly, which seems lame because it puts the entity logic into the repository.
Is there something I'm missing here? I can see in the entity model designer that I can specify custom insert, update, delete SPs but I don't see any way to use select SPs to actually retrieve the data.

I agree with Tim here...any solution you come up with isn't going to fully leverage the ORM and will be a potential nightmare to maintain. I would suggest creating a model, in code, that is framed in the manner in which you want to develop.
In the Data Access layer of your app, you can map your data objects that use SPs to hydrate your model objects (have a look at AutoMapper). Your app will only know about your model objects.
Doing this will give you a consistency in how you interact with the objects, and you can begin to apply pressure to the powers that be to allow more fine grained access to the tables, at which point, you can adjust your Data Access Layer to support EF and remove SPs. At this point you would be able to consider migrating the objects you created to POCO objects that are persisted via EF.
We had a similar issue in that granting raw DB access was "forbidden". We overcame this problem by using a model in which we only grant access to tables as they are used, not the entire database, and by ensuring the DBA that EF uses parameterized SQL, eliminating the concern of SQL Injection.