We Keep Coding

Is it appropriate for a Factory class to also include functionality of extracting data from a database

One of the main aspects of software development that I struggle with is delegating the correct responsibility into classes within my program. Working at my first junior role I'm also being exposed to a lot of different design patterns and ideas, and sometimes the information can be overwhelming.
Obviously when we are building software we tend to state that a class should be responsible for one thing and one thing only. It should do that thing well and nothing more. So in the case of the Factory pattern the Factory class should be responsible for building the product and exposing an interface that allows the director to extract the product from the factory.
However the factory class obviously needs to receive the data to build the product from somewhere, without input data we have no output product. Therefore I'd like to know whether including functionality for a factory to query a database is appropriate? My rationale for this is that if the factory is tasked with building a particular product, then it should also be responsible for retrieving the data required to build that product. But I'm not 100% sure if this is correct.
Alternatively, should there be a repository class who's responsibility is to retrieve the data in question from the database, which can be then passed to the factory for assembly into the required product? The use of a repository class seems a bit excessive in this case as we have a class that will hold a large number of different pieces of data which then must be shipped into the factory class.
If we also bear in mind Uncle Bob's teachings that state that functions and methods should have absolutely no more than three parameters then we will be breaking this rule by passing in a large amount of data to the factory. If we first assemble the data into an encompassing class before passing to the factory then we are essentially doing the factory's job within the repository class.
Some guidance would be really appreciated on this, as in my head the lines are very blurry and I'm not sure how I should proceed.

You shouldn't use the factory pattern to object building that extracted from a database. There are the Repository pattern and the Data Mapper pattern for this goal. Those patterns must encapsulate all logic of work with the data storage. Those patterns must have the following responsibility:
the Repository must give an interface to business logic for work with data storage
the Data Mapper must convert data from database to concrete object
The algorithm of cooperation between objects can look like:
business logic uses a repository to read/persist objects.
the repository uses a Data Mapper to convert objects to INSERT or UPDATE queries and to convert data from data storage to object
Also, you can read more details about the repository pattern in C# on the site of Microsoft and you can see C# example of the repository pattern

Use 2 different classes.
A Data Access Object (DAO) provides an abstract interface to the database, and hides its details.
A factory abstracts and hides the details of the creation of your objects. For example, for unit testing you might want to configure the factory so that it doesn't use the Database at all.
To reduce the number of parameters between the DAO and the factory, wrap your many pieces of data in a few logically related classes.

Is it apropriate for a Factory class to also include functionality of extracting data from a database
My rationale for this is that if the factory is tasked with building a particular product, then it should also be responsible for retrieving the data required to build that product. But I'm not 100% sure if this is correct.
A product to retrieve from database is not a trivial object, it is a domain model.
A domain model (aka business model, aka entity(which might indicate a particular instance of it)) belongs to your domain layer (aka business layer).
In this regard, there are some patterns you should be a minimum familiar with...
(Active Record) VS (Data Mapper + Repository) VS (Table Data Gateway + Factory)
Active record pattern kind of violate the Single Responsibility Principle by leading you to implement database access logic inside your domain model and tightly couples them.
Ideally, to avoid the cons above for the cost of a slightly increased complexity (on short term only), we separate the database access logic into a supplementary layer, the data access layer. One of the main component of this layer being the data mapper which in our context (READ operation) is in charge to retrieve data from database and map it to a new domain model instance, your specific product (entity). More generally it encapsulates CRUD operations to the database abstracting this database. Its API inputs and outputs are entity objects and possibly Query Objects.
Optionally, a featured data mapper would make use of patterns such as:
Unit Of Work
Lazy Loading
Identity Map
Transaction
Lock Strategies
Metadata Mapping
Alternatively, should there be a repository class who's responsibility is to retrieve the data in question from the database, which can be then passed to the factory for assembly into the required product? The use of a repository class seems a bit excessive in this case as we have a class that will hold a large number of different pieces of data which then must be shipped into the factory class.
Repository is not a part of your data access layer, but of your domain layer. So it is client of your data access layer. It doesn't encapsulate any database access logic but uses the data mapper.
A repository encapsulates query logic for a particular domain model plus a collection of in-memory entities you've previously retrieved.
A very basic example:
class ProductRepository
{
private $productCollection;
public function findById($id)
{
if (!$this->productCollection->has($id)) {
$product = $this->dataMapper->get(new Query(Product::class, $id));
$this->productCollection->add($product);
return $product;
}
return $this->productCollection->get($id);
}
}
Finally, we can encapsulate database access logic in a table data gateway and use it in a factory. This would result in a solution similar to Gonen I's one. It is simple to implement but there might be cons compared to the data mapper solution. I've never implemented, used or even studied this approach so I can't tell much...
You'd definitely learn a lot by attempting to implement all that by yourself and I'd encourage you to, but keep in mind that if you need a serious solution, ORMs might be interesting for you.
If you're keen to learn more about all this, I recommend Martin Fowler's Patterns of Enterprise Application Architecture book which is summarized here: https://www.martinfowler.com/eaaCatalog/index.html

Entities cannot be tracked by more than one context, interferes with N-tier design

I have a colleague who insists I develop ASP.Net MVC websites using an n tier data access layer and MVVM.
He has a background in Silverlight and WPF and I have attempted to create a solution but it is causing problems.
I created and DAL and logic layers (using generics):
Dal - containing a repository pattern that wraps up entity frame work.
Logic - pass through layer, no generic logic to apply currently.
I am still using the MVC pattern and am passing the Entity framework model to View unless the need additional properties or methods - in which case I create a view model and an interface to map between the two.
The n tier data access layer has locked the entity framework context at the bottom of the stack and the main problem I have that entities can't be tracked on more than one context. ({"An entity object cannot be referenced by multiple instances of IEntityChangeTracker."})
I recently came across a problem where context coming up with this error despite my attempts despite to use models, interfaces, deep copying.
The issue here is this was the approach I was going to take to try and put a model in the DAL layer and map the data between entity framework entity and model.
I understand the underlying issue: that even though you dispose of a context it doesn't release the entities that were attached to it. Is there anyway to get this n tire data access approach to work with MVC? or am right this will never work and I should stick to using the entity framework context in the controller method (or underlying class implemented using dependency injection).

There's a lot to unpack here.
If you are looking for an MVVM implementation, then you are going to want to look at something like knockout.js. Or some other framework that is going to do declarative databinding in the client side. there are a number of articles that you can read to understand this.
https://learn.microsoft.com/en-us/aspnet/core/client-side/knockout
but let's start with the assumption that you aren't going down that road and want to stick with MVC.
I would stop passing entities to your views. You are going to cause yourself endless grief. Create a view model for each corresponding view. you use of an interface seems like overkill. You can project linq queries for your entities/repos right into the the view models. you could also use something like Automapper to map from entity to view model. Appending some of your code to the questions might help. And my view models tend to be pretty flat. I don't see why you would need to perform deep copies of entity trees out to the view. Those can be gotten latter using partial views or some other method.
Now that you aren't sending entities out to the view, when you post information for updating, you are going to have to instatiate a new object for fetch the object you are looking to update from ef again. make your changes and save.
To me that simplest implementation is the best. I would pass you context to the controller method (preferably using an IoC) and querying the context directy in the controller action and projecting directly into the view model. If you've got a lot of shared code between controller actions, maybe move stuff into a service and then inject the service into the controller and inject the context into the service. but i'd start with just injecting the context into the controller.

Using View-Models with Repository pattern

I'm using Domain driven N-layered application architecture with EF code first in my recent project, I defined my Repository contracts, In Domain layer.
A basic contract to make other Repositories less verbose:
public interface IRepository<TEntity, in TKey> where TEntity : class
{
TEntity GetById(TKey id);
void Create(TEntity entity);
void Update(TEntity entity);
void Delete(TEntity entity);
}
And specialized Repositories per each Aggregation root, e.g:
public interface IOrderRepository : IRepository<Order, int>
{
IEnumerable<Order> FindAllOrders();
IEnumerable<Order> Find(string text);
//other methods that return Order aggregation root
}
As you see, all of these methods depend on Domain entities.
But in some cases, an application's UI, needs some data that isn't Entity, that data may made from two or more enteritis's data(View-Models), in these cases, I define the View-Models in Application layer, because they are closely depend on an Application's needs and not to the Domain.
So, I think I have 2 way's to show data as View-Models in the UI:
Leave the specialized Repository depends on Entities only, and map the results of Repositories's method to View-Models when I want to show to user(in Application Layer usually).
Add some methods to my specialized Repositories that return their results as View-Models directly, and use these returned values, in Application Layer and then UI(these specialized Repositories's contracts that I call them Readonly Repository Contracts, put in Application Layer unlike the other Repositories'e contract that put in Domain).
Suppose, my UI needs a View-Model with 3 or 4 properties(from 3 or 4 big Entities).
It's data could be generate with simple projection, but in case 1, because my methods could not access to View-Models, I have to fetch all the fields of all 3 or 4 tables with sometimes, huge joins, and then map the results to View-Models.
But, in case 2, I could simply use projection and fill the View-Models directly.
So, I think in performance point of view, the case 2 is better than case 1. but I read that Repository should depend on Entities and not View-Models in design point of view.
Is there any better way that does not cause the Domain Layer depend on the Application layer, and also doesn't hit the performance? or is it acceptable that for reading queries, my Repositories depend on View-Models?(case2)

Perhaps using the command-query separation (at the application level) might help a bit.
You should make your repositories dependent on entities only, and keep only the trivial retrieve method - that is, GetOrderById() - on your repository (along with create / update / merge / delete, of course). Imagine that the entities, the repositories, the domain services, the user-interface commands, the application services that handles those commands (for example, a certain web controller that handles POST requests in a web application etc.) represents your write model, the write-side of your application.
Then build a separate read model that could be as dirty as you wish - put there the joins of 5 tables, the code that reads from a file the number of stars in the Universe, multiplies it with the number of books starting with A (after doing a query against Amazon) and builds up a n-dimensional structure that etc. - you get the idea :) But, on the read-model, do not add any code that deals with modifying your entities. You are free to return any View-Models you want from this read model, but do trigger any data changes from here.
The separation of reads and writes should decrease the complexity of the program and make everything a bit more manageable. And you may also see that it won't break the design rules you have mentioned in your question (hopefully).
From a performance point of view, using a read model, that is, writing the code that reads data separately from the code that writes / changes data is as best as you can get :) This is because you can even mangle some SQL code there without sleeping bad at night - and SQL queries, if written well, will give your application a considerable speed boost.
Nota bene: I was joking a bit on what and how you can code your read side - the read-side code should be as clean and simple as the write-side code, of course :)
Furthermore, you may get rid of the generic repository interface if you want, as it just clutters the domain you are modeling and forces every concrete repository to expose methods that are not necessary :) See this. For example, it is highly probable that the Delete() method would never be used for the OrderRepository - as, perhaps, Orders should never be deleted (of course, as always, it depends). Of course you can keep the database-row-managing primitives in a single module and reuse those primitives in your concrete repositories, but to not expose those primitives to anyone else but the implementation of the repositories - simply because they are not needed anywhere else and may confuse a drunk programmer if publicly exposed.
Finally, perhaps it would be also beneficial to not think about Domain Layer, Application Layer, Data Layer or View Models Layer in a too strict manner. Please read this. Packaging your software modules by their real-world meaning / purpose (or feature) is a bit better than packaging them based on an unnatural, hard-to-understand, hard-to-explain-to-a-5-old-kid criterion, that is, packaging them by layer.

Well, to me I would map the ViewModel into Model objects and use those in my repositories for reading / writing purposes, as you may know there are several tools that you can do for this, in my personal case I use automapper which I found very easy to implement.
try to keep the dependency between the Web layer and Repository layers as separate as possible saying that repos should talk only to the model and your web layer should talk to your view models.
An option could be that you can use DTO's in a service and automap those objects in the web layer (it could be the case to be a one to one mapping) the disadvantage is that you may end up with a lot of boilerplate code and the dtos and view models could feel duplicated.
the other option is to return partial hydrated objects in your model and expose those objects as DTOs and map those objects to your view models this solution could be a little bit obscure but you can make the projections you want and return only the information that you need.
you can get rid of the of the view models and expose the dtos in your web layer and use them as view models, less code but more coupled approach.

I Kindof agree with Pedro here. using a application service layer could be benificial. if your aiming for an MVVM type of implementation i would advise to create a Model class that is responsible for holding the data that is retrieved using the service layer. Mapping the data with automapper is a really good idea if your entities, DTO's and models are named consistently (so you don't have to write a lot of manual mappings).
In my experience using your entities/poco in viewmodels to display data will result in big balls of mud. Different views have different needs en will all add a need to add more properties to an entity. Slowly making your queries more complex and slower.
if your data is not changing that often you might want to consider introducing (sql/database) views that will transfer some of the heavy lifting to the database (where it is highly optimized). EF handles database views fairly well. Then retrieving the entity and mapping the data (from the views) to the model or DTO becomes fairly straightforward.

Running custom queries in ASP.NET MVC

I'm new to using the ASP.net MVC framework (and C# too). I have more experience using JSP + JSTL MVC framework. I want to know how to run custom queries if I need to do more complex sql.
In the JSP framework, I'm used to creating DAOs to query sql tables which doesn't seem to be the case in ASP.net. For ASP.net, I noticed that we use LINQ to do the querying over a DBSet object from a model.
My Questions:
Where do I write the custom query code? In the model?
How do I write it? Is it through some sort of db.execute statements?
Are there any configuration changes I need to make?
Examples and other resources would be nice.
I tried looking through the asp.net MVC tutorials on asp.net, but I couldn't find anything.

How you execute SQL queries depends entirely on what data access framework you are using. You can write plain SQL with ADO.NET and execute queries in a manner much like that you would expect.
In answer to the second part of your question, this should be done in a data access layer and be completely unrelated to your ASP.NET MVC application itself. Any necessary logic for instantiating and communicating with classes in that abstracted layer should reside in your controller (and, ideally, be loosely coupled from your application via the use of interfaces and/or service layers).
In this sense, yes, it belongs in the model. However, bear in mind that, conceptually, the 'M' in ASP.NET MVC is the domain model (i.e. the aforementioned data access layer), not the view model. The view model is just an independent wrapper that is designed to store relevant data for a given view and should - in most cases - be completely devoid of logic whatsoever.

In the JSP framework, I'm used to creating DAOs to query sql tables
which doesn't seem to be the case in ASP.net.
It is very much the same, you should have a separate DAL project with your DAO objects and reference those from your Business Layer (another separate project). The ASP.NET MVC project itself will add a reference to the Business Layer project which in turn will add a reference to the DAL. That way, you can keep a 3-tier architecture:
UI (ASP.NET MVC project)
|
|
Business Layer (POCO objects (= to POJO), validation, biz logic) - separate project(s), depending on whether you put your POCO objects in the same project or not.
|
|
Data Access Layer (You can use ADO.NET, EF (LINQ), etc.)
An ASP.NET MVC app usually is composed of 3 folders: Models, Views, Controllers but everything that's typically placed in the Models folder is really your Business Objects so I tend to get rid of that folder completely and I am left with just the Views and the Controllers folders. The views are just the html markup, pretty much. The Controllers classes simply call the methods from the business layer and passes the results to the views. So for example, a UserController class will look something like this:
public class UserController : Controller
{
public ActionResult Index()
{
var allUsers = UserBusLayer.User.GetAll();
return View(allUsers);
}
}

Repository, Pipeline, business logic and domain model - how do I fit these together?

I'm designing N-tier application and I came across a difficulty which you might have a solution to. Presentation layer is MVC.
My ORM is carried out using LinqToSQL - it's a seperate project which serves repositories.
Each reporsitory has an interface and at least 1 concrete implementation.
Repositories have the following methods: FindAll(), Save(T entity), Delete(int id)
FindAll() returns IQueryable of some type, which means that it returns queries to which I can apply filters.
ORM mapping has been carried out using Database First methodology, where tables were created first and then classes were generated by SQL Metal.
I have added a Pipeline layer which works with repositories. It applies further filters to queries. E.g. OrderRepository.FindAll().Where(o => o.CustomerId == 10)
Pipeline also returns IQueryable of some type, which means that I can pass it further up the layer and do more stuff with it.
At this point I would like to move to the BusinessLogic layer, but I don't want to work with entity models any longer, I want to convert entity model to a domain model. This means that I can add validation to a model and use that model in the presentation layer. Model can't be defined in MVC project as it would be dependant on the presentation layer, so that's a no.
I'm fairly certain that business logic (behaviour) and model must be stored seperate from pipeline, data and presentation layer. The question is where?
For example, a pipeline has three methods:
1. FindByCustomerId
2. FindByOrderId
3. FindBySomethingElse
All these methods return IQueryable of Order. I need to convert this to a domain model, but I don't want to do it per each method as it won't be mainteinable.
I feel that this model is fairly robust and scalable. I just don't see what is the best place for mapping from entities to domain model and vise versa.
Thank you

First of all, if you are applying Domain Driven Design principles here, you must not have BusinessLogic layer in your application. All business logic should live inside your domain model.
But it is quite hard to achieve using LinqToSQL because it does not support inheritance mapping and you would have to deal with partial classes to put business logic into your domain. So I would strongly recommend to consider moving from LinqToSQL to NHibernate or Entity Framework Code First .In this case you also won't have to convert your persistence model into your domain model and vice versa.
If you still want to do conversion, you could take a look at Automapper

From a domain driven point of view you would need a factory to convert your 'database entity' into a domain model entity.
When you're thinking of turning the 'database entities' to domain model entities at the end of your pipeline you should realize that after the conversion to domain model entities (a projection) you won't be able to use the IQueryable functionality as the projection will trigger execution of your expression tree. For example if you call FindAll for you customer database entity and then convert the IQueryable to (or project it onto) a customer domain entity it will execute (requesting the contents of your entire table).

This is how I do my N-Tier projects. This architecture has great separation of concerns. It sounds like you are already headed in this direction.
In the Mvc project is all your usual objects (Controllers, ViewModels, Views, helpers, etc). Fairly straight forward. All views are strongly typed. Plus my modified T4 templates that generate Controllers, Views, and ViewModels.
In the Business Model project I have all my business objects and rules, Included are the interfaces that define functionality of the data repositories. Instead of having one repository for every business object / table I prefer to group mine by functionality. All objects related to a blog are in one repository while all objects related to a photo gallery are in a separate repository, and logging may be in a third.
You could place your pipeline layer here.
In the Data project I implement those data repository interfaces. You can use Linq2SQL without having to use partial classes. Extending those Linq2SQL partial classes means you have tied your ORM to your domain model. Something you really don't want to do. You want to leave those generated data classes in the data domain. Here is an example Linq2SQL select that returns a BusinessModel object.
from t in Table
where t.Field == keyField
select new BusinessModel.DataObject
{
Id = t.Id,
Field1 = t.Field1,
Field2 = t.Field2
}
If I were you I would look at EntityFramework 4.1 using the CodeFirst approach or use NHibernate. Either of these will map your data model to the domain model. Then to map the domain models to the view models you could use AutoMapper or write custom code or write a T4 template that would generate the mapping code for you.
You could take the code generated by the dbml file as a starting point for your business objects.

Further to xelibrion's comments you could have a look at LightSpeed for your ORM needs. You are currently using LinqToSQL so you should find Lightspeed very straight-forward since it uses the same idea.
http://www.mindscapehq.com/products/lightspeed
If you can get your data to map Models that more match the form your higher levels want then hopefully you can simplify things. The less complexity in the system the less scope for bugs.

All these methods return IQueryable of
Order. I need to convert this to a
domain model, but I don't want to do
it per each method as it won't be
mainteinable.
This is not a true assessment and is probably blocking you from seeing the proper solution.