We Keep Coding

C# MVC - Dynamic database architecture

I have an idea for a web app where I will want the user to create their own database through a web application, with their own table names and field types.
I thought about creating a database structure using Object Oriented Programming so that a pre-made database will support all kinds of Entities with custom properties. Something like this:
CustomType
{
public long TypeId {get;set;}
public string ActiveType {get;set;}
}
CustomProperty
{
public int wholeNumber {get;set;}
public string text {get;set;}
public bool boolean {get;set;}
public decimal dec {get;set;}
//Choosen Id of the type to work with
public long TypeId {get;set;}
public bool wholeNumber_ACTIVE {get;set;}
public bool text_ACTIVE {get;set;}
public bool boolean_ACTIVE {get;set;}
public bool dec_ACTIVE {get;set;}
}
CustomEntity
{
public string TableName {get;set;}
public CustomProperty Prop01 {get;set;}
public CustomProperty Prop02 {get;set;}
public CustomProperty Prop03 {get;set;}
public CustomProperty Prop04 {get;set;}
public CustomProperty Prop05 {get;set;}
}
The idea behind this is to let the user decide what they want their database to store, on a pre-made database for them to work with, without having to create it during runtime since this is a web app.
I believe I can manage it like this for them to store whatever they need, but I'm also thinking about the following issues:
How will I manage relationships when the user needs to link tables with Ids and foreign keys.
(I though about managing a public long ForeignId {get;set;} and just store the Id they need to associate).
How will I manage queries since tables will have CodeNames and each will have a different meaning for each person that sets it up.
(I though about, renaming the table during Runtime, but I'm afraid of errors and DB corruption).
Also thought about sending direct queries to create the database according to user's need, but then again non-experienced users can really mess up here or find it hard to manage.
How can I manage migration or DB changes with code instead of the use of PowerShell console.
If we have multiple users each with a unique database, but the same web app how can we manage webconfigs to work with this idea.
I know there's a lot of questions here, I'm looking for the best way to achieve this, having multiple users own their small web app through the internet using MVC pattern and lots of options through a browser.

I would recommend an Entity Attribute Value (EAV) pattern as a solution. With the EAV pattern, rather than creating new tables with new columns for every custom property you wish to store, you store those properties in rows. For example. Instead of every custom table being defined like this:
You define them like this instead:
This allows for flexible creation of multiple entities with multiple properties. The classes in your business logic will then be Entity classes with a collection of Property objects.
In case you haven’t spotted the trade-offs already, the limitation of using the EAV model is the inability to specify field types (int, varchar, decimal etc.), infact, all your property values will be stored as a single type (usually strings).
There are a number of ways to address this. Some handle all the validation etc. in the business logic, other create Field tables per type, so based on my example, rather than having just one EntityFields table, you’ll have multiple, separated by type.

Better practise to hide sensitive fields in EF model or create a new class?

I'm using code-first with entity framework for modeling and creating my db.
I'm just wondering - when I return objects to the user - I don't want to return data sensitive fields like id.
Should I add an atttibute like [DoNotReturn] combined with a filter to remove this fields when returning to the user, or should I just make a whole new class which doesn't contain these fields?
Example:
public class UserAccount
{
//Option 1 - Hide fields
[DoNotReturn]
public int Id { get; set; }
[Index(IsUnique = true)]
[MaxLength(50)]
public string Username { get; set; }
public decimal Cash { get; set; }
[DoNotReturn]
public Account Account { get; set; } //Contains Email / Password
//Option 2 - return a `safe` version
public User UserAccountToFriendly() {
return new FriendlyUser(this.Username, this.Cash);
}
}

Keep your database model separate from your view model that's the approach I have taken and doing it for a long time. it will give you a good separation. Once you start dealing with ViewModel then you can use a library like Automapper or custom mapping classes to convert ViewModel to database model or vice-versa. I hope it helps

Never use your database models as result for end-users and keep it separate from Presentation/Application layer.
There are so many problems that you will encounter:
disclosure of sensitive data (you've mentioned about);
performance issues and waste of RAM and CPU (for instance, you have Order entity with dozens of properties, it would be better to load only those properties that is required instead all);
problems with serialization (with enabled lazy-loading, for instance MVC could try to serialize whole object with navigation properties... );
etc...
I'd like to recommend the following:
return original database entity from Repository layer if necessary, but don't forget to cast it on Presentation layer to another completely brand new xxxModel, xxxViewModel, xxxResponse, etc;
return xxxView from Repository layer if you want to achieve best optimizations, but don't forget to cast it on Presentation layer to brand new object. (any changes on one layer shouldn't affect others, especially end-users);

ServiceStack Request and Response Objects

Is it ok (read good practice) to re-use POCO's for the request and response DTO's. Our POCO's are lightweight (ORM Lite) with only properties and some decorating attributes.
Or, should I create other objects for the request and/or response?
Thanks,

I would say it depends on your system, and ultimately how big and complex it is now, and has the potential to become.
The ServiceStack documentation doesn't specify which design pattern you should use. Ultimately it provides the flexibility for separating the database model POCO's from the DTOs, but it also provides support for their re-use.
When using OrmLite:
OrmLite was designed so that you could re-use your data model POCOs as your request and response DTOs. As noted from the ServiceStack documentation, this was an intentional design aim of the framework:
The POCOs used in Micro ORMS are particularly well suited for re-using as DTOs since they don't contain any circular references that the Heavy ORMs have (e.g. EF). OrmLite goes 1-step further and borrows pages from NoSQL's playbook where any complex property e.g. List is transparently blobbed in a schema-less text field, promoting the design of frictionless Pure POCOS that are uninhibited by RDBMS concerns.
Consideration:
If you do opt to re-use your POCOs, because it is supported, you should be aware that there are situations where it will be smarter to use separate request and response DTOs.
In many cases these POCO data models already make good DTOs and can be returned directly instead of mapping to domain-specific DTOs.
^ Not all cases. Sometimes the difficulty of choosing your design pattern is foreseeing the cases where it may not be suitable for re-use. So hopefully a scenario will help illustrate a potential problem.
Scenario:
You have a system where users can register for your service.
You, as the administrator, have the ability to list users of your service.
If you take the OrmLite POCO re-use approach, then we may have this User POCO:
public class User
{
[PrimaryKey, AutoIncrement, Alias("Id")]
public int UserId { get; set; }
public string Username { get; set; }
public string Password { get; set; }
public string Salt { get; set; }
public bool Enabled { get; set; }
}
When you make your Create User request you populate Username and Password of your User POCO as your request to the server.
We can't just push this POCO into the database because:
The password in the Password field will be plain text. We are good programmers, and security is important, so we need to create a salt which we add to the Salt property, and hash Password with the salt and update the Password field. OK, that's not a major problem, a few lines of code will sort that before the insert.
The client may have set a UserId, but for create this wasn't required and will cause our database query to fail the insert. So we have to default this value before inserting into the database.
The Enabled property may have been passed with the request. What if somebody has set this? We only wanted the deal with Username and Password, but now we have to consider other fields that would effect the database insert. Similarly they could have set the Salt (though this wouldn't be a problem because we would be overriding the value anyway.) So now you have added validation to do.
But now consider when we come to returning a List<User>.
If you re-use the POCO as your response type, there are a lot of fields that you don't want exposed back to the client. It wouldn't be smart to do:
return Db.Select<User>();
Because you don't have a tight purpose built response for listing Users, the Password hash and the Salt would need to be removed in the logic to prevent it being serialised out in the response.
Consider also that during the registration of a user, that as part of the create request we want to ask if we should send a welcome email. So we would update the POCO:
public class User
{
// Properties as before
...
[Ignore] // This isn't a database field
public bool SendWelcomeEmail { get; set; }
}
We now have the additional property that is only useful in the user creation process. If you use the User POCO over and over again, you will find over time you are adding more and more properties that don't apply to certain contexts.
When we return the list of users, for example, there is now an optional property of SendWelcomeEmail that could be populated - it just doesn't make sense. It can then be difficult to maintain the code.
A key thing to remember is that when sharing a POCO object such that it is used as both a request and response object: Properties that you send as a response will be exposed in a request. You will have to do more validation on requests, ultimately the sharing of the POCO may not save effort.
In this scenario wouldn't it be far easier to do:
public class CreateUserRequest
{
public string Username { get; set; }
public string Password { get; set; }
public bool SendWelcomeEmail { get; set; }
}
public class UserResponse
{
public int UserId { get; set; }
public string Username { get; set; }
public bool Enabled { get; set; }
}
public class User
{
[PrimaryKey, AutoIncrement, Alias("Id")]
public int UserId { get; set; }
public string Username { get; set; }
public string Password { get; set; }
public string Salt { get; set; }
public bool Enabled { get; set; }
}
We know now when we create a request (CreateUserRequest) that we don't have to consider UserId, Salt or Enabled.
When returning a list of users it's now List<UserResponse> and there is no chance the client will see any properties we don't want them to see.
It's clear to other people looking at the code, the required properties for requests, and what will be exposed in response.
Summary:
Sorry, it's a really long answer, but I think this addresses an aspect of sharing POCOs that some people miss, or fail to grasp initially, I was one of them.
Yes you can re-use POCOs for requests and response.
The documentation says it's OK to do so. In fact it is by design.
In many cases it will be fine to re-use.
There are cases where it's not suitable. (My scenario tries to show this, but you'll find as you develop your own real situations.)
Consider how many additional properties may be exposed because your shared POCO tries to support multiple actions, and how much extra validation work may be required.
Ultimately it's about what you are comfortable maintaining.
Hope this helps.

We have other approach, and my answer is opinionated.
Because we work not only with C# clients, but mainly with JavaScript clients.
The request and response DTO's, the routes and the data entities, are negotiated between
the customer and the front-end analyst. They are part of the specs in a detailed form.
Even if "customer", in some cases, is our product UI.
These DTO's don't change without important reason and can be reusable in both sides.
But the objects in the data layer, can be the same or partial class or different,
They can be changed internally, including sensitive or workflow information,
but they have to be compatible with the specification of the API.
We start with the API first , not the database or ORM.
Person { ... }
Address { ... }
ResponceDTO
{
public bool success {get; set;}
public string message {get; set;}
public Person person {get; set;}
public List<Address> addresses {get; set;}
//customer can use the Person & Address, which can be the same or different
//with the ones in the data-layer. But we have defined these POCO's in the specs.
}
RequestDTO
{
public int Id {get; set;}
public FilteByAge {get; set;}
public FilteByZipCode {get; set;}
}
UpdatePersonRequest
{
public int Id {get; set;}
public bool IsNew {get; set;}
public Person person {get; set;}
public List<Address> addresses {get; set;}
}
We don't expose only Request or Response DTOs.
The Person and Address are negotiated with the customer and are referenced in the API specs.
They can be the same or partial or different from the data-layer internal implementation.
Customer will use them to their application or web site, or mobile.
but the important is that we design and negotiate first the API interface.
We use also often the requestDTO as parameter to the business layer function,
which returns the response object or collection.
By this way the service code is a thin wrapper in front of the business layer.
ResponseDTO Get(RequestDTO request)
{
return GetPersonData(request);
}
Also from the ServiceStack wiki , the API-First development approach

This will not be a problem given you are OK with exposing the structure of your data objects (if this is a publicly consumed API). Otherwise, Restsharp is made to be used with simple POCOs :)

I think it all depends on how you're using your DTO's, and how you want to balance re-usability of code over readability. If both your requests and responses both utilize a majority of properties on your DTO's, then you'll be getting a lot of re-usability without really lowering readability. If, for instance, your request object has 10 properties (or vice-versa), but your response only needs 1 of them, someone could make an argument that it's easier to understand/read if your response object only had that 1 property on it.
In summary, good practice is just clean code. You have to evaluate your particular use case on whether or not your code is easy to use and read. Another way to think of it, is to write code for the next person who will read it, even if that person is you.

Should I use separate models for domain and EF?

Scenario: I am writing a program that handles report generation.
I have the report stored in a database, mapped to an EF model. There are some non-database fields (i.e. some fields are auto-calculated based on other fields that ARE in the db). Would it make sense to have one class that solely maps to the DB, and another class that takes that information and additionally has the other calculating fields?
i.e. a sample class to interact with the codefirst database would be
public class Report{
public int CategoryOneSeverity {get; set;}
public int CategoryTwoSeverity {get;set;}
public string Title {get;set;}
}
Would it make sense to make another class, like:
public class ReportModel{
public int CategoryOneSeverity;
public int CategoryTwoSeverity;
public string Title;
public int RiskRating{
get{ return CategoryOneSeverity + CategoryTwoSeverity; }
}
}
Or should the RiskRating property be in the EF model.

Yes, I absolutely believe you should have different classes to model your domain than your DB. Unless your application is extremely trivial, if you try to map your domain objects directly, you invariably have to change them to match what you need your data structure to be, and possibly expose things you don't want to expose. Think of it as a violation of the Single Responsibility principle; your class has two reasons to change if you make it your domain object and map it directly. One is in response to changing business requirements, the other is in response to changing data storage schema.

"Would it make sense to have one class that solely maps to the DB, and
another class that takes that information and additionally has the
other calculating fields?"
Most likely yes. Usually I would create a new class suffixed with "ViewModel" such as HumanResourcesReportViewModel if my entity class was HumanResourcesReport.
There's lots of variations on how to use ViewModels, and we could get into a pedantic debate about terminology, but conceptually, take your entity and create a new class with that data plus whatever additional information you need to process the report. In this case the report generation is in a way the View of the MVC model, so I don't think it's offensive to call the class holding the data a ViewModel.

Are you using Code First or DB First?
You can have auto calculated fields in your model, which are not mapped to fields in the database.
It also depends on your architecture. If you're using DB first, refreshing your EF model would update your EF classes, losing your mapped fields. In the DB-First scenario, an alternative would be to use the EF model class as your base class and inherit from it for your report class.
public class ReportModel
{
public int CategoryOneSeverity;
public int CategoryTwoSeverity;
public string Title;
}
public class ReportClass : ReportModel
{
public int RiskRating
{
get { return CategoryOneSeverity + CategoryTwoSeverity; }
}
}

I dont want my object to know about my business layer - so how to load it?

I posted this:
Object depending on another object - when to load its properties
After reading this and researching some more I've come to realize that lazy loading would be ideal for my situation. However, some background info. The class I posted Department gets its properties data from a database (Sql Server 2k5). Its your typical setup:
Front End->BOL->DAL
So I wanted to keep my Department class basically just hold information pertinent to it. I did not / do not want to expose my business object class to this class. So how would I fill my department object without having to make a call to my business object layer.
I think code would help:
public class Employee
{
public int ID { get; set; }
public string FirstName { get; set; }
public Department d { get; set; } //todo: implement lazy load in get.
public Employee(int ID, string FirstName)
{
this.ID = ID;
this.FirstName = FirstName;
}
}
class Department
{
public string DepartmentID { get; set;}
public string CostCenter { get; set; }
public bool hasManager { get; set; }
//more code
//constructor for department
}
And then say my BOL is used to call my DAL to do something like:
//some bol class that simply calls the data access layer class
DAL.GetDepartments("SomeDepartment");
And the DAL does some code to return some data to fill a department object...
However if I lazy load my get property for the department then it would need to know about my BOL so I'd have to add a using statement to include that class as well. That can't be the right way to do this...Any tips or pointers?

If you really want to do lazy loading of components, take a look at some ORM like NHibernate, EF4, etc.
To create your own lazy load solution is not a trivial task and it involves some complex concepts like dynamic proxing, maybe IL generation and other advanced topics.
Anyway, regarding your question on dependencies and use or not use your BOL from your DAL, check if you can apply dependency inversion.

In my current project, almost all my domain business objects have at least two constructors. (1) takes an id for that object, so that it can self-construct, pulling data from the db itself. The other constructor (2) takes the data entity required for that domain object to function. In this way, I can eager load or lazy load, whichever is best at the time.
So, if someone calls your Employee business object Department property, the property could check if it already had that. If not, you could get the DepartmentID from the employee data, instantiate a department, and (store if and) return it. In this way, it's easy to get whatever domain objects you need.
But you still want the eager-loading option there. Let's say you have a Department business object, with an Employees property to return a List(Of Employee). The Employees property would directly get the data from the database, and instantiate each Employee using the data constructor. So you'd have your cool Employee business objects, 80 of them, but with just one data query.
And then to take it up a notch, your objects could accept multi-layer data. For example, you can construct an Employee with an Employee entity from the DAL, that also includes the data for the Department, Supervisor, etc. The "ID" constructor could also get this data from the get-go.
Eventually you'll have some decisions to make about what to pre-load and what to lazy-load. For example, maybe 90% of the time when you construct an Employee, you also need the Department data. Then you might decide to just get the Department data when you construct an employee using the EmployeeID constructor. But the, say, supervisor data is only used 8% of the time. You might decide to always lazy load that.

The simple answer is, obviously, it can't be done. Either you're going to have to let your business objects know about your data layer (make sure to use DI if you go this route) or you can't lazy load from within the business class. A happy medium might be to create a service class that knows about both DAL workings and your business layer and knows how to load and create business objects. Of course, as Juanma stated, ORMs were built to do this kind of thing.