We Keep Coding

C# MVC - Dynamic database architecture

I have an idea for a web app where I will want the user to create their own database through a web application, with their own table names and field types.
I thought about creating a database structure using Object Oriented Programming so that a pre-made database will support all kinds of Entities with custom properties. Something like this:
CustomType
{
public long TypeId {get;set;}
public string ActiveType {get;set;}
}
CustomProperty
{
public int wholeNumber {get;set;}
public string text {get;set;}
public bool boolean {get;set;}
public decimal dec {get;set;}
//Choosen Id of the type to work with
public long TypeId {get;set;}
public bool wholeNumber_ACTIVE {get;set;}
public bool text_ACTIVE {get;set;}
public bool boolean_ACTIVE {get;set;}
public bool dec_ACTIVE {get;set;}
}
CustomEntity
{
public string TableName {get;set;}
public CustomProperty Prop01 {get;set;}
public CustomProperty Prop02 {get;set;}
public CustomProperty Prop03 {get;set;}
public CustomProperty Prop04 {get;set;}
public CustomProperty Prop05 {get;set;}
}
The idea behind this is to let the user decide what they want their database to store, on a pre-made database for them to work with, without having to create it during runtime since this is a web app.
I believe I can manage it like this for them to store whatever they need, but I'm also thinking about the following issues:
How will I manage relationships when the user needs to link tables with Ids and foreign keys.
(I though about managing a public long ForeignId {get;set;} and just store the Id they need to associate).
How will I manage queries since tables will have CodeNames and each will have a different meaning for each person that sets it up.
(I though about, renaming the table during Runtime, but I'm afraid of errors and DB corruption).
Also thought about sending direct queries to create the database according to user's need, but then again non-experienced users can really mess up here or find it hard to manage.
How can I manage migration or DB changes with code instead of the use of PowerShell console.
If we have multiple users each with a unique database, but the same web app how can we manage webconfigs to work with this idea.
I know there's a lot of questions here, I'm looking for the best way to achieve this, having multiple users own their small web app through the internet using MVC pattern and lots of options through a browser.

I would recommend an Entity Attribute Value (EAV) pattern as a solution. With the EAV pattern, rather than creating new tables with new columns for every custom property you wish to store, you store those properties in rows. For example. Instead of every custom table being defined like this:
You define them like this instead:
This allows for flexible creation of multiple entities with multiple properties. The classes in your business logic will then be Entity classes with a collection of Property objects.
In case you haven’t spotted the trade-offs already, the limitation of using the EAV model is the inability to specify field types (int, varchar, decimal etc.), infact, all your property values will be stored as a single type (usually strings).
There are a number of ways to address this. Some handle all the validation etc. in the business logic, other create Field tables per type, so based on my example, rather than having just one EntityFields table, you’ll have multiple, separated by type.

Xamarin SQLite using LINQ

I got a sqlite table in xamarain (native android / pcl):
[Table("Customer")]
public class Customer
{
[PrimaryKey, AutoIncrement]
public int Id { get; set; }
public Address Address{ get; set; }
}
"Address" represents a second table.
1) Is it possible to automatically create the "Address" Table when I call
connection.CreateTable<CustomerDto>();
because it is it's dependency?
2) Is it possible to use a LINQ expression which automatically maps the correct "Address" to this "Customer?

In my .NET Standard library I'm using:
"sqlite-net": "1.0.8"
"sqlite-net-pcl": "1.3.1"
My approach was to create "initial state models" of all the tables, marked as abstract (so there is no risk that somebody could instantiate them), defining only the fields necessary in the database and the primary keys (GUID in my case), used only to create tables at the beginning. Following modification to the data structures always with ALTER instructions.
In another namespace a duplication of all the models, this time with getters/setters and other utilities, and I used these as "real models".
For representing linked models I used a field as Id and another one as model (refreshed when necessary):
public int IdAddress { get; set; }
[Ignore]
public Address Address { get; set; }
I don't think sqlite-net can do what you are asking because it's a very lightweight orm, and even if it could I prefer don't automate too much because of my past experiences with Hibernate.
https://github.com/praeclarum/sqlite-net
https://components.xamarin.com/view/sqlite-net

It sounds like you should look at using Entity Framework because that will allow you to use LINQ with SQLite. The standard library on the web (not Entity framework) is very light and doesn't have much functionality for the ORM like functionality you are looking for.
If you're looking for a more lightweight library, you can use this, but it will not allow you to write LINQ expressions without writing your own ORM:
https://github.com/MelbourneDeveloper/SQLite.Net.Standard

Saving data into multiple collections in DocumentDb

In DocumentDb, what is the best way and place to decouple data in order to save them in separate collections?
So far, most of the examples of how to manage data with DocumentDb use simple objects but in real life, we hardly ever do. I just want to understand how and where I need to handle my complex classes before I save them as Json objects in DocumentDb.
Let's look at the following example. I'll be saving my project information into the Projects collection but I do NOT want to save full names of people in the project team within a project document. I just want to save their EmployeeId's in the project document. I have a separate Employees collection where I want to save person/employee specific information. My project object looks like this:
public class Project
{
[JsonProperty(PropertyName="id")]
public int ProjectId {get; set;}
[JsonProperty(PropertyName="projectName")]
public string ProjectName {get; set;}
[JsonProperty(PropertyName="projectType")]
public string ProjectType {get; set;}
[JsonProperty(PropertyName="projectTeam")]
public List<TeamMember> ProjectTeam {get; set}
}
My TeamMember class inherits from Employee object and looks like this:
public class TeamMember : Employee
{
[JsonProperty(PropertyName="position")]
public string Position {get; set;}
}
My Employee class looks like this:
public class Employee
{
[JsonProperty(PropertyName="id")]
public int EmployeeId {get; set;}
[JsonProperty(PropertyName="firstName")]
public string FirstName {get; set;}
[JsonProperty(PropertyName="lastName")]
public string LastName {get; set;}
[JsonProperty(PropertyName="gender")]
public string Gender {get; set;}
[JsonProperty(PropertyName="emailAddress")]
public string EmailAddress {get; set;}
}
Before saving to Projects collection, here's an example of what my Project document should look like:
{
id: 12345,
projectName: "My first project",
projectType: "Construction Project",
projectTeam: [
{ id: 7777, position: "Engineer" },
{ id: 8998, position: "Project Manager" }
]
}
As you can see, I decoupled my project information from the employee data in order to store them in their own collections, Projects and Employees collections respectively.
Let's not get into why I should or should not decouple data. I just want to see how and where I should handle decoupling in order to produce fastest results. I want to follow the best practices so I just want to see how experts working with DocumentDb handle this scenario.
I can think of two places to handle this but I want to understand if there's a better, more direct way to do this:
I can convert my Project class into a JSON object within my C# code and pass the JSON object to DocumentDb for storage.
Alternatively, I can pass my Project object directly to DocumentDb, into a JavaScript stored procedure and I can handle decoupling and storing data in two or more collections within DocumentDb.
Here's what I'd like to know:
Which is the right place to handle decoupling data?
Which would provide better performance?
Is there a better way to handle this? I keep reading about how I can just pass my POCO classes to DocumentDb and it will just handle them for me. Would DocumentDb handle this more complex scenarios? If so, how?
I appreciate your help. Thank you.

in a NoSql store such as this, you can store different types of documents with different schemas in the same collection.
please do not treat collections as tables. think of collections rather as units of partition and boundaries for execution of queries, transactions etc.
so with that in mind, there is nothing wrong with storing your project document as shown and including the employee documents in to the same collection.
now saying all of that; if you still wanted to do this, then you can ...
in order to achieve this your project object would have to change.
instead of having TeamMember : Employee (which would include the entire Employee object) have the TeamMember object mimic what you want from your JSON ... i.e.
class TeamMember
{
int id {get;set;}
string position {get;set;}
}
Now when DocumentDB serializes your project object you would end up with JSON that resembled what you wanted. And then you could separately save your Employee object somewhere else.
If you don't want to do this, or can't do this because you don't control the definition of the Model or because other parts of the system are built to depend on this already then you could investigate building a custom JSON converter for your Project object that would spit out the JSON you wanted.
Then decorate your Project object with that JsonConverter and when DocumentDB does the conversion the correct result would be created each time.

ServiceStack Request and Response Objects

Is it ok (read good practice) to re-use POCO's for the request and response DTO's. Our POCO's are lightweight (ORM Lite) with only properties and some decorating attributes.
Or, should I create other objects for the request and/or response?
Thanks,

I would say it depends on your system, and ultimately how big and complex it is now, and has the potential to become.
The ServiceStack documentation doesn't specify which design pattern you should use. Ultimately it provides the flexibility for separating the database model POCO's from the DTOs, but it also provides support for their re-use.
When using OrmLite:
OrmLite was designed so that you could re-use your data model POCOs as your request and response DTOs. As noted from the ServiceStack documentation, this was an intentional design aim of the framework:
The POCOs used in Micro ORMS are particularly well suited for re-using as DTOs since they don't contain any circular references that the Heavy ORMs have (e.g. EF). OrmLite goes 1-step further and borrows pages from NoSQL's playbook where any complex property e.g. List is transparently blobbed in a schema-less text field, promoting the design of frictionless Pure POCOS that are uninhibited by RDBMS concerns.
Consideration:
If you do opt to re-use your POCOs, because it is supported, you should be aware that there are situations where it will be smarter to use separate request and response DTOs.
In many cases these POCO data models already make good DTOs and can be returned directly instead of mapping to domain-specific DTOs.
^ Not all cases. Sometimes the difficulty of choosing your design pattern is foreseeing the cases where it may not be suitable for re-use. So hopefully a scenario will help illustrate a potential problem.
Scenario:
You have a system where users can register for your service.
You, as the administrator, have the ability to list users of your service.
If you take the OrmLite POCO re-use approach, then we may have this User POCO:
public class User
{
[PrimaryKey, AutoIncrement, Alias("Id")]
public int UserId { get; set; }
public string Username { get; set; }
public string Password { get; set; }
public string Salt { get; set; }
public bool Enabled { get; set; }
}
When you make your Create User request you populate Username and Password of your User POCO as your request to the server.
We can't just push this POCO into the database because:
The password in the Password field will be plain text. We are good programmers, and security is important, so we need to create a salt which we add to the Salt property, and hash Password with the salt and update the Password field. OK, that's not a major problem, a few lines of code will sort that before the insert.
The client may have set a UserId, but for create this wasn't required and will cause our database query to fail the insert. So we have to default this value before inserting into the database.
The Enabled property may have been passed with the request. What if somebody has set this? We only wanted the deal with Username and Password, but now we have to consider other fields that would effect the database insert. Similarly they could have set the Salt (though this wouldn't be a problem because we would be overriding the value anyway.) So now you have added validation to do.
But now consider when we come to returning a List<User>.
If you re-use the POCO as your response type, there are a lot of fields that you don't want exposed back to the client. It wouldn't be smart to do:
return Db.Select<User>();
Because you don't have a tight purpose built response for listing Users, the Password hash and the Salt would need to be removed in the logic to prevent it being serialised out in the response.
Consider also that during the registration of a user, that as part of the create request we want to ask if we should send a welcome email. So we would update the POCO:
public class User
{
// Properties as before
...
[Ignore] // This isn't a database field
public bool SendWelcomeEmail { get; set; }
}
We now have the additional property that is only useful in the user creation process. If you use the User POCO over and over again, you will find over time you are adding more and more properties that don't apply to certain contexts.
When we return the list of users, for example, there is now an optional property of SendWelcomeEmail that could be populated - it just doesn't make sense. It can then be difficult to maintain the code.
A key thing to remember is that when sharing a POCO object such that it is used as both a request and response object: Properties that you send as a response will be exposed in a request. You will have to do more validation on requests, ultimately the sharing of the POCO may not save effort.
In this scenario wouldn't it be far easier to do:
public class CreateUserRequest
{
public string Username { get; set; }
public string Password { get; set; }
public bool SendWelcomeEmail { get; set; }
}
public class UserResponse
{
public int UserId { get; set; }
public string Username { get; set; }
public bool Enabled { get; set; }
}
public class User
{
[PrimaryKey, AutoIncrement, Alias("Id")]
public int UserId { get; set; }
public string Username { get; set; }
public string Password { get; set; }
public string Salt { get; set; }
public bool Enabled { get; set; }
}
We know now when we create a request (CreateUserRequest) that we don't have to consider UserId, Salt or Enabled.
When returning a list of users it's now List<UserResponse> and there is no chance the client will see any properties we don't want them to see.
It's clear to other people looking at the code, the required properties for requests, and what will be exposed in response.
Summary:
Sorry, it's a really long answer, but I think this addresses an aspect of sharing POCOs that some people miss, or fail to grasp initially, I was one of them.
Yes you can re-use POCOs for requests and response.
The documentation says it's OK to do so. In fact it is by design.
In many cases it will be fine to re-use.
There are cases where it's not suitable. (My scenario tries to show this, but you'll find as you develop your own real situations.)
Consider how many additional properties may be exposed because your shared POCO tries to support multiple actions, and how much extra validation work may be required.
Ultimately it's about what you are comfortable maintaining.
Hope this helps.

We have other approach, and my answer is opinionated.
Because we work not only with C# clients, but mainly with JavaScript clients.
The request and response DTO's, the routes and the data entities, are negotiated between
the customer and the front-end analyst. They are part of the specs in a detailed form.
Even if "customer", in some cases, is our product UI.
These DTO's don't change without important reason and can be reusable in both sides.
But the objects in the data layer, can be the same or partial class or different,
They can be changed internally, including sensitive or workflow information,
but they have to be compatible with the specification of the API.
We start with the API first , not the database or ORM.
Person { ... }
Address { ... }
ResponceDTO
{
public bool success {get; set;}
public string message {get; set;}
public Person person {get; set;}
public List<Address> addresses {get; set;}
//customer can use the Person & Address, which can be the same or different
//with the ones in the data-layer. But we have defined these POCO's in the specs.
}
RequestDTO
{
public int Id {get; set;}
public FilteByAge {get; set;}
public FilteByZipCode {get; set;}
}
UpdatePersonRequest
{
public int Id {get; set;}
public bool IsNew {get; set;}
public Person person {get; set;}
public List<Address> addresses {get; set;}
}
We don't expose only Request or Response DTOs.
The Person and Address are negotiated with the customer and are referenced in the API specs.
They can be the same or partial or different from the data-layer internal implementation.
Customer will use them to their application or web site, or mobile.
but the important is that we design and negotiate first the API interface.
We use also often the requestDTO as parameter to the business layer function,
which returns the response object or collection.
By this way the service code is a thin wrapper in front of the business layer.
ResponseDTO Get(RequestDTO request)
{
return GetPersonData(request);
}
Also from the ServiceStack wiki , the API-First development approach

This will not be a problem given you are OK with exposing the structure of your data objects (if this is a publicly consumed API). Otherwise, Restsharp is made to be used with simple POCOs :)

I think it all depends on how you're using your DTO's, and how you want to balance re-usability of code over readability. If both your requests and responses both utilize a majority of properties on your DTO's, then you'll be getting a lot of re-usability without really lowering readability. If, for instance, your request object has 10 properties (or vice-versa), but your response only needs 1 of them, someone could make an argument that it's easier to understand/read if your response object only had that 1 property on it.
In summary, good practice is just clean code. You have to evaluate your particular use case on whether or not your code is easy to use and read. Another way to think of it, is to write code for the next person who will read it, even if that person is you.

SQLBulk Copy With Related Entities (Entity Framework)

Say I have the below entities. (Heavily Simplified for brevity, but the key properties are included)
public class Crime
{
[Key]
public int CrimeId {get;set;}
public virtual ICollection<Victim> Victims {get;set;}
public virtual CrimeDetail Detail {get;set}
}
public class Victim
{
[Key]
public int VictimId {get;set;}
public string VictimCategory {get;set;}
}
public class CrimeDetail
{
[Key]
public int id {get;set;}
public string DetailText {get;set;}
}
I have 600,000+ of these records to insert into SQL Server 2008 Express R2, which takes quite some time using Entity Framework 4.4.
Ideally I'd like to use SQLBulkCopy to batch insert these records, but since that doesn't support complex types (At least not out of the box), then I'm trying to find some other potential solutions.
I'm open to changing my model if necessary.
EDIT: would the AsDataReader Extension method from MSDN help in this instance?

When having the same issue we end up having code-first generated database with EF and strongly typed generated datasets to be used for SQLBulkCopy.
(We never really coded those classes, they were generated using xsd util from xsd definition of 1-10gb xml file. I'm trying to recall right now when we havent generated typed datasets from the same xsd, but that seems irrelevant to your issue.)
Depending on how you are getting those 600k+ records you either can change the code to use generated strongly-typed datasets or use some object-to-object mapper to map your EF POCO objects to datasets as properties going to be named the same.
Here is a link on generating strongly typed datasets.
Here is an example how to use SqlBulkInsert.