We Keep Coding

How to handle multiple mongodb Collections in an API for .net

I'm trying to make a backend for my .Net webApp using mongoDB for this purpose.
I'm new to mongoDb and quite frankly I feel lost in all the documentation.
Until now I've followed the Microsoft guide on how to make the first steps in building an "onedimensional" api.
I could potentially build everything using only one collection, but I feel like this will quite hard to handle down the road.
That's why I thought it would be wise to split everything into smaller collections.
The Api is written with C#.
My code so far
appsetting:
"FantaTrainerDatabaseSettings": {
//"UsersCollectionName": "Users",
"FantaTrainerCollectionName": "Trainers",
//"TeamsCollectionName": "Teams",
"SoccerPlayersCollectionName": "SoccerPlayers",
"ConnectionString": "mongodb://localhost:27017",
"DatabaseName": "FantaTrainerDb"
}
}
The Startup.cs file has this method where the Controllers are instanciated:
public void ConfigureServices(IServiceCollection services)
{
services.Configure<FantaTrainerDatabaseSettings>(
Configuration.GetSection(nameof(FantaTrainerDatabaseSettings)));
services.AddSingleton<IFantaTrainerDatabaseSettings>(sp =>
sp.GetRequiredService<IOptions<FantaTrainerDatabaseSettings>>().Value);
services.AddSingleton<FantaTrainerService>();
services.AddControllers();
}
What I tried was
public void ConfigureServices(IServiceCollection services)
{
services.Configure<FantaTrainerDatabaseSettings>(
Configuration.GetSection(nameof(FantaTrainerDatabaseSettings)));
services.AddSingleton<IFantaTrainerDatabaseSettings>(sp =>
sp.GetRequiredService<IOptions<FantaTrainerDatabaseSettings>>().Value);
services.AddSingleton<FantaTrainerService>();
services.Configure<SoccerPlayersDatabaseSettings>(
Configuration.GetSection(nameof(SoccerPlayersDatabaseSettings)));
services.AddSingleton<ISoccerPlayersDatabaseSettings>(sp =>
sp.GetRequiredService<IOptions<SoccerPlayersDatabaseSettings>>().Value);
services.AddSingleton<SoccerPlayersService>();
services.AddControllers();
}
But as I supposed It doesn't work this way..
I'm not going to paste all the other code since it's more or less a copy-paste from the microsoft guide, with renamed variables. But let me know if you need more details.
To make it short, I don't get where I need to put the reference to access the other collections.
Do I need like a big controller class that handles all the controllers, or do I need to make the ConfigureServices() dynamic and figure out a way to handle the different collections?
Is there a right way to do that?
Let me know if you need further details or maybe reformulate the question to make it clearer what the problem is

data & collection modelling:
don't put everything in one collection. even though you can. it get's extremely difficult to query and update deeply nested entities as your app grows in complexity due to the c# mongodb driver having limitations on what it can do (without jumping through hoops).
i usually have one collection per logical entity. such as Book,Author,Publisher and have references between them when needed to define relationships. some may say that modelling your entities like a relational db beats the purpose of a nosql db but i don't agree with that becuase it helps with ease of development and maintaining your app in the longrun. you do lose a bit of performance doing lookups/joins but it won't be much worse than mysql/sql-server (with the proper use if indexes).
also keep in mind that mongodb has a hard limit on how big a single document can get, which is 16mb in size. so it would be a bad idea to embed millions of entities inside an array field of an entity. my personal rule of thumb is; if there's going to be more than a few hundred entities embeded in a field, store it in it's own collection. even if it's going to be less than a few hundred, something may get it's own collection if it's a complex (enough) entity.
sometimes, you'll be duplicating data to make queries fast. for ex: you could choose to embed a list of author names inside of the publisher entity. but that has the downside of you having to manually update that embeded list when there's a change to a name of one of the authors. the more places you duplicate in, the more work you have to do to keep data consistent.
in the end, it all really depends on how your app's views/ui/api is going be querying your data. when doing complex apps, you will have to choose the right balance of embedded vs. referenced.
app architecture & layering:
following microsoft tutorials is fine for just getting a lay of the land but they are just too basic when you need to figure out how to build complex systems.
my suggestion is to find some open source projects on github and study how people do things when building real-world apps. but choose wisely what you look at, because there's tendancy in the industry to over-engineer and over-complicate things due to trends/ hype of certain patterns, frameworks and technologies. a couple of such things would be: dependency injection, mocking, ddd, etc. i can suggest the following youtube videos if you'd like to cut through the crap and get to the heart of what really matters.
Core Principles Of API Design
Responsibilities Of a Controller
Dependency Injection? No Thank you!
Mocking? No Thank you!
Interfaces? No Thank you!
you might also find this mongodb web-api starter template interesting where i try to simplify things as much as possible in order to increase ease of development, readability and maintainability.

API architecture - obfuscating/mapping/encrypting id's

I am developing a RESTful API for my company, but a couple of people have some issues regarding the exposure of the entity id's, which I can definitely see as a problem regarding securing our data.
My data is scoped, meaning, you cannot see data that doesn't belong to you in the first place.
I am using Web API and EF6.
What have you done about this issue? is this even an issue (why/why not)?
If it is an issue;
Do I encrypt or otherwise obfuscate the id's?
Do I internally map to different id's? - any good frameworks for this?
Do I add a column to all my tables with an uuid and expose that instead?
What is deemed "good practice" or "secure" in this manner?
The edit of this answer seems like a good solution, but I would still like to see what is considered good/bad/great and maybe other solutions to the 'problem'
Not an issue according to this, I can see why it shouldn't be a problem, as long as
The data is securely scoped

"Do I encrypt or otherwise obfuscate the id's?"
If you have to do this then you probably shouldn't be returning them.
"Do I internally map to different id's? - any good frameworks for this?"
This seems like it would add a high level of complication to your app.
"Do I add a column to all my tables with an uuid and expose that instead?"
Some thing to remember when exposing id's is if you have a certain permission to view something at endpoint: /api/user/1 what is to stop you from "walking" the url and changing that to /api/user/2 to view someone else's data. One thing you can do is use Guids as id's to prevent walking the url, but in general if you do not need to return the id's then don't. If you have to return any data at all that is sensitive then it should ALWAYS be over SSL.

I'm getting Confused over Classes

I bet this is the dumbest question anyone could ever ask but I can't seem to wrap my head around this topic. I understand a Class is used to create an object (when you instantiate it) but what is confusing me is, "When do you know if it is viable to create a Class in a program?".
Say hypothetically your creating a program that gathers an input from the user (be it a name, a number or any other details), use those details to do some calculations and then storing all of it locally. Its basic but would you need to use a class to make the program smoother/faster or more maintainable?
Am I confusing myself?
EDIT: I am mostly using programs like Visual Studio and NetBeans IDE.

As you say a class is a construct that allows you to describe a type, which has properties methods and events on it.
In simple situations you can easily get away with not using classes, but in larger, more complex projects having a properly thought out object model makes things so much easier. Easier to maintain, extend, reuse, read.
It may feel like more work at the time (and in most cases it probably is), but it is definitely not wasted effort if you're creating something that will need to be supported.

In Java you don't have a choice: you can't write code that's outside of a class.
A class is a template for instances. It encapsulates state and behavior together into a single software component. You write programs by creating instances of classes that interact together to accomplish your goals.
would you need to use a class to make the program smoother/faster or
more maintainable?
I can't speak for C#, but in Java you don't have a choice. You either create a single class that does all that in a main class or you break it up into several classes that handle different parts of the problem (e.g. I/O, calculations, persistence, etc.) You have to have one or more classes.
You write classes and create objects from them because they map well to the kind of problems that you want to solve. They're either real objects that model physical things in the world (e.g. Person, Car, Bank, etc.) or reifications of ideas (e.g. PersonFactory, Account, etc.) You choose to write object-oriented code because objects model the problem you'd like to solve well.
Some problems lend themselves to functional programming. There are more than one way to write programs to solve problems.

here is a very simple / easy to understand Tutorial that will help you in learning / understanding C# especially Classes also look at Structs as well
C# Class Tutorial

Classes are primarily for us humans to help us organize code. They don't necessarily make software run faster on a computer. When I first learned about classes, I found it helpful to model "real-world" objects. For example, if I wanted to write a program that calculates the area and perimeter of geometrical shapes, I would create a simple Shape class which defines the abstract methods which do the calculations. Then I extend this class to create different kinds of shapes, say Circle, Square, and Triangle. By starting with simple applications of classes and through more programming experience, I have been able to gradually see other places to use classes.

For a simple program, like your example, to gather input from the user (name, number, comment), you can easily do this without creating a new class, excluding the fact that languages like C# and Java require a static class to put your main function inside. If you used a non-object oriented language like C, you could easily do it without a class.
For a simple class like you describe, in fact, it might seem like a little extra work to have to create the class (although it is probably negligible). However, that is only because you are referring to a small set of data.
The point where it gets useful to create classes is when you have many pieces of data. The class acts as a way to group that data together, and possibly store it locally as one. You can also add other related methods onto each class that are directly related to the class members.
When you app becomes more complex, say you need to keep track of customers, products, billing information (Credit Card, PayPal, ...), addresses (ship to, and billing), is when classes become extremely valuable in keeping each bit of information together as well as relating each of those larger "bundles" (classes) of information to each other.
You could have a customer who has an order who has a bill to and a shipping address. Each of these classes, itself has many fields inside of it. But you can relate the larger concept of customer to a target shipping address a lot easier with classes.
As far as "When do you know if it is viable to create a Class in a program?", the answer is not always easy, but any time you see data fields which naturally fit together (well, like an address, or a product description or billing information). I wish I had a more concrete answer, but it really depends on what you are building, and what type of data you are working with.
And no, it's not the dumbest question anyone could ever ask! I hope this helps your understanding.

1. When do you know if it is viable to create a Class in a program?
I think just about everyone who answers this question will have a slightly different response, but typically, you should use Classes to modularize your program's functionality. In your example, you said that your program might take input from the user, use input to perform some calculations, and persist the user data and calculation results somewhere. You could have the following three classes:
UserInput - handles keyboard input and converting it into some easier-to-process format
Calculator - processes all input after it has been converted
DataPersistence - handles reading/writing from/to disk, or database, or whatever you need.
This way, all of your code isn't just piled up inside a massive Java/C# main() call, you can focus on the smaller parts independently of each other. The interaction between these components is what determines your program's behavior.
2. Its basic but would you need to use a class to make the program smoother/faster or more maintainable?
Classes may actually end up adding overhead to your program because of how objects are referenced in languages like Java or C#, but they make your code much easier to read and modify than if your program was written inside one gigantic function. This is sort of analogous to dividing up math textbooks into chapters. If Algebra, Calculus, and Differential Equations were all condensed into a single chapter of the text book, then the text book wouldn't be very useful to those who want to skip ahead to the Calculus part. The overhead of adding chapter headings is negligible because it allows the author/reader to focus on certain portions of the book.
Likewise, Classes help you divide up your work so it's easier to maintain later. Speed/Performance are generally not affected by how you divide your program into classes, provided you do it intelligently - this is where the real artistry of Object Oriented Design manifests itself :)

Issues with Orchard CMS

I have researched for some time the Orchard CMS and I'm pleased with some of his futures but also I have some issues that I don't know how to deal with them:
All the items (content type) are linear and they don't support a tree like data structure
(Ex: books > titles > web-links)
One of the big problem (depending how you see things) is that the model and the view for the items are coupled (content part > driver with display / editor views)
So for a new page the model, view and position are locked and you can have only one view of the model.
Use of advance language futures the are not suited for beginner developers and are not very clear (dynamic functions, clay objects - nice future, ...)
// Creating table VPlayerRecord
SchemaBuilder.CreateTable("VPlayerRecord", table => table
.ContentPartRecord()
.Column("Title", DbType.String)
.Column("VideoUrl", DbType.String)
.Column("WidthPx", DbType.Double)
.Column("HeightPx", DbType.Double)
);
This syntax is not very clear for beginner developers and is a bit over engineered. Also because the model is a dynamic object in the view we don't have any intellisense support.
To build a new page we have something like three degree of separation (3 projects)
Build a content part module
Build a content type
Build a theme module
How do you overcome these issues in your projects with Orchard CMS? and what other issues have you found and fixes :)

read this: http://orchardproject.net/docs/Creating-1-n-and-n-n-relations.ashx and this: http://orchardproject.net/docs/Creating-lists.ashx
How is this a problem and why do you see this as coupling? What alternative do you see?
Where do you see this as a problem and how has it blocked you?
edit on 2: it is not true that you can have only one view of the model. You can have any number of display types. For example, the summary view of items is handled this way. You also have display types for admin views, and you can add your own.
Not sure what you mean by "position is locked". If we mean the same thing by position, I'm puzzled by how you could have gotten such an idea. Relative positioning of parts and fields can be changed through placement.info.
edit on 3: even with this example, I'm not sure what would be difficult here. This is fairly expressive imo. Were you confused yourself or are you just assuming people would be?
You are claiming that this is over-engineered. How would you simplify it then? What feature do you think is not needed?
You don't get IntelliSense in views on model objects but the flexibility you gain by doing so justifies it by a very large margin. Ask anyone who's been making real use of it.
new 4th point: I can't see a reason why you would separate that into three modules or why you think you should. I've certainly never seen an example of that. I would also point out that creating a part and a type are often done by two different people (a type creator is often just a consumer of existing parts). But again you don't have to separate them into different modules.
A theme is clearly a different concern from the two others and makes sense to be a separate project but a theme can come with code and can actually in principle do everything a module is doing. So if you want to package a part, type and theme into a single package, you could do that. It wouldn't make a lot of sense but you could.
Finally, I don't see how any of those four points are related to page creation.

Orchard has to be taken as a challenge. As a beginner I have quickly built a few sites with ease. After that the learning curve became steeper. I've read many articles on the subject, numerous times.
I have used some CMS's before and had some knowledge what is the nature of managing content. Learning Orchard opened a whole new definition of content management. Now I can think of solving some everyday tasks and various business processes by implementing Orchard.
The whole thing is built in a very abstract layer, forcing you to think abstract too. If you follow this way, there are many blog posts, as well as official documentation to help you.
There are few basic building blocks and concepts that can be used like a bricks. Sounds like a phrase, I've heard it hunderts of times. I have also seen thousands of houses built from red square bricks, and they were all different, while the bricks were all equal. Such things can be accomplished with Orchard.
Read and understand the programming patterns. They are essential part of knowledge that will help you in solving Orchard based tasks. They will also help you change the way you are accomplishing your non Orchard related tasks.
I would say, there are two basic areas one need to understand. Storing and retreiving the piece of content is one, while presenting it to the crowd is the other. It might look difficult, it is difficult, but the goodies behind are delightfull. Not to mention great guys, some from evil empire, some not, that will certainly help you along the way. Not to forget, git's are your best friend. There are many wheels already invented. Caution, neither comes with free lunch.
P.S. I haven't write such a long post since usenet times. It might not be suitable for a site like this. It's kind a way to give a thanks to this French guy, and to all other Orchard evangelists from Poland, over Cyprus to the States. They saved my ass in many occasions.

SOA/WCF dissecting the system & service boundaries

I'm building a system which will have a few channels feeding different clients (MonoDroid, MonoTouch, Asp.Net Mvc, REST API)
I'm trying to adopt an SOA archetecture and also trying to adopt the persistence by reachability pattern (http://www.udidahan.com/2009/06/29/dont-create-aggregate-roots/)
My question relates to the design of the archetecture. How best to split the system into discreet chunks to benefit from SOA.
In my model have a SystemImplementation which represents the an installation of the system iteself. And also an Account entity.
The way I initially thought about designing this was to create the services as:
SystemImplementationService - responsible for managing things related to the actual installation itself such as branding, traffic logging etc
AccountService - responsible for managing the users assets (media, network of contacts etc)
Logically the registration of a new user account would happen in AccountService.RegisterAccount where the service can take care of validating the new account (duped username check etc), hashing the pw etc
However, in order to achieve persistence by reachability I'd need to add the new Account to the SystemImplementation.Accounts collection for it to save in the SystemImplementation service automatically (using nhibernate i can use lazy=extra to ensure when i add the new account to the collection it doesn't automatically load all accounts)
For this to happen I'd probably need to create the Account in AccountService, pass back the unsaved entity to the client and then have the client call SystemImplementation.AssociateAccountWithSystemImplementation
So that I don't need to call the SystemImplementation service from the AccountService (as this, correct me if I'm wrong - is bad practise)
My question is then - am i splitting the system incorrectly? If so, how should I be splitting a system? Is there any methodology for defining the way a system should be split for SOA? Is it OK to call a WCF service from in a service:
AccountService.RegisterAccount --> SystemImplementation.AssociateAccountWithSystemImplementation
I'm worried i'm going to start building the system based on some antipatterns which will come to catch me later :)

You have a partitioning issue, but you are not alone, everyone who adopts SOA comes up against this problem. How best to organize or partition my system into relevant pieces?
For me, Roger Sessions is talking the most sense around this topic, and guys like Microsoft are listening in a big way.
The papers that changed my thinking in this can be found at http://www.objectwatch.com/whitepapers/ABetterPath-Final.pdf, but I really recommend his book Simple Architectures for Complex enterprises.
In that book he introduces equivalence relations from set theory and how they relate to the partitioning of service contracts.
In a nutshell,
The rules to formulating partitions can be summarized into five laws:
Partitions must be true partitions.
a. Items live in one partition only, ever.
Partitions must be appropriate to the problem at hand.
a. Partitions only minimize complexity when they are appropriate to the problem
at hand, e.g. a clothing store organized by color would have little value to
customers looking for what they want.
The number of subsets must be appropriate.
a. Studies show that there seems to be an optimum number of items in a
subset, adding more subsets, thus reducing the number of items in each
subset, has very little effect on complexity, but reducing the number of
subsets, thus increasing the number of elements in each subset seems to
add to complexity. The number seems to sit in the range 3 – 12, with 3 – 5
being optimal.
The size of the subsets must be roughly equal
a. The size of the subsets and their importance in the overall partition must be
roughly equivalent.
The interaction between the subsets must be minimal and well defined.
a. A reduction in complexity is dependent on minimizing both the number and
nature of interactions between subsets of the partition.
Do not stress to much if at first you get it wrong, the SOA Manifesto tell us we should value Evolutionary refinement over pursuit of initial perfection .
Good luck

With SOA, the hardest part is deciding on your vertical slices of functionality.
The general principles are...
1) You shouldn't have multiple services talking to the same table. You need to create one service that encompasses an area of functionality and then be strict by preventing any other service from touching those same tables.
2) In contrast to this, you also want to keep each vertical slice as narrow as it can be (but no narrower!). If you can avoid complex, deep object graphs, all the better.
How you slice your functionality depends very much on your own comfort level. For example, if you have a relationship between your "Article" and your "Author", you will be tempted to create an object graph that represents an "Author", which contains a list of "Articles" written by the author. You would actually be better off having an "Author" object, delivered by "AuthorService" and the ability to get "Article" object from the "ArticleService" based simply on the AuthorId. This means you don't have to construct a complete author object graph with lists of articles, comments, messages, permissions and loads more every time you want to deal with an Author. Even though NHibernate would lazy-load the relevant parts of this for you, it is still a complicated object graph.