I have been struggling with this for some time, and can't seem to find any info.
I have created an online point of sales system using an SQL database, EntityFramework, ASP and .Net4 in C#. My previous programming experience has been with OpenGL and DirectX, so this is all very new to me.
Users from a single company will log into the site using a single account. They can then use the online till and backoffice. Thus far the software has been developed using a single SQL database. What I would like to do is have the application create and use a different instance of the SQL database per account.
Does this sound sensible/possible, or would you expect to have a single huge database for all users? - Note: there could potentially be a huge number of users which each store a lot of information and each user will only need access to their own database.
Any suggestions would be greatly appreciated
Thanks,
I think this is not a good approach, as if there will be any scheme changes in you will have to keep synchronize each database instance.
Does this sound sensible/possible, or would you expect to have a
single huge database for all users? - Note: there could potentially be
a huge number of users which each store a lot of information and each
user will only need access to their own database.
This is possible, but it is not at all sensible. Users don't access databases, applications do. Separating a full database per user only makes sense if your application is offering users the capability of creating their own data structures within the application (such as a hosting provider would do). If it is simply meant to house multiple identical structures keyed off each user, then you'll be better off spending your time normalizing a proper relational structure (an assumption since you mentioned you're using SQL) and optimizing queries/stored procedures to make data retrieval/insertion fast.
RDBMS is perfectly capable of handling huge amounts of users and large subsets of data if you normalize it effectively keying each structure appropriately (in most cases by the userkey).
The best approach is to have a single huge database for all users.
SQL Server will only support 32,767 databases per instance.
http://msdn.microsoft.com/en-us/library/ms143432.aspx
Related
I am making an application in C#, which is the first professional project I will be adding to my portfolio. The application connects to a remote SQL Server and queries specific tables to get data such as Student information, details on courses, and data of student enrollment in each course.
My question is, even though the query will return a single record most of the time, is there a need to cache the tables? There are only 3 tables (Students, Courses, Enrollment), and Courses is the only table to doesn't change all that often, at least in comparison to the other two.
In a nutshell, the app is a CLI that lets the user view school courses, registered students and the student's respective enrollment in those courses. The app has the functionality of entering student info such as their name, their mailing address and contact information, which is then persisted to the SQL Server. The same goes for the Course details like the CourseID, Name and description. As well as the enrollment, which is where the server joins the StudentID and CourseID in the record to show that the specified student is enrolled in that course.
I am currently running a local instance of MSSQL, but plan to create a lightweight Virtual Machine to hold the SQL server to replicate a remote access scenario.
I figure that if the application is ever deployed to a large scale environment, the tables will grow to a large size and a simple query may take some time to execute remotely.
Should I be implementing a cache system if I envision the tables growing to a large size? Or should I just do it out of good practice?
So far, the query executes very quickly. However, that could be due to the fact the MSSQL installation is local or the fact that the tables currently only have 2-3 records of sample data. I do in the future plan to create more sample data to see if the execution time is managable.
Caching is an optimisation tool. Try to avoid premature optimisation, especially in cases when you do not know (and can't even guess) what you are optimising for (CPU, Network, HD speed etc).
Keep in mind that databases are extremely efficient at searching and retrieving data. Provided adequate hardware is available, a database engine will always outperform C# cache structures.
Where caching is useful is in scenarios where network latency (between DB and the app) is an issue or chatty application designs (multiple simple DB calls into small tables in one interaction/page load).
Well for a C# app (Desktop/mobile), cache system is a good practice.But you can make a project for a school without cache system because it doesn't weaken your app performance a lot. Its up to you whether you want to use it or not.
Caching is a good option for the kind of data which is to be accessed frequently but does not change so frequently. In your case it would apply to 'Courses' for which you said that data won't change frequently.
However, for data that is going to grow in size in future and will have frequent inserts/updates to it, it is better to think about optimizing the way they are stored and retrieved from the data stores. In your case, the tables 'Student'and 'Enrollment' are such tables where it is expected to have lots of inserts/updates over time.
So it is better to write optimized procedures to perform CRUD operations on these tables and keep indexes of the right sort on the tables as well. It will not only provide better manageability of data but also give the performance that you are looking for, when compared to caching the results.
I'm part of a small team that currently uses an Access database for scheduling a larger team's availability. This has presented some issues with corruption of the Access database. Additionally, I want to implement additional functionality over time.
I've set out to create an application for the 4-5 of us to use that will solve the concurrent database issue, as well as give the team more functionality.
Since this is a shared network drive, I won't have access to SQL Server (from my guess). I thought maybe a web service would be the way to go, but I don't really want to front the bill for this. Additionally, when I eventually leave the team I don't want to maintain this.
Some ideas I've come up with is an application written in C# that acts as the front-end with SQLite embedded as the back-end. However, I've spent days trying to get the Entity Framework to work with SQLite and am at the point of giving up.
I'm trying to decide what else I can do to solve this issue. Is there another technology I can use?
As was said, it sounds like you try to reinvent the DMBS wheel.
If you have a Database that multiple clients can use at the same time, "sharing a access file on a network share" will simply not cut it. You need a proper DBMS. You have simply outgrown the scale Access was designed for. propably even the scale it was intended for.
You said cost might be an issue, but it is not really: There are dozens of DBMS out there, with a number being Freeware. MySQL is a shining example of a free DBMS. Conver that whole Access thing into a MySQL Database. Write a frontend for the MySQL Database. Done.
If you already have a computer providing the share across the network, that same computer can provide the MySQL Server. Setting up a DBMS with 1+ instances can be a bit more involved then just enabling a share, but not much more then programming a WebService.
We have a relative large scale application that uses relational DB (MSSQL).
After a lot of reading I've decided that I want to examine using MongoDB and not MSSQL, mainly because performance and scale issues.
I read and study about Mongo and couldn't figure out the answer for the following questions:
Should we do it? Bare in mind we have the time to invest, the only question is "is it good for us?"
How to model our data?
My problem with mongo is that we have a lot of one to many relations in our DB.
After reading this great post (and the second part as well), I've realized a good practice will be to divide the decision into 3 scenarios:
1 to few
1 to many
1 to squillions.
In our db, most of the times we use one-to-many, but the problem is that most of the times it's the same "one".
For example, we have users and transactions tables.
Each user can perform a transaction, so basically what I should do is to model the user as following:
{
"name": "John",
...,
"Transactions" : [ObjectId("..."), ObjectId("..."),...]
}
So far it's fine, the problem is that we have a lot more than just transactions, for example we could have: posts, requests and many more features like transactions, and then, my users collection becomes huge (more then 25 "columns"). And also when I want to retrieve a data set I have to do several queries unlike MSSQL in which I'm just using Join statement.
Another issue is that I'll have to save a lot of extra data, for example, for each transaction I have to save the terminal ID, and in the report I'll have to show the terminal name, in that case (as for my understanding) I have 2 choices, the one is to do 2 queries and the other is to save the terminal name as well. In relational DB this is a simple join.
So maybe for schemes like ours, Mongo(or any other document based DB) is not the best choice?
I know those are a newbie questions :)
We use c# for our server side (ASP.Net Web API)
Thanks in advance!
You can face with some serious issues while modeling your data with 2 and 3 approaches:
For One to many you may face with data inconsistency or/and eventual consistency. Here, you store inside document an index (array of references) to external documents. So, for your example to add a new transaction you need two requests: create a transaction and add its reference to a user (update document). Mongo DB has ACID transactions only on document level, so for your case application for some reason can create a transaction but doesn’t add its reference to user. It can be app failures, network problems, bugs and so on. Of course, you can simulate db transaction in app with try/catch block making data cleanup when an error occurs. It will help but not in fully because app can fall down between requests.
So, if your app is high loaded after some time you can have some number of “dad” transactions which are not linked to any user. It couldn’t be a big problem if your app doesn’t query transactions directly – only via users, you will have only useless data in db. Otherwise you will have data inconsistency.
To fix that you need to create background job which will make proper cleanup. So, some period of time your data can be inconsistent – eventual consistency. For some applications, it can be ok, for another – not.
The same problem you can face while deleting transactions.
I agree, that a document with 25 arrays of references (columns) looks not very good. Working with such objects manually will be harder (testing, manual data fixes and so on.
One to squillions doesn’t have this affect but you need indexes to query efficiently. For large and shared db you can have bad performance.
In general, I’d like to say document dbs are pretty good if your app works mostly with one document (aggregate) and don’t have a lot of references to another docs and you don’t need transactions between docs. Denormalization can also be a source of inconsistency.
Key-value data is very easy to scale. Document dbs – it’s one step closer to key-value data-store. Column-oriented dbs are even more closed to key-value and so they can be scaled even better.
Also, I recommend you to consider the next measures to improve your SQL Server db performance:
Caching – perhaps you can cache some your app aggregates instead of gathering (making joins) them in SQL db all the time. For instance, Stack Overflow uses SQL Server db and Redis for caching aggregates (questions with answers, comments and so on).
Tune query performance within indexes, db structure, demoralization and so on.
If your db is hosted in on premise SQL Server then additional memory, SSD disk, table partitioning, data compressions, replication can help. As a rule, SQL Server gives a good performance with these approaches for dbs up to 1 TB.
CQRS approach.
Consider storing your app data in different databases. Every type of dbs has its own strong and weak sides. Document DB is good for storing aggregates, SQL db – for relational data and so on. Complex apps as a rule use a few db types.
In a current project of mine I need to manage and store a moderate number (from 10-100 to 5000+) of users (ID, username, and some other data).
This means I have to be able to find users quickly at runtime, and I have to be able to save and restore the database to continue statistics after a restart of the program. I will also need to register every connect/disconnect/login/logout of a user for the statistics. (And some other data as well, but you get the idea).
In the past, I saved settings and other stuff in encoded textfiles, or serialized the needed objects and wrote them down. But these methods require me to rewrite the whole database on each change, and that's increasingly slowing it down (especially with a growing number of users/entries), isn't it?
Now the question is: What is the best way to do this kind of thing in C#?
Unfortunately, I don't have any experience in SQL or other query languages (except for a bit of LINQ), but that's not posing any problem for me, as I have the time and motivation to learn one (or more if required) for this task.
Most effective is highly subjective based on who you ask even if narrowing down this question to specific needs. If you are storing non-relational data Mongo or some other NoSQL type of database such as Raven DB would be effective. If your data has a relational shape then an RDBMS such as MySQL, SQL Server, or Oracle would be effective. Relational databases are ideal if you are going to have heavy reporting requirements as this allows non-developers more ease of access in writing simple SQL queries against it. But also keeping in mind performance with disk cache persistence that databases provide. Commonly accessed data is stored in memory to save the round trips to the disk (with hybrid drives I suppose accessing some files directly accomplishes the same thing however SSD's are still not as fast as RAM access). So you really need to ask yourself some questions to identify the best solution for you; What is the shape of your data (flat, relational, etc), do you have reporting requirements where less technical team members need to be able to query the data repository, and what are your performance metrics?
I have been asked to setup a course leaflet system for a college. For whatever reason in the past their current system is not linked to their actual course file, they wish to close this link so course leaflets are related to actual course codes. Unfortunately their course file is a ms access database linked to many of their existing systems (cannot easily be upgraded/moved). Since the course leaflets are going on the web it is a requirement with their hosting to use a sql server database.
This means I need to query between the two internally so they can work out what courses they have without a leaflet, I would not like to add ad hoc queries to the access database to do this.
What is the best way to do this in C#, I think LINQ can do it but have not learnt it yet, should I learn it for this project or is there an easier way?
I thought about a linked server to the ms access db but this would require moving the db to the sql server. Another difficult task as from what I can tell links to the database are hard coded.
Just how often does the course file change? Fifty times a day? Once a month?
What about creating the appropriate tables in the SQL Server database? Then every so often (as often as necessary to stay reasonably current), clear those tables out and repopulate them from the Access database. You could set this to run every morning at 3 a.m. or whatever. Or you could just do it periodically whenever the tables change significantly.
Why do you need the Access file to the SQL server to create a Linked Server? Just put it on a network share with appropriate security and create your linked server like that.
To add, LINQ has nothing to do with SQL or Access or anything else, it's for querying in memory object collections. Some linq providers allow you to use that to access your DB in question, but they won't be much help in this situation, I think.