What are good priciples for creating a scalable website predominantly C#? What design patterns are more common for a C# based website? Links to good books or articles are welcome.
I think these apply to all websites, not just C#
Set proper expectations
Scaling means different things at different times. Are you trying to scale up from your 1000 beta users, to 100,000 active users on launch day? Are you trying to handle constant growth without refactoring? Do you just want to make sure if there's a good old "Slashdot" effect on your site, you can handle it? These all require scalability, but some are very different than others.
Calculate the value of data
Often times people freak out about data loss. But really what people mean to freak out about is data consistency. I won't be really mad if the account I created 30 seconds ago disappears. I will be mad if the photo I uploaded of my trip to Prague is replaced by Halloween Harlots downing beer bongs. So if you are able to calculate the risk of data loss, its a fairly easy process to then calculate the impact, and make a real business decision on whether or not its ok, and if so, how much before it hurts you.
Simplicity trumps Coolness
I love block diagrams as much as anyone else, but how many times have you heard someone go "DAMN, this TV is SO WELL DESIGNED I have to buy it." More often than not they'll say things like "Sony hasn't let me down, and this TV LOOKS great, I have to have it!" Keeping things simple and modular, even if it means NOT using some really cool, ultra-abstracted infinitely scalable pattern, will allow you to scale when and where you need it.
You will not always be alone
The toughest part of your website (and any business) to scale is always people. Sure, by the time you need more engineers, you should be rolling in money so you can just hire 10 people and they'll rewrite the site to be maintainable. However, if you can just hire 2, and they don't immediately surf to the daily wtf to post your entire code base in a 7 part exposé ... you should come out ahead.
This is hard to answer without more details about your architecture.
Which version of .NET and C# are you using?
Are you using MVC or webforms?
Do you have multiple webservers connecting to one database?
When designing a website for scalability I start with having a lot of javascript that will be the controller, pulling information from the webservice, either a WCF app or .asmx service. This way the webserver serves out the pages, but after that the rendering is done by the javascript. This helps relieve stress on the webserver.
If you can have the webserver have all the static content and for any business logic have the code-behind call to another server that will do the processing, get info from the database and return back to the code-behind.
By having this separation, if you need to add more servers in one layer you can determine which part needs extra horsepower, and ensure that the webserver is only really doing one task, interfacing with the browsers.
If you could go into more detail about your architecture it would be helpful, as well as how much of a load you are planning for, then it would be easier to give a more detailed response.
Martin Fowler's Patterns of Enterprise Application Architecture (summaries on his website) are a good place to start. There's an awful, awful lot of components and technologies that can go into building a scalable website... load balancing, caching, application serving, database setup, networking, and somewhere in there is the actual code being written.
Related
I have a pawnshop CRUD app written 20 years ago with INFORMIX-SQL/SE (DOS) which is currently running on DOS 6.22 within Microsoft Virtual PC 2007 on Windows Vista. I would like to modernize this app with a GUI, SQL-based engine and retain its existing functionality. It doesn't require any networking or multi-user capability. I would prefer a product which is royalty-free.
I also would like to quickly re-write it with as little effort possible. Which tool would you recommend?
I'm debating whether to re-write my INFORMIX-SQL app with I4GL (character-based) or another Windows/GUI-based tool.
My app is very robust and has some incredible features which my users are very happy with. Only obstacle which is keeping me from effectively acheiving market penetration is, believe it, my app is char-based and I would like to duplicate the same functionality with a GUI. My feeling is that its quicker for a user to process a transaction with my char-based app vs. having to focus a cursor with a mouse, but cosmetics is hurting me!
I would like to know specific instances of limitations, bugs or drawbacks of using another development tool before I invest considerable amount of time evaling another product. Answers to this question could save me a lot of time and money!
If you visit www.frankcomputer.com you can view a video-demo of my pawnshop app. (CAVEAT: The website's in Spanish, use google translate to get a more-or-less decent translation of the text. Start the video at the two-minute mark, with 720p resolution and full-screen to best comprehend my app.)
If I were doing it, I would probably choose to write a WPF GUI in C# with a SQL Server Express backend database. An embedded database like SQLite might work as well. But the main reason I would choose that is because that's what I'm most familiar with. Someone else would likely choose something else...
I might also choose ASP.NET MVC and make it a web application if that were an option (you say that multi-user is not required, but I say it's not required yet).
Also, if you're not the one who's going to be developing it (i.e. you're going to hire someone to build it for you) then I would say that you should find the developer first and let them choose (or at least have a say in) the technology. If you choose the technology up-front then you're simply limiting the field of developers who'll be able to work with you and there's really not much point in that.
I'd recommend you use Python with a PostgreSQL backend. Now some will think this is overkill, but after watching your video and reading your site (I had to use a translator), I suspect the added flexibility is something you will truly enjoy by going this route.
The reasons I'd argue for this solution are:
Python and PostgreSQL are both great products with amazing communities when you need them.
Both products have a bright outlook in their development paths. Since you obviously spent a lot of time and effort tweaking SPACE, I'm betting you will do the same over the next 40 years. So, the tools you choose now need to be there for you as you continue your development cycle.
They are both free with friendly licenses.
Cross-platform support.
Scalability. You can use PostgreSQL installed locally and connect via socket or scale it all the way up to several servers using load balanced connection pooling.
Security.
Data integrity. This includes how easy it is to make your whole environment easy to backup and thus easy to restore in the event of a catastrophe.
Whatever tools you end up choosing. I wish you the best in this project. I can tell you are working on something you truly love and that is something more of us should strive for!!
Based upon your answers and your emphasis upon time to make the changes and that you don't seem to want to change the Application at all but it is being forced upon you by then you should certainly evaluate Genero from 4js.
This will allow you to utilise your existing code but provide a nicer looking front-end. You can also maintain a single codebase supporting both character and "Gui" clients.
Choose whatever language and technology is easiest for you. If you need DB access and a short lead time it sounds like Java or Visual Basic would be best. Both have plenty of free tools to get you started.
The top languages tags in StackOverflow are C# (by a long margin), then Java, PHP and DotNet, followed by C++ and Python. Some of that will be skewed by the Joel & Jeff origin of the site, but any of those is more than capable of the task. Personally, I'd go with Java or Python but I don't like being tied to the Microsoft stack.
wxWidgets and QT might be options for the GUI components.
Of the databases, mysql, SQL Server Express or Oracle Express Edition are all free and robust. SQLite is good enough for most single user applications though. I'd put this at the bottom of the 'importance' list. For small-scale single user apps, you should be able to chop and change DB platforms without much hassle. The biggest relevance would be in how you actually backup/copy/restore data in the event of disk failure or corruption.
My question is not technical. It's more of a philosophical and really down to individual preference. I am designing and developing an application (web + desktop) and this just occurred to me and was wondering if you guys (programmers and designers) ever came across this before:
Some designers believe in making applications that will run 3-5 years down the line and any changes that come to them will be reflected on them without any need to resort to system core changes. As a programmer I know for a fact that this is never the case. Small cosmetic changes do occur but usually they die away after a year or two, as time progresses there will be changes that will require core changes and eventually you will make a new application.
Given the fast paced changes of technologies, designing an application for 5 years down the line is rather absurd, IMHO. Well I mean not designing, but the idea that this application will run for 5 years and the belief that we will not need to create a new one I think is living in a fool's paradise. I mean really, fellow programmers, most mission critical or basic small application that have a running flow usually are re-made/re-structured/re-organized/re-coded few years down the line anyway.
So my question is why keep to this attitude of having this perfect application that will run for a decade. It's stupid really, because you know for a fact that technology will change every year; new frameworks, new methods, new technologies will emerge and your client will want them. So, if you forgive my use of this phrase, WTF is the point?
I keep telling my designer that the application will be redesigned in a few years anyway, there is no point in trying to make it shoot lighting from its #ss because it just won't, ever. There is no such thing as a perfect application.
I hope you guys get my drift. Have you guys felt the same way too. BTW I have been in the software programming business for about 7 years now. If you really think about it, do you really think Facebook will remain the same 5 years down the line, for sure the design will change every year or so to remain "funky" but the core will change every couple of years. I am dead sure of that. Am I paranoid or what? Please tell me there are other programmers in the same road as I am. Anyone?
My approach is to design for change. This means writing the most maintainable code I can, keeping things loosely coupled and modular, trying to do things in as standard a way as possible so that other developers can quickly pick up on the code, etc.
I usually put a little more effort into future-proofing the database design, as changes there can be a lot more difficult than code changes in many cases.
I have worked at two different companies which had software products that were nearing 10+ years old. Although they had been extended with tons of new functionality and had been given many face-lifts, the core of the application was essentially untouched since the first stable release. This may not be typical, but if the architect is skilled enough, the system can be built to be modular and extensible enough to accommodate a surprising amount of growth.
"The biggest roadblock to a great plan, is the dream of a perfect plan."
I agree - designing the perfect system which can be elegantly resilient to every possible future change, is fruitless. Every successful project is a tradeoff: building flexibility where you're confident it will be needed (or where it's easy to do anyway); and building somewhat quick-n-dirty code where you believe flexibility/change is unlikely. If you analyze the system well and the client has a good idea of their needs/requirements (not always a given), you'll get that balance right at least most of the time.
However, the idea that the whole system will be replaced every 3-5 years by some newer technology is also a fallacy. For every client who wants the latest, newest, sexiest system, there's 5 clients that are afraid to part with (or can't afford to replace) their legacy COM/VB/MS-Access/whatever system that is a morass of spaghetti logic built-up haphazardly without regard to maintainability, flexibility, or extensibility. You don't want to be the one building that system; if you are, then you're doing your client/employer a disservice.
Personally, I would never set out designing any sort of app assuming it is going to be replaced within a few years. It is all to often those 'updates' and 'rewrites' get pushed off for quick fixes desired by customers who don't want to wait for an entire new application. Sure the requirements change, features are needed, but often they will be desired in the current iteration.
I mean, there are a lot of apps, languages, and design patterns out there that thought the same thing, and are still in use today. One that pops into my head is the y2k bug. Programmers in the 70's never thought their code would last 30 years, and surely someone would expand those year values to 4 numbers before the turn of the century. We all remember how that thinking turned out...
I beleive that if the core of the system is built on principles that haven't changed much in many many years, then the system fundamentally will not change much, most changes would be primarially aesthetically.
Some tried and tested principles that remain strong to this day are for example database normalisation, modularisation of code etc etc.
So it depends what you define as the core. For me, the core means the design of the system, which if done properly probably wont change much at all in the future.
It really depends on the environment. I find that our internal corporate applications are fully overhauled (often retired) within 18 months of launch. It isn't anyone's fault - business priorities change, requirements change, new systems come online. At the same time other system run for a very long time.
We certainly don't develop any application with the expectation that it will soon be retired but there are business needs that need immediate solutions and sometimes it is better to get the app online and in the hands of the end-users as soon as possible. We update, iterate, and determine the next best steps.
The data and data structures (often) need to be legitimately designed to last for decades. The algorithms, UI and everything else is expected to evolve rapidly.
If your data represents legal documents, financial records, you might be required to hold on to it for decades.
This, too, though can be taken to extremes. There are subsets of data that probably no one will care about in 50 years, like memory performance counters that might have been stored in the database.
I remember a quote from James Kovac I believe, in one of the .Net Rocks' podcasts where he described our industry as:
one where the only constant is change
which is why you should design for flexibility so that when changes come, and it inevitably will in most cases* you will have an easier job adapting/updating your application. That's not to say you shouldn't try to build your application on solid grounds, but just more important to have a flexible solution which you can change easily than the perfect solution the first time around.
*I know in lots of banks people are still using ancient applications because it's too risky to change them and they simply don't have the expertise to change them anymore..
Well I think the ideal design is completely orthogonal, but you just have to accept that it rarely works out the way you envisage. If you've never read The Pragmatic Programmer, it talks a lot about future-proofing your code.
We have been having a lot of internal debate regarding staging tables. Some are view staging tables as archaic and will prevent the ability to build re-usable services, etc. It is also being communicated that these will prohibit the business to grow and handle expanded business channels.
I am not necessarily for or against either option, but I do know that having the staged data has been a life saver in many occasions and has made it really easy to re-import orders we have had issues with.
Just wanted to see what others thought about staging data and what other methods are being used to handle scenarios similar to ours (Taking orders from external partners, Amazon, etc and importing them into our ERP system).
Thanks,
S
Some places I've worked I've used staging tables, others I've used other techniques.
Each one has its own advantages and disadvantages.
That said, don't worry about it.
If some data feed comes along that requires some method other than what you are doing, then you'll come up with a new solution.
Change is driven by requirements.
(personally, when someone comes to me and says "We have to change to X because what we do now is inefficient and bad and witches will come and eat our children", they have this image in their minds that on tuesday, we will have an opportunity to triple our client base but only if we do this new thing, but if we don't get cracking on it now, then we'll miss the opportunity because none of those potential clients is willing to wait even a minute and they'll all demand the exact same thing and we can build exactly what they want right now even though we have no idea what they want HURRY HURRY HURRY AND DON'T BREAK ANYTHING. Which, of course, isn't how anything works. A single client (or whatever) comes along and says "Hey, we want your services, can you accept our XML?" to which the response is always "Sure thing", and then you get tasked with it and can make intelligent decisions, and plan things out. As opposed to the "chicken with its head cut-off" methodology preferred by people who like technical words but hate knowing anything tehcnical)
There is no reason for a debate - you have a working system. Anyone who thinks their "re-usable services" theory can do it better should put up or shut up.
Let them build a test implementation on your development servers for a common high volume scenario, and compare it to the current system - including criteria for recovery and re-import after a failure.
I hear this all the time where I work as well (usually from managers who just read an article about SOA and XML) and in situations dealing with large amounts of data - bulk imports into staging tables can handle a much higher data volume than any type of web service.
I am starting work on my first business (application + database) type application, using c# and sql I'm completely new to this,
What tips do you have for me?
What should I look for?
What concepts should I understand?
My tip is to get started and come back when you actually have a concrete question. If it makes you feel more prepared, go read some more C# and SQL books first.
While your question is broad, here is the number one thing I 'wish I knew' when I started out with business apps, kinda specific but it's pretty universal:
Writing business apps is made considerably easier by setting up a solid DAL (Data access layer) and abstracting your data access out and away from the rest of your application. That way all of your SQL is in one place and not strewn throughout your code.
Some golden ideas to read up on in this area are 'ORM' (object-relational mapping, as you're using C#, Linq to SQL could be a nice place to start) - this maps your database access to actual classes. If you have a good database design you might even find you have very little SQL work to do at all.
Another nice practice is using the Repository pattern, which effectively encapsulates all your data access into a single class (at least in the simple case - of course in bigger apps you might have multiple). Then in order to access any data, you always go via the repository. This is typically done via an interface which defines the repository, which then allows you to implement multiple concrete implementations. For example, you might want to fetch your data directly from an SQL server, but later on or in an alternative application you might want to use a web service to fetch data instead - no need to re-write everything, just drop in a new repository class! The interface stays the same, so the rest of your application doesn't know any different :D)
This is a pretty broad overview (and a bit of a mind dump sorry), and I'm certainly no expert, but believe me, good data access practices certainly make your life easier!
Just start writing code. You're going to have to throw is away later when you figure out what going on, but that's alright.
Well, I'd say you've come to the right site if you start asking specific questions.
Some of our highest rated questions, however, will give you tons and tons of reading material, books, link to other sites, etc. Here is the URL
https://stackoverflow.com/questions?sort=votes
General tips:
Get some help from somebody who has done this before. There is no way you're going to pull this off by yourself unless you allow for plenty of time to learn on the job.
Don't get distracted by the technical details -- make sure you understand the business. If you don't know why you're building the app (or your clients don't know why they need it) no good can come from it.
As far as what you should look for or how much you need to understand, I don't know the scope of the application you are trying to build -- thus I can't give any intelligent advice. A real-time financial system used by thousands of concurrent users is different from a small retail site that gets hit by hundreds. So my only look for/understand advice is this: don't overengineer your solution.
I am building a prototype for a web-based application and was considering building the front-end in HTML, which can then be reused later for the actual application. I had done a Flash-based prototype earlier, which embedded the .swf into a C# executable. Flash made for rapid turnaround time while the Windows application provided unlimited access to fancy API's for DB access and sound.
I want to consider something similar for this one too. Does this approach make sense? I am particularly concerned about the way the HTML would communicate with the container app. From what I understand out of preliminary research, it would be only through JavaScript, which might quickly get unwieldy. This is especially so because unlike the Flash-based prototype which implemented a lot of its functionality in the .swf, the HTML UI will depend entirely upon the shell to maintain state. Also, I don't need anything more than access to a database. So a desktop application might be overkill.
Another alternative that comes to mind is to build the prototype using PHP and deploy it with a portable server stack such as Server2Go or XAMPP. But I've never done something like this before. Anybody here shed some light on drawbacks of this approach?
The key requirement is rapid iterations of the UI, reusable front-end code and simplified deployment without any installations or configuration.
Some of the best programming advice I've seen came from Code Complete, and was along the lines of, "evolutionary prototypes are fine things, and throwaway prototypes are fine things, but you run into trouble when you try to make one from the other." That is, know which type of prototype you're developing, and respect it. If you're developing a throwaway prototype, don't permit yourself to use any of it, however tempting it may be, in the production system. And if you're developing an evolutionary prototype - one intended to become the production system - don't compromise quality in any way.
It sounds like you're trying to get both, the rapid development of a throwaway and the reusability of an evolutionary prototype - and you can't. Make up your mind, and stand by it. You can't have your cake and eat it, too.
I think you off to the wrong start, here. Why would you want your prototype to be fully functional? A prototype is intended to be throw-away and to help flesh out requirements and UI. If you need full functionality, why not just skip to the final product? If prototyping is really something you want to do, I suggest looking into a specialized prototyping tool.
Are you prototyping the user interface for a customer? If you are, consider something less unwieldy like paper prototypes or presentation software (like PowerPoint) until you get the UI nailed down. If you can establish the UI and are clear about the customer's requirements, you can then develop the application in whatever the actual platform is going to be with a clear model in mind.
In my current project, I prototyped the UI in PowerPoint first. In a subsequent iteration, I used static web pages and some jQuery plugins to simulate actual user interaction. That proved to be very effective in demonstrating the interface, and I didn't have to build the application first.
I would join in on folks suggesting paper prototyping as the "idea", but not necessarily the implementation. The biggest point here is that tools such as HTML or Flash let you get "bogged down" in the details - what does this color look like? What's the text on this thing? Lots of time can pass by that way. Instead, what you should be focusing on is user flows.
One tool that keeps the spirit of paper prototyping without all the "paper" drawbacks is Balsamiq: http://www.balsamiq.com/demos/mockups/Mockups.html. It was covered by Jeff and Joel in one of the Stack Overflow podcasts; I've been using it for my own projects for a while. It's freeware, and it does its job magnificently.
If you know C# then another option you can look at is Silverlight. You can then leverage your knowledge of C# and/or JavaScript and interact with a rich object model.
Would that do what you are looking for? The installation would be minimal on the part of the client - download and install the Silverlight plugin
If prototyping is something you truly wish to accomplish here, paper and pencil will be your best friends. You can draw out as many iterations as necessary. While none of this is ultimately useful later on once you begin coding, it is as quick and rapid is it goes.
As mentioned previously, there are many prototyping tools which have a bit of a learning curve, but an alternative to consider would be using a framework such as CakePHP or Ruby on Rails which make for fast application logic and leave customizing the front end being the primary hard work left. And plus, you're left with a mostly functional application when you're done with your prototyping which can be tweaked as needed.
In either scenario, you're paying with your time either upfront (in the case with learning a new framework), of over time in payments (with the case of prototyping on paper or coding by hand).