I am new to this concept so i need guidance that what will be best to use in following scenario.
I have to make a desktop application that contains many features like parts Stock ,Employees Data,Company Cars Data etc etc.
now the problem is that many users would be using the application and offices situated are in different cities in which this application is installed.
I want a scheme that if one uploads any data to database other gets its reflection and other instantly gets updated.for example if more cars are added everyone using gets their cars list updated.
I was having idea to use webservices and data should be stored somewhere on website database so that everyone's application refreshes lists every say 20 seconds or so.
Any help is appreciated
You wouldn't reload all your data constantly; there are a couple of common approaches here:
keep a list of changes; if you add new data you add the primary data record and you write the fact that the change happened (essentially an "events" list). Then you can query the change log periodically to get and additions/updates/deletes simply by asking for all events after (x)
if the infrastructure allows, some kind of pub/sub framework - same approach really but typically using middleware for the changes, rather than the main DB
re how you get the data; polling is simple and effective; active pushing is harder to setup but may reduce latency - not sure it is worth it here
Another approach, though, is to design it as a web app - then all your data lives at the server-farm and is trivial to update immediately. Your "desktop" app could be a web page using ajax
Try Cloud Computing and store your data into cloud
OK trying to recover my points here after the downvote.
The cloud (windows Azure especially) is a great fit for this project. Web services would help too as they can be easily scaled out to a number of webservers (Instances in Azure speak). Having many desktop clients talking directly to a database is not a good idea and often results in scalability issues.
Output cacheing could help a great deal here if you are refreshing your client side data frequently, this can be implemented with almost no code. This makes it much easier to do than managing lists of changes.
Related
I'm looking for a design solution for a pattern that I am going to have to repeat quite a lot throughout a website I am designing. It is going to be ASP.NET MVC front-end, with C# WCF web services connecting using NHibernate to SQL database.
It's a social networking site so imagine facebook here to get a conceptual idea. What I'm looking for is an efficient and performant way to return paginated results of large datasets, for example a user may have 150 emails. I want to return them 10 at a time depending on what page theyre on, obviously only returning the 10 that relate to the page rather than having to load all 150 items into memory and only displaying 10 at a time as I think the user experience would be better to have a slightly longer delay in changing pages compared to a faster initial load. After all when do you look at emails 6 months old? The usual case is you only care about the first page of results anyway. Similarly a user may have had a number of interactions since their last login (eg your notifications feed on facebook) but again I only want to load n number of results at a time, but in this instance rather than having pages, you would click the "Display more" button which would then fetch the next N results, display them with another "display more" link and so forth you can keep clicking until you reach the end of the dataset. I can imagine they would both use the same design though as they are technically both paginated results, just with different UI output and flow.
Can anyone offer some advice on a good design to use for this, bearing in mind my data retrieval is using NHibernate Queryable or Enumerables? Would I want to be loading all data from DB in one hit then using an interator pattern to only return N rows from the service layer, keeping the rest of the list held in memory on the server open in the users session context so if I made another call to retrieve the next N rows, it would be held in place and keep returning N rows until the iterator finished, or would it be best to simply retrieve N rows from the database and return those, holding nothing in session context? I can see how to return top 10 results from Queryable as
var results = (from email in emails where email.UserId = userId).Take(10);
But I'm not sure how efficient this is, is this the fastest way of doing it? And furthermore I don't see how to start at a certain position, this will always only return the first 10, not say the second 10, or third 10 etc.
So I'm a bit unsure how the best way to proceed is and was hoping for some pointers and advice from people who have done something similar. Bearing in mind with my website performance is going to be of the essence so the user experience needs to be pretty sharp and interactive with refreshing new results. Basically if you were trying to simulate a facebook news feed/wall - how would you implement it with the above architecture?
Thanks!
You can use Skip in combination with Take:
var results = (from email in emails where email.UserId = userId)
.Skip((currentPage - 1) * 10)
.Take(10);
About the web service: You really should make it a stateless web service. You could use the ASP.NET Web API for this. This enables you to build a RESTful web service.
Do I want to be loading all the in one hit...
Definitely not, you only want to pull down the records you need, not the ones you may need.
...using an interator pattern to only return N rows from the service layer, keeping the rest of the list held in memory on the server open in the users session context...
Scalability goes right out the window with that idea.
...or would it be best to simply retrieve N rows from the database and return those, holding nothing in session context?
Now your starting to get on the right track...
In general, you want to let the database do as much as the querying as possible i.e. you don't want to hit the database to then have to further query the results (however, that's not always avoidable). In other words, you want to delegate most, if not all, the heavy lifting to the database.
You mentioned you are using NHibernate which is a pretty powerful ORM. The good news is that do a lot of the work for you in terms of query optimization/caching data etc. Like most ORM's nowadays, NHibernate uses deferred execution with it's queries so just watch out for things like hitting the database too early & choosing when to eager load data instead of performing multiple queries. There is a lot to learn with NHibernate, if you haven't already, it's worth taking the time to read up about it before diving in it will save you a lot of hassle in the long run.
Bearing in mind with my website performance is going to be of the essence so the user experience needs to be pretty sharp and interactive with refreshing new results
In terms of the performance (I assume you mean page load speeds) you would just want to ajaxify your site i.e. load what needs to be loaded with the page, pull the rest in the background & update the page dynamically. To achieve the "refreshing new results" part you need to look at polling the server and pulling down new data. I am pretty sure Facebook use a technique called long polling which essentially keeps an active request open with the server for a set amount of time so the data appears to happen "instantly". Polling is a different ball game all together though, it's about striking the balance of server load vs how "fresh" the data needs to be - that's something you would need to decide yourself and the answer to that is usually dependant on the type of data vs the hardware capabilities of the server.
There are some links about it (like this) out there but I liked this guy approach. I don't know if I'd use his PagedQueryable, but his IPageable, IPagedEnumerable and PagedEnumerable are really interesting. Besides, his project introduction page may give you some ideas on how to roll your own pagination.
I need to decide on the optimal way to write a C# client application to view the dataset in a number of different views. One, some or all views may be visible at once and must be coherent.
A simplified illustration of the dataset would be something like this, assume around 10000 items.
Based on this dataset a number of aggregates must be calculated, such as the sum of values for each ItemId and for each ClientId. The actual calculations are a bit more complicated, but assume that around 30 different aggregates must be calculated.
There will be around 10 clients that will view the data at any one time. Each user will decide if the data is continuously updated or refreshed automatically.
The data is stored in SQL Server 2008 R2 and all clients have access to this directly and are on the same LAN.
The UI needs to be non-blocking, so that new data can be read in the background and the active views refreshed when all aggregates have been calculated.
What architecture/technology/pattern is best suited to this sort of scenario?
Should I use WPF, Windows Forms or Silverlight?
Should the views be pre calculated on the server or should the client do this processing?
Should the client connect directly to the database or via a WCF service?
Unfortunately I think the answer for most of your questions is "It Depends".
1 - MVVM makes sense given your requirements of many views of the same sets of data.
2 - Which technology are you most familiar with now, and what is the timeframe of your application. If you're very familiar with WinForms and have a tight schedule that makes sense. If you're not and you have time learn, then Silverlight may make more sense. I'm kind of torn on WPF, since nowadays it seems more like Silverlight++ instead of the other way around. In other words, if you need to do a Line of Business app, pick Silverlight UNLESS theres a requirement that can only be fulfilled with WPF.
3 - The answer to this question depends on 2 things: How often is the data being updated and how complex/suited to SQL are the calculations. I would generally prefer to handle aggregating on the server side, but depending on the exact calculations you're performing that may or may not be feasible.
4 - This will really be made for you depending on your choice of technologies. Silverlight can't connect to the database directly, so you have to use a service. WinForms and WPF can directly connect to the database. Even so, going with a data access service can help if you find that having every client directly call the database could be a performance issue.
A lot of these decisions are trade offs, and you might not even know that you've traded something until it becomes an issue.
TLDR: It depends.
I mostly agree with Barry: MVVM for design pattern, Silverlight makes sense, going through a Web service can also help with authorization / providing different features to different clients (even though this was not a specific requirement)
The only place I disagree is where the aggregations are made: I personally think that you should send the minimum amount of data necessary to the client, and let the clients do as many calculations as possible. For example, in your table, the 'Value' field looks like it's a simple product of the 'Quantity' and 'Price'. I don't really see the reason for calculating this in the database, then loading it to the web service, then passing it to the client, such that the client only displays the data. This means that your database is doing 95% of the work, but you can make the client do some work (10-15%) and reduce the workload on your database.
What architecture/technology/pattern is best suited to this sort of scenario?
As this sounds like mostly a read-only application and not something that will be doing data entry and validation, I'd probably opt for WPF (10 clients isn't much). Silverlight would be my second choice. Either tech will give you asyn callbacks. I've done both; WPF is going to be more responsive.
Should I use WPF, Windows Forms or Silverlight?
Avoid WinForms like the plague.
Should the views be pre calculated on the server or should the client do this processing?
It depends on how much work you want the database to do. Personally, I hate putting business logic in the database... I'd probably write a lightweight Service Facade over the DB, put my business logic in there, and then expose the results as a WCF Service to a WPF client (or as a RIA Service if I opted for Silverlight).
Should the client connect directly to the database or via a WCF service?
Connecting directly to the DB means you're going to HAVE to put all the business logic either in the DB in views/Sprocs or all of the calculations will have to be on the client (bad, IMO). Or spread out over the two (worse).
I, personally, would want all of the business logic in one place, on the server, so that clients who are fed the data are guaranteed to get the SAME results. I would not let the clients do calculations of the data.
EDIT: Also, having the clients do ANY sort of business calculations makes deployment and versioning more difficult. If you KNOW all of the business logic sits on a server, then all you have to do to update things is release a new version of the service. Then you don't care how many clients you have and who is on what version of the client app - you can still guarantee that the clients are getting the newest data via the one service.
I'd probably either construct views on the DB for the result sets (if the data calculations were simple and lent themselves to views easily) or I'd have a service on the server-side that did the calculations every X minutes and then cached those results (cache to other tables, or a different DB, or no-sql, whatever). Clients would then be free to pull data as often as they want, but the calculation of the data would actually be controlled by the service, not the clients. Letting the clients drive the updating of the data could be a mistake... The only way I'd find that acceptable really is if you put some kind of throttling mechanism on your messages so you could tell clients, "Don't ping me so fast for results, the server is lagging". Because the server needs to have some control over the load.
There's a lot to think about in this scenario, and these are just my initial thoughts.
I am making a member based web app in ASP MVC3 and I am trying to plan ahead, at first our user base will not be huge, but as with any software the potential for a sudden volume spike is always a possibility.
Thinking ahead to this scenario, I know that the database is the bottleneck area on most web apps. We are using MSSQL 2008RS we will have dedicated servers with several client databases each client has there own database so if one server begins to bottle neck we can scale vertically or move some of the databases to a new server and begin filling it up.
To access the databases we use primarily LINQ 2 SQL and are currently re-factoring some of our code to make use of the IQueryable mechanisms to do a lazy load of content. but each page contains quite a bit of content from various parts of the database.
We also have a few large databases that are used for widgets in the program that rarely change but have millions of rows. The goal with those is to somehow sync them to the primary source and distribute them across several machines and then load balance those servers.
With this layout should I even worry about caching, or will the built-in caching mechanisms in MSSQL be sufficient?
If so where should I begin? I have looked briefly at app fabric but it looks as tho it is for Azure only?
Resources:
How to cache data in a MVC application
http://stephenwalther.com/blog/archive/2008/08/28/asp-net-mvc-tip-39-use-the-velocity-distributed-cache.aspx
http://stephenwalther.com/blog/archive/2008/08/29/asp-net-mvc-tip-40-don-t-cache-pages-that-require-authentication.aspx
Lazy loading is a performance killer. Its better to load the entire object graph with one join than to lazy load other properties. This is especially the case with a list of objects. If you iterate you'll end up lazy loading for each item in the list. Furthermore every call to the db has overhead. Less calls = better performance.
SO was a top 1000 website before it needed two database servers. I think you'll be ok.
If your revenue model says "each client will have its own database" than your scaling issues should be really easy to solve. Sounds like you already have a plan to scale up with more servers as your client base increases. Whats the problem?
Caching on the web tier is usually the first scaling fix you'll have to worry about. You probably don't need to do a fresh db call with each page request.
Overall this sounds like a lot of premature optimization. Your traffic hasn't reached a point where you need to be worried about scaling. Make these kinds of decisions at the last second possible.
The database cache is different to most caches - it can if course load used data into memory and re-use query plans, but that isn't really a cache as such.
AppFabric is definitely not just azure; after all, I it was you wouldnt be able to install it (and use it) locally :) but in truth there is little between AppFabroc, redis and memcached (the latter lacks persistance, of course).
But I think you should initially look at using the inbuilt asp.net caching; both data caching via HttpContext.Cache, and caching of entire responses (or, in MVC 3, partials). Obviously you should have a broad idea of what data is used heavily by lots of requests, and is safe to re-use : cache that!
Just make sure you treat all cached FAA as immutable (if you need to update the cache, re-add a modified value; don't modify the existing objects) - reason: it won't work the same if you start needing to use distributed caching, as that uses serialization, and any changes you make won't be seen by the next request.
What would you recommended re an approach to persist data for the following situation:
WPF application (desktop)
Will be capturing information ever second (approx) and will need to store about 5 values per second effectively.
Will need to save data for up to say 1 month
Usage will be both (a) real time viewing of last few hours data, + (b) ability to view historic view of data for other data, kind of like an ad hoc query to view the data. There would need to be some limited filtering or querying on the data store (whatever it is) prior to it being presented
What approach would be recommended here, ideally that is easiest and keep the WPF installation simple?
You could do it using the new SQL CE of Microsoft (which allows multithread). It's easy to deploy (I think it's just a matter of including a DLL)...
http://en.wikipedia.org/wiki/SQL_Server_Compact
It should handle the load fine, assuming not a gezillion people will be using it on the same machine. Even then it would be fairly easy to upgrade.
So you would run a timer or something to push your data mining into it every 5 seconds, then the client polls as the user loads screens in the client.
Later, you might want to separate it into two apps, one for the data mining, maybe running as a service and the other one as 'the client'. In that case having a server dedicated to the data mining would help.
I am working on a Sometimes Connected CRUD application that will be primarily used by teams(2-4) of Social Workers and Nurses to track patient information in the form of a plan. The application is a revisualization of a ASP.Net app that was created before my time. There are approx 200 tables across 4 databases. The Web App version relied heavily on SP's but since this version is a winform app that will be pointing to a local db I see no reason to continue with SP's. Also of note, I had planned to use Merge Replication to handle the Sync'ing portion and there seems to be some issues with those two together.
I am trying to understand what approach to use for the DAL. I originally had planned to use LINQ to SQL but I have read tidbits that state it doesn't work in a Sometimes Connected setting. I have therefore been trying to read and experiment with numerous solutions; SubSonic, NHibernate, Entity Framework. This is a relatively simple application and due to a "looming" verion 3 redesign this effort can be borderline "throwaway." The emphasis here is on getting a desktop version up and running ASAP.
What i am asking here is for anyone with any experience using any of these technology's(or one I didn't list) to lend me your hard earned wisdom. What is my best approach, in your opinion, for me to pursue. Any other insights on creating this kind of App? I am really struggling with the DAL portion of this program.
Thank you!
If the stored procedures do what you want them to, I would have to say I'm dubious that you will get benefits by throwing them away and reimplementing them. Moreover, it shouldn't matter if you use stored procedures or LINQ to SQL style data access when it comes time to replicate your data back to the master database, so worrying about which DAL you use seems to be a red herring.
The tricky part about sometimes connected applications is coming up with a good conflict resolution system. My suggestions:
Always use RowGuids as your primary keys to tables. Merge replication works best if you always have new records uniquely keyed.
Realize that merge replication can only do so much: it is great for bringing new data in disparate systems together. It can even figure out one sided updates. It can't magically determine that your new record and my new record are actually the same nor can it really deal with changes on both sides without human intervention or priority rules.
Because of this, you will need "matching" rules to resolve records that are claiming to be new, but actually aren't. Note that this is a fuzzy step: rarely can you rely on a unique key to actually be entered exactly the same on both sides and without error. This means giving weighted matches where many of your indicators are the same or similar.
The user interface for resolving conflicts and matching up "new" records with the original needs to be easy to operate. I use something that looks similar to the classic three way merge that many source control systems use: Record A, Record B, Merged Record. They can default the Merged Record to A or B by clicking a header button, and can select each field by clicking against them as well. Finally, Merged Records fields are open for edit, because sometimes you need to take parts of the address (say) from A and B.
None of this should affect your data access layer in the slightest: this is all either lower level (merge replication, provided by the database itself) or higher level (conflict resolution, provided by your business rules for resolution) than your DAL.
If you can install a db system locally, go for something you feel familiar with. The greatest problem I think will be the syncing and merging part. You must think of several possibilities: Changed something that someone else deleted on the server. Who does decide?
Never used the Sync framework myself, just read an article. But this may give you a solid foundation to built on. But each way you go with data access, the solution to the businesslogic will probably have a much wider impact...
There is a sample app called issueVision Microsoft put out back in 2004.
http://windowsclient.net/downloads/folders/starterkits/entry1268.aspx
Found link on old thread in joelonsoftware.com. http://discuss.joelonsoftware.com/default.asp?joel.3.25830.10
Other ideas...
What about mobile broadband? A couple 3G cellular cards will work tomorrow and your app will need no changes sans large pages/graphics.
Excel spreadsheet used in the field. DTS or SSIS to import data into application. While a "better" solution is created.
Good luck!
If by SP's you mean stored procedures... I'm not sure I understand your reasoning from trying to move away from them. Considering that they're fast, proven, and already written for you (ie. tested).
Surely, if you're making an app that will mimic the original, there are definite merits to keeping as much of the original (working) codebase as possible - the least of which is speed.
I'd try installing a local copy of the db, and then pushing all affected records since the last connected period to the master db when it does get connected.