We Keep Coding

Using LINQ in C# for paginated results

I'm looking for a design solution for a pattern that I am going to have to repeat quite a lot throughout a website I am designing. It is going to be ASP.NET MVC front-end, with C# WCF web services connecting using NHibernate to SQL database.
It's a social networking site so imagine facebook here to get a conceptual idea. What I'm looking for is an efficient and performant way to return paginated results of large datasets, for example a user may have 150 emails. I want to return them 10 at a time depending on what page theyre on, obviously only returning the 10 that relate to the page rather than having to load all 150 items into memory and only displaying 10 at a time as I think the user experience would be better to have a slightly longer delay in changing pages compared to a faster initial load. After all when do you look at emails 6 months old? The usual case is you only care about the first page of results anyway. Similarly a user may have had a number of interactions since their last login (eg your notifications feed on facebook) but again I only want to load n number of results at a time, but in this instance rather than having pages, you would click the "Display more" button which would then fetch the next N results, display them with another "display more" link and so forth you can keep clicking until you reach the end of the dataset. I can imagine they would both use the same design though as they are technically both paginated results, just with different UI output and flow.
Can anyone offer some advice on a good design to use for this, bearing in mind my data retrieval is using NHibernate Queryable or Enumerables? Would I want to be loading all data from DB in one hit then using an interator pattern to only return N rows from the service layer, keeping the rest of the list held in memory on the server open in the users session context so if I made another call to retrieve the next N rows, it would be held in place and keep returning N rows until the iterator finished, or would it be best to simply retrieve N rows from the database and return those, holding nothing in session context? I can see how to return top 10 results from Queryable as
var results = (from email in emails where email.UserId = userId).Take(10);
But I'm not sure how efficient this is, is this the fastest way of doing it? And furthermore I don't see how to start at a certain position, this will always only return the first 10, not say the second 10, or third 10 etc.
So I'm a bit unsure how the best way to proceed is and was hoping for some pointers and advice from people who have done something similar. Bearing in mind with my website performance is going to be of the essence so the user experience needs to be pretty sharp and interactive with refreshing new results. Basically if you were trying to simulate a facebook news feed/wall - how would you implement it with the above architecture?
Thanks!

You can use Skip in combination with Take:
var results = (from email in emails where email.UserId = userId)
.Skip((currentPage - 1) * 10)
.Take(10);
About the web service: You really should make it a stateless web service. You could use the ASP.NET Web API for this. This enables you to build a RESTful web service.

Do I want to be loading all the in one hit...
Definitely not, you only want to pull down the records you need, not the ones you may need.
...using an interator pattern to only return N rows from the service layer, keeping the rest of the list held in memory on the server open in the users session context...
Scalability goes right out the window with that idea.
...or would it be best to simply retrieve N rows from the database and return those, holding nothing in session context?
Now your starting to get on the right track...
In general, you want to let the database do as much as the querying as possible i.e. you don't want to hit the database to then have to further query the results (however, that's not always avoidable). In other words, you want to delegate most, if not all, the heavy lifting to the database.
You mentioned you are using NHibernate which is a pretty powerful ORM. The good news is that do a lot of the work for you in terms of query optimization/caching data etc. Like most ORM's nowadays, NHibernate uses deferred execution with it's queries so just watch out for things like hitting the database too early & choosing when to eager load data instead of performing multiple queries. There is a lot to learn with NHibernate, if you haven't already, it's worth taking the time to read up about it before diving in it will save you a lot of hassle in the long run.
Bearing in mind with my website performance is going to be of the essence so the user experience needs to be pretty sharp and interactive with refreshing new results
In terms of the performance (I assume you mean page load speeds) you would just want to ajaxify your site i.e. load what needs to be loaded with the page, pull the rest in the background & update the page dynamically. To achieve the "refreshing new results" part you need to look at polling the server and pulling down new data. I am pretty sure Facebook use a technique called long polling which essentially keeps an active request open with the server for a set amount of time so the data appears to happen "instantly". Polling is a different ball game all together though, it's about striking the balance of server load vs how "fresh" the data needs to be - that's something you would need to decide yourself and the answer to that is usually dependant on the type of data vs the hardware capabilities of the server.

There are some links about it (like this) out there but I liked this guy approach. I don't know if I'd use his PagedQueryable, but his IPageable, IPagedEnumerable and PagedEnumerable are really interesting. Besides, his project introduction page may give you some ideas on how to roll your own pagination.

Performance Issues in Real-time data visualization application

Here's the scenario:
I'm working on decision-support system which is being developed as windows forms client and MySQL database server. There are sources which update data in real-time in the MySQL database and my application is supposed to read and refresh the latest data every second and present them to user in graphically rich forms of presentations like graphs, bars, grids etc.
Now, we are facing problems and have performance bottlenecks. We found reading directly from database is a main problem and we want to boost this transaction. I came across memcached but I'm not sure if I can use this in this scenario. Can you please help me in removing this bottleneck? Can memcahced be used in this scenario? What are the different alternatives that can come to help in this situation?
Preferably, I would want to perform everything in memory and keep taking the database snapshots every 15 minutes to the disk. Is there a way I can do this?

Single Database and Many Users Desktop Application

I am new to this concept so i need guidance that what will be best to use in following scenario.
I have to make a desktop application that contains many features like parts Stock ,Employees Data,Company Cars Data etc etc.
now the problem is that many users would be using the application and offices situated are in different cities in which this application is installed.
I want a scheme that if one uploads any data to database other gets its reflection and other instantly gets updated.for example if more cars are added everyone using gets their cars list updated.
I was having idea to use webservices and data should be stored somewhere on website database so that everyone's application refreshes lists every say 20 seconds or so.
Any help is appreciated

You wouldn't reload all your data constantly; there are a couple of common approaches here:
keep a list of changes; if you add new data you add the primary data record and you write the fact that the change happened (essentially an "events" list). Then you can query the change log periodically to get and additions/updates/deletes simply by asking for all events after (x)
if the infrastructure allows, some kind of pub/sub framework - same approach really but typically using middleware for the changes, rather than the main DB
re how you get the data; polling is simple and effective; active pushing is harder to setup but may reduce latency - not sure it is worth it here
Another approach, though, is to design it as a web app - then all your data lives at the server-farm and is trivial to update immediately. Your "desktop" app could be a web page using ajax

Try Cloud Computing and store your data into cloud
OK trying to recover my points here after the downvote.
The cloud (windows Azure especially) is a great fit for this project. Web services would help too as they can be easily scaled out to a number of webservers (Instances in Azure speak). Having many desktop clients talking directly to a database is not a good idea and often results in scalability issues.
Output cacheing could help a great deal here if you are refreshing your client side data frequently, this can be implemented with almost no code. This makes it much easier to do than managing lists of changes.

Live Data Feed: Oracle + C#/ASP.Net

I have to monitor some data during the day (basically a log stored in a Oracle table), and it has to be as close to real-time as possible. So what I need is:
A good way to get incremental data from the table (replication packets?); and
A good way to display it on both a web page and a C# GUI monitor.
Explaining the existence of two displays: they're aimed for different users, role-based - embedded question, any advice on middle layer filtering method?
Rgs,
Arthur

Wouldn't you simply have an auto-refreshing (meta tag) asp.net page, which does a select straight from your log table. Refreshing say every 5 seconds wouldn't put undue load on your oracle server i'd think, especially with only 2 users (i assume the GUI version is conceptually similar).

Marking as answered, only today finished something that worked well.
All the data I need comes from a legacy query, ugly and slow. It uses a dozen of tables, and I need to keep it updated just because of two of them (the others change like three or four times a day only).
The solution was to refactor (and exhaustively test) the query so I could divide the tables in three groups:
Tables that don't change at all;
Tables that rarely change over the day;
Tables that change frequently.
This way I was able to query the high-frequency ones every second, and keep the others updated in a reasonable manner. Data merging was not an issue when done programatically.
Thanks for the help,

Data-Driven Websites for Very Small Businesses

I have a client who has a product-based website with hundreds of static product pages that are generated by Microsoft Access reports and pushed up to the ISP via FTP (it is an old design). We are thinking about getting a little more sophisticated and creating a data-driven website, probably using ASP.NET MVC.
Here's my question. Since this is a very small business (a handful of employees), I'd like to avoid enterprise patterns like web services if I can. How does one push updated product information to the website, batch-style? In a SQL Server environment, you can't just push up a new copy of the database, can you?
Clarification: The client already has a system at his facility where he keeps all of his product information and specifications. I would like to refresh the database at the ISP with this information.

You don't mention what exactly the data source is, but the implication is that it's not already in SQL Server. If that's the case, have a look at SSIS.
If the source data is in SQL Server, then I think you'd want to be looking at either transactional replication or log shipping to sync the two databases.

If you are modernizing, and it is a handful of employees, why would you push the product info out batch style?
I don't know exactly what you mean by "data driven", but why not allow the ASP.NET app to query the SQL Server product catalog database directly? Why generate static pages at all?
UPDATE: ok, I see, the real question is, how to update the SQL database running at the ISP.

You create an admin panel so the client can edit the data directly on the server. It is perfectly reasonable to have the client keep all their records on the server as long as the server is backed up nightly. Many cloud and virtual services offer easy ways to do replicated backups.
The additional benefit of this model is that more than one user can be adding or updating records at a time, making the workforce a lot more scalable. Likewise, the users can log in from anywhere they have a web browser to add new records, fix mistakes made in old records, etc.
EDIT: This approach assumes you can convince the client to abandon their current data entry system in favor of a centralized web-based management panel. Even if this isn't the case, the SQL database can be hosted on the server and the client's application could be made to talk to that so you're only ever using one database. From the sounds of it, it's a set of Access forms and macros which you should have source access to.

Assuming that there is no way to sync the data directly between your legacy system DB (is it in Access, or is Access just running the reports) and the SQL Server DB on the website (I'm not aware of any):
The problem with "pushing" the data directly into the SQL server will be that "old" (already in the DB) records won't be updated, but instead removed and then recreated. This is a big problem with foreign keys. Plus, I really don't like the idea of giving the client any access to the db at all.
So considering that, I find that the best is to write a relatively simple page that takes an uploaded file and updates the database. The file will likely be CSV, possibly XML. After a few iterations of writing these pages over the years, here's what I've come up with:
Show file upload box.
On next page load, save file to temp location
Loop through each line (element in XML) and validate all the data. Foreign keys, especially, but also business validations. You can also validate that the header row exists, etc. Don't update the database.
3a. If invalid data exists, save an error message to an array
At the end of the looping, show the view.
4a. If there were errors, show the list of error messages and tell them to re-upload the file.
4b. If there were no errors, create a link that has the file location from #2 and a confirmation flag
After the file location and confirm flag have been submitted run the loop in #3 again, but there's an if (confirmed) {} statement that actually makes the updates to the db.
EDIT: I saw your other post. One of the assumptions I made is that the databases won't be the same. ie, the legacy app will have a table or two. Maybe just products. But the new app will have orders, products, categories, etc, etc. This will complicate "just uploading the file".

Why do you need to push anything?
You just need to create a product management portion of the webpage and a secondly a public facing portion of the webpage. Both portions would touch the same SqlServer database.

.Net has the ability to monitor a database and check for updates. then you can run a query to [push] the data elsewhere.
or use sql to push the data with a trigger on the table(s) in question.
Is this what you were looking for?

You can try Dynamic Data Web Application.

You should have a service that regularly updates the data in the target DB. It will probably run on your source data machine (where the Access-DB is)
The service can use SSIS or ADO.NET to write the data. You can do this over the web, because you have access via TCP/IP to the server I assume.
Please check when the updates are done and how long it takes. If you can do the updates during the night you are fine. If not you should check, if you can still access the web during the import. That is sometimes not the case.

Use wget to push the new data file to the mvc app and once the data is received by the action, the mvc app invokes the processing/importing of the data (maybe in a worker process if you dont want long requests).